SIGSEGV: segmentation violation in OVH provider because of capitalization of domains

[kevinvalk]

WORKAROUND: Make sure to lowercase all domains and subdomains!

What happened:

I have been using external dns for a couple of weeks now and all was working great. However, just today I saw that it was crashing with the following segmentation violation.

....
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATEOLD dns:\"u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:A"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATEOLD dns:\"u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:AAAA"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATEOLD dns:\"yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] / type:AAAA"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATEOLD dns:\"yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] / type:A"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATEOLD dns:\"xxxxxxxx-a-u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:\"eWLouCLoc8A55px2aZB6pF+P4QScADOs3zzElDwkJpFXfecQztVM3ujTRzTD5+gU7s4UXj59qjJMj0DS7IbhZRoOEVMHv5944m6NNkppB4FYpc2f3j4R31O1R91J6jfKIent4yoQECbqTMYNAg8Sm7cOlbWeLzIFAzRmHY2HcP1Py+EVQNh3DzfvJQFifQ==\" / type:TXT"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATEOLD dns:\"xxxxxxxx-aaaa-u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:\"OKBfBOP0kwvaZ+IXT3xoeGTpcrfZ1k7JdE2+nwONK1Po7ZJGXvM5UitFU/mypeN6DkbbrfpYFIgAiwULQKOPj4nSJKi8qHqdaVMzYGKFCPDpWE3NAtMq+7kGO8l+9WtnKMJcILezhdv6HZ7x0MQruvgIORVJIEMYlMwxs1uo3lHvZ/DiHfwm0zbVJHlLkg==\" / type:TXT"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATEOLD dns:\"xxxxxxxx-aaaa-yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:\"PoyU3C6M/CtDPRT+1DeKNp0KVtWIMkBwlWPznXZcQ/ANSlOfA21LQ2Zpjils9BdMMDS0W4AsXthRKWR21A9hjpRzM/0fFb2IHsEguJd3JftAGUYMmQdzENezd7jyQqm4BglM1iP2dS9Mo9jgPgNMxLE/4Y8zUp1HALmNUAyqOy7XyUteIeuLusQeg00LuQ==\" / type:TXT"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATEOLD dns:\"xxxxxxxx-a-yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:\"oL0YYMJV+Fp0sjcsYwABXzbAqDicPArQRSxBGmtRBrvD3uwtOzTkX//IKpFLwEVu2sL6CGp40nLydYED0ozpwMMsrD17EBhJsi3qJNdKuSdYsUltBkCfAb2PGC0KUV4F406Y5YXRdtlYb8MYQvZDCyfqsf6fG3+WhDiLzgNHniLOkWcYpBM2REu574xCLw==\" / type:TXT"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATENEW dns:\"u1BTO2YJ6soaxrAvHo0svLy1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:A"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATENEW dns:\"u1BTO2YJ6soaxrAvHo0svLy1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:AAAA"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATENEW dns:\"yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] / type:AAAA"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATENEW dns:\"yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] / type:A"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATENEW dns:\"xxxxxxxx-a-u1BTO2YJ6soaxrAvHo0svLy1.xxxxxxxxxxxxxxxxxxxxx\" / targets:\"zT/eXU0uCOhmQ6Iwz/CIA0dVggvHdcE1zLjuu4p/lBgFFxA4CbDB5lq2ztzKZUeXxHGyDR8q6vTcMQWK2FwPF8gffQqrCDCFnTj5Z1FHzgE9Ra8RPd9siq28FBt07zIC4pGQW0+c/IyLJTQY+hdGHnOJj2ozm1jDNgkHOrzTUxFBKp7tnWnUKOWIPaZ5+Q==\" / type:TXT"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATENEW dns:\"xxxxxxxx-aaaa-u1BTO2YJ6soaxrAvHo0svLy1.xxxxxxxxxxxxxxxxxxxxx\" / targets:\"QICrQtqEjnOgsElmoeScDGXQBR/c/iUh8pP08eyLHRbSLZ5ubd+cyw/0YgdR3uiBfB/S826oh5uIZdUruO8I7fIOHtyibfO4V03P6JtS8aZlJ9cEz3fZcFt8lR8q5c+eqG/4OHQqsMRVCm6jRQZUIR3iR9ZUseXfDuTg+MLz+srRJZT57aeZ/9aMV8TrtA==\" / type:TXT"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATENEW dns:\"xxxxxxxx-aaaa-yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:\"n0PxkUifztEXak9hKunQfCs+9MDrO7eRBkh4MMBW9fbYjgFzFCRG1z01/blB8td9dWxqAecV3VCpQkzPDIiImKb8wOzzQjsN6ceYKhhmgDDB7/7tOt5x81otloYCtEwAdG88qouqk4X23otb2SAD4ctaEdx3gtkCWdpE2wF0u4X6l4l8md7ShXUkzTrPgA==\" / type:TXT"
time="2025-05-12T12:05:07Z" level=debug msg="OVH: changes UPDATENEW dns:\"xxxxxxxx-a-yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:\"4os1HKnXpc6X7y6E43f/LLr7IO8XZjOUD8Baac9cAG2UAjQVh2wQUpqOfh9ufiT0d5DRjoFqLEcyvZr5DIfPcnCB1DfRqoTYdsne0wD1uRXjWFflyrp2B4DD8DuEze4oCB7Ib0vVnVM8aFomI9tAaX4Lf0n2Is6ZW7WtnvFlAjPFWipLRrTikK3QEAcx6g==\" / type:TXT"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x293bc68]

goroutine 290 [running]:
sigs.k8s.io/external-dns/provider/ovh.OVHProvider.newOvhChangeUpdate({{}, {0x3f79620, 0x4000bff8c0}, {0x3f2ecc0, 0x40010208c0}, {{0x4000408850, 0x1, 0x1}, {0x0, 0x0, ...}, ...}, ...}, ...)
    sigs.k8s.io/external-dns/provider/ovh/ovh.go:590 +0x688
sigs.k8s.io/external-dns/provider/ovh.OVHProvider.computeSingleZoneChanges({{}, {0x3f79620, 0x4000bff8c0}, {0x3f2ecc0, 0x40010208c0}, {{0x4000408850, 0x1, 0x1}, {0x0, 0x0, ...}, ...}, ...}, ...)
    sigs.k8s.io/external-dns/provider/ovh/ovh.go:207 +0x234
sigs.k8s.io/external-dns/provider/ovh.(*OVHProvider).handleSingleZoneUpdate(0x40010342c0, {0x3f791a8, 0x4000652870}, {0x4000bbde80, 0x10}, {0x400028e008?, 0x0?, 0x0?}, 0x0?)
    sigs.k8s.io/external-dns/provider/ovh/ovh.go:214 +0x98
sigs.k8s.io/external-dns/provider/ovh.(*OVHProvider).ApplyChanges.func2()
    sigs.k8s.io/external-dns/provider/ovh/ovh.go:266 +0x44
golang.org/x/sync/errgroup.(*Group).Go.func1()
    golang.org/x/sync@v0.13.0/errgroup/errgroup.go:79 +0x54
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
    golang.org/x/sync@v0.13.0/errgroup/errgroup.go:76 +0x94
stream closed EOF for external-dns/external-dns-54f8cd75d7-7q5bz (external-dns)

What you expected to happen:

Not to crash.

How to reproduce it (as minimally and precisely as possible):

See below as to what it is and hence how to reproduce.

Anything else we need to know?:

Did a quick read on what it could be, and I think it is due to capitalization:

UPDATEOLD

u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx
u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx
yyyyyyyyyyyyyyyyyyyyyyyy
yyyyyyyyyyyyyyyyyyyyyyyy
xxxxxxxx-a-u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx
xxxxxxxx-aaaa-u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx
xxxxxxxx-aaaa-yyyyyyyyyyyyyyyyyyyyyyyy
xxxxxxxx-a-yyyyyyyyyyyyyyyyyyyyyyyy

UPDATENEW

u1BTO2YJ6soaxrAvHo0svLy1.xxxxxxxxxxxxxxxxxxxxx
u1BTO2YJ6soaxrAvHo0svLy1.xxxxxxxxxxxxxxxxxxxxx
yyyyyyyyyyyyyyyyyyyyyyyy
yyyyyyyyyyyyyyyyyyyyyyyy
xxxxxxxx-a-u1BTO2YJ6soaxrAvHo0svLy1.xxxxxxxxxxxxxxxxxxxxx
xxxxxxxx-aaaa-u1BTO2YJ6soaxrAvHo0svLy1.xxxxxxxxxxxxxxxxxxxxx
xxxxxxxx-aaaa-yyyyyyyyyyyyyyyyyyyyyyyy
xxxxxxxx-a-yyyyyyyyyyyyyyyyyyyyyyyy

This causes the unchecked lookup on line 586 to return null. This is not checked and as such, line 590 throws.

external-dns/provider/ovh/ovh.go

Lines 584 to 590 in 0cc2c50

	for id := range oldEndpointByTypeAndName {
	oldRecords := slices.Clone(oldRecordsInZone[id])
	endpointsNew := newEndpointByTypeAndName[id]
	var toInsertTarget []string
	for _, target := range endpointsNew.Targets {

Weirdly enough from a quick glance on https://datatracker.ietf.org/doc/html/rfc4343 it seems that case should be kept by DNS providers, but in the case of OVH, this is NOT done.

To conclude:

• To avoid these kind of issues, maybe it is good to lowercase all DNS zones/domains/labels throughout external DNS?
• An extra check in the OVH provider (maybe others) to avoid assuming records exist?

Environment:

• External-DNS version (use external-dns --version): external-dns:v20250508-v0.16.1-138-g8d8f8136
• DNS provider: OVH
• Others:

[ivankatliarchuk]

Hi @rbeuque74

Would you be able to have a look?

We planning to do a release soon-ish #5362

[kevinvalk]

Mmm, after lowercasing all domain definitions, I hit the following panic. From a glance at the code I could not figure out exactly what is happening here. It might be fixable by removing all "old" records from OVH and trying again...

time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"iecbweqpa2dasidntjdea8aw.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:A"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"iecbweqpa2dasidntjdea8aw.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:AAAA"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:A"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:AAAA"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] / type:A"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] / type:AAAA"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"zzzzzzzz-a-iecbweqpa2dasidntjdea8aw.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] /i4M9GxuhcVzDIQw4GC+3MuMMdzOFdD5jqqwz8pDHsP56crB9+Pp5aBoSoiOOQH96EMCyz0peEqG4SIw9tl+w4KqHAP9hoyqKh92084kVOpI1b7dLxOigpgS3a0z283Up9pb5DMjnDJXPwxK4koph33Klukz570r6d5PWCGu6+nfkvj5TOqBcs/fK14Q==\" / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"zzzzzzzz-aaaa-iecbweqpa2dasidntjdea8aw.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] /eISTk6lH5Fuse5z5Gcu/fzrpDR46NWtontLR4sPTfhU4vGwoD38MlqRt34V4gn8vV+CQ0anHLD1F3u6jUCD0AJIGpkOwbhYPmvZpWeAkJ6LVw==\" / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"zzzzzzzz-a-u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"zzzzzzzz-aaaa-u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] /mypeN6DkbbrfpYFIgAiwULQKOPj4nSJKi8qHqdaVMzYGKFCPDpWE3NAtMq+7kGO8l+9WtnKMJcILezhdv6HZ7x0MQruvgIORVJIEMYlMwxs1uo3lHvZ/DiHfwm0zbVJHlLkg==\" / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"zzzzzzzz-a-yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] /1NQ0df+6ZqDgK8dbvKbCGayMVQk9XXZRQQ==\" / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATEOLD dns:\"zzzzzzzz-aaaa-yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] /SS9+9ajTSeb+8kiN+aDOgQiGEMdsnJpQsnLYc+awqm/7is/hCAJ4dOELTtXzuFQtWIFaopKhUDKXPMDsrDSnGNksQfD4Zr5TStI4YuHhu3euDDGUFJWroYCFb8FcfIsQFRlI3HrwKOA==\" / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"iecbweqpa2dasidntjdea8aw.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:A"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"iecbweqpa2dasidntjdea8aw.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:AAAA"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:A"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] / type:AAAA"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] / type:A"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] / type:AAAA"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"zzzzzzzz-a-iecbweqpa2dasidntjdea8aw.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] /qeVveS8/aA1YEqmZnulquqRyzxZE47gCi/IO3p47bjRXi4aXoGjFxPClznue28eu0yQRgbCvTnMIn0LLwYFKd65WpCl1p7lSsQZsTprx+XYhMKDqUz208P4vlxxLNhvXn9Y3VDc6dSC0RfrOwRlLSjUL3yNy+n+02OMdJx+5yfwHkU+XMOdnSOxezF+q4HP3zTAA==\" / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"zzzzzzzz-aaaa-iecbweqpa2dasidntjdea8aw.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] /oI7G+Vda8Cae9hSiTNOnk7WjbfWSrO837YD1W11VxMWJI++rxz7aARVs71vkI2/destQdZ0mSW1XDEqNmtcCbuO24FyYVP/dP3Oh3PNt+VW0xMZrrFb4bxNBw==\" / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"zzzzzzzz-a-u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] /TKsv+OOySj/OHi93BV1eW4A76Ir4bZvxkgIJxgLkQZTxCrIIlq7dmL173V4Kn+jJ30m5fawo5pbQOQSDksf+7utU1fQmaL5p/BF4gfAZvRS13+jP2DuYgMjyfBUUlfEY99HKcZQttrjht4c4iFSGp8MBP\" / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"zzzzzzzz-aaaa-u1bto2yj6soaxravho0svly1.xxxxxxxxxxxxxxxxxxxxx\" / targets:[redacted] /JqEW4qWiUmjIBvLDPkS/htQea7z6Cx9nLLrv9jZROJLAAPvxx2PnTMGOF5qd\" / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"zzzzzzzz-a-yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] /8wb7grAYK9f6QNtWW0zA4IwFrBJ/k1jGqVxJ+geh9qiqsba4zm/JVDu3mK/Qn4TRnDTEa5CfvqoUjiyMKCtT+xfrILNqhvkbq7Fkt1yCHSk0k2jKgmZr4mMwwfZqFNeDMgDAX+14Z22ACLCQ8vdsD3gyR5M82wtldc\" / type:TXT"
time="2025-05-13T08:15:55Z" level=debug msg="OVH: changes UPDATENEW dns:\"zzzzzzzz-aaaa-yyyyyyyyyyyyyyyyyyyyyyyy\" / targets:[redacted] /2tA13r//bta0rT4W4I6v8QdqdVxsj/437JwwGiZfWb8xAWGl0tsisihfGww9r7DWl8nlDr9eifhZ/O4tMl0nWHQM7MDxgPcM0fi+5K8dnLSze9uBVc4u2cYZVBapMX8qhkVlAJu/fwZXBbrU6wzuss3GARbe8cEz09rjScHSng3aJf39TqcMZpW7Jny\" / type:TXT"
panic: runtime error: slice bounds out of range [:2:1]

goroutine 264 [running]:
slices.Delete[...](...)
    slices/slices.go:221
sigs.k8s.io/external-dns/provider/ovh.OVHProvider.newOvhChangeUpdate({{}, {0x3f79620, 0x40008cb380}, {0x3f2ecc0, 0x40007e9220}, {{0x400007a020, 0x1, 0x1}, {0x0, 0x0, ...}, ...}, ...}, ...)
    sigs.k8s.io/external-dns/provider/ovh/ovh.go:632 +0x13f0
sigs.k8s.io/external-dns/provider/ovh.OVHProvider.computeSingleZoneChanges({{}, {0x3f79620, 0x40008cb380}, {0x3f2ecc0, 0x40007e9220}, {{0x400007a020, 0x1, 0x1}, {0x0, 0x0, ...}, ...}, ...}, ...)
    sigs.k8s.io/external-dns/provider/ovh/ovh.go:207 +0x234
sigs.k8s.io/external-dns/provider/ovh.(*OVHProvider).handleSingleZoneUpdate(0x400083bad0, {0x3f791a8, 0x4000d740a0}, {0x4000d6a0c0, 0x10}, {0x40003e8408?, 0x4000d74050?, 0x4000d760c0?}, 0x13ff197bd?)
    sigs.k8s.io/external-dns/provider/ovh/ovh.go:214 +0x98
sigs.k8s.io/external-dns/provider/ovh.(*OVHProvider).ApplyChanges.func2()
    sigs.k8s.io/external-dns/provider/ovh/ovh.go:266 +0x44
golang.org/x/sync/errgroup.(*Group).Go.func1()
    golang.org/x/sync@v0.13.0/errgroup/errgroup.go:79 +0x54
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
    golang.org/x/sync@v0.13.0/errgroup/errgroup.go:76 +0x94
stream closed EOF for external-dns/external-dns-54f8cd75d7-h8mv8 (external-dns)

[kevinvalk]

Update: even after deleting all records from OVH zone and lowercasing all (sub)domains used within K8s. I am still hitting the slice bounds out of range [:2:1], so not sure what is up with that...

[rbeuque74]

Hi @rbeuque74

Would you be able to have a look?

We planning to do a release soon-ish #5362

Hi,

I will have a look on that.
Thanks for the report and the mention ;)

[rbeuque74]

#5390 for the fix