Error extracting availability zone list on v0.12.2

[theboringstuff]

/kind bug

What steps did you take and what happened:

• Install CAPI v1.9.6 with CAPO 0.12.2
• Create cluster
• CAPO fails to reconcile with following error

E0331 09:13:49.341181       1 controller.go:316] "Reconciler error" err="error extracting availability zone list: unable to verify compatible server version: Expected HTTP response code [200 300] when accessing [GET http://ip:port/v2/8b15e6751b4747b8ba0db5616392682f/], but got 404 instead: {\"message\": \"The resource could not be found.<br /><br />\\n\\n\\n\", \"code\": \"404 Not Found\", \"title\": \"Not Found\"}" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="4207ae4a-d059-4b3e-bc1f-bff041a55128"

What did you expect to happen:

• I expect reconciliation to succeed, since it work on CAPO 0.11.5 (and below)

Anything else you would like to add:

• I suspect the path to list availability zones is incorrect, since openstack CLI uses something like following (which works), but I am not sure

REQ: curl -g -i -X GET http://ip:port/v2/8b15e6751b4747b8ba0db5616392682f/os-availability-zone -H "Accept: application/json" -H "User-Agent: openstacksdk/4.4.0 keystoneauth1/5.10.0 python-requests/2.32.3 CPython/3.10.12" -H "X-Auth-Token: ..."

• The very same usecase works on CAPO v0.11.5

Environment:

• Cluster API Provider OpenStack version (Or git rev-parse HEAD if manually built): v0.12.2
• Cluster-API version: v1.9.6
• OpenStack version:
• Minikube/KIND version:
• Kubernetes version (use kubectl version):
• OS (e.g. from /etc/os-release):

[EmilienM]

Our CI doesn't reproduce this issue it seems and we test multiple AZs.
I'm wondering if you're using a specific microversion in the client.

[theboringstuff]

I do not select a specific microversion and, as I mentioned, the very same thing works on CAPO v0.11.5, but fails on v0.12.2. So maybe v0.12.2 dropped support for some Openstack APIs verisons? On our OpenStack we have min=2.1 and max=2.38 microversions for compute service

[mdbooth]

@theboringstuff I just spent some time looking for any reason this might happen, but afaict neither the CAPO code which gets AZs or (my first suspect from your description) gophercloud changed at all between v0.11.5 and v0.12.2. As @EmilienM said we do test this in our E2E tests, although admittedly not against such an old OpenStack version.

Without anything more concrete to go on we're unlikely to be able to progress this. Are you able to debug to determine if some different code path is being followed between the 2 versions?

[theboringstuff]

@mdbooth I will try to collect more information and will let you know

[theboringstuff]

Collected verbose logs where we can see different request path/headers

On v0.11.5

I0403 05:16:31.430777       1 provider.go:251] "OpenStack Request URL: GET http://ip:port/v2/8b15e6751b4747b8ba0db5616392682f/os-availability-zone" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="730bfd46-f99b-478a-90f9-84bf9fc7f387" cluster="test"
I0403 05:16:31.430839       1 provider.go:251] "OpenStack Request Headers:\nAccept: application/json\nOpenstack-Api-Version: compute 2.60\nUser-Agent: cluster-api-provider-openstack/v0.11.5 gophercloud/v2.6.0\nX-Auth-Token: ***\nX-Openstack-Nova-Api-Version: 2.60" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="730bfd46-f99b-478a-90f9-84bf9fc7f387" cluster="test"
I0403 05:16:31.498996       1 provider.go:251] "OpenStack Response Code: 200" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="730bfd46-f99b-478a-90f9-84bf9fc7f387" cluster="test"
I0403 05:16:31.499058       1 provider.go:251] "OpenStack Response Headers:\nContent-Length: 173\nContent-Type: application/json\nDate: Thu, 03 Apr 2025 05:16:31 GMT\nX-Compute-Request-Id: req-e2178e1f-bc18-40b6-aff2-e1cb671befe0" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="730bfd46-f99b-478a-90f9-84bf9fc7f387" cluster="test"

On v0.12.2

I0403 05:45:46.023705       1 provider.go:250] "OpenStack Request URL: GET http://ip:port/v2/8b15e6751b4747b8ba0db5616392682f/" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="113b2d9d-cdca-4f54-9c53-54d11cb950f2" cluster="test"
I0403 05:45:46.023747       1 provider.go:250] "OpenStack Request Headers:\nAccept: application/json\nUser-Agent: cluster-api-provider-openstack/v0.12.2 gophercloud/v2.6.0\nX-Auth-Token: ***" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="113b2d9d-cdca-4f54-9c53-54d11cb950f2" cluster="test"
I0403 05:45:46.059059       1 provider.go:250] "OpenStack Response Code: 404" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="113b2d9d-cdca-4f54-9c53-54d11cb950f2" cluster="test"
I0403 05:45:46.059151       1 provider.go:250] "OpenStack Response Headers:\nContent-Length: 112\nContent-Type: application/json; charset=UTF-8\nDate: Thu, 03 Apr 2025 05:45:46 GMT\nX-Compute-Request-Id: req-89c82144-34fa-4185-a941-1d8030312103" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="113b2d9d-cdca-4f54-9c53-54d11cb950f2" cluster="test"
I0403 05:45:35.067540       1 provider.go:250] "OpenStack Response Body: {\n  \"code\": \"404 Not Found\",\n  \"message\": \"The resource could not be found.\\u003cbr /\\u003e\\u003cbr /\\u003e\\n\\n\\n\",\n  \"title\": \"Not Found\"\n}" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="113b2d9d-cdca-4f54-9c53-54d11cb950f2" cluster="test"
E0403 05:45:35.068563       1 controller.go:316] "Reconciler error" err="error extracting availability zone list: unable to verify compatible server version: Expected HTTP response code [200 300] when accessing [GET http://ip:port/v2/8b15e6751b4747b8ba0db5616392682f/], but got 404 instead: {\"message\": \"The resource could not be found.<br /><br />\\n\\n\\n\", \"code\": \"404 Not Found\", \"title\": \"Not Found\"}" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="4f4546b0-7a3b-493e-8516-2c3ebb5b1417"

The difference, as I can see:

• On v0.12.2 request does not contain /os-availability-zone in path
• On v0.12.2 request headers do not contain Openstack-Api-Version: compute 2.60 and X-Openstack-Nova-Api-Version: 2.60

I am not familiar with CAPO/gophercloud code, so any more details will take time for me to collect

[theboringstuff]

It seems the error comes from this code https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/v0.12.2/pkg/clients/compute.go#L94

It was not present in v0.11.5, this is why we see this error only on v0.12.2 https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/v0.11.5/pkg/clients/compute.go#L65

Since on v0.11.5 cluster provisioning works, I suppose our version should be supported? We have max=2.38 for compute. But on v0.12.2 it is now considered unsupported. Actually, since it fails with 404 error, I suppose it is not necessarily unsupported, CAPO just can not collect required information to check compatibility

[stephenfin]

That's a really old version of OpenStack. I suspect the issue here might be the trailing slash, as there was a point in the distant past where that wasn't supported.

Could you curl that URL with and without a trailing slash and see what happens? You should get the same version document for both.

❯ curl http://ip:port/v2/8b15e6751b4747b8ba0db5616392682f
❯ curl http://ip:port/v2/8b15e6751b4747b8ba0db5616392682f/

You might need to pass a token too, since the version doc also required authentication a long time ago. If you do, you can generate that with the openstack token issue command. Or you can use openstacksdk instead of curl:

❯ python3 -m venv
❯ source venv/bin/activate
❯ pip install openstacksdk
❯ python

>>> import openstack
>>> conn = openstack.connect()
>>> openstack.enable_logging(debug=True)
>>> conn.compute.get('').json()
>>> conn.compute.get('/').json()

[stephenfin]

Also, just to confirm: do you see the unable to verify compatible server version error in your logs?

[theboringstuff]

>>> conn.compute.get('').json()
>>> conn.compute.get('/').json()

These both commands give the same 404 error, judging by debug logs, openstacksdk adds trailing slash in both cases

>>> conn.compute.get('').json()
...
REQ: curl -g -i -X GET http://ip:port/v2/8b15e6751b4747b8ba0db5616392682f/ -H "User-Agent: openstacksdk/4.4.0 keystoneauth1/5.10.0 python-requests/2.32.3 CPython/3.12.9" -H "X-Auth-Token: ***"

---

Also, just to confirm: do you see the unable to verify compatible server version error in your logs?

Yes, I added this log in my above comments, it is like this in the end:

E0403 05:45:35.068563       1 controller.go:316] "Reconciler error" err="error extracting availability zone list: unable to verify compatible server version: Expected HTTP response code [200 300] when accessing [GET http://ip:port/v2/8b15e6751b4747b8ba0db5616392682f/], but got 404 instead: {\"message\": \"The resource could not be found.<br /><br />\\n\\n\\n\", \"code\": \"404 Not Found\", \"title\": \"Not Found\"}" controller="openstackcluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="OpenStackCluster" OpenStackCluster="default/test" namespace="default" name="test" reconcileID="4f4546b0-7a3b-493e-8516-2c3ebb5b1417"

Compute mircoversions are as following, as I understand 2.38 should be supported

$ openstack --os-cloud openstack versions show | grep compute
| RegionOne   | compute        | 2.0     | SUPPORTED  | http://ip:port/v2/   | None             | None |
| RegionOne   | compute        | 2.1     | CURRENT    | http://ip:port/v2.1/ | 2.1              | 2.38 |

Actually, I am not sure 2.38 is really supported, I judge by this code, but here it states 2.60, I am not sure which is true. If our openstack is too old that is fine, I can use 0.11.5, I just want to confirm that 2.38 is not supported in 0.12.2

[stephenfin]

>>> conn.compute.get('').json()
>>> conn.compute.get('/').json()

These both commands give the same 404 error, judging by debug logs, openstacksdk adds trailing slash in both cases

>>> conn.compute.get('').json()
...
REQ: curl -g -i -X GET http://ip:port/v2/8b15e6751b4747b8ba0db5616392682f/ -H "User-Agent: openstacksdk/4.4.0 keystoneauth1/5.10.0 python-requests/2.32.3 CPython/3.12.9" -H "X-Auth-Token: ***"

And if you run the curl command manually without the trailing slash? I'd just like to nail down whether that's the issue.

Actually, I am not sure 2.38 is really supported, I judge by this code, but here it states 2.60, I am not sure which is true. If our openstack is too old that is fine, I can use 0.11.5, I just want to confirm that 2.38 is not supported in 0.12.2

Assuming the trailing slash is the issue, this is one of those things that microversions don't capture: if I run the exact same command on e.g. a Wallaby cluster using API microversion 2.38, it will work for me. That's because the fix happened earlier in the stack.

[theboringstuff]

And if you run the curl command manually without the trailing slash? I'd just like to nail down whether that's the issue.

I get 401 error, not sure how to authenticate with curl. I copy curl command from CAPO/openstacksdk debug output which contains auth token, but it still fails with 401 (both with and without trailing slash)

[stephenfin]

The easiest way is to use openstack token issue to generate a token. For example:

❯ openstack token issue                                                                                                                                                                                                                                                                                                                                                                                                                    
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                                                                                                                                                                                                                   
| Field      | Value                                                                                                                                                                                   |                                                                                                                                                                                                                                   
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                                                                                                                                                                                                                   
| expires    | 2025-04-04T10:42:04+0000                                                                                                                                                                |                                                                                                                                                                                                                                   
| id         | gAAAAABn76ls6kEX8RSFWB8aKMiZ_le9Rr6vWI_2Hd8-B8iZRMKO2cJYVbMX9XF185i-Ml64EDXCtW5I4CW0gCrNiiLNo4Haz27O7HSitryiNwY2lzfNRtr-9i69fnwMErCzrenlttsmmgkKvwcSM5MM9PXmtjEOzR7en2g2Dct1rUnbGRVwXs8 |                                                                                                                                                                                                                                   
| project_id | e11b30c8ea0d418bac3c7c43c0509721                                                                                                                                                        |                                                                                                                                                                                                                                   
| user_id    | b6e34c015c624bcb9483b9431a4b3562                                                                                                                                                        |                                                                                                                                                                                                                                   
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

The id field is the token. Pass this via the X-Auth-Token header:

❯ curl -g -i -X GET "http://10.0.110.96/compute/v2.1/servers" -H "X-Auth-Token: gAAAAABn76ls6kEX8RSFWB8aKMiZ_le9Rr6vWI_2Hd8-B8iZRMKO2cJYVbMX9XF185i-Ml64EDXCtW5I4CW0gCrNiiLNo4Haz27O7HSitryiNwY2lzfNRtr-9i69fnwMErCzrenlttsmmgkKvwcSM5MM9PXmtjEOzR7en2g2Dct1rUnbGRVwXs8"

Once you're done, the token can be revoked:

❯ openstack token revoke gAAAAABn76ls6kEX8RSFWB8aKMiZ_le9Rr6vWI_2Hd8-B8iZRMKO2cJYVbMX9XF185i-Ml64EDXCtW5I4CW0gCrNiiLNo4Haz27O7HSitryiNwY2lzfNRtr-9i69fnwMErCzrenlttsmmgkKvwcSM5MM9PXmtjEOzR7en2g2Dct1rUnbGRVwXs8

Alternatively, you can generate the token from scratch too. That cluster is old enough that you probably still have the keystone v2 API around?

❯ curl -X POST http://10.0.110.96/identity/v2.0/tokens -d '{"auth":{"passwordCredentials":{"username": "admin", "password":"password"}, "tenantId":"my-tenant-id"}}' -H 'Content-type: application/json'

The return value can be used directly.