Hetzner DNS Provider: Let's Encrypt DNS Record Fails w/ multiple Hetzner DNS Zones: "HTTP Error 422: Unprocessable Entity"

[karstengresch]

Trying to create a new cluster w/ under the following conditions:

• host == localhost, i.e. installing from the Hetzner instances
• Ansible updated (cf. ansible-navigator - KeyError: 'time_stamp'  #229 )
• DNS records at Hetzner already deleted (cf. Hetzner DNS Api not idempotent - playbook cannot be rerun #241 )

Running

ansible-navigator run -m stdout ./ansible/setup.yml

results in this error:

TASK [letsencrypt : Create letsencrypt DNS record at Hetzner] ******************
failed: [host -> localhost] (item=[{'key': '_acme-challenge.apps.ocp4.devworkshop.cc', 'value': ['Px7SpNn1fcVGMhz1sehue1OTu82onJCXkBn-wiQGbUM']}, 'Px7SpNn1fcVGMhz1sehue1OTu82onJCXkBn-wiQGbUM']) => {"access_control_allow_origin": "*", "ansible_loop_var": "item", "changed": false, "connection": "close", "content": "{\"record\":{\"id\":\"\",\"type\":\"\",\"name\":\"\",\"value\":\"\",\"zone_id\":\"\",\"created\":\"\",\"modified\":\"\"},\"error\":{\"message\":\"zone not found\",\"code\":422}}\n", "content_length": "140", "content_type": "application/json; charset=utf-8", "date": "Thu, 26 Jan 2023 21:48:55 GMT", "elapsed": 0, "item": [{"key": "_acme-challenge.apps.ocp4.devworkshop.cc", "value": ["Px7SpNn1fcVGMhz1sehue1OTu82onJCXkBn-wiQGbUM"]}, "Px7SpNn1fcVGMhz1sehue1OTu82onJCXkBn-wiQGbUM"], "json": {"error": {"code": 422, "message": "zone not found"}, "record": {"created": "", "id": "", "modified": "", "name": "", "type": "", "value": "", "zone_id": ""}}, "msg": "Status code was 422 and not [200]: HTTP Error 422: Unprocessable Entity", "ratelimit_limit": "240", "ratelimit_remaining": "234", "ratelimit_reset": "5", "redirected": false, "status": 422, "url": "https://dns.hetzner.com/api/v1/records", "vary": "Origin", "x_ratelimit_limit_minute": "240", "x_ratelimit_remaining_minute": "234"}
failed: [host -> localhost] (item=[{'key': '_acme-challenge.api.ocp4.devworkshop.cc', 'value': ['HfgMxhyRf5zF7_JRBf_pNPTjUWJmtUypxRBy9JZ8B7s']}, 'HfgMxhyRf5zF7_JRBf_pNPTjUWJmtUypxRBy9JZ8B7s']) => {"access_control_allow_origin": "*", "ansible_loop_var": "item", "changed": false, "connection": "close", "content": "{\"record\":{\"id\":\"\",\"type\":\"\",\"name\":\"\",\"value\":\"\",\"zone_id\":\"\",\"created\":\"\",\"modified\":\"\"},\"error\":{\"message\":\"zone not found\",\"code\":422}}\n", "content_length": "140", "content_type": "application/json; charset=utf-8", "date": "Thu, 26 Jan 2023 21:48:56 GMT", "elapsed": 0, "item": [{"key": "_acme-challenge.api.ocp4.devworkshop.cc", "value": ["HfgMxhyRf5zF7_JRBf_pNPTjUWJmtUypxRBy9JZ8B7s"]}, "HfgMxhyRf5zF7_JRBf_pNPTjUWJmtUypxRBy9JZ8B7s"], "json": {"error": {"code": 422, "message": "zone not found"}, "record": {"created": "", "id": "", "modified": "", "name": "", "type": "", "value": "", "zone_id": ""}}, "msg": "Status code was 422 and not [200]: HTTP Error 422: Unprocessable Entity", "ratelimit_limit": "240", "ratelimit_remaining": "233", "ratelimit_reset": "4", "redirected": false, "status": 422, "url": "https://dns.hetzner.com/api/v1/records", "vary": "Origin", "x_ratelimit_limit_minute": "240", "x_ratelimit_remaining_minute": "233"}

PLAY RECAP *********************************************************************
host                       : ok=33   changed=1    unreachable=0    failed=1    skipped=27   rescued=0    ignored=0

Strange enough, I could curl the DNS API from Hetzner:

curl "https://dns.hetzner.com/api/v1/zones" -H 'Auth-API-Token: <my-token>'

which returns results like:

{"zones":[{"id":"<id>","name":"devworkshop.cc","ttl":86400,"registrar":"","legacy_dns_host":"","legacy_ns":[],"ns":["hydrogen.ns.hetzner.com","oxygen.ns.hetzner.com","helium.ns.hetzner.de"],"created":"2020-04-07
 01:24:03 +0000 UTC","verified":"2020-04-07 01:54:11.243070434 +0000 UTC
 m=+643.369442731","modified":"2022-09-06 09:21:56.346 +0000 
UTC","project":"","owner":"","permission":"","zone_type":{"id":"","name":"","description":"","prices":null},"status":"verified","paused":false,"is_secondary_dns":false,"txt_verification":{"name":"","token":""},"records_count":3},{"id":"<i>","name":"opsworkshop.cc","ttl":86400,"registrar":"","legacy_dns_host":"","legacy_ns":["robotns3.second-ns.de.opsworkshop.cc.","ns1.first-ns.de.opsworkshop.cc.","robotns2.second-ns.de.opsworkshop.cc."],"ns":["hydrogen.ns.hetzner.com","oxygen.ns.hetzner.com","helium.ns.hetzner.de"],"created":"2022-11-18
 16:00:25.59 +0000 UTC","verified":"","modified":"2022-11-18 16:00:25.59
 +0000 
UTC","project":"","owner":"","permission":"","zone_type":{"id":"","name":"","description":"","prices":null},"status":"verified","paused":false,"is_secondary_dns":false,"txt_verification":{"name":"","token":""},"records_count":13}],"meta":{"pagination":{"page":1,"per_page":100,"previous_page":1,"next_page":1,"last_page":1,"total_entries":2}}}

It seems, the problem is that there are two DNS zones setup in my Hetzner account:

After deleting the second zone, the playbook runs w/o errors.

Though the easy workaround is to delete the 2nd zone, it'd be better to consider just the zone with the specified zone name in cluster.yml.

[rbo]

Feel free to test the new hetzner dns implementation: https://github.com/RedHat-EMEA-SSA-Team/hetzner-ocp4/tree/hetzner_dns

[rbo]

Fixed in devel branch, will be merged into master with #252