OKD 4.x Install Fails Despite DNS Resolution from Bootstrap Node success

[shashankmotruri]

I am installing OKD on Google Cloud Platform (GCP) using a preconfigured VPC network. For the domain setup, I have delegated a GCP subdomain to an AWS Route 53 main hosted zone. According to the documentation, the OKD installer "creates records in a pre-existing public zone." I have a pre-existing public DNS-managed zone in GCP delegated to AWS Route 53. The records were updated in this zone by the OKD installer as mentioned here.

I was able to resolve the host from the bootstrap node.

[core@okd-cluster-gh848-bootstrap ~]$ curl -k https://api.okd-cluster.okd.dev.example.com:6443/version
{
  "major": "1",
  "minor": "30+",
  "gitVersion": "v1.30.6-dirty",
  "gitCommit": "5a24e858060a609366c5ff1402ec1424e9f7e403",
  "gitTreeState": "dirty",
  "buildDate": "2024-11-19T02:46:50Z",
  "goVersion": "go1.22.7 (Red Hat 1.22.7-1.el9_5) X:strictfipsruntime",
  "compiler": "gc",
  "platform": "linux/amd64"
}

But, surprisingly, the OKD installation failed to resolve the host

DEBUG E0419 17:08:31.945864 1721882 controller.go:324] "Reconciler error" err="failed to get client: failed to create cluster accessor: error creating http client and mapper for remote cluster \"openshift-cluster-api-guests/okd-cluster-gh848\": error creating client for remote cluster \"openshift-cluster-api-guests/okd-cluster-gh848\": cluster is not reachable: Get \"https://api.okd-cluster.okd.dev.example.com:6443/?timeout=5s\": dial tcp: lookup api.okd-cluster.okd.dev.example.com on 100.100.100.100:53: no such host" controller="machine" controllerGroup="cluster.x-k8s.io" controllerKind="Machine" Machine="openshift-cluster-api-guests/okd-cluster-gh848-master-2" namespace="openshift-cluster-api-guests" name="okd-cluster-gh848-master-2" reconcileID="457f696c-c9d4-48d1-a34d-03ea72131ee2"

DEBUG E0419 17:08:43.501025 1721882 controller.go:324] "Reconciler error" err="failed to get client: failed to create cluster accessor: error creating http client and mapper for remote cluster \"openshift-cluster-api-guests/okd-cluster-gh848\": error creating client for remote cluster \"openshift-cluster-api-guests/okd-cluster-gh848\": cluster is not reachable: Get \"https://api.okd-cluster.okd.dev.example.com:6443/?timeout

OKD install config used:

  apiVersion: v1
  baseDomain: okd.dev.example.com
  credentialsMode: Manual
  metadata:
    name: okd-cluster
  platform:
    gcp:
      projectID: <project>
      region: us-east1
      network: okd-vpc
      controlPlaneSubnet: okd-subnet
      computeSubnet: okd-subnet
      defaultMachinePlatform:
        serviceAccount: [email protected]
  pullSecret: '{"auths":{"fake":{"auth":"aWQ6cGFzcwo="}}}'
  sshKey: <sshKey>
  controlPlane:
    name: master
    replicas: 3
    hyperthreading: Enabled
    platform:
      gcp:
        type: n2-standard-8
  compute:
    - name: worker
      replicas: 6
      hyperthreading: Enabled
      platform:
        gcp:
          type: n2-standard-8
  networking:
    networkType: OVNKubernetes
    clusterNetwork:
      - cidr: 10.128.0.0/14
        hostPrefix: 23
    serviceNetwork:
      - 172.30.0.0/16
    machineNetwork:
      - cidr: 10.0.0.0/24

Verions:
openshift-install 4.17.0-okd-scos.0

[shashankmotruri]

The same happened for 4.15.0-0.okd-2024-02-23-163410

DEBUG Built from commit e8c70b7db9e64cafa7fe3182c365257c3e53fbb6
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG   Loading SSH Key...
DEBUG   Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Cluster Name...
DEBUG     Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Pull Secret...
DEBUG   Loading Platform...
DEBUG Using Install Config loaded from state file
DEBUG Reusing previously-fetched Install Config
INFO Waiting up to 20m0s (until 7:38PM UTC) for the Kubernetes API at https://api.okd-cluster.okd.dev.example.com:6443...
DEBUG Loading Agent Config...
DEBUG Still waiting for the Kubernetes API: Get "https://api.okd-cluster.okd.dev.example.com:6443/version": dial tcp: lookup api.okd-cluster.okd.dev.example.com on 100.100.100.100:53: no such host
DEBUG Still waiting for the Kubernetes API: Get "https://api.okd-cluster.okd.dev.example.com:6443/version": dial tcp: lookup api.okd-cluster.okd.dev.example.com on 100.100.100.100:53: no such host

From bootstrap node

core@okd-cluster-zcw4z-bootstrap:~$ curl -I -k https://api.okd-cluster.okd.dev.example.com:6443/version
HTTP/2 403
audit-id: 10b0ea7d-674a-44b4-a201-f4e45d61172f
cache-control: no-cache, private
content-type: application/json
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-content-type-options: nosniff
x-kubernetes-pf-flowschema-uid: 918af5f5-07b2-42f0-ae85-852ffb0693db
x-kubernetes-pf-prioritylevel-uid: 1a46b4b9-0d54-4d23-b108-af2ef2520669
content-length: 225
date: Sat, 19 Apr 2025 19:24:24 GMT

core@okd-cluster-zcw4z-bootstrap:~$ nslookup api.okd-cluster.okd.dev.example.com
Server:		169.254.169.254
Address:	169.254.169.254#53

Non-authoritative answer:
Name:	api.okd-cluster.okd.dev.example.com
Address: 35.237.60.213

The only thing that was changed during the installation process was I had to delete the private zone created during okd installation. since the endpoint was resolving to an internal IP.

# Before deleting the privately created GCP DNS managed zone
core@okd-cluster-zcw4z-bootstrap:~$ dig api.okd-cluster.okd.dev.example.com

; <<>> DiG 9.18.21 <<>> api.okd-cluster.okd.dev.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43720
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;api.okd-cluster.okd.dev.example.com. IN A

;; ANSWER SECTION:
api.okd-cluster.okd.dev.example.com. 60 IN A	10.0.0.54

;; Query time: 4 msec
;; SERVER: 169.254.169.254#53(169.254.169.254) (UDP)
;; WHEN: Sat Apr 19 19:23:38 UTC 2025
;; MSG SIZE  rcvd: 81

# After deleting the privately created GCP DNS managed zone
core@okd-cluster-zcw4z-bootstrap:~$ dig api.okd-cluster.okd.dev.example.com

; <<>> DiG 9.18.21 <<>> api.okd-cluster.okd.dev.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55765
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;api.okd-cluster.okd.dev.example.com. IN A

;; ANSWER SECTION:
api.okd-cluster.okd.dev.example.com. 60 IN A	35.237.60.213

;; Query time: 63 msec
;; SERVER: 169.254.169.254#53(169.254.169.254) (UDP)
;; WHEN: Sat Apr 19 19:24:14 UTC 2025
;; MSG SIZE  rcvd: 81

[shashankmotruri]

Am I missing anything here? 🤔

[shashankmotruri]

This was fixed, my machine was not. able to resolve the host. Updating the nameservers fixed the issue.