Installer can't get managedZones while service account and gcloud cli can on GCP

[gngeorgiev]

Version

$ openshift-install version
openshift-install 4.8.13
built from commit 450e95767d89f809cb1afe5a142e9c824a269de8
release image quay.io/openshift-release-dev/ocp-release@sha256:5d396ad7d5f3cb527580c735e87dfd3b853bbb531e7f03e3a184d0accc223cdf

Platform:

GCP
IPI

What happened?

Running openshift-install create install-config fails to list the managed zones in GCP project:

DEBUG OpenShift Installer 4.8.13
DEBUG Built from commit 450e95767d89f809cb1afe5a142e9c824a269de8
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG   Loading SSH Key...
DEBUG   Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Cluster Name...
DEBUG     Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Networking...
DEBUG     Loading Platform...
DEBUG   Loading Pull Secret...
DEBUG   Loading Platform...
DEBUG   Fetching SSH Key...
DEBUG   Generating SSH Key...
? SSH Public Key /Users/georgin.georgiev/.ssh/id_ed25519.pub
DEBUG   Fetching Base Domain...
DEBUG     Fetching Platform...
DEBUG     Generating Platform...
? Platform gcp
INFO Credentials loaded from gcloud CLI defaults
? Project ID group-verify (group-verify-df9383)
? Region europe-west4
DEBUG   Generating Base Domain...
FATAL failed to fetch Install Config: failed to fetch dependency of "Install Config": failed to generate asset "Base Domain": could not retrieve base domains: Get "https://dns.googleapis.com/dns/v1/projects/<OUR-PROJECT>/managedZones?alt=json&prettyPrint=false": context canceled

I am sure that the service account used by the gcloud cli has the correct permissions, it's also the only one authenticated with the CLI:

gcloud auth list
                    Credentialed Accounts
ACTIVE  ACCOUNT
*       openshift@<OUR-PROJECT>.iam.gserviceaccount.com

I can also confirm that I can list zones:

❯ gcloud dns managed-zones list                                                                                                                                                                                       04:22:28
NAME                          DNS_NAME                         DESCRIPTION                VISIBILITY
<LISTS ZONES>

What you expected to happen?

Installer to work or get an indication on why I have no permissions, what account it's trying to use etc.

How to reproduce it (as minimally and precisely as possible)?

In a new shell:

$ gcloud auth activate-service-account  gcloud.json
$ gcloud auth revoke <all other accounts>
$ openshift-install create install-config --log-level=debug

Anything else we need to know?

N/A

References

N/A

[gngeorgiev]

Turns out it reads the file $HOME/.config/gcloud/application_default_credentials.json, which apparently gcloud cli doesn't delete when the account is revoked. However, this isn't documented anywhere in the installer's documentation and I had to attach a debugger to figure it out.

What worked for me was to set GOOGLE_CLOUD_KEYFILE_JSON to an absolute path to the service account's json file.

[staebler]

The OpenShift installer uses FindDefaultCredentials. If that is different than what the gcloud cli uses to find credentials, then that is unfortunate.

What is the output of gcloud info?

[wallrj]

I am also encountering this error.

$ openshift-install version
openshift-install 4.8.0-0.okd-2021-10-10-030117
built from commit e0b12188a18a84be721116ddda74bb3510bfe03f
release image quay.io/openshift/okd@sha256:1d3f75529b141333939987ba03bf4ad76d83ae31d3b17df9a12c1f1ef67feff2


$ gcloud auth list
     Credentialed Accounts
ACTIVE  ACCOUNT
*       ***@***


$ openshift-install create install-config --log-level=debug
DEBUG OpenShift Installer 4.8.0-0.okd-2021-10-10-030117 
DEBUG Built from commit e0b12188a18a84be721116ddda74bb3510bfe03f 
DEBUG Fetching Install Config...                   
DEBUG Loading Install Config...                    
DEBUG   Loading SSH Key...                         
DEBUG   Loading Base Domain...                     
DEBUG     Loading Platform...                      
DEBUG   Loading Cluster Name...                    
DEBUG     Loading Base Domain...                   
DEBUG     Loading Platform...                      
DEBUG   Loading Networking...                      
DEBUG     Loading Platform...                      
DEBUG   Loading Pull Secret...                     
DEBUG   Loading Platform...                        
DEBUG   Fetching SSH Key...                        
DEBUG   Generating SSH Key...                      
? SSH Public Key /home/richard/.ssh/id_rsa.pub
DEBUG   Fetching Base Domain...                    
DEBUG     Fetching Platform...                     
DEBUG     Generating Platform...                   
? Platform gcp
INFO Credentials loaded from gcloud CLI defaults  
? Project ID <MY-PROJECT>
? Region europe-west1
DEBUG   Generating Base Domain...                  
FATAL failed to fetch Install Config: failed to fetch dependency of "Install Config": failed to generate asset "Base Domain": could not retrieve base domains: Get "https://dns.googleapis.com/dns/v1/projects/<MY-PROJECT>/managedZones?alt=json&prettyPrint=false": context canceled 

I tried to logout and login again to recreate the application_default_credentials.json file but it did not help

gcloud auth application-default revoke
gcloud auth application-default login

[gngeorgiev]

The OpenShift installer uses FindDefaultCredentials. If that is different than what the gcloud cli uses to find credentials, then that is unfortunate.

What is the output of gcloud info?

As long as application_default_credentials.json exists, gcloud info doesn't care about my service account in any way. This behavior is partially fine, it's out of openshift-installer's control (if indeed we use the same account resolution as other Google products), it just needs to be mentioned in the docs.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.