[OKD-SCOS v4.17] UPI install with agent installer fails for 4.17.0-0.okd-scos-2024-06-27-091238

[titou10titou10]

(this is in relation to discussion #1960)

Context

Trying to install a SNO (Single Node) cluster:

• with OKD-SCOS v4.17.0-0.okd-scos-2024-06-27-091238 (an "Accepted" version)
• on baremetal with UPI
• using agent based installer (ABI)
• on vm: 40G RAM, 100G SSD, 10 vCPUS on Proxmox (qemu)

It is important to note that the install works perfectly well with the exact same agent and install config files for OKD-FCOS v4.15.0-0.okd-2024-03-10-010116

I also have tried for a multi-node cluster, it fails the same way

Summary

It fails with the following error:

Failed to register cluster with assisted-service: AssistedServiceError Code: 400 Href:  ID: 400 Kind: Error Reason: pull secret for new cluster is invalid: pull secret must contain auth for \"registry.ci.openshift.org\""

Details

install-config.yaml:

  apiVersion: v1
  baseDomain: denis.prive
  metadata:
    name: okd-sno
  controlPlane:
    architecture: amd64
    hyperthreading: Enabled   
    name: master 
    replicas: 1
  compute:
    - architecture: amd64
      hyperthreading: Enabled   
      name: worker
      replicas: 0 
  networking:
    networkType: OVNKubernetes
    clusterNetwork:
    - cidr: 10.132.0.0/16
      hostPrefix: 23 
    machineNetwork:
    - cidr: 192.168.5.0/24
    serviceNetwork: 
    - 172.31.0.0/16
  platform:
    none: {} 
  fips: false 
  sshKey: 'ssh-rsa AAAA{...}' 
  pullSecret: '{"auths": {"empty": {"auth": "aWQ6cGFzcwo="}}}'
  additionalTrustBundle: |
    -----BEGIN CERTIFICATE-----
    {...}
    -----END CERTIFICATE-----

agent-config.yaml

  apiVersion: v1beta1
  kind: AgentConfig
  metadata:
    name: okd-sno
  rendezvousIP: 192.168.5.71
  additionalNTPSources:
  - 0.pool.ntp.org
  - 1.pool.ntp.org
  - 2.pool.ntp.org
  - 3.pool.ntp.org
  hosts: 
    - hostname: okd-sno
      role: master
      interfaces:
        - name: ens18
          macAddress: xx:xx:xx:xx:xx:71
      rootDeviceHints: 
        deviceName: /dev/sda
      networkConfig: 
        interfaces:
          - name: ens18
            type: ethernet
            state: up
            mac-address: xx:xx:xx:xx:xx:71
            ipv4:
              enabled: true
              dhcp: true
              #auto-dns: true
              auto-dns: false
              auto-gateway: true
              auto-routes: true
            ipv6:
              enabled: false
        dns-resolver:
          config:
            search:
              - denis.prive
            server:
              - 192.168.5.18
              - 8.8.8.8

The install fails after a few minutes the node boots, but the process fails, looping forever

ssh core@okd-sno
sudo su -

journalctl -f
{...}
Jul 04 16:30:47 okd-sno agent-register-cluster[12524]: time="2024-07-04T16:30:47Z" level=info msg="SERVICE_BASE_URL: http://192.168.5.71:8090/"
Jul 04 16:30:47 okd-sno podman[12490]: time="2024-07-04T16:30:47Z" level=info msg="SERVICE_BASE_URL: http://192.168.5.71:8090/"
Jul 04 16:30:47 okd-sno agent-register-cluster[12524]: time="2024-07-04T16:30:47Z" level=info msg="Registering cluster"
Jul 04 16:30:47 okd-sno podman[12490]: time="2024-07-04T16:30:47Z" level=info msg="Registering cluster"
Jul 04 16:30:47 okd-sno systemd[1]: agent-register-infraenv.service: Scheduled restart job, restart counter is at 30.
Jul 04 16:30:47 okd-sno systemd[1]: Stopped Service that registers the infraenv.
Jul 04 16:30:49 okd-sno agent-register-cluster[12524]: time="2024-07-04T16:30:49Z" level=info msg="releaseImage: registry.ci.openshift.org/origin/release-scos@sha256:881ff38766ab4e59d4e20bd49e63cccf9388f023093fd1ffe7786b0fa1dfbaa4"
Jul 04 16:30:49 okd-sno agent-register-cluster[12524]: time="2024-07-04T16:30:49Z" level=info msg="releaseImage version 4.17.0-0.okd-scos-2024-06-27-091238 cpuarch x86_64"
Jul 04 16:30:49 okd-sno podman[12490]: time="2024-07-04T16:30:49Z" level=info msg="releaseImage: registry.ci.openshift.org/origin/release-scos@sha256:881ff38766ab4e59d4e20bd49e63cccf9388f023093fd1ffe7786b0fa1dfbaa4"
Jul 04 16:30:49 okd-sno podman[12490]: time="2024-07-04T16:30:49Z" level=info msg="releaseImage version 4.17.0-0.okd-scos-2024-06-27-091238 cpuarch x86_64"
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=info msg="Register cluster: okd-sno with id 712cd40a-5ef3-44e8-a420-6478536d25c0 and params {\"api_vips\":[],\"base_dns_domain\":\"denis.prive\",\"cluster_networks\":[{\"cidr\":\"10.132.0.0/16\",\"host_prefix\":23}],\"cpu_architecture\":\"x86_64\",\"high_availability_mode\":\"None\",\"ingress_vips\":[],\"machine_networks\":[{\"cidr\":\"192.168.5.0/24\"}],\"name\":\"okd-sno\",\"network_type\":\"OVNKubernetes\",\"olm_operators\":null,\"openshift_version\":\"4.17.0-0.okd-scos-2024-06-27-091238\",\"platform\":{\"type\":\"none\"},\"pull_secret\":\"***\",\"schedulable_masters\":false,\"service_networks\":[{\"cidr\":\"172.31.0.0/16\"}],\"ssh_public_key\":\"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPQSXLHzwM93ANj4oy8M8uT199fiCGtj9/3AKpnE/5FNpYSioh0WALtmjnLHtgnENbej6whcuphMI57ls3VEXyLIpkH55C+aKWPLdKTaNEWcJLkhH5a5BHUDNa1/BPHZM9j9j4tYH0Eq//iHsJ3wAtNjSx6U3E04l7BqNvM66JvDECi3ei6OrvnKQ9o6mbIIivPZvSa6EDM/z1612rmLZ5zUSHXPe5irORn64zK/ULIhcdMc6p9/Uuu2aJtQfXdbRorQ3/QCX4g80wUMzRxMnZ0tdb8kzHfwqp48C8wLc3c1hmLUXJecKkUVVZXhfVBvRd2SdF5cBIqK608C6Tlee1 [email protected]\",\"user_managed_networking\":true,\"vip_dhcp_allocation\":false}" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).RegisterClusterInternal" file="/src/internal/bminventory/inventory.go:511" cluster_id=712cd40a-5ef3-44e8-a420-6478536d25c0 go-id=2367 pkg=Inventory request_id=c5b8c970-8aaf-4a2c-82de-4b44dc655d94
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=info msg="Verifying cluster platform and user-managed-networking, got platform=none and userManagedNetworking=true" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).setDefaultRegisterClusterParams" file="/src/internal/bminventory/inventory.go:333" go-id=2367 pkg=Inventory request_id=c5b8c970-8aaf-4a2c-82de-4b44dc655d94
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=info msg="Cluster high-availability-mode is set to None, setting platform type to none and user-managed-networking to true" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).setDefaultRegisterClusterParams" file="/src/internal/bminventory/inventory.go:342" go-id=2367 pkg=Inventory request_id=c5b8c970-8aaf-4a2c-82de-4b44dc655d94
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=info msg="selected cluster release image: arch=x86_64, openshiftVersion=4.17.0-0.okd-scos-2024-06-27-091238, url=registry.ci.openshift.org/origin/release-scos@sha256:881ff38766ab4e59d4e20bd49e63cccf9388f023093fd1ffe7786b0fa1dfbaa4" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).RegisterClusterInternal" file="/src/internal/bminventory/inventory.go:569" cluster_id=712cd40a-5ef3-44e8-a420-6478536d25c0 go-id=2367 pkg=Inventory request_id=c5b8c970-8aaf-4a2c-82de-4b44dc655d94
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc001bd1290 cluster_id=712cd40a-5ef3-44e8-a420-6478536d25c0 cluster_version=0xc001bd1280 pkg=Inventory support_level_feature="Openshift OVN"
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc001bd1360 cluster_id=712cd40a-5ef3-44e8-a420-6478536d25c0 cluster_version=0xc001bd1350 pkg=Inventory support_level_feature="Skip MCO reboot"
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc001bd1430 cluster_id=712cd40a-5ef3-44e8-a420-6478536d25c0 cluster_version=0xc001bd1420 pkg=Inventory support_level_feature="User Managed Networking"
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc001bd1500 cluster_id=712cd40a-5ef3-44e8-a420-6478536d25c0 cluster_version=0xc001bd14f0 pkg=Inventory support_level_feature="None Platform Integration"
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc001bd15d0 cluster_id=712cd40a-5ef3-44e8-a420-6478536d25c0 cluster_version=0xc001bd15c0 pkg=Inventory support_level_feature="Single Node OpenShift"
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=info msg="feature does not implement SupportLevelFeatureValidator, skipping validation" func=github.com/openshift/assisted-service/internal/featuresupport.ValidateActiveFeatures file="/src/internal/featuresupport/common.go:44" arch=0xc001bd16a0 cluster_id=712cd40a-5ef3-44e8-a420-6478536d25c0 cluster_version=0xc001bd1690 pkg=Inventory support_level_feature="Platform managed networking"
Jul 04 16:30:49 okd-sno service[2776]: time="2024-07-04T16:30:49Z" level=error msg="Failed to registered cluster okd-sno with id 712cd40a-5ef3-44e8-a420-6478536d25c0" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).RegisterClusterInternal.func1" file="/src/internal/bminventory/inventory.go:535" cluster_id=712cd40a-5ef3-44e8-a420-6478536d25c0 error="pull secret for new cluster is invalid: pull secret must contain auth for \"registry.ci.openshift.org\"" go-id=2367 pkg=Inventory request_id=c5b8c970-8aaf-4a2c-82de-4b44dc655d94
Jul 04 16:30:49 okd-sno agent-register-cluster[12524]: time="2024-07-04T16:30:49Z" level=fatal msg="Failed to register cluster with assisted-service: AssistedServiceError Code: 400 Href:  ID: 400 Kind: Error Reason: pull secret for new cluster is invalid: pull secret must contain auth for \"registry.ci.openshift.org\""
Jul 04 16:30:49 okd-sno podman[12490]: time="2024-07-04T16:30:49Z" level=fatal msg="Failed to register cluster with assisted-service: AssistedServiceError Code: 400 Href:  ID: 400 Kind: Error Reason: pull secret for new cluster is invalid: pull secret must contain auth for \"registry.ci.openshift.org\""
Jul 04 16:30:49 okd-sno systemd[1]: libpod-0d440b5f976a2112e51da53004f10eed1c19c44997ab60dbad9a01776fbfffec.scope: Deactivated successfully.
Jul 04 16:30:49 okd-sno podman[12490]: 2024-07-04 16:30:49.492679016 +0000 UTC m=+2.807777396 container died 0d440b5f976a2112e51da53004f10eed1c19c44997ab60dbad9a01776fbfffec (image=registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-27-091238@sha256:6799d75859b42efc2fbf471a168512cbe3853558d53bfce4c67b931c3b012a5b, name=agent-register-cluster, io.openshift.build.commit.id=6ff36b21a63599b9781686107c3f2d4ebeeef07a, io.openshift.ci.from.origin_scos-4.17_base-stream9=sha256:8b37063a013d4cff4a80a45520852f69ca72513454a8da34df346b9166ea4f9c, io.k8s.display-name=OpenShift Base RHEL9, io.buildah.version=1.32.2, vcs-url=https://github.com/openshift/assisted-service, io.openshift.build.commit.message=, io.openshift.build.source-context-dir=, vcs-ref=6ff36b21a63599b9781686107c3f2d4ebeeef07a, io.k8s.description=This is the base image from which all OpenShift RHEL9 images inherit., io.openshift.build.commit.date=, io.openshift.ci.from.pre-repo-base-stream9=sha256:cb6ebe1964a863618478b887fc4cd134b46afdac118dab77e3764b40bf825d33, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, PODMAN_SYSTEMD_UNIT=agent-register-cluster.service, org.label-schema.license=GPLv2, vcs-type=git, io.openshift.build.commit.ref=master, io.openshift.build.source-location=https://github.com/openshift/assisted-service, io.openshift.build.commit.author=, io.openshift.build.name=, io.openshift.ci.from.ocp_builder_stream9=sha256:0e4c909afa1eda8e63001acdc6fb388be07b2a33bfee036b4487db33296144aa, io.openshift.release.operator=true, io.openshift.build.namespace=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image)
Jul 04 16:30:49 okd-sno systemd[1]: libpod-0d440b5f976a2112e51da53004f10eed1c19c44997ab60dbad9a01776fbfffec.scope: Consumed 2.526s CPU time.
Jul 04 16:30:49 okd-sno systemd[1]: var-lib-containers-storage-overlay-18e1bdcb079e9bd97527e3fb3ced70a3ff82d7933bc4043341e83f920a559c80-merged.mount: Deactivated successfully.
Jul 04 16:30:49 okd-sno podman[12587]: 2024-07-04 16:30:49.574595853 +0000 UTC m=+0.070167766 container remove 0d440b5f976a2112e51da53004f10eed1c19c44997ab60dbad9a01776fbfffec (image=registry.ci.openshift.org/origin/4.17-okd-scos-2024-06-27-091238@sha256:6799d75859b42efc2fbf471a168512cbe3853558d53bfce4c67b931c3b012a5b, name=agent-register-cluster, pod_id=55982eee506a7948d600cecdff5639b10c2733adfe16d8f1f05e9107f354fbd9, io.openshift.ci.from.origin_scos-4.17_base-stream9=sha256:8b37063a013d4cff4a80a45520852f69ca72513454a8da34df346b9166ea4f9c, io.openshift.build.commit.message=, io.openshift.build.source-context-dir=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.openshift.ci.from.pre-repo-base-stream9=sha256:cb6ebe1964a863618478b887fc4cd134b46afdac118dab77e3764b40bf825d33, vcs-type=git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, io.openshift.build.commit.date=, io.openshift.build.commit.ref=master, PODMAN_SYSTEMD_UNIT=agent-register-cluster.service, io.openshift.build.commit.author=, io.openshift.build.source-location=https://github.com/openshift/assisted-service, io.openshift.build.namespace=, io.openshift.release.operator=true, io.openshift.ci.from.ocp_builder_stream9=sha256:0e4c909afa1eda8e63001acdc6fb388be07b2a33bfee036b4487db33296144aa, io.buildah.version=1.32.2, io.openshift.build.commit.id=6ff36b21a63599b9781686107c3f2d4ebeeef07a, io.openshift.build.name=, vcs-url=https://github.com/openshift/assisted-service, io.k8s.display-name=OpenShift Base RHEL9, vcs-ref=6ff36b21a63599b9781686107c3f2d4ebeeef07a, io.k8s.description=This is the base image from which all OpenShift RHEL9 images inherit.)
Jul 04 16:30:49 okd-sno systemd[1]: agent-register-cluster.service: Main process exited, code=exited, status=1/FAILURE
Jul 04 16:30:49 okd-sno systemd[1]: agent-register-cluster.service: Ignoring invalid environment assignment '2bvY7J4Wq6pGIUAOWlxgsYs67r04sz3ygBM6EHqkaGEOtr3Yj92Em1bKCA==': /usr/local/share/assisted-service/assisted-service.env
Jul 04 16:30:49 okd-sno systemd[1]: agent-register-cluster.service: Ignoring invalid environment assignment '7r04sz3ygBM6EHqkaGEOtr3Yj92Em1bKCA==': /usr/local/share/assisted-service/assisted-service.env
Jul 04 16:30:49 okd-sno systemd[1]: agent-register-cluster.service: Failed with result 'exit-code'.
Jul 04 16:30:49 okd-sno systemd[1]: Failed to start Service that registers the cluster.
Jul 04 16:30:49 okd-sno systemd[1]: agent-register-infraenv.service: Ignoring invalid environment assignment '2bvY7J4Wq6pGIUAOWlxgsYs67r04sz3ygBM6EHqkaGEOtr3Yj92Em1bKCA==': /usr/local/share/assisted-service/assisted-service.env
Jul 04 16:30:49 okd-sno systemd[1]: agent-register-infraenv.service: Ignoring invalid environment assignment '7r04sz3ygBM6EHqkaGEOtr3Yj92Em1bKCA==': /usr/local/share/assisted-service/assisted-service.env
Jul 04 16:30:49 okd-sno systemd[1]: Starting Service that registers the infraenv...
Jul 04 16:30:49 okd-sno systemd[1]: agent-register-infraenv.service: Ignoring invalid environment assignment '2bvY7J4Wq6pGIUAOWlxgsYs67r04sz3ygBM6EHqkaGEOtr3Yj92Em1bKCA==': /usr/local/share/assisted-service/assisted-service.env
Jul 04 16:30:49 okd-sno systemd[1]: agent-register-infraenv.service: Ignoring invalid environment assignment '7r04sz3ygBM6EHqkaGEOtr3Yj92Em1bKCA==': /usr/local/share/assisted-service.env

So the "agent-register-infraenv.service" fails to start because it needs a pull secret for "registry.ci.openshift.org"?

level=fatal msg="Failed to register cluster with assisted-service: AssistedServiceError Code: 400 Href:  ID: 400 Kind: Error Reason: pull secret for new cluster is invalid: pull secret must contain auth for \"registry.ci.openshift.org\""

Adding the following inassisted-service.envfile does not change anything. Either

PUBLIC_CONTAINER_REGISTRIES=registry.ci.openshift.org
PUBLIC_CONTAINER_REGISTRIES=quay.io,registry.ci.openshift.org

Also, it seems the service is not able to parse variablesEC_PUBLIC_KEY_PEM(?) andEC_PRIVATE_KEY_PEM(?) from its own generated (?) /usr/local/share/assisted-service/assisted-service.env file?

RemovingEC_PUBLIC_KEY_PEMandEC_PRIVATE_KEY_PEMvalues fromassisted-service.envfile maked the warning disappear but the install still fails

Original content of the/usr/local/share/assisted-service/assisted-service.envfile on the node

ssh core@okd-sno
sudo su -
more /usr/local/share/assisted-service/assisted-service.env

AUTH_TYPE=none
DB_HOST=127.0.0.1
DB_NAME=installer
DB_PASS=admin
DB_PORT=5432
DB_USER=admin
DEPLOY_TARGET=onprem
DISK_ENCRYPTION_SUPPORT=true
DUMMY_IGNITION=false
ENABLE_SINGLE_NODE_DNSMASQ=true
EPHEMERAL_INSTALLER_CLUSTER_TLS_CERTS_OVERRIDE_DIR=/opt/agent/tls
HW_VALIDATOR_REQUIREMENTS=[{"version":"default","master":{"cpu_cores":4,"ram_mib":16384,"disk_size_gb":100,"installation_disk_speed_threshold_ms":10,"network_latency_threshold_ms":      100,"packet_loss_percentage":0},"worker":{"cpu_cores":2,"ram_mib":8192,"disk_size_gb":100,"installation_disk_speed_threshold_ms":10,"network_latency_threshold_ms":1000,"packet_loss      _percentage":10},"sno":{"cpu_cores":8,"ram_mib":16384,"disk_size_gb":100,"installation_disk_speed_threshold_ms":10}}]
INSTALL_INVOKER=agent-installer
IPV6_SUPPORT=true
NTP_DEFAULT_SERVER=
PUBLIC_CONTAINER_REGISTRIES=quay.io
RELEASE_IMAGES=[{"openshift_version":"4.17.0-0.okd-scos-2024-06-27-091238","cpu_architecture":"x86_64","cpu_architectures":["x86_64"],"url":"registry.ci.openshift.org/origin/releas      e-scos@sha256:881ff38766ab4e59d4e20bd49e63cccf9388f023093fd1ffe7786b0fa1dfbaa4","version":"4.17.0-0.okd-scos-2024-06-27-091238"}]
OPENSHIFT_INSTALL_RELEASE_IMAGE_MIRROR=
STORAGE=filesystem
INFRA_ENV_ID=bf0b49ad-1ded-4942-b772-d2ea043a613e
EC_PUBLIC_KEY_PEM=-----BEGIN EC PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEaVwxvVynsQflYVXAb192gx/4C/aw
2bvY7J4Wq6pGIUAOWlxgsYs67r04sz3ygBM6EHqkaGEOtr3Yj92Em1bKCA==
-----END EC PUBLIC KEY-----

EC_PRIVATE_KEY_PEM=-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIGLxQcDfQ075dVIBXlJbdnoEHWV84f63c6iHkM6oqjgDoAoGCCqGSM49
AwEHoUQDQgAEaVwxvVynsQflYVXAb192gx/4C/aw2bvY7J4Wq6pGIUAOWlxgsYs6
7r04sz3ygBM6EHqkaGEOtr3Yj92Em1bKCA==
-----END EC PRIVATE KEY-----

[JaimeMagiera]

Thank you. We'll surface this.

[izderadicka]

Similar issue here - trying to install SNO with assisted Installer with config from this file:
https://github.com/openshift/assisted-service/blob/master/deploy/podman/okd-configmap.yml

Here OKD version is older - 4.16 , but got same error - first I tried real pull secret generated at RH Hybrid Cloud Console , then dummy pull secret as in this case, none worked.

I'm rather new to OKD installation, hoped that assisted installer could be easiest way ...

[flauschi1985]

I agree with this. I also get the error "pull secret for new cluster is invalid: pull secret must contain auth for "registry.ci.openshift.org"" with the variant "Installing single-node OKD using the Assisted Service".

[flauschi1985]

Now it seems to be working again. There are no more problems with the latest version.

I use Podman and Fedora to start the pods for the "Assisted Service".

[titou10titou10]

Same failure with v4.17.0-0.okd-scos-2024-07-16-072426 that contains openshift/assisted-service#6552 that "seems" to address this problem

[danmanor]

@titou10titou10 @izderadicka @flauschi1985 Thank you for bringing it up. The reason you see this behavior is because of the way assisted-service validates pull-secret - basically going over all images in RELEASE_IMAGES which are not in PUBLIC_CONTAINER_REGISTRIES and tries to find a matching token in the user's pull secret. If not found, the service returns this validation failure you see. Assisted-service does this validation on registries even if the images are public.

@titou10titou10 Notice that the images specified in PUBLIC_CONTAINER_REGISTRIES should be comma separated (no whitespace).
I believe specifying PUBLIC_CONTAINER_REGISTRIES=quay.io,registry.ci.openshift.org should resolve the issue

[titou10titou10]

@titou10titou10 Notice that the images specified in PUBLIC_CONTAINER_REGISTRIES should be comma separated (no whitespace). I believe specifying PUBLIC_CONTAINER_REGISTRIES=quay.io,registry.ci.openshift.org should resolve the issue

It did not. Check my initial post, I tried many variations of it in the generated"/usr/local/share/assisted-service/assisted-service.env"file on the node without success (without space, it was a typo):

PUBLIC_CONTAINER_REGISTRIES=registry.ci.openshift.org
PUBLIC_CONTAINER_REGISTRIES=quay.io,registry.ci.openshift.org

As the agent pod if being destroyed/recreated it takes the new values each time
Wrong! see below

But maybe it is not the right place to set this variable? I do not have access to anassisted-service.envfile before ssh-ing in the node and did not even know the existence of it before

[titou10titou10]

I got it to go further !
Manually changingPUBLIC_CONTAINER_REGISTRIESin/usr/local/share/assisted-service/assisted-service.envis not sufficient
The service has to be restarted after the change:

ssh core@okd-sno
sudo su -
vi /usr/local/share/assisted-service/assisted-service.env
> change PUBLIC_CONTAINER_REGISTRIES=quay.io,registry.ci.openshift.org

systemctl restart assisted-service-pod.service

Verify that the env variables are correct for the assisted-service pod:

podman pod ls
podman kube generate <pod of of assisted-service>

[ArthurVardevanyan]

This is the workaround I went with that seems to work, I cache stable builds, so I just added an entry for the CIs to see if it would skip the check:

In the install Config

ImageDigestSources:
  - mirrors:
      - <REGISTRY>/openshift
    source: quay.io/openshift
  - mirrors:
      - <REGISTRY>/okd
    source: quay.io/okd
  - mirrors:
      - <REGISTRY>/origin
    source: registry.ci.openshift.org/origin