Image pull from factory.talos.dev does not work on Talos node

[aeleks]

Hi,

my Talos OS can pull docker images normally from any registry, but factory.talos.dev and I don't know how to debug the issue further. My laptop can docker pull the exact image.

Talos:
`
➜ talosctl images pull nginx -n node01.prod.gallifrey.local

➜ talosctl images pull factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1 -n node01.prod.gallifrey.local
Laptop:
➜ docker pull factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1
v1.10.1: Pulling from metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68
Digest: sha256:c2f263a4773ad32ec4a6743b7ecf898dad50625ac1c86111f78fc0d0b374dd97
Status: Image is up to date for factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1
factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1
`

When using this image with

➜ talosctl upgrade -n node01.prod.gallifrey.local --image factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1 --force

the dashboard shows

user: warning: [2025-05-09T21:36:07.039181137Z]: level=info msg=fetch failed error=failed to do request: Head "https://images.factory.talos.dev/v2/image-factory/metal-installer/ def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68/manifests/sha256:c2f263a4773ad32ec4a6743b7ecf898dad50625ac1c86111f78fc0d0b374dd97": net/http: TLS handshake timeout host=factory.talos.dev image=factory.talos.dev/metal-installer/def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1 user: warning: [2025-05-09T21:36:07.042248137Z]: [talos] retrying error: failed to pull image "factory.talos.dev/metal-installer/ def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1": failed to resolve reference "factory.talos.dev/metal-installer/ def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68:v1.10.1": failed to do request: Head "https://images.factory.talos.dev/v2/image-factory/metal-installer/ def336ea9dcf27176acacc021de46490035de9d7750fd260806e0a6d8d7a6f68/manifests/sha256:c2f263a4773ad32ec4a6743b7ecf898dad50625ac1c86111f78fc0d0b374dd97": net/http: TLS handshake timeout

Which does't make any sense, because my Kubernetes pods can resolve the domain, and Talos and my Laptop use the same dns servers (1 local hosted - which can actually resolve this domain, but also 8.8.8.8)

Do you have any further clues or an idea what else I can provide to you?

Thank you in advance!

[aeleks]

Background why I want that: I have an intel cpu with intel arc and i want to use hardware transcoding with the intel i915 extension

[rothgar]

Was this the first time you tried to pull that image? Images are built by the image factory and then cached in a CDN but sometimes the first pull times out.

I was able to pull the image from my computer, but if you are getting repeat errors it would be something to look into.

[aeleks]

Thank you for your quick response!

I have tried it multiple times (5-6 times) distributed over about 1 hour, since I really wanted it to work.

I also tried it right now 3 times, for about 10 minutes. Same error I am afraid.

[rothgar]

TCP Retransmissions happen when the sender (your node) doesn't receive an ACK for a packet. In this case the problem is the SNI packet isn't being received by the server which is likely due to a firewall or MTU fragmentation issue. Because it works from within the cluster I assume it's not a firewall issue and probably an issue with a hypervisor or MTU.

Where are you running the node? You mention you're pulling the metal image, but I'm not sure if you're running on bare metal or on a hypervisor.
Can you also let me know what CNI you're using?

[aeleks]

I really appreciate your help, btw!

I am running bare-metal (it is an Intel NUC) without any Hypervisor involved.

Currently running on cilium-1.17.3 with the following values:

➜ helm get values cilium -n kube-system
USER-SUPPLIED VALUES:
cgroup:
  autoMount:
    enabled: false
  hostRoot: /sys/fs/cgroup
ingressController:
  default: true
  enabled: true
  enforceHttps: false
  loadbalancerMode: shared
  service:
    annotations:
      lbipam.cilium.io/sharing-cross-namespace: '*'
      lbipam.cilium.io/sharing-key: gallifrey-prod
ipam:
  mode: kubernetes
k8sServiceHost: kube-apiserver.prod.gallifrey.local
k8sServicePort: 6443
kubeProxyReplacement: true
operator:
  replicas: 1
securityContext:
  capabilities:
    ciliumAgent:
    - CHOWN
    - KILL
    - NET_ADMIN
    - NET_RAW
    - IPC_LOCK
    - SYS_ADMIN
    - SYS_RESOURCE
    - DAC_OVERRIDE
    - FOWNER
    - SETGID
    - SETUID
    cleanCiliumState:
    - NET_ADMIN
    - SYS_ADMIN
    - SYS_RESOURCE

Although I don't think that is an issue - but not best practice ^^ - it is a single node cluster.

It seems weird that i can pull a generic nginx image, but not any image from factory.talos.dev

[rothgar]

Because you're replacing the kube-proxy and unless you disabled KubePrism you should use localhost for the k8sServiceHost and port 7445 for the k8sServicePort. See the documentation at https://www.talos.dev/v1.10/kubernetes-guides/network/deploying-cilium/#without-kube-proxy

I would try without replacing kube-proxy first and see if it works.

[aeleks]

Ah I didn't notice, thank you. I will try different settings and let you know the outcome.

[aeleks]

The issue persists. I tried both:

1. Fixing k8sServiceHost and port to what the documentation says
2. re-enabling kube-proxy and disabeling kube proxy replacement on cilium

both did not work - which would surprise me given that i can pull from other repos. I will investigate the issue further in the following days.

Thank you for your help so far!