"cluster create" fails behind corporate proxy

[miphip]

Hi, using talosctl v 1.0.3.

> talosctl cluster create
validating CIDR and reserving IPs
generating PKI and tokens
downloading ghcr.io/siderolabs/talos:v1.0.3
creating network talos-default
creating master nodes
creating worker nodes
waiting for API
bootstrapping cluster
◳ waiting for etcd to be healthy: 1 error occurred:
    * 10.5.0.2: service "etcd" not in expected state "Running": current state [Preparing] Running pre state

From " docker logs talos-default-master-1":

[talos] 2022/04/20 08:37:29 service[etcd](Failed): Failed to run pre stage: failed to pull image "gcr.io/etcd-development/etcd:v3.5.3": 1 error(s) occurred:
	failed to pull image "gcr.io/etcd-development/etcd:v3.5.3": failed to resolve reference "gcr.io/etcd-development/etcd:v3.5.3": failed to do request: context canceled

talos-default-master-1.log

I'm behind a corporate proxy with a self-signed ca-cert. The proxy env vars are injected into docker containers through "~/.docker/config.json" but I suspect it fails because of the proxy ca-cert. How can I provide it to the containers?

[smira]

Probably the error you're seeing is before the bootstrap, and it's not the real error. You might need to wait more to see the pull error (if it fails).

I suspect you might need to have HTTP_PROXY env vars injected as well as the self-signed CA cert.

So you probably need to do:

• https://www.talos.dev/v1.0/talos-guides/configuration/certificate-authorities/
• https://www.talos.dev/v1.0/reference/configuration/ and machine.env

You can create a config patch which looks like that:

- op: add
  path: /machine/env
  value:
      http_proxy: ...
      https_proxy: ...
- op: add
  path: /machine/files
  value: 
    - content: |
         -----BEGIN CERTIFICATE-----
         ...
         -----END CERTIFICATE-----        
      permissions: 0644
      path: /etc/ssl/certs/ca-certificates
      op: append 

And pass it to talosctl cluster create as --config-patch @your-patch.yaml

[smira]

So this adds to the confusion, docker-based as you asked initially doesn't do an install. And config patches work just fine.

The problem is that install phase runs before the user files are set up (as /var is actually not mounted yet). So to handle pulling an installer image you have two options:

• mirror the image somewhere local which doesn't go through the proxy, and use some kind of air-gapped scenario to redirect the image pulling (https://www.talos.dev/v1.0/advanced/air-gapped/)
• try specifying your MitM proxy certificate CA directly in the registry config: https://www.talos.dev/v1.0/reference/configuration/#registrytlsconfig

[smira]

And last but not the least, docker cp doesn't handle correctly getting files out of the container, as Talos performs bind mounts over readonly rootfs, use talosctl read to verify file contents.

[miphip]

I'm at loss.

> talosctl cluster create --config-patch @mip.yaml
validating CIDR and reserving IPs
generating PKI and tokens
creating network talos-default
creating master nodes
creating worker nodes
renamed talosconfig context "talos-default" -> "talos-default-10"
waiting for API
bootstrapping cluster

> talosctl read -n 10.5.0.2 /etc/mip
rpc error: code = Unknown desc = stat /etc/mip: no such file or directory

[frezbo]

/etc is read only and the patch would have failed when you check the console logs, use /var for updating content, regarding extra CA's talos appends the CA and then mounts as read-only

[miphip]

Ok. So config-patching ca-certificates works. I could verify that my ca-cert is appended to /etc/ssl/certs/ca-certificates using "talosctl read". Still, "cluster create" fails in the end with:

◳ waiting for etcd to be healthy: 1 error occurred:
    * 10.5.0.2: service "etcd" not in expected state "Running": current state [Preparing] Running pre state
context deadline exceeded

and the container log talos-default-master-1.log contains several similar:

failed to pull image "gcr.io/etcd-development/etcd:v3.5.3": failed to resolve reference "gcr.io/etcd-development/etcd:v3.5.3": failed to do request: Head "https://gcr.io/v2/etcd-development/etcd/manifests/v3.5.3": x509: certificate signed by unknown authority

When I curl to the same url using the ca-certificates file read from the container, it verifies the server correctly using our ca-cert ("ZScaler").

> talosctl read -n 10.5.0.2 /etc/ssl/certs/ca-certificates > ca.crt
> curl -v https://gcr.io/v2/etcd-development/etcd/manifests/v3.5.3 --cacert ca.crt
* Uses proxy env variable no_proxy == 'localhost,127.0.0.1,::1,.schneider-electric.com,schneider-electric.com,.se.com,se.com,10.0.0.0/8'
* Uses proxy env variable https_proxy == 'http://gateway.schneider.zscaler.net:9480'
*   Trying 165.225.72.40:9480...
* Connected to gateway.schneider.zscaler.net (165.225.72.40) port 9480 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to gcr.io:443
> CONNECT gcr.io:443 HTTP/1.1
> Host: gcr.io:443
> User-Agent: curl/7.82.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection Established
< Proxy-Agent: Zscaler/6.1
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: ca.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.gcr.io
*  start date: Apr 16 05:23:51 2022 GMT
*  expire date: Apr 30 05:23:51 2022 GMT
*  subjectAltName: host "gcr.io" matched cert's "gcr.io"
*  issuer: C=US; ST=California; O=Zscaler Inc.; OU=Zscaler Inc.; CN=Zscaler Intermediate Root CA (zscaler.net) (t) 
*  SSL certificate verify ok.