How to properly init node network identities / primary address

[Xnyle]

TLDR;

apply-config --insecure should also accept -e and -n, not only -n

-n some.internal.ip.that.does.not.yet.exist -e insecure.endpoint.that.is.never.used.again

After some debugging it basically comes down to this:

Nodes are unhealty/do not boot up cause they are (automatically) configured to use the wrong interface.

I already disabled external registries and manually configured ip networks at several points in the config:

machine:
  nodeAnnotations:
    networking.talos.dev/self-ips: 172.20.20.1
  certSANs:
    - 172.20.0.0/16
  kubelet:
    nodeIP:
      validSubnets:
        - 172.20.0.0/16
cluster:
  controlPlane:
    endpoint: https://172.20.20.1:6443
  discovery:
    enabled: true
    registries:
      kubernetes:
        disabled: false
      service:
        disabled: true
        endpoint: https://do.not.phone.home/
  apiServer:    
    certSANs:
      - 172.20.0.0/16
  etcd:
    advertisedSubnets:
      - 172.20.0.0/16

But this all still does not help, I can't get rid of the unwanted external node ip which is used as primary id:

On a fresh 1 node install with blocked firewall access to external ip (after init) health shows the following:

discovered nodes: ["9.10.11.12"]
[...]
waiting for all k8s nodes to report: can't find expected node with IPs ["9.10.11.12" "172.20.20.1"]

This is already wrong, why did the apply-config --insecure followed by a bootstrap with the above config snippets create a node with an address from a different network interface? How do I suppress that behavior?

I need the node to identify as 172.20.20.1 in that case. And then I suspect bootstrap would continue.

Question: How do i bootstrap a node via external ip but do NOT set that ip/interface as node / endpoint identity (cause it will be immediately blocked via firewall config)

The config successfully sets up wg networking (without / independent of kubespan).

So all networking / node identities must be from that wg network after initial config application.

---

Same problem on an already running 2 node cluster after blocking network access to external IP(s):

waiting for etcd members to be control plane nodes: etcd member ips ["172.20.20.1" "5.6.7.8"] are not subset of control plane node ips ["1.2.3.4" "172.20.20.1"]

172.20.20.1 is the correct ip for node1, node 2 would be 172.20.20.2.
The external interfaces should not be in there, they are only used during initial, insecure apply

Question: How do I get rid of those external IPs (1.2.3.4, 5.6.7.8) in all required configs that came in via some external registry lookup (in case of node2) and initial insecure endpoint config (node1).

[smira]

Talos has no way to guess which of the addresses (if you have many) should be used to communicate across nodes - depending on the environment specifics it might be a public or private IP.

There is not feature today to set the "main" IP.

But all commands which try to discover addresses automatically have a better, fixed way to set the expected cluster topology (vs. the discovered one which might be incorrect, e.g. missing a node), see #10848 (comment)

[Xnyle]

This question is not about the health command, it is about the cluster not being able to boot up.
It's nice that you hinted at the options mentioned but that doesn't change anything about the bootstrap problem.

"Talos has no way to guess"

Thats my question, how do i make talos NOT guess, that's the problem I have, that it apparently does that!

"You need to start using hardcoded IPs for your clusters"

Again, I'm trying exactly that. But I need initial access via another interface. And then Talos stuffs the address from that interface "somewhere" and afterwards tries to connect etcd on that interface (which is then forbidden)

[smira]

See https://www.talos.dev/v1.10/introduction/prodnotes/#multihoming

[Xnyle]

Did you take a look at the config provided above?
The topics mentioned in https://www.talos.dev/v1.10/introduction/prodnotes/#multihoming are covered in my config, still during bootstrap the external IP address is being tried (and failed)

[smira]

If you have a specific issue, bug, create an issue with targeted details, a way to reproduce, and a support bundle.