/etc/hosts does not contain container name

[ngarratt]

We're migrating some workloads from an old Nomad/Docker setup to Nomad/Podman, but seeing a mismatch between jobs invoked directly by 'podman run' and those invoked by Nomad. Both are being tested in bridge mode to the same cni-podman0 bridge.

Podman is setting the following which all work fine:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
<bridge_ip> host.containers.internal host.docker.internal
<container_ip> <container_id> <container_name>

Nomad is setting the following

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
<host_ip> host.containers.internal host.docker.internal

Setting the host.containers.internal to the host IP is causing some introspection issues when doing reverse lookup against that IP, and no entry for <container_id> causes different Java binding failures because it can't resolve the hostname to an IP.

Is there a setting in the driver (or Nomad) that influences this behavior?

OS: RHEL 8
Nomad: 1.8.1
Nomad CNI: 1.5.1
Podman: 4.9.4
nomad-driver-podman: 0.6.1

[ngarratt]

This looks to be an issue with Nomad invoking CNI directly instead of allowing podman to handle networking.

The podman/libnetwork code that populates /etc/hosts is making decisions based on 'network.Driver'. If the Nomad task is not specifying a network mode, it defaults to "nsmode":"bridge" when invoking against libpod. That works fine.

But if the Nomad task has a network mode specified, it engages CNI to build the network then uses "nsmode":"path" pointing to the netns from the CNI when invoking against libpod. This turns off all the etchost logic in podman/libnetwork for bridge networks.

In our case we can just work around it by not specifying a network mode for bridge networks, and by modifying the /etc/hosts file in our host network containers during prestart. This podman driver isn't directly building the /etc/hosts file, I'm not sure what the correct solution is.

[tomwardill-payoneer]

I've just run into this, specifically with the java binding failures.
In our case, we don't appear to be able to work around it by not specifying the mode as we also need to expose health checks, which isn't valid without an explicit mode being set.