DNS leakage on Ubuntu 18.10.

I'm experiencing DNS leakage using this script in which the IP addresses of my ISP's DNS servers are visible. Strangely, the IP addresses of my VPN's DNS servers are also visible. I'm running an OpenVPN server connected to a Comcast router.

From DNS Leak Test:

IP	Hostname	ISP
208.67.219.70	m41.pao.opendns.com	OpenDNS, LLC
208.67.219.14	m4.pao.opendns.com	OpenDNS, LLC
76.96.15.73	sjos-cns05.nlb.sjc1.comcast.net	Comcast Cable
208.67.219.29	m21.pao.opendns.com	OpenDNS, LLC
...	...	...

systemd-resolve --status reports that everything is fine with the tunnel, so I assume I'm leaking traffic to some other interface:

Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 4 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 208.67.220.220
         DNS Servers: 208.67.222.222
                      208.67.220.220
          DNS Domain: ~.

Link 3 (wlp4s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 75.75.76.76
         DNS Servers: 75.75.75.75
                      75.75.76.76
                      2001:558:feed::1
                      2001:558:feed::2
          DNS Domain: ~.
                      hsd1.ca.comcast.net

Link 2 (enp0s31f6)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

OpenVPN client configuration:

...
script-security 2
setenv PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
up /etc/openvpn/update-systemd-resolved
down /etc/openvpn/update-systemd-resolved
down-pre

# Prevent DNS leakage.
dhcp-option DOMAIN-ROUTE .

I know very little about networking, so I apologize if this is a trivial issue. Any insight would be greatly appreciated.

[piotr-dobrogost]

I guess it is due to DNS Domain: ~. set on link wlp4s0 (wireless I guess).
Is there any reason for this setting?
If not then removing it should not break anything and prevent leakage here.

Thanks for the reply @piotr-dobrogost.

I'm not sure what the DNS Domain: ~. directive actually does, but it was referenced in this comment to specify that the VPN's DNS servers should take priority.

I don't know why DNS Domain: ~. appears in the wireless interface when the VPN is not running and when the VPN is run through the OpenVPN client using sudo openvpn --config ./client.ovpn. When run using NetworkManager, there is no such directive and no DNS leakage:

Global
       LLMNR setting: no
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 14 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 208.67.222.222
         DNS Servers: 208.67.222.222
                      208.67.220.220
          DNS Domain: ~.

Link 3 (wlp4s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 75.75.75.75

This doesn't seem like a problem with update-systemd-resolved.

[jonathanio]

DNS Domain: ~. is the search configuration - especially systemd-resolved has been told that the servers configured on this link are responsible for searches on all domains (. is in effect the root domain, so all domains will match). Your tun0 interface has the same search query, so you will be sending out requests to via interfaces to both sets of DNS servers and the first successful response back will be the one provided to your application.

My current configuration is:

Global
       LLMNR setting: yes
MulticastDNS setting: yes
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: no
Fallback DNS Servers: 1.1.1.1
                      9.9.9.10
                      8.8.8.8
                      2606:4700:4700::1111
                      2620:fe::10
                      2001:4860:4860::8888
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 4 (roadwarrior)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes

Link 3 (enp12s0u1u2)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes
  Current DNS Server: 2001:4860:4860::8844
         DNS Servers: 8.8.8.8
                      8.8.4.4
                      2001:4860:4860::8844
                      2001:4860:4860::8888

Link 2 (wlp2s0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 8.8.8.8
         DNS Servers: 8.8.8.8
                      8.8.4.4

As you can see, I do not have any DNS Domain configuration by default, so when I enable a VPN with DOMAIN-ROUTE . it will take over. Something on your system is overriding this configuration for some reason. What do you use to set up your networking configuration on your system?

[piotr-dobrogost]

I don't know why DNS Domain: ~. appears in the wireless interface when the VPN is not running and when the VPN is run through the OpenVPN client using sudo openvpn --config ./client.ovpn. When run using NetworkManager, there is no such directive and no DNS leakage:

That's interesting as it's Network Manager which likes to set ~. according to systemd/systemd#10125

I haven't manually configured systemd-resolved in /etc/systemd/resolved.conf or any other place. file /etc/resolv.conf reports /etc/resolv.conf: symbolic link to ../run/systemd/resolve/stub-resolv.conf, so direct DNS requests should be routed to systemd-resolved's DNS stub. /etc/resolv.conf/ contains the following:

nameserver 127.0.0.53
search hsd1.ca.comcast.net

Thank you for explaining the purpose of the DNS Domain: ~. directive. I don't want to take any more of your time because this is unrelated to update-systemd-resolved. Is there a more appropriate place for this question, perhaps Networking Stack Exchange or systemd?

[jonathanio]

I have never been a fan of NetworkManager - much like systemd at times, it tries to do too much with a number of questionable decisions under the surface which often result in difficult to detect and resolve failure modes. There is, as in this example, absolutely no reason to demand for systemd-resolved.service that all DNS queries should be always directed to the local DNS server on the LAN when other, better, configurations may be provided.

Ultimately, and as you suspected, the reason why it works in your case with dhcp-option DOMAIN-ROUTE . on your VPN connection is that systemd, when faced with multiple possible options to route a query (explicit, or implicit), will send the same query to all servers. The result that is returned is either the first successful response (which can lead to some unpredictability without proper routing) or the last failed response (i.e. no server could resolve). With both links configured for ~. you'll probably always get one failure (LAN) and one success (VPN), so the success will be returned.

I'm uncertain how to remove ~. from your LAN interface, but ultimately this will help you if you can manage this configuration.

[bohlstry]

If have written a Bash-Script to get rid of ~. from all non-VPN-devices, if VPN is active. If VPN goes down, ~. is written to the default device (identified by the default route).
Important: as mentioned above, dhcp-option DOMAIN-ROUTE . is required in VPN config.

Save the following code at:
/etc/NetworkManager/dispatcher.d/10-check-dns-domain, don't forget to give execute rights.

#!/bin/bash

# Process only TUN (i.e. VPN) devices!

if [[ $1 == tun* ]]; then

  case $2 in

    up)
      logger -t "check-dns-domain" "TUN up: $1"
      # Remove from all but the current TUN interface the default domain search flag '~.'...
      for device in $(nmcli -t device | cut -d: -f1 | sort | grep -vx "$1"); do
        # Get current domain entries...
        domain=$(resolvectl domain "$device" | cut -d: -f2 | sed -e 's/^\s//')
        # Check for an '~.' entry (surrounded by spaces or at begin/end of string)...
        if echo "$domain" | grep -qE '(^|\s)~\.(\s|$)'; then
          # Remove '~.', trim whitespace...
          domain=$(echo "$domain" | sed -E -e 's/(^|\s)~\.(\s|$)/ /g' -e 's/^\s+//' -e 's/\s+$//')
          logger -t "check-dns-domain" "Setting domain for '$device': '$domain'"
          if [ -z "$domain" ]; then
            # Domain is now empty...
            resolvectl domain "$device" ""
          else
            # Set new domain value(s). Hint: don't quote last argument, we need individual arguments...
            resolvectl domain "$device" $domain
          fi
        fi
      done
      ;;

    down)
      logger -t "check-dns-domain" "TUN down: $1"
      # Get device name of default route...
      device=$(ip route ls | perl -ne 'do {print $1; last} if /^default\s.*\sdev\s(\S+)/')
      # Get current domain entries...
      domain=$(resolvectl domain "$device" | cut -d: -f2 | sed -e 's/^\s//')
      # Append '~.', separated by space. If domain was empty, just set value (without space)...
      domain=${domain:+$domain }~.
      logger -t "check-dns-domain" "Setting domain for '$device': '$domain'"
      # Set new domain value(s). Hint: don't quote last argument, we need individual arguments...
      resolvectl domain "$device" $domain
      ;;

  esac

fi

To check the log, you can use journalctl -f -t check-dns-domain

[dannyk81]

Thanks @bohlstry for sharing the script, works great for me.

I tried every possible way to get rid of ~. from non-VPN (Wifi) device but it keeps coming back, I suppose by NetworkManager (?)

Using this method seems like the only way to fix DNS leakage...

EDIT: The script handles the case of a single VPN well, however I'm normally connected to multiple VPNs, so when I disconnected from just one it pushes back ~.

EDIT 2: slightly revised version of @bohlstry's script, works in case there are multiple VPN connections:

#!/bin/bash

function remove_default_search_flag() {
  # Remove from all but the current TUN interface the default domain search flag '~.'...
  for device in $(nmcli -t device | cut -d: -f1 | sort | grep -vx "$1"); do
    # Get current domain entries...
    domain=$(resolvectl domain "$device" | cut -d: -f2 | sed -e 's/^\s//')
    # Check for an '~.' entry (surrounded by spaces or at begin/end of string)...
    if echo "$domain" | grep -qE '(^|\s)~\.(\s|$)'; then
      # Remove '~.', trim whitespace...
      domain=$(echo "$domain" | sed -E -e 's/(^|\s)~\.(\s|$)/ /g' -e 's/^\s+//' -e 's/\s+$//')
      logger -t "check-dns-domain" "Setting domain for '$device': '$domain'"
      if [ -z "$domain" ]; then
        # Domain is now empty...
        resolvectl domain "$device" ""
      else
        # Set new domain value(s). Hint: don't quote last argument, we need individual arguments...
        resolvectl domain "$device" $domain
      fi
    fi
  done
}

function add_default_search_flag() {
  # Get device name of default route...
  device=$(ip route ls | perl -ne 'do {print $1; last} if /^default\s.*\sdev\s(\S+)/')
  # Get current domain entries...
  domain=$(resolvectl domain "$device" | cut -d: -f2 | sed -e 's/^\s//')
  # Append '~.', separated by space. If domain was empty, just set value (without space)...
  domain=${domain:+$domain }~.
  logger -t "check-dns-domain" "Setting domain for '$device': '$domain'"
  # Set new domain value(s). Hint: don't quote last argument, we need individual arguments...
  resolvectl domain "$device" $domain
}

# Process only TUN (i.e. VPN) devices!
if [[ $1 == tun* ]]; then

  case $2 in

    up)
      logger -t "check-dns-domain" "TUN up: $1"
      remove_default_search_flag "$1"
      ;;

    down)
      logger -t "check-dns-domain" "TUN down: $1"
      # Check if there are other active TUN devices
      active_tuns=$(nmcli connection show --active|grep -q tun && echo 0 || echo 1)
      if [ $active_tuns -eq 0 ]; then
        logger -t "check-dns-domain" "Other TUN devices are active"
        remove_default_search_flag "$1"
      else
        logger -t "check-dns-domain" "No additional active TUN devices"
        add_default_search_flag
      fi
      ;;

  esac

fi

[Edu4rdSHL]

Hello, you just need to use the following option in your .network file:

[DHCP]
UseDNS=no

The "leakage" that you're getting is because if you don't specify that option, the DNS servers received from the DHCP server will be used and take precedence over any statically configured ones.

See https://www.freedesktop.org/software/systemd/man/systemd.network.html#%5BDHCP%5D%20Section%20Options

[jonathanio]

@bohlstry and @dannyk81,

Thank you both for your scripts. It's still a shame that NetworkManager has not looked into this and doesn't seem to be any closer to resolving this regression. I'm much more a fan of systemd-networkd, despite even its issues at times.

Nonetheless, I'll keep this open waiting for a resolution from NetworkManager. In the meantime, I've added into the documentation notes about the issue of leakage when using NetworkManager, and linked to this issue.

[Edu4rdSHL]

I don't understand what have to be network-manager in a script for systemd-resolved... network-manager provides their own DNS resolver implementation, nothing to be with systemd-resolved.

[vaskokj]

How do I go about removing the DNS Domain: ~. from my primary connection? I have no idea how that is even getting there in the first place? Is this something that my DHCP server adding this? If not, how do I go about removing it?