How to get consul-agent and systemd.resolvd to co-exist peicefully and still be able to resolve *.consul hostsnames from within docker

readme.md

I see many people struggeling to make consul-agent work with systemd.resolvd and eventually give up and go with dnsmasq or a similar approach.

Here's a reasonably simple way to make everything play nicely together.

If you found this useful, say thanks. And as much as i'd love your support via patreon, go and donate to the EFF.

Here's an exerpt from the install_consul_agent.sh that my base packer builder runs for all my systemd hosts:

# Binary is in place, secured. Deploy the systemd components to make it useful
##
echo "moving systemd components into place..."
# deploy the service file
mv systemd/consul-agent.service /etc/systemd/system/consul-agent.service
mv systemd/dummy0.netdev /etc/systemd/network/dummy0.netdev
mv systemd/dummy0.network /etc/systemd/network/dummy0.network

# Then get the interface created
echo "reconfiguring network for dummy0..."
systemctl restart systemd-networkd

# Ok, now we're ready!
echo "Attempting to bring consul up for POST..."
systemctl enable consul-agent
systemctl start consul-agent

# Confirm that consul actually came up...
if [ `systemctl is-failed consul-agent.service` == 'failed' ];
then
    echo "Consul failed to start"
    # Bail, packer should fail this build...
    exit 1
fi

It really is that simple. Systemd.resolved will happily do split-zone DNS. I think this functionality was intended for VPN users, but we can take advantage of this for our purposes. Rather than tell systemd.resolved that $host.internal.corp.com can be reached via ppp01, we tell resolvd that $host.consul can be resolved via dummy0 and bind consul-agent to dummy0.

Oh the fun to be had with virtual interfaces.... :).

dummy0.netdev

# Creates a "dummy" network interface
# we'll configure this interface with a link-local address
# See: https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
##

[NetDev]
Name=dummy0
Kind=dummy

dummy0.network

# We "find" the dummy0 interface we created and configure it with a link-local address
# See: https://www.freedesktop.org/software/systemd/man/systemd.network.html
##
[Match]
Name=dummy0

[Network]
Address=169.254.1.1/32

# We want these domains
Domains= ~consul.

# To go to the consul DNS server we'll bind to this address
DNS=169.254.1.1

network.hcl

##
# This file is one of a few in `/etc/consul.d/config` and the consul-agent binary is launched with 
#   something that looks like this in the `consul-agent.service` file:
#
#   ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/ -config-dir=/etc/consul.d/services/ ... etc
##
#   
# IN order to get split-zone DNS working properly with Consul and systemd resolver and docker,
# we have to slightly tweak the consul network configuration.
##
# The address that consul will bind to for INTER-CLUSTER COMMS.
# By default, this is "0.0.0.0", meaning Consul will bind to all addresses on the local machine and will 
#   advertise the first available private IPv4 address to the rest of the cluster.
#
# However, with the dummy0 and (possibly) docker0 interfaces, consul will find multiple private addresses
#   so we must explicitly tell consul to bind to the *actual* IP/Interface that is linked to amazon's network
#   this is almost always going to be the `eth0` equivelent on a ubuntu host.
##
# See: https://www.consul.io/docs/agent/options.html#_bind
bind_addr = "{{ GetPrivateIP }}"


# See https://www.consul.io/docs/agent/options.html#_client
##
# The address that consul will bind CLIENT SERVIDES (http/dns) to is very different from the 
#   address we want to accept clients on! As the agent is running locally, we want to accept 
#   clients *ONLY* on our link-local / dummy0 address!
#
# Luckily, this is pretty easy. Hashi has go-sockaddr which lets us write golang style template
#   statements to filter out various host network things. We'll use this shortly...
#
# See: https://godoc.org/github.com/hashicorp/go-sockaddr/template
##
# Currently, this is broken. 
# See: https://github.com/hashicorp/consul/issues/5371
# See: https://github.com/hashicorp/go-sockaddr/issues/31
#
# The fix is to hard-code the IP address. Normally, a sin punishable by death
#   but in this case, it's just a paddelin' , as it's a link local and very unlikely
#   to every need changing. I still feel like this is going to bite me one day...
# client_addr = "{{ GetInterfaceIP \"dummy0\" }}"
client_addr = "169.254.1.1"

# the ssyd.resolvd conf does not allow you to set ports, unfortunately. Consul must run on 53 for
#   clients expectations about DNS to work.
ports = {
  dns = 53
}

# Docker runtime can not use 127.* addresses for DNS when the container is on it's own network name space
# This is because the docker runtime hosts it's own resolver in the 127.X space
#
# Docker can't forward DNS queries to 127.0.0.53, but it *can* forward to 169.254.1.1.
# However, consul will not know what to do with DNS queries it gets that do not end in *.consul
#
# We fix this by telling consul to forward queries to the system resolver
# See: https://www.consul.io/docs/agent/options.html#recursors
##
# Forward non *consul domains to the systemd resolver so consul will play nice w/ docker
recursors = ["127.0.0.53"]

Awesome, thanks! 🏅

Do you happen to have that whole packer script out in the open somewhere? Your setup seems fairly similar to mine, but written in a better and more structured way. Would love to see what other tricks you've applied. 😄

Do you happen to have that whole packer script out in the open somewhere?
@sandstrom

No, it's not something that i can share. It grew from a few simple bash scripts and has gotten pretty big and somewhat messy. I keep each script small and. - if possible - at LEAST a 50/50 split between comments/docs and code. Scripts do one thing, and are named appropriately. I keep everything in either a scripts/ folder or a files folder and one of the first things that I do once packer has connected to the machine is upload the entire files/ folder to a temp location. Every script that is later executed usually starts with. a cd $WORKDIR where WORKDIR is a variable that packer passes along to the script and is set to the same place that all the files were uploaded.

@kquinsland Much appreciated, thanks!

This helped a lot, thank you!

I presume you run CONSUL as ROOT. I run consul as user consul, i tried your solution, but consul can not bind to the dummy network interface.

@iahim, you might want to add

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE

in consul unit if you want to bind to port 53

@iahim, you might want to add

[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE

in consul unit if you want to bind to port 53

Thank you! i will try it tomorrow.

Note: systemd 246 supports port in DNS (ref) - so if you use Ubuntu 20.10 (which ships systemd 246) it works without this hack.

# /etc/systemd/resolved.conf.d/00-consul.conf
[Resolve]
DNS=127.0.0.1:8600
Domains=~consul

# just restart systemd-resolved service

However, probably 20.04 probably won't get systemd 246. See https://answers.launchpad.net/ubuntu/+source/systemd/+question/692909

See the systemd version of ubuntu releases at https://packages.ubuntu.com/search?keywords=systemd

@chulkilee, thanks for the update! Glad that systemd shipped an update to enable ports. That'll significantly simplify things going forward for those that can afford to use a non LTS release.

Event if we can set DNS port for systemd-resolve in systemd 246+, ex on ubuntu 20.10+, this hack is still need for docker. Or not?
Because we can NOT set dns port when run other container. Ex:

• run consul in port 8600

$ cat /etc/systemd/resolved.conf.d/00-consul.conf
[Resolve]
DNS=172.22.101.101:8600
Domains=~consul

$ echo '{"ID": "redis1", "Name": "redis", "Tags": ["primary", "v1"], "Address": "127.0.0.1", "Port": 8000,  "Meta": {"redis_version": "4.0"}, "EnableTagOverride": false, "Weights": {"Passing": 10, "Warning": 1}}' > payload.json

$ curl -XPUT --data @payload.json http://127.0.0.1:8500/v1/agent/service/register?replace-existing-checks=true

$ dig redis.service.dc1.consul. A

; <<>> DiG 9.16.8-Ubuntu <<>> redis.service.dc1.consul. A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60229
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;redis.service.dc1.consul.	IN	A

;; ANSWER SECTION:
redis.service.dc1.consul. 0	IN	A	127.0.0.1

;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sat Jul 03 09:09:43 UTC 2021
;; MSG SIZE  rcvd: 69

$ docker run --rm -it alpine sh -c 'apk add bind-tools && dig redis.service.dc1.consul. A'

; <<>> DiG 9.16.16 <<>> redis.service.dc1.consul. A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 56936
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;redis.service.dc1.consul.	IN	A

;; AUTHORITY SECTION:
.			86400	IN	SOA	a.root-servers.net. nstld.verisign-grs.com. 2021070300 1800 900 604800 86400

;; Query time: 79 msec
;; SERVER: 10.0.2.3#53(10.0.2.3)
;; WHEN: Sat Jul 03 09:13:15 UTC 2021
;; MSG SIZE  rcvd: 128

$  docker run --rm -it alpine sh -c 'cat /etc/resolv.conf'
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 172.22.101.101
nameserver 10.0.2.3
search lan

You can see that the nameserver setting in /etc/resolv.conf in the container don't have port!