Notes on what I had to do it setup an IPv6 only Ubuntu bionic 20.04 host.
updated: 2020-04-06 (virus times, whatcha gonna do?)
updated: 2020-10-24 (still virus times...)
2022-03-22, Posted as a Gist because it always should've been...
I doubt these are the best way. Is there a best way? IPv6-only is not an out-of-the-box configuration for Linux distros. Despite many enterprises running their intranets in this manner.
My hardware? nVidia Jetson Nano running Ubuntu bionic. Any hardware will do. Cloud providers seem to finally be getting into the IPv6 routed to your VMs game? If so, consider setting up a couple of those!
- Produce automation for this config so it can be easily adopted and applied.
- Update the
dnscrypt-proxysection for modern distros.
This was quite a challenge to figure out. The Linux kernel and systemd are
hardcoded to bring lo up with this address. (!)
Create a file called /etc/systemd/system/disable-ipv4-lo.service containing
[Unit]
Description=Disabling the IPv4 loopback lo interface.
After=network.target
After=network-online.target
[Service]
User=root
Type=oneshot
ExecStart=/sbin/ip -4 addr del 127.0.0.1/8 dev lo
[Install]
WantedBy=multi-user.target
Don't forget to sudo systemctl enable disable-ipv4-lo.
Obviously IPv4 localhost existed for a while during system boot, so some things will wound up having bound to that address. systemd on port 2947, sshd, rpcbind and avahi-daemon according to lsof on my system. I'll ignore that for now.
For a true IPv6-only experience we don't want to allow software to even be able to use an IPv4 stack for anything. We need to ferret out those evil 127.0.0.1 references as being a problem.
A corporate intranet probably keeps 127.0.0.1 and the IPv4 stack around as so much of the world's software assume it exists.
How do we connect to IPv4-only servers in the real 2020 world such as pathetic Github?
Individual protocol proxy on a local network IPv6 host supporting both protocols work, use /etc/hosts to direct those hostnames there... But this is far from ideal as you must manually configure these for all required combos. I could get away with it for a buildbot's needs... but yuck.
NAT64 is designed for this purpose. We need to use special DNS64 servers that return IPv6 addresses pointing within our NAT64's /96 gateway address for this to work. But after that it is a transparent proxy.
It by default runs a local resolver on 127.0.0.53:53, you will need to get rid of this before you make 127.0.0.1 go away if you want DNS...
https://medium.com/@niktrix/getting-rid-of-systemd-resolved-consuming-port-53-605f0234f32f
systemctl stop systemd-resolved
Edit /etc/systemd/resolved.conf to contain
[Resolve]
DNS=2001:4860:4860::6464 2001:4860:4860::64
DNSStubListener=no
DNSSEC=no
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
You'll notice above we listed two DNS64 resolvers (Google DNS64). But that isn't enough. That list is not exclusive, systemd will merge the DNS servers it hears from the IPv6 Router Advertisement and/or DHCP servers in with those. As my IPv6-only host is on a dual stack network not configured to force everyone to use DNS64 and NAT64... This is bad, it was picking up the normal DNS servers! We gotta disable that behavior:
To accomplish this, create an /etc/systemd/network/dns64.network file:
[Match]
Name=eth0
[Network]
Description=Ethernet configured for IPv6-only
DNS=2001:4860:4860::6464 2001:4860:4860::64
DNSSEC=no
[IPv6AcceptRA]
UseDNS=no
[DHCPv6]
UseDNS=no
[DHCPv4]
UseDNS=no
[route]
destination=64:ff9b::/96
scope=link
This importantly configures systemd to ignore externally provided DNS server
lists on the eth0 network interface. Substitute your interface name or other
matching criteria as appropriate. This also hardcodes the same DNS servers (I
doubt we need them listed in both files... but that's what I've got right now).
After the above you need to make sure systemd-networkd is enabled rather than
NetworkManager. (I'm on a wired host, I don't know the implications of this
change for other network configuration situations)
sudo systemctl enable systemd-networkd
sudo systemctl start systemd-networkd
sudo systemctl disable NetworkManager
sudo systemctl stop NetworkManager
TODO(gps@2020-04-06): now figure out why systemd-networkd-wait-online is
failing and timing out before network-online.target can be satisfied.
# I bet I need to tell it to ignore other network interfaces:
greg@jetson-nano:~$ sudo networkctl
IDX LINK TYPE OPERATIONAL SETUP
1 lo loopback carrier configured
2 dummy0 ether degraded configured
3 eth0 ether routable configured
4 rndis0 ether no-carrier configuring
5 usb0 ether no-carrier configuring
6 l4tbr0 ether no-carrier configured
6 links listed.
greg@jetson-nano:~$ sudo /lib/systemd/systemd-networkd-wait-online
managing: eth0
ignoring: lo
Resolved: I had to edit /etc/systemd/system/network-online.target.wants/systemd-networkd-wait-online.service
to add -i eth0 to the command line as the other interfaces are unimportant
and were getting in the way.
You need a dual-stack host on your network to run a NAT64 gateway daemon.
I chose WrapSix. There are others. An older popular one is Tayga; untested by me. Whatever you choose, you will need to provide it a unique unused IPv4 address on your LAN beyond what the host already uses. Don't let your local IPv4 DHCP server hand this one out!
The only line in my wrapsix.conf is the ipv4_address 192.168.x.y one.
Depending on your system, you may also need to specify the network interface.
On your +NAT64 gateway host (not your IPv6-only host!), create an
/etc/systemd/system/wrapsix-nat64.service config:
[Unit]
Description=wrapsix IPv6 NAT64 translation daemon.
After=network.target
After=network-online.target
[Service]
User=root
Type=simple
# Give it a boost as it does network routing for others.
Nice=-1
ExecStart=/usr/local/sbin/wrapsix
[Install]
WantedBy=multi-user.target
After that, fire it up:
sudo systemctl start wrapsix-nat64
sudo systemctl status wrapsix-nat64
sudo systemctl enable wrapsix-nat64
Our IPv6-only host needs to how how to route to that reserved NAT64 network as it isn't provided by the internet at large (unless your ISP is amazing).
A static route will do the trick.
Did you see that I already added that in the IPv6-only host's dns64.network
config above? If you try to do it any other way while systemd-networkd is the
interface, it'll wind up being removed. Use this form of [Route] section:
[Route]
destination=64:ff9b::/96
scope=link
(DEPRECATED - doesn't work) Create a /etc/systemd/system/add-nat64-route.service file:
[Unit]
Description=Adds route to IPv6 NAT64 gateway on local lan.
After=network.target
After=network-online.target
[Service]
User=root
Type=oneshot
ExecStart=/sbin/ip -6 route add 64:ff9b::/96 dev eth0
[Install]
WantedBy=multi-user.target
Don't forget to sudo systemctl enable add-nat64-route.
We'd need a IPv6 Router Advertisement. I tried to get this going with radvd on the NAT64 host and had mixed success. It is possibly due to my radvd config?
Here's my radvd config on my NAT64 gateway host running wrapsix:
# For reference only. radvd was not reliable for me.
interface enp1s0 {
AdvSendAdvert on;
# We are NOT a default router gateway, this disables that concept.
AdvDefaultLifetime 0;
# Advertise reasonably often.
MaxRtrAdvInterval 42;
prefix 64:ff9b::/96 {
AdvOnLink off;
AdvAutonomous off;
# Short lifetime
AdvValidLifetime 300;
AdvPreferredLifetime 120;
};
};
Don't forget to sudo systemctl enable radvd.service and sudo service radvd start.
Comment out or remove the 127.0.0.1 lines in /etc/hosts and make sure ::1 lines for those exist.
In the time since I configured the above, I started playing with a DNS-molesting setting on my router (Eero "Secure" service). That, unsurprisingly, works purely by hijacking your DNS. So those traditional UDP port 53 queries to the remote Google Public DNS DNS64 servers were being intercepted and replaced with responses who's answers clearly were not coming from those servers. Result: The internet appears broken on my IPv6 box. "It's always DNS" I've long since kicked their the "Eero Secure" DNS mangling service out of my house.
To fix this and survive malicious DNS-mangling environments, my first thought was to move to DoH or DoT (DNS over HTTPS or DNS over TLS). Every good DNS service out there support those.
In the process of configuring my way around this, I found my way to dnscrypt-proxy as the software of choice. It can do way more than we need; potential configuration confusion? Maybe. It works well and is written in network server safe language (Golang).
Do not use the dnscrypt-proxy package in Ubuntu 18.04, it is quite old.
Build or Get a modern dnscrypt-proxy version yourself. v2.0.44 at the time of
this writing. Debian Buster has a more recent 2.0.x package, that one might
work for you... But beware of the distro package shenanigans. I noticed
Ubuntu 18.04's old package tries to add a "dnscrypt-proxy-resolveconf" systemd
service into the mix which is really just a command line attempting to extract
the address dnscrypt-proxy is listening on from its config file in such a way
it would not understand an IPv6 address... (you can't cut -d ":" -f 1 an IPv6
address to strip the trailing :port people!). Installing our own with a
simpler config was easier to control.
I was thinking of running dnscrypt-proxy on my NAT64 gateway. Alas, no reason
for that. The nat64 gateway is an opaque host that my IPv6 only host never
needs to be configured to know the specifics of. Run dnscrypt-proxy locally on
your IPv6 host itself. Here's how:
Create an /etc/dnscrypt-proxy/dnscrypt-proxy.toml config file:
# An IPv6-only 6to4 config derived from the dnscrypt-proxy 2.0.44 example
# and https://github.com/DNSCrypt/dnscrypt-proxy/wiki/Configuration
server_names = ['google-ipv6'] # Yes, not even the DNS64 ones.
listen_addresses = ['[::1]:53']
ipv4_servers = false
ipv6_servers = true
dnscrypt_servers = true
doh_servers = true
## These are normal, non-encrypted DNS resolvers, that will be only used
## for one-shot queries when retrieving the initial resolvers list.
## No user application queries will ever be leaked through these resolvers.
fallback_resolvers = ['[2001:4860:4860::6464]:53', '[2001:4860:4860::64]:53']
## Always use the fallback resolver before the system DNS settings.
## (because we _are_ the system DNS setting)
ignore_system_dns = true
## Address and port to try initializing a connection to, just to check
## if the network is up. It can be any address and any port.
netprobe_address = '[2001:4860:4860::64]:53'
# Because this would be dumb...
block_ipv6 = false
block_unqualified = true
block_undelegated = true
cache = true
cache_size = 4096
cache_min_ttl = 2400
cache_max_ttl = 86400
cache_neg_min_ttl = 60
cache_neg_max_ttl = 600
#########################
# Servers #
#########################
# I got this from https://dnscrypt.info/public-servers/
# explore the configuration, you can use a dynamically updated maintained list
# of these rather than a hardcoded static one like this.
[static]
[static.'google-ipv6']
stamp = 'sdns://AgUAAAAAAAAAFlsyMDAxOjQ4NjA6NDg2MDo6ODg4OF2gHvYkz_9ea9O63fP92_3qVlRn43cpncfuZnUWbzAMwbkgdoAkR6AZkxo_AEMExT_cbBssN43Evo9zs5_ZyWnftEUKZG5zLmdvb2dsZQovZG5zLXF1ZXJ5'
# The key to the whole thing... We don't need a DNS64 server, we are one.
# Thanks dnscrypt-proxy!
[dns64]
prefix = ["64:ff9b::/96"]
There's an example config included with dnscrypt-proxy that goes into much more detail on the plethora of config options.
Once you have a build of dnscrypt-proxy, install it:
sudo mkdir /opt/dnscrypt-proxy
sudo cp /path/to/my/dnscrypt-proxy-build/dnscrypt-proxy /opt/dnscrypt-proxy/
sudo /opt/dnscrypt-proxy -service install
The latter command should create an
/etc/systemd/system/dnscrypt-proxy.service file for you. If you previously
installed an OS package and have on laying around or an uninstall symlink to
/dev/null, remove that first.
Now edit the generated config to fix a few things. This is what I wound up with:
[Unit]
Description=Encrypted/authenticated DNS proxy
ConditionFileIsExecutable=/opt/dnscrypt-proxy/dnscrypt-proxy
[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/opt/dnscrypt-proxy/dnscrypt-proxy -config /etc/dnscrypt-proxy/dnscrypt-proxy.toml
WorkingDirectory=/opt/dnscrypt-proxy
Restart=always
RestartSec=120
[Install]
WantedBy=multi-user.target
You may be able to skip that spooky -service install command and just plop
that file into place directly? I didn't investigate what all it did, if you
are curious it's action appear to come from
https://github.com/kardianos/service/blob/master/service_systemd_linux.go.
Like your config? Fire it up:
servicectl start dnscrypt-proxy
Confirm all is well via servicectl status dnscrypt-proxy. If not, happy
debugging.
Re-edit both /etc/systemd/resolved.conf and
/etc/systemd/network/dns64.network files and change the DNS= line from the
2001:4860:4860::64 settings we configured earlier... Replace those with:
DNS=::1
Now force the system to refresh its config:
servicectl restart systemd-resolved
servicectl restart systemd-networkd
After that, you should see /etc/resolv.conf is still your symlink to
/run/systemd/resolve/resolv.conf and only contains a single nameserver ::1
line.
Normal software should route queries to your local dnscrypt-proxy dns server
which in turn is both proxying your traffic over DNS over HTTPS to the remote
DNS server (avoiding man in the middle DNS-rewriting) and it is locally
creating IPv6 AAAA records for A-only hosts to your NAT64 64:ff9b::/96
address range.
If you are doing this on a desktop... note that the Trustworthy Brand web browsers such as Mozilla Firefox and Google Chrome may have their own ideas about DNS and speak to their own secure resolvers directly to avoid man in the middle shenanigans, including our special system config. DoH! They can likely be configured not to, but as I'm not doing this to a desktop I'll leave discovering how to do that up to you.
As my purpose is to setup a buildbot in this configuration I need to have it speak with github and with the buildbot master. Both of which sadly do not support IPv6. (really Github? In 2019? Sad).
You will need another host on your local network to run a proxy. Get creative
with local address and port mappings. IPv6 has plenty of room in the local
address space you can make it work. A proxy that appears to work is:
https://github.com/elgs/gproxy - aka gpr
I tested a gpr json config looking like this.
{
"github.com" : {
"dstPort" : 443,
"localPort" : 443,
"localAddr" : "[fdb8:ab16:abe5:1:204:4bff:fe25:bffa]",
"dstAddr" : "github.com"
},
"buildbot.blah.blah" : {
"dstPort" : 9020,
"localPort" : 9020,
"localAddr" : "[fdb8:ab16:abe5:1:204:4bff:fe25:bffa]",
"dstAddr" : "buildbot.blah.blah"
}
}
Thankfully you don't actually need gpr. I just left those notes in for reference.