Skip to content

Instantly share code, notes, and snippets.

@Ravenstine

Ravenstine/how-to-run-yggdrasil-in-docker.md

Created September 29, 2023 20:48
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save Ravenstine/707180ef29e9d37a8f816e019ca32dbf to your computer and use it in GitHub Desktop.
Save Ravenstine/707180ef29e9d37a8f816e019ca32dbf to your computer and use it in GitHub Desktop.
Download ZIP
How To Run Yggdrasil In Docker
Raw
how-to-run-yggdrasil-in-docker.md

How To Run Yggdrasil In Docker

Want to run Yggdrasil in a Docker container? This is how you can do it.

The keys to getting it working are the following:

  • Give the container access to the TUN interface of the host (or the VM guest in the case of Docker Machine or Docker for Mac)
  • Enable IPv6
  • Assign a dedicated MAC address

It took me a while to figure this stuff out, so hopefully this helps you. When the requirements aren't met, Yggdrasil exits with a barely helpful "permission denied" message.

Here is an example Dockerfile:

FROM alpine:3.18.3

RUN \
  apk add \
  --no-cache \
  --no-check-certificate \
  --allow-untrusted \
  --no-scripts \
  yggdrasil=0.4.7-r9

CMD yggdrasil -useconffile /etc/yggdrasil.conf

At this point, you will likely want to create a yggdrasil.conf file that configures the host machine as a peer.

Assuming that your host is running Yggdrasil and listening for peers on TCP port 8008, this is how you can generate a new yggdrasil.conf file for your container:

echo "{ \"Peers\": [ \"tcp://host.docker.internal:8008\" ] }" | yggdrasil -useconf -normaliseconf > yggdrasil.conf

The specific port number doesn't matter so long as it's the one that the peer running on your host is listening for other peers on. The host.docker.internal domain is automatically set up by Docker.

Here is an example of how to run a container:

docker build -t yggdrasil .
docker run \
--rm \
--cap-add "NET_ADMIN" \
--device "/dev/net/tun" \
--volume "./yggdrasil.conf:/etc/yggdrasil.conf" \
--sysctl "net.ipv6.conf.all.disable_ipv6=0" \
--mac-address "1E:FC:B2:8D:DF:D4" \
yggdrasil

Here is an example with Docker Compose:

version: '3.9'
services:
  yggdrasil:
    build: .
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun
    volumes:
      - "./yggdrasil.conf:/etc/yggdrasil.conf"
    sysctls:
      - "net.ipv6.conf.all.disable_ipv6=0"
    mac_address: 1E:FC:B2:8D:DF:D4

In which case, you would run docker-compose up. To spin it down, run docker-compose down.

Check the log of the running container to see whether Yggdrasil is successfully reaching the host peer. If you are unsure, you can check on your host machine by running yggdrasilctl getPeers.

Ok, this is all fine, but now you want to make another containerized service available from your Yggdrasil peer container, right?

The following is an approach using Docker Compose, with a Redis service as an example.

We will need to change the Dockerfile so that it will support forwarding the Redis port.

FROM alpine:3.18.3

RUN \
  apk add \
  --no-cache \
  --no-check-certificate \
  --allow-untrusted \
  --no-scripts \
  yggdrasil=0.4.7-r9 \
  socat=1.7.4.4-r1 \
  supervisor=4.2.5-r2

CMD supervisord -c /etc/supervisord.conf

We will also need a supervisord.conf file:

[supervisord]
logfile=/dev/null
nodaemon=true
user=root

[program:yggdrasil]
command=yggdrasil -useconffile /etc/yggdrasil/yggdrasil.conf
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

[program:redis-forwarder]
command=socat TCP6-LISTEN:6379,fork,forever,reuseaddr TCP4:redis:6379

And now let's add a volume for the supervisord configuration:

version: '3.9'
services:
  yggdrasil:
    build: .
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun
    volumes:
      - "./yggdrasil.conf:/etc/yggdrasil.conf"
      - "./supervisord.conf:/etc/supervisord.conf"
    sysctls:
      - "net.ipv6.conf.all.disable_ipv6=0"
    mac_address: 1E:FC:B2:8D:DF:D4
 
  redis:
    image: docker.io/redis:7.2.1-alpine3.18
    ports:
      - "6379:6379"

Run docker-compose up and use nc -v <yggdrasil peer ipv6 address> 6379 or redis-cli -h <yggdrasil peer ipv6 address> to confirm that your containerized peer has joined your network and that you can reach the Redis service through it.

Alternative Setup (if all else fails)

Can't get IPv6 to work in Docker? /dev/net/tun doesn't exist?

There's a less optimal alternative that can work, and that's to run a virtual machine within a Docker container. The reason this works is that it emulates a full network stack, bypassing any limitations on the host.

Here is an example alternative Dockerfile that runs Yggdrasil within a VM:

FROM alpine:3.18.3 AS initramfs

WORKDIR /

RUN \
  apk add --no-cache \
    coreutils \
    curl

RUN mkdir -p /root

ENV initramfsDir=/initramfs

## Make default directory structure
RUN \
  mkdir -p \
  "$initramfsDir" \
  "$initramfsDir/bin" \
  "$initramfsDir/etc" \
  "$initramfsDir/mnt" \
  "$initramfsDir/proc" \
  "$initramfsDir/root" \
  "$initramfsDir/sbin" \
  "$initramfsDir/sys" \
  "$initramfsDir/var/run"

# Add base system
RUN \
  apk add \
  --root "$initramfsDir" \
  --no-cache \
  --initdb \
  --no-check-certificate \
  --allow-untrusted \
  --repository "https://dl-cdn.alpinelinux.org/alpine/v3.18/main" \
  busybox=1.36.1-r2 \
  supervisor \
  socat

# Add yggdrasil
RUN \
  apk add \
  --root "$initramfsDir" \
  --initdb \
  --no-cache \
  --no-check-certificate \
  --allow-untrusted \
  --no-scripts \
  --repository "https://dl-cdn.alpinelinux.org/alpine/v3.18/main" \
  --repository "https://dl-cdn.alpinelinux.org/alpine/v3.18/community" \
  yggdrasil=0.4.7-r9

ENV linuxLtsDir=/linux-virt

RUN mkdir -p $linuxLtsDir

RUN \
  curl -o linux-virt.tar -sSL https://dl-cdn.alpinelinux.org/alpine/v3.18/main/aarch64/linux-virt-6.1.54-r0.apk && \
  tar xvf linux-virt.tar -C $linuxLtsDir && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.alias $initramfsDir/lib/modules/6.1.54-0-virt/modules.alias && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.alias.bin $initramfsDir/lib/modules/6.1.54-0-virt/modules.alias.bin && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.builtin $initramfsDir/lib/modules/6.1.54-0-virt/modules.builtin && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.builtin.alias.bin $initramfsDir/lib/modules/6.1.54-0-virt/modules.builtin.alias.bin && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.builtin.bin $initramfsDir/lib/modules/6.1.54-0-virt/modules.builtin.bin && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.builtin.modinfo $initramfsDir/lib/modules/6.1.54-0-virt/modules.builtin.modinfo && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.dep $initramfsDir/lib/modules/6.1.54-0-virt/modules.dep && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.dep.bin $initramfsDir/lib/modules/6.1.54-0-virt/modules.dep.bin && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.devname $initramfsDir/lib/modules/6.1.54-0-virt/modules.devname && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.order $initramfsDir/lib/modules/6.1.54-0-virt/modules.order && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.softdep $initramfsDir/lib/modules/6.1.54-0-virt/modules.softdep && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.symbols $initramfsDir/lib/modules/6.1.54-0-virt/modules.symbols && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/modules.symbols.bin $initramfsDir/lib/modules/6.1.54-0-virt/modules.symbols.bin && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/drivers/net/ethernet/intel/e1000/e1000.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/drivers/net/ethernet/intel/e1000/e1000.ko.gz && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/net/9p/9pnet.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/net/9p/9pnet.ko.gz && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/net/packet/af_packet.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/net/packet/af_packet.ko.gz && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/drivers/net/tun.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/drivers/net/tun.ko.gz && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/fs/9p/9p.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/fs/9p/9p.ko.gz && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/net/9p/9pnet_virtio.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/net/9p/9pnet_virtio.ko.gz && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/fs/fscache/fscache.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/fs/fscache/fscache.ko.gz && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/fs/netfs/netfs.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/fs/netfs/netfs.ko.gz && \
  install -Dm644 $linuxLtsDir/lib/modules/6.1.54-0-virt/kernel/net/ipv6/ipv6.ko.gz $initramfsDir/lib/modules/6.1.54-0-virt/kernel/net/ipv6/ipv6.ko.gz


## Add host nameserver
RUN cat > $initramfsDir/etc/resolv.conf <<EOF
nameserver 10.0.2.3
EOF


## Add init file for initramfs
RUN cat > $initramfsDir/init <<EOF
#!/bin/sh

mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs dev /dev

echo /sbin/mdev > /proc/sys/kernel/hotplug

/sbin/mdev -s

# Shared Directory
modprobe virtio_pci
modprobe 9pnet
modprobe 9pnet_virtio
modprobe 9p

mkdir -p /etc/yggdrasil

mount -t 9p -o trans=virtio host0 /etc/yggdrasil

# Networking
modprobe e1000
# modprobe virtio_net
modprobe tun
modprobe ipv6

echo "nameserver 10.0.2.3" > /etc/resolv.conf

ip link set lo up
ip link set eth0 up
udhcpc -i eth0
ifconfig eth0 10.0.2.15
route add default gw 10.0.2.2 eth0

mkdir -p /var/run

socat TCP6-LISTEN:6379,fork,forever,reuseaddr TCP4:redis:6379 &
/usr/bin/yggdrasil -useconffile /etc/yggdrasil/yggdrasil.conf

# exec /bin/busybox sh
EOF

RUN chmod +x "$initramfsDir/init"

## Bundle initramfs file and copy vmlinuz
RUN \
  cd $initramfsDir && \
  find . | sort | cpio --quiet --renumber-inodes -o -H newc | gzip -9 > /root/initramfs && \
  cp "$linuxLtsDir/boot/vmlinuz-virt" "/root/vmlinuz"




FROM alpine:3.18.3 AS primary

COPY --from=initramfs /root /root

WORKDIR /root

RUN apk add --no-cache \
  qemu-system-aarch64

CMD qemu-system-aarch64 \
  -m "128M" \
  -machine "virt" \
  -cpu "cortex-a76" \
  -smp "1" \
  -kernel "vmlinuz" \
  -initrd "initramfs" \
  -serial "mon:stdio" \
  -append "ip=dhcp" \
  -device "e1000,mac=1E:FC:B2:8D:DF:D4,netdev=net0" \
  -netdev "user,id=net0" \
  -virtfs "local,path=/etc/yggdrasil,mount_tag=host0,security_model=passthrough,id=host0" \
  -nographic

This is just an example. It's not as efficient because it emulates a CPU, but I believe it's workable if there's something about the host's Docker or network setup that makes it impossible to run Yggdrasil the normal way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment