Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
LXD containers networked on VLANs shared with the (ancient) physical world

LXD containers networked on VLANs shared with the (ancient) physical world

Topology

Starting from a Debian testing base install on host system ...


Let's start with networking

Install OvS on host

$ sudo aptitude install openvswitch-switch
$ sudo aptitude versions openvswitch-switch
i   2.11.0+2019.06.25+git.9ebe795035+ds1-7    testing   500

Add new switches to host

  • C-3PO is the distribution layer switch.
  • sw-vlan100 is an access layer switch. All of its ports belong to VLAN 100.
  • sw-vlan105 is an access layer switch. All of its ports belong to VLAN 105.
  • host system is networked on IPv4 via the Switched Virtual Interface (SVI) sw-vlan100 which has the lowest metric default route.
  • host system is networked on IPv6 via Stateless Address Autoconfiguration (SLAAC). IPv6 supports multiple default routes.

Here is a copy of the /etc/network/interfaces file. Don't forget to edit this file to fit your context.

  • Physical interface name enp0s6 may have a different name on your hardware
  • IPv4 and IPv6 network prefix may be different for each VLAN
  • In the example below, IPv6 addressing is done through SLAAC
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-ovs C-3PO
iface C-3PO inet manual
        ovs_type OVSBridge
        ovs_ports enp0s6 sw-vlan100 sw-vlan105
        up ip link set dev $IFACE up
        down ip link set dev $IFACE down

allow-C-3PO enp0s6
iface enp0s6 inet manual
        ovs_type OVSPort
        ovs_bridge C-3PO
        up ip link set dev $IFACE up
        down ip link set dev $IFACE down

allow-C-3PO sw-vlan100
iface sw-vlan100 inet static
        ovs_type OVSBridge
        ovs_bridge C-3PO
        ovs_options C-3PO 100
        address 172.17.64.254/25
        gateway 172.17.64.129
        metric 1

iface sw-vlan100 inet6 auto
        ovs_type OVSBridge
        ovs_bridge C-3PO
        ovs_options C-3PO 100

allow-C-3PO sw-vlan105
iface sw-vlan105 inet static
        ovs_type OVSBridge
        ovs_bridge C-3PO
        ovs_options C-3PO 105
        address 10.6.9.254/23
        gateway 10.6.8.1
        metric 2

iface sw-vlan105 inet6 auto
        ovs_type OVSBridge
        ovs_bridge C-3PO
        ovs_options C-3PO 105

Turn IPv(4|6) routing on at the kernel level

Uncomment the following lines in /etc/sysctl.conf

$ egrep -v '(^#|^$)' /etc/sysctl.conf 
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

Make it happen !

$ sudo sysctl --system
sudo sysctl --system
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/99-sysctl.conf ...
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.log_martians = 1
* Applying /etc/sysctl.d/protect-links.conf ...
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.conf ...
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.log_martians = 1

Go on with containers

Install LXD

$ sudo apt install snapd
$ sudo snap install lxd
$ sudo adduser etu lxd

User etu is the UNprivileged user running the containers and must belong to lxd group. Log out and log back in to make it effective.

$ id | grep -o lxd
lxd

List the installed snaps

$ snap list
Name  Version    Rev    Tracking  Publisher   Notes
core  16-2.42.4  8213   stable    canonical✓  core
lxd   3.18       12631  stable    canonical✓  -

Run LXD initial configuration and create profiles for each VLAN

We have to start with the command lxd init at user level in order to configure a storage pool and a repository. On the networking side we choose to bridge on the VLAN 100, but we will define a different profile for each VLAN afterwards.

Here is a copy of the resulting initial configuration.

$ lxd init --dump
config: {}
networks: []
storage_pools:
- config:
    source: /var/snap/lxd/common/lxd/storage-pools/default
    volatile.initial_source: /var/snap/lxd/common/lxd/storage-pools/default
  description: ""
  name: default
  driver: btrfs
profiles:
- config: {}
  description: Default LXD profile
  devices:
    eth0:
      name: eth0
      nictype: bridged
      parent: sw-vlan100
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: default

Copy default profile to create a new profile for each VLAN.

lxc profile copy default vlan100
lxc profile copy default vlan105

Edit each new profile to set the proper parent bridge with a command like this : lxc profile edit vlan105 for VLAN 105.

Then we can check the two profiles.

$ lxc profile show vlan100
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: sw-vlan100
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: vlan100
used_by: []
$ lxc profile show vlan105
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: sw-vlan105
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: vlan105
used_by: []

Launch the new containers in their respective VLANs

$ lxc launch images:debian/bullseye c100 -p vlan100
Creating c100
Starting c100
$ lxc launch images:debian/bullseye c105 -p vlan105
Creating c105
Starting c105

The container list is now available. We can check VLAN numbers from the IPv6 addressing scheme.

$ lxc ls
+------+---------+------+-------------------------------------------+------------+-----------+
| NAME |  STATE  | IPV4 |                   IPV6                    |    TYPE    | SNAPSHOTS |
+------+---------+------+-------------------------------------------+------------+-----------+
| c100 | RUNNING |      | 2001:678:3fc:64:216:3eff:febf:2e86 (eth0) | PERSISTENT | 0         |
+------+---------+------+-------------------------------------------+------------+-----------+
| c105 | RUNNING |      | 2001:678:3fc:69:216:3eff:fe37:d8d7 (eth0) | PERSISTENT | 0         |
+------+---------+------+-------------------------------------------+------------+-----------+

IPv6 networking just lacks DNS name resolution.

$ lxc exec c100 -- /bin/bash -c "echo nameserver 2620:fe::fe >/etc/resolv.conf"
$ lxc exec c105 -- /bin/bash -c "echo nameserver 2620:fe::fe >/etc/resolv.conf"

Finally, the two containers launched can reach the Internet

$ for container in c100 c105; do echo ---------------- $container; lxc exec $container -- apt update; done
---------------- c100
Hit:1 http://deb.debian.org/debian bullseye InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
---------------- c105
Hit:1 http://deb.debian.org/debian bullseye InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

Final check with networking tracepath

Install iputils-tracepath package in each container

$ for container in c100 c105
do 
echo ---------------- $container
lxc exec $container -- apt install iputils-ping iputils-tracepath
done

Run a tracepath test and check the default gateway of each container

  • On the VLAN 100, the default gateway address is 2001:678:3fc:64::1
  • On the VLAN 105, the default gateway address is 2001:678:3fc:69::1
$ for container in c100 c105
do
echo ---------------- $container
lxc exec $container -- tracepath inetdoc.net
done
---------------- c100
 1?: [LOCALHOST]                        0.010ms pmtu 1420
 1:  2001:678:3fc:64::1                                    1.379ms
 1:  2001:678:3fc:64::1                                    1.247ms
 2:  2001:678:3fc:1::2                                     0.688ms
 3:  no reply
 4:  no reply
 5:  no reply
 6:  no reply
 7:  no reply
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
12:  no reply
13:  2a03:7220:8083:c300::1                               32.045ms reached
     Resume: pmtu 1420 hops 13 back 5
---------------- c105
 1?: [LOCALHOST]                        0.011ms pmtu 1500
 1:  2001:678:3fc:69::1                                    1.586ms
 1:  2001:678:3fc:69::1                                    1.230ms
 2:  2001:678:3fc:1::2                                     0.636ms
 3:  2001:678:3fc:1::2                                     0.659ms pmtu 1420
 3:  no reply
 4:  no reply
 5:  no reply
 6:  no reply
 7:  no reply
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
12:  no reply
13:  2a03:7220:8083:c300::1                               32.328ms reached
     Resume: pmtu 1420 hops 13 back 5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment