Skip to content

Instantly share code, notes, and snippets.

@platu

platu/ovs+lxd-shared-vlans.md

Last active December 5, 2023 12:58
Show Gist options
  • Save platu/fc0c22a42d002a00382c20e023658688 to your computer and use it in GitHub Desktop.
Save platu/fc0c22a42d002a00382c20e023658688 to your computer and use it in GitHub Desktop.
Download ZIP
LXD containers networked on VLANs shared with the (ancient) physical world
Raw
ovs+lxd-shared-vlans.md

LXD containers networked on VLANs shared with the (ancient) physical world

Topology

Starting from a Debian testing base install on host system ...

Let's start with networking

Install OvS on host

$ sudo aptitude install openvswitch-switch
$ sudo aptitude versions openvswitch-switch
i   3.1.0-2                                   stable    500

Add new switches to host

  • C-3PO is the distribution layer switch.
  • sw-vlan100 is an access layer switch. All of its ports belong to VLAN 100.
  • sw-vlan105 is an access layer switch. All of its ports belong to VLAN 105.
  • host system is networked on IPv4 via the Switched Virtual Interface (SVI) sw-vlan100 which has the lowest metric default route.
  • host system is networked on IPv6 via Stateless Address Autoconfiguration (SLAAC). IPv6 supports multiple default routes.

Here is a copy of the /etc/network/interfaces file. Don't forget to edit this file to fit your context.

  • Physical interface name enp0s1 may have a different name on your hardware
  • IPv4 and IPv6 network prefix may be different for each VLAN
  • In the example below, IPv6 addressing is done through SLAAC
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto C-3PO
iface C-3PO inet manual
        ovs_type OVSBridge
        ovs_ports enp0s1 sw-vlan100 sw-vlan105
        up ip link set dev $IFACE up
        down ip link set dev $IFACE down

allow-C-3PO enp0s1
iface enp0s1 inet manual
        ovs_type OVSPort
        ovs_bridge C-3PO
        up ip link set dev $IFACE up
        down ip link set dev $IFACE down

allow-C-3PO sw-vlan100
iface sw-vlan100 inet static
        ovs_type OVSBridge
        ovs_bridge C-3PO
        ovs_options C-3PO 100
        address 172.17.64.254/25
        gateway 172.17.64.129
        metric 1

iface sw-vlan100 inet6 auto
        ovs_type OVSBridge
        ovs_bridge C-3PO
        ovs_options C-3PO 100

allow-C-3PO sw-vlan105
iface sw-vlan105 inet static
        ovs_type OVSBridge
        ovs_bridge C-3PO
        ovs_options C-3PO 105
        address 10.6.9.254/23
        gateway 10.6.8.1
        metric 2

iface sw-vlan105 inet6 auto
        ovs_type OVSBridge
        ovs_bridge C-3PO
        ovs_options C-3PO 105

Turn IPv(4|6) routing on at the kernel level

Uncomment the following lines in /etc/sysctl.conf

$ egrep -v '(^#|^$)' /etc/sysctl.conf 
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

Make it happen !

$ sudo sysctl --system
sudo sysctl --system
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/99-sysctl.conf ...
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.log_martians = 1
* Applying /etc/sysctl.d/protect-links.conf ...
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.conf ...
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.all.log_martians = 1

Go on with containers

Install LXD

$ sudo apt install snapd
$ sudo snap install lxd
$ sudo adduser etu lxd

User etu is the UNprivileged user running the containers and must belong to lxd group. Log out and log back in to make it effective.

$ id | grep -o lxd
lxd

List the installed snaps

$ snap list
Name  Version    Rev    Tracking  Publisher   Notes
core  16-2.42.4  8213   stable    canonical✓  core
lxd   3.18       12631  stable    canonical✓  -

Run LXD initial configuration and create profiles for each VLAN

We have to start with the command lxd init at user level in order to configure a storage pool and a repository. On the networking side we choose to bridge on the VLAN 100, but we will define a different profile for each VLAN afterwards.

Here is a copy of the resulting initial configuration.

$ lxd init --dump
config: {}
networks: []
storage_pools:
- config:
    source: /var/snap/lxd/common/lxd/storage-pools/default
    volatile.initial_source: /var/snap/lxd/common/lxd/storage-pools/default
  description: ""
  name: default
  driver: btrfs
profiles:
- config: {}
  description: Default LXD profile
  devices:
    eth0:
      name: eth0
      nictype: bridged
      parent: sw-vlan100
      type: nic
    root:
      path: /
      pool: default
      type: disk
  name: default

Copy default profile to create a new profile for each VLAN.

lxc profile copy default vlan100
lxc profile copy default vlan105

Edit each new profile to set the proper parent bridge with a command like this : lxc profile edit vlan105 for VLAN 105.

Then we can check the two profiles.

$ lxc profile show vlan100
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: sw-vlan100
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: vlan100
used_by: []

$ lxc profile show vlan105
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: sw-vlan105
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: vlan105
used_by: []

Launch the new containers in their respective VLANs

$ lxc launch images:debian/bullseye c100 -p vlan100
Creating c100
Starting c100
$ lxc launch images:debian/bullseye c105 -p vlan105
Creating c105
Starting c105

The container list is now available. We can check VLAN numbers from the IPv6 addressing scheme.

$ lxc ls
+------+---------+------+-------------------------------------------+------------+-----------+
| NAME |  STATE  | IPV4 |                   IPV6                    |    TYPE    | SNAPSHOTS |
+------+---------+------+-------------------------------------------+------------+-----------+
| c100 | RUNNING |      | 2001:678:3fc:64:216:3eff:febf:2e86 (eth0) | PERSISTENT | 0         |
+------+---------+------+-------------------------------------------+------------+-----------+
| c105 | RUNNING |      | 2001:678:3fc:69:216:3eff:fe37:d8d7 (eth0) | PERSISTENT | 0         |
+------+---------+------+-------------------------------------------+------------+-----------+

IPv6 networking just lacks DNS name resolution.

$ lxc exec c100 -- /bin/bash -c "echo nameserver 2620:fe::fe >/etc/resolv.conf"
$ lxc exec c105 -- /bin/bash -c "echo nameserver 2620:fe::fe >/etc/resolv.conf"

Finally, the two containers launched can reach the Internet

$ for container in c100 c105; do echo ---------------- $container; lxc exec $container -- apt update; done
---------------- c100
Hit:1 http://deb.debian.org/debian bullseye InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
---------------- c105
Hit:1 http://deb.debian.org/debian bullseye InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

Final check with networking tracepath

Install iputils-tracepath package in each container

$ for container in c100 c105
do 
echo ---------------- $container
lxc exec $container -- apt install iputils-ping iputils-tracepath
done

Run a tracepath test and check the default gateway of each container

  • On the VLAN 100, the default gateway address is 2001:678:3fc:64::1
  • On the VLAN 105, the default gateway address is 2001:678:3fc:69::1
$ for container in c100 c105
do
echo ---------------- $container
lxc exec $container -- tracepath inetdoc.net
done
---------------- c100
 1?: [LOCALHOST]                        0.010ms pmtu 1420
 1:  2001:678:3fc:64::1                                    1.379ms
 1:  2001:678:3fc:64::1                                    1.247ms
 2:  2001:678:3fc:1::2                                     0.688ms
 3:  no reply
 4:  no reply
 5:  no reply
 6:  no reply
 7:  no reply
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
12:  no reply
13:  2a03:7220:8083:c300::1                               32.045ms reached
     Resume: pmtu 1420 hops 13 back 5
---------------- c105
 1?: [LOCALHOST]                        0.011ms pmtu 1500
 1:  2001:678:3fc:69::1                                    1.586ms
 1:  2001:678:3fc:69::1                                    1.230ms
 2:  2001:678:3fc:1::2                                     0.636ms
 3:  2001:678:3fc:1::2                                     0.659ms pmtu 1420
 3:  no reply
 4:  no reply
 5:  no reply
 6:  no reply
 7:  no reply
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
12:  no reply
13:  2a03:7220:8083:c300::1                               32.328ms reached
     Resume: pmtu 1420 hops 13 back 5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment