Skip to content

Instantly share code, notes, and snippets.

@FatihBAKIR

FatihBAKIR/networking.md

Last active June 23, 2022 23:52
Show Gist options
  • Save FatihBAKIR/b77c3a94109a7e3b669beb38f8a651c8 to your computer and use it in GitHub Desktop.
Save FatihBAKIR/b77c3a94109a7e3b669beb38f8a651c8 to your computer and use it in GitHub Desktop.
Download ZIP
Ambience VM Networking
Raw
networking.md

Ambience VM Networking

This tutorial assumes you can already build Ambience images and only focuses on networking.

Setup

Install the following packages

Fedora 35/36

dnf install bridge-utils tuntap qemu-system-x86 dhcp-server

Add the user to the netdev group

usermod -a -G netdev USERNAME

Network topology

We assume we have multiple machines with 2 network interfaces each. One network interface on each machine has a publicly accessible IP address, while the others are connected to each other directly or via a switch (and not a router).

Creating a tap device

A tap device is essentially a virtual ethernet card. You have to create at least 1 tap device per VM.

This command creates a tap device called tap0 (-t, accessible by users in the netdev group (-g):

tunctl -g netdev -p -t tap0

You must be root the execute this command

-p means this device should get packets with ethernet headers.

Once you have a tap device, you can launch a VM with it by adding the following to your qemu line:

-device virtio-net,netdev=network0,mac=66:66:66:66:66:66,host_mtu=65535 \  
-netdev tap,id=network0,ifname=tap0,script=no,downscript=no

Note that you must have a unique MAC address for each VM in your network.

However, nothing much can be done with it since the tap0 is basically an ethernet card that is not plugged in to.

Creating a bridge network

For our purposes, a bridge network is a virtual ethernet switch. A number of physical and virtual ethernet cards (interfaces) can be plugged into it and it will forward packets to the correct interfaces.

A bridge can be created with the following command:

brctl addbr br0

You must be root to execute this command

To be able to interface with the bridge from the host machine (for instance, to send requests to ambience servers), we have to assign the bridge some network details:

ifconfig ambience0 10.0.1.5 netmask 255.255.255.0 up

Note that we will be connecting bridges on different machines, and each bridge must have a different IP address! For each different machine on the same private network, use a different IP address like 10.0.1.6, 10.0.1.7, 10.0.1.4 etc.

Once you have a bridge and an interface you want to add to the bridge, you can do so with the following command:

brctl addif br0 tap0

The interface does not have to be virtual either, you can add the private interface (called eno1 here) to the bridge using the same command:

ifconfig eno1 down
brctl addif ambience0 eno1

We first disable the interface with the ifconfig line. tap devices come disabled, so you don't need to down it first.

Once all interfaces are on the bridge, you can up them:

ifconfig tap0 up
ifconfig eno1 up

Great! Now your VM can communicate with the host machine, as well as other machines and VMs running on the same or other machines!

However, you would have to manually assign IP addresses to every single VM this way, and ambience does not even allow static IP addresses.

Running a DHCP server

DHCP is a protocol for assigning necessary networking configuration to machines dynamically. Once you set it up, it simplifies network management greatly.

We'll use dhcpd, an existing dhcp server. All we need is a configuration file. Create a file called dhcpd.conf and put the following in it:

subnet 10.0.1.0 netmask 255.255.255.0 {
  option routers 10.0.1.5;
  option subnet-mask 255.255.255.0;
  option domain-search "example.com";
  option domain-name-servers 10.0.1.5;
  option time-offset -18000;
  range 10.0.1.10 10.0.1.100;
}

Note that this network definition matches our bridges' networks

Also create an empty file called leasefile.

Once you have all the files ready, just run the following command:

dhcpd -d -4 -f -cf dhcpd.conf -lf leasefile -pf dhcpd.pid --no-pid br0

You have to be root to run this command

This command runs in the foreground and will serve DHCP requests on our bridge. You must run 1 and only 1 DHCP server on the entire private network, even if you have 100 machines on it.

Tearing down the network

To shutdown a network, first kill all the VMs running, then kill the DHCP server by hitting CTRL-C on it, and then down all interfaces in the bridge, including the bridge:

ifconfig br0 down
ifconfig tap0 down
ifconfig eno1 down

Then, delete the bridge and all virtual interfaces:

brctl delbr br0
tunctl -d tap0
# tunctl -d tap...
# tunctl -d tapN

That's it! It'll be as if your network never existed.

Image building

You must specify the (private) IP address of the nodes in Ambience node manifests. However, to be able to know the address, you have to run the VM for the DHCP server to assign an IP address to it, which might seem cyclical. However, the DHCP server will try to assign the same IP address to the same MAC address every time, so you can a dummy ambience image, which could by anything, since we are only interested in getting an IP address, let the server assign an IP address to the MAC address, and put that IP address in the Ambience manifest.

Just make sure to run the same node image with the same MAC address every time!

Yes, DNS would fix this. You are welcome to implement support for it in ambience!

TLDR

Create the necessary files

netup.sh:

#!/bin/bash

brctl addbr br0

### IMPORTANT: ### 
### Change this line on each machine to have a different IP address!!!
##################
ifconfig br0 10.0.1.5 netmask 255.255.255.0 up

tunctl -g netdev -p -t tap0 || true

brctl addif br0 tap0

ifconfig eno1 down
brctl addif br0 eno1

ifconfig tap0 up
ifconfig eno1 up

netdown.sh:

#!/bin/bash
set -o pipefail
set -e

ifconfig br0 down || true
ifconfig tap0 down || true
ifconfig eno1 down || true

brctl delbr br0 || true
tunctl -d tap0

dhcpd.conf:

subnet 10.0.1.0 netmask 255.255.255.0 {
  option routers 10.0.1.5;
  option subnet-mask 255.255.255.0;
  option domain-search "example.com";
  option domain-name-servers 10.0.1.5;
  option time-offset -18000;
  range 10.0.1.10 10.0.1.100;
}

leasefile:

Intentionally empty

Make the scripts executable by running chmod +x netup.sh, chmod +x netdown.sh

Create the networks in each machine

Run sudo ./netup.sh on each machine to create the virtual networks.

Run the DHCP server

On the 10.0.1.5 machine, run the dhcp server:

dhcpd -d -4 -f -cf dhcpd.conf -lf leasefile -pf dhcpd.pid --no-pid br0

Run VMs

Run a VM on any machine with the following command:

qemu-system-x86_64 \
 -machine accel=kvm,type=q35 \
 -smp cpus=1,cores=1 \
 -m 2G \
 -cdrom /tmp/sierra/mydeployment/cmake-build-barex64/iso/app_vm-iso.iso \
 -serial stdio -display none \
 -no-reboot -no-shutdown \
 -device virtio-net,netdev=network0,mac=66:66:66:66:66:66,host_mtu=65535 \
 -netdev tap,id=network0,ifname=tap0,script=no,downscript=no

If you want more than 1 VM per machine, create more tap devices in your netup script. Also, delete them in your netdown script. Change ifname=tap0 to the correct tap device for each VM.

It does not have to be an ambience image, you can run a Linux VM on the same network as well.

Tear down everything

  • Kill all VMs
  • Kill the DHCP server by hitting CTRL-C
  • Run sudo ./netdown.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment