The goal of this article is to explain how to install one or many Raspberry Pi 3 B+ devices connected with one to four Raspberry Pi Zero 1.3 devices each via USB OTG networking.
Each device - Pi and Zero - will be connected to the network 192.168.0.0/24. Each Zero will be connected via USB OTG to the Pi. That will establish a network between the two devices. On each Pi a network bridge will be configured to forward the traffic between the Pi ethernet port and the connected USB ports.
Their individual hostname/IP pattern will be:
10.lab | 192.168.0.10 (1st Pi)
20.lab | 192.168.0.20 (2nd Pi)
11.lab | 192.168.0.11 (1st Zero on 1st Pi)
12.lab | 192.168.0.12 (2nd Zero on 1st Pi)
22.lab | 192.168.0.22 (2nd Zero on 2nd Pi)And so on. Here is a small diagram that shows the setup for one Pi:
┌────────┐ 192.168.0.11
│ Zero 1 │ 00:22:82:ff:ff:01 usb0 ──────────────┐
└────────┘ 11.lab │
│ ┌──────────── br0 ────────────┐
┌────────┐ 192.168.0.12 │
│ Zero 2 │ 00:22:82:ff:ff:02 usb0 ──────┐ │
└────────┘ 12.lab │ ethusb1 00:22:82:ff:ff:11 ┌────────┐ 192.168.0.10
└── ethusb2 00:22:82:ff:ff:12 │ Pi 1 │ eth0 ────────────────────
┌────────┐ 192.168.0.13 ┌── ethusb3 00:22:82:ff:ff:13 └────────┘ 10.lab
│ Zero 3 │ 00:22:82:ff:ff:03 usb0 ──────┘ ethusb4 00:22:82:ff:ff:14
└────────┘ 13.lab │
│
┌────────┐ 192.168.0.14 │
│ Zero 4 │ 00:22:82:ff:ff:04 usb0 ──────────────┘
└────────┘ 14.labOn all SD cards - for the Pi and the Zero - perform the following steps before booting the devices:
Connect an empty SD card (at least 2GB). Figure out the device e.g. via watching the dmesg -w on your Linux host output while inserting it.
Assuming that your SD card device is /dev/sdb you can now perform the following step do download the Raspbian Lite image and burn it to your SD card:
wget -O - https://downloads.raspberrypi.org/raspbian_lite_latest | dd of=/dev/sdb bs=4096That command may take several minutes. After that your SD card has two partitions: One for /boot and one for /.
Mount the /boot partition with mkdir -p /media/boot && mount /dev/sdb1 /media/boot
Then enable the SSH daemon by createing an empty ssh file on the /boot partition by typing touch /boot/ssh.
The following instructions should only be executed on the SD cards for the Zero before booting.
The dwc2 and g_ether kernel modules are required to be loaded for this setup. Therefore change the file /boot/cmdline.txt from:
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwaitto:
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait modules-load=dwc2,g_ether g_ether.host_addr=00:22:82:ff:ff:01 g_ether.dev_addr=00:22:82:ff:ff:11 quietEach OTG networking connection requires both MAC address setups. One address for the Zero (host) and a second one for the Pi (dev).
The pattern is:
host_addr=00:22:82:ff:ff:01 dev_addr=00:22:82:ff:ff:11 (1st Zero)
host_addr=00:22:82:ff:ff:02 dev_addr=00:22:82:ff:ff:12 (2nd Zero)
host_addr=00:22:82:ff:ff:03 dev_addr=00:22:82:ff:ff:13 (3rd Zero)
host_addr=00:22:82:ff:ff:04 dev_addr=00:22:82:ff:ff:14 (4th Zero)Now the modules need to be activated. To enable the OTG overlay and provide a bit more memory edit the /boot/config.txt file and add at the end of the file:
dtoverlay=dwc2
gpu_mem=16To enable ssh add an empty file to the /boot folder:
touch /boot/sshConfigure the IP address statically by editing the file /etc/dhcpcd.conf:
interface usb0
static ip_address=192.168.0.11/24
static routers=192.168.0.10
static domain_name_servers=9.9.9.9 1.1.1.1The following instructions are required only to be done on the Raspberrey Pi, not the Zero.
We require network packet forwarding between the ethernet port and the USB OTG ports. As it is disabled by default we need to enable it by typing sysctl -w net.ipv4.ip_forward=1. You might want to add this property to the /etc/sysctl.conf file as well to keep it persistent (run sudo sysctl -p /etc/sysctl.conf afterwards).
Check also iptables for blocking forwarding rules with iptables -S and iptables -t filter -L.
Adjust iptables to accept forwarding. By default it is set to DROP:
iptables -P FORWARD ACCEPT
iptables-saveNow we need to give static device names to USB OTG network devices. Sometimes they would show up as ethX, sometimes as usbX. We don't want them to toggle so we fix their names by creating a /etc/udev/rules.d/90-pi-network.rules file with the following content:
SUBSYSTEM=="net", ATTR{address}=="00:22:82:ff:ff:01", NAME="ethusb1"
SUBSYSTEM=="net", ATTR{address}=="00:22:82:ff:ff:02", NAME="ethusb2"
SUBSYSTEM=="net", ATTR{address}=="00:22:82:ff:ff:03", NAME="ethusb3"
SUBSYSTEM=="net", ATTR{address}=="00:22:82:ff:ff:04", NAME="ethusb4"The USB network devices will be named ethusbX. The MAC addresses will be explicitly defined by setting up the Zeros in the next chapter.
Each USB OTG network requires two MAC addresses. One for each side. The USB OTG host (Zero) and the connected device (Pi). The link between the 1st Zero and the Pi will 00:22:82:ff:ff:11 on the Zero side and 00:22:82:ff:ff:01 on the Pi side. The second one 00:22:82:ff:ff:12/00:22:82:ff:ff:02 etc.
Now we need to setup the bridge on the Raspberry Pi. All the connected USB network devices together with eth0 need to be added.
The first step is to make sure that the /etc/network/interfaces file contains the following line:
source /etc/network/interfaces.d/*Some older versions contain a different syntax which won't work any more.
Then we add our bridge configuration to the file /etc/network/interfaces.d/bridge.conf:
auto eth0
auto ethusb1
auto ethusb2
auto ethusb3
auto ethusb4
iface eth0 inet manual
iface ethusb1 inet manual
iface ethusb2 inet manual
iface ethusb3 inet manual
iface ethusb4 inet manual
auto br0
iface br0 inet dhcp
bridge_ports eth0 ethusb1 ethusb2 ethusb3 ethusb4
bridge_stp off
bridge_fd 0
bridge_maxage 1In addition we need to modify the dhcpcd configuration to ignore the devices and to setup a static IP configuration for the bridge. This is done in the /etc/dhcpcd.conf file:
denyinterfaces eth0 ethusb1 ethusb2 ethusb3 ethusb4
interface br0
static ip_address=192.168.0.10/24
static routers=192.168.0.1
static domain_name_servers=9.9.9.9 1.1.1.1Now the bridge is configured. We just need to change the host name in /etc/hostname to 10 (or 20, 30 according to the pattern described above).
The hostname should also be added to the /etc/hosts file like:
127.0.1.1 10.lab 10Now you can boot your devices. On all of them - Pi and Zero - you should perform the following steps:
To expand the filesystem to the full size of the SD card execute the raspi-config command. A wizard shows up. Go to advanced settings and there you will find an option to expand the filesystem to the full SD card size.
Execute passwd to set an individual password.
Since each Raspbian image has pre-configured SSH host keys they need to be re-created individually. Therefore execute the following:
rm /etc/ssh/ssh_host_*
dpkg-reconfigure openssh-serverIn addition you might want to add your SSH public key to /home/pi/.ssh/authorized_keys to enable logins via public key.
In case you don't want to use IPv6 you can turn it off with sysctl -w net.ipv6.conf.all.disable_ipv6=1. You might want to add this property to the /etc/sysctl.conf file as well to keep it persistent (run sudo sysctl -p /etc/sysctl.conf afterwards).
To watch the OTG network devices becoming ready type the following command on the Pi:
dmesg -ewTo get more information about the configured bridge install the bridge-utils and have a look at its current state:
apt install bridge-utils
brctl show br0
brctl showstn br0
brctl showmac br0
ip a
ip rThis document is based on the Raspberry Pi OTG network article. That describes a similar setup but with a NAT network configuration instead of a bridged one.
Also an interesting read: Build your PiZero Swarm with OTG networking
You might want to install your own DNS server like dnsmasq to resolve hosts by the pattern described above e.g. 10.lab or 21.lab.
A very easy solution is to install a pi-hole on one one Pi. That comes with a dnsmasq tool which you can configure to resolve your hosts. More information: https://pi-hole.net/