Skip to content

Instantly share code, notes, and snippets.

@G-UK

G-UK/Netbooting Raspberry Pi 3B's.md

Last active June 19, 2023 15:35
Show Gist options
  • Save G-UK/779b344d534296ad26db32adcafff781 to your computer and use it in GitHub Desktop.
Save G-UK/779b344d534296ad26db32adcafff781 to your computer and use it in GitHub Desktop.
Download ZIP
Setting up multiple Raspberry Pi3's with Netboot and NFS Root partitions
Raw
Netbooting Raspberry Pi 3B's.md

Introduction

The objective of these instructions is to run a number of Raspberry Pi 3b's without SD cards. This is achieved through providing the Bootloader and Linux Kerenel to the Pi's via PXE network boot using a TFTP server.

Notes:

  • Please ensure your Server and Clients all have reserved IP's, you don't want them changing on you.
  • Netboot will only work over an Ethernet connection. Wireless netboot is not supported on the Raspberry Pi3.

Client set-up

By default the Raspberry Pi 3 has netbooting disabled so before we start lets enable it on each Pi.

  1. Add the following line to the end of "/boot/config.txt"
program_usb_boot_mode=1
  1. Reboot the Pi
sudo reboot
  1. Check that the config has been set by running:
vcgencmd otp_dump | grep 17:

The output should read:

17:3020000a
  1. Remove the program_usb_boot_mode line from "/boot/config.txt" ensuring you don't leave a blank line at the end of the file.

Thats it for client set-up, now when the Pi is booted without an SD card it will try and netboot from your network.

Server set-up

To get you completely off of your Pi's SD card we need both a TFTP server to boot the Pi and an NFS server to provide the root filesystem.

First job then is to install DNSMasq and NFS-Kernel-Server from the apt repositories.

sudo apt install dnsmasq nfs-kernel-server

DNSMasq Set-up

"/etc/dnsmasq.conf" is the dnsmasq config file and needs to be edited to contain something like the following to boot the Raspberry Pi's:

interface=eth0
enable-tftp
tftp-root=/srv/boot
tftp-no-blocksize
pxe-service=0,"Raspberry Pi Boot"

"interface" is the network interface you have connected to your LAN. "tftp-root" is the location of your boot files, "/srv/boot" in my case.

Your "tftp-root" directory can now be set-up as follows.

  • First copy the bootcode.bin file from one of you Pi's existing boot folder into your new tftp-root folder.
  • Next create a folder for each of your Pi's, the folder for each Pi is named after the last 8 digits of the serial number of the Pi it is responsible for.
  • You can find the serial numbeer by running "cat /proc/cpuinfo" on each Pi.
  • You can also see this on your server syslog when a Pi attempts to netboot.
    • (See the log output at the end of the instructions)

Your tftp-boot folder should now look something like this

# ls -la /srv/boot
drwxr-xr-x  .
drwxr-xr-x  ..
drwxr-xr-x  12345678
drwxr-xr-x  87654321
lrwxrwxrwx  bootcode.bin
  • Now copy the remaining files from your Pi's boot folder into the respective tftp-boot sub-folder for that Pi
    • (So the files for the Pi with serial 12345678 go in the sub-folder named 12345678)

Your tftp-boot sub-folders should now look something like this

# ls -la /srv/boot/0e3dee9d
drwxr-xr-x  .
drwxr-xr-x  ..
-rwxr--r--  bcm2708-rpi-0-w.dtb
-rwxr--r--  bcm2708-rpi-b.dtb
-rwxr--r--  bcm2708-rpi-b-plus.dtb
-rwxr--r--  bcm2708-rpi-cm.dtb
-rwxr--r--  bcm2709-rpi-2-b.dtb
-rwxr--r--  bcm2710-rpi-3-b.dtb
-rwxr--r--  bcm2710-rpi-3-b-plus.dtb
-rwxr--r--  bcm2710-rpi-cm3.dtb
-rwxr-xr-x  config.txt
-rwxr--r--  COPYING.linux
-rwxr-xr-x  cmdline.txt
-rwxr--r--  fixup_cd.dat
-rwxr--r--  fixup.dat
-rwxr--r--  fixup_db.dat
-rwxr--r--  fixup_x.dat
-rwxr--r--  kernel8.img
-rwxr--r--  LICENCE.broadcom
drwxr-xr-x  overlays
-rwxr--r--  start_cd.elf
-rwxr--r--  start_db.elf
-rwxr--r--  start.elf
-rwxr--r--  start_x.elf

Note: Personally I keep my boot files in a Master boot directory and create symlinks to them in each of the folders as only cmdline.txt and config.txt would normally vary between Pi's.

NFS-Kernel-Server Set-up

"/etc/exports" is the nfs config file and specifys which folders are shared with the network and which machines on your network are allowed to access them. You will need to edit the file so it look something like this:

/srv/RPi1 192.168.0.101(rw,sync,no_subtree_check,no_root_squash)
/srv/RPi2 192.168.0.102(rw,sync,no_subtree_check,no_root_squash)

Each line is in the form : "Pi Root Folder" "IP address of the Pi"("options") Note: There is not a space between the IP address and the options!

Now lets create a folder for each of your Pi's Here's an example (note the boot folder we created earlier when setting up dnsmasq)

# ls -la /srv/
drwxr-xr-x  .
drwxr-xr-x  ..
drwxr-xr-x  boot
drwxr-xr-x  RPi1
drwxr-xr-x  RPi2

Next copy the entire filesystem from each Pi's SD card into it's new folder (assuming you've mounted the sd to /mnt/sd)

sudo cp -a /mnt/sd/* /srv/G-RPi1/

Pointing your Pi's at the new filesystem

Now we need to tell each Pi where to look for it's new filesystem. To do this we need to edit the following two files for each Pi:

sudo nano /srv/boot/12345678/cmdline.txt

Here's mine as an example

dwc_otg.lpm_enable=0 console=tty1 root=/dev/nfs rw nfsroot=192.168.0.2:/srv/RPi1,tcp,vers=3 ip=dhcp rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

The importent bits are:

  • root=/dev/nfs -- This tells the Pi its root partition will be over NFS
  • nfsroot=xxxx -- This is in the form "Servers IP Address":"Pi Root Folder","Use TCP","Use NFS Version 3"
  • ip=dhcp -- This tells the Pi to get it's IP address from the DHCP server

Everything else can be left the same as it was, just make sure you dont have any blank lines in the file!! Note: Depending on which NFS versions you have built into your Kernel you can use NFS vers=4 or vers=4.2 instead, I've used Vers=3 in the example as that should work out of the box for the standard Raspbian or Debian Kernels.

sudo nano /srv/RPi1/etc/fstab

Remove the lines starting "/" and "/boot" they are no longer needed. This will leave you with only the /proc line unless you have any other drives mapped.

Testing

To test first reboot your server to ensure everything has started up and then open two console windows.

In the first run:

sudo tail -F /var/log/syslog

This will show you your syslog output.

In the second run:

sudo watch -n0.5 "netstat -vepotu"

This will show your servers network activity updated every 0.5 sec.

Now start a Pi with no SD card in and you should see something like the following in the syslog output.

    May  9 01:48:43 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/bootsig.bin not found
    May  9 01:48:43 G-RPi12 dnsmasq-tftp[594]: sent /srv/boot/bootcode.bin to 192.168.0.104
    May  9 01:48:43 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/dfe614dd/autoboot.txt not found
    May  9 01:48:43 G-RPi12 dnsmasq-tftp[594]: error 0 Early terminate received from 192.168.0.104
    May  9 01:48:43 G-RPi12 dnsmasq-tftp[594]: failed sending /srv/boot/dfe614dd/start.elf to 192.168.0.104
    May  9 01:48:43 G-RPi12 dnsmasq-tftp[594]: sent /srv/boot/dfe614dd/config.txt to 192.168.0.104
    May  9 01:48:43 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/dfe614dd/recovery.elf not found
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: sent /srv/boot/dfe614dd/start_cd.elf to 192.168.0.104
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: sent /srv/boot/dfe614dd/fixup_cd.dat to 192.168.0.104
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/dfe614dd/recovery.elf not found
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: sent /srv/boot/dfe614dd/config.txt to 192.168.0.104
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/dfe614dd/dt-blob.bin not found
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/dfe614dd/recovery.elf not found
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: sent /srv/boot/dfe614dd/config.txt to 192.168.0.104
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/dfe614dd/bootcfg.txt not found
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: sent /srv/boot/dfe614dd/cmdline.txt to 192.168.0.104
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/dfe614dd/armstub8.bin not found
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: error 0 Early terminate received from 192.168.0.104
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: failed sending /srv/boot/dfe614dd/kernel8.img to 192.168.0.104
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/dfe614dd/armstub8-32.bin not found
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/dfe614dd/armstub7.bin not found
    May  9 01:48:45 G-RPi12 dnsmasq-tftp[594]: file /srv/boot/dfe614dd/armstub.bin not found
    May  9 01:49:28 G-RPi12 dnsmasq-tftp[594]: sent /srv/boot/dfe614dd/kernel8.img to 192.168.0.104
    May  9 01:49:28 G-RPi12 dnsmasq-tftp[594]: sent /srv/boot/dfe614dd/bcm2710-rpi-3-b.dtb to 192.168.0.104
    May  9 01:49:28 G-RPi12 dnsmasq-tftp[594]: sent /srv/boot/dfe614dd/config.txt to 192.168.0.104

Note: The "not found" messages and errors are perfectly normal, the Raspberry Pi only needs some of the files that the server can send to boot.

Once the last file has been sent keep an eye on your netstat output and you should see an NFS connection start up to your Pi and data should start transferring.

Give it a minute or two, netbooting can be sluggish and you should now be able to access your Pi.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment