Skip to content

Instantly share code, notes, and snippets.

@cywf

cywf/pxe_rpi.md

Last active November 9, 2022 02:03
Show Gist options
  • Save cywf/20b104cd36cc91e1ddad22d801e85e17 to your computer and use it in GitHub Desktop.
Save cywf/20b104cd36cc91e1ddad22d801e85e17 to your computer and use it in GitHub Desktop.
Download ZIP
Conceptual Overview of setting up a Raspberry Pi to boot from images hosted on a network share.
Raw
pxe_rpi.md

Screen Shot 2022-11-08 at 7 51 11 PM

PXE / Network Booting Raspberry Pi's

Overview

  • PXE/Netbooting Pis is generally considered beta

    • stable once everything is working

  • Pi boots off the network

Benefits

  • No SD Card, or other local drive

    • SD Cards can be unreliable, stolen, lost, hard to track

  • Security

    • Pi's are cheapo, data is not

  • Reliability

    • Server storage is more durable and reliable than SD Cards

  • Centralized storage for all Pi's

    • Backup all Pi's from a central location, offloading the work

  • Shared binaries and systems

    • Share the same OS, configuration, and/or settings

  • Scaliability

    • Practical way to manage quantities of Pi's

  • Use the SD Care slot

    • Installation/Re-Installation of OS

Requirements

Non-Standard PXE

  • Does not use PXE bootloaders

  • Does not use common DHCP PXE options

    • ISC DHCP: filename "pxelinux.0"
    • dnsmasq: dhcp-boot=pxelinux.0

  • Pi specific DHCP Options

    • ISC DHCP: option vendor-class-identifier "PXEClient"; option vendor-encapsulated-options "Raspberry Pi Boot";
    • dnsmasq: pxe-service=0, "Raspberry Pi Boot"

  • Short window for booting

    • Around 30 seconds, does not retry
    • Requires portfast on switches

Scalable DHCP

  • Adding options to lots of hosts is unweildy

  • Use class feature of ISC DHCP

### PXE Netboot Pi's

class "rpi-pxe" {
	match if ((
		(binary-to-ascii(16,8,":",substring(hardware,1,3)) = "b8:27:eb") or
 		(binary-to-ascii(16,8,":",substring(hardware,1,3)) = "dc:a6:32")
 		) and
 		(option vendor-class-identifier = "PXEClient:Arch:00000:UNDI:002001")
 		);
			option tftp-server-name "10.1.10.1";
			option vendor-class-identifier "PXEClient";
			option vendor-encapsulated-options "Raspberry Pi Boot";
}

### DHCP Netboot Pi's

class "rpi-dhcp" {
	match if (
		(binary-to-ascii(16,8,":",substring(hardware,1,3)) = "b8:27:eb") or
 		(binary-to-ascii(16,8,":",substring(hardware,1,3)) = "dc:a6:32")
 		);
			option root-path "10.1.10.1:/srv/nfs/pi,vers=3";
}

TFTP Directory Layout

  • Requires all files in /boot directory to be in /tftproot

    • creates a messy tftp root directory

  • Option to use device specific directory

    • folder named using device serial number in /tftproot
    • only way to get serial is cat /proc/cpuinfo
    • can limit boot files to device specific

  • Copy all boot files from Github repo

  • Must edit cmdline.txt

    • change root path: root=/dev/nfs
    • add nfs specific options: nfsroot=x.x.x.x:/path/to/distro,vers=3
    • add rootfstype=nfs
    • add ip=dhcp
    • optionally disable quiet and splash to show details
console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=10.1.10.1:/srv/nfs/pi,vers=3 rootfstype=nfs rw ip=dhcp
elevator=deadline rootwait noquiet nosplash plymouth.ignore-serial-consoles

NFS

  • Single directory per system, single export

    • /srv/nfs/pi 10.1.10.2 (rw,async,no_root_squash,no_subtree_check,fsid=1001)

  • Bind mounted shared system, multiple exports

    • only exporting bind mounted directories
    • needs crossmount for main export, nohide for bind mounts
    • /srv/nfs/pi *(rw,async,crossmnt,nohide,no_root_squash,no_subtree_check,fsid=100)
    • /srv/nfs/pi/bin *(rw,async,nohide,no_root_squash,no_subtree_check,fsid=101)
    • /srv/nfs/pi/boot *(rw,async,nohide,no_root_squash,no_subtree_check,fsid=102)
    • /srv/nfs/pi/home *(rw,async,nohide,no_root_squash,no_subtree_check,fsid=103)
    • /srv/nfs/pi/lib *(rw,async,nohide,no_root_squash,no_subtree_check,fsid=104)
    • /srv/nfs/pi/opt *(rw,async,nohide,no_root_squash,no_subtree_check,fsid=105)
    • /srv/nfs/pi/root *(rw,async,nohide,no_root_squash,no_subtree_check,fsid=106)
    • /srv/nfs/pi/sbin *(rw,async,nohide,no_root_squash,no_subtree_check,fsid=107)
    • /srv/nfs/pi/usr *(rw,async,nohide,no_root_squash,no_subtree_check,fsid=108)

  • Either way, single mount point for boot purposes

Bind mount fstab example

  • This is from a Pi, sharing binaries for other Pi's

Extracting and copying Raspbian

  • Download and unpack img file from zip

  • Use fdisk to see partition offset of root partition (usually 532480)

    • calculate partition offset, multiply by 512 (usually 272629760)

  • Loopback mount the image using partition offset

  • Copy files, preserving ownership and permissions

Before first PXE/net boot

  • Edit /etc/fstab, comment out /boot and / mounts

    • filesystem will already be mounted

  • Optionally, copy ssh keys

  • Optionally, change default.target

    • toggle between graphical and CLI aka multi-user

  • Optionally, enable/disable services

Troubleshooting

  • tcpdump is best for both DHCP and TFTP

    • tcpdump -n -v -i eth0 port 68 (dhcp)
    • tcpdump -n -v -i eth0 port 69 (tftp)
    • best to use two terminals and seprate commands

  • Pi will request lots of files not found via tftp

    • this is normal and makes debugging tftp difficult

  • Limit CPUs for kernal panic details

    • add maxcpus=2 to kernel options

  • If boot hangs or crashes: ensure latest boot files

  • Try disabling any service that hangs

    • add systemd.confirm_spawn=true to kernel options

  • Bind mounted NFS shares require additional exports

    • mount share on another system
    • check for empty directories
    • export any directory that is empty

  • General NFS issues: view server logs

  • Systemd is not your friend

Whats next?

  • Install Gentoo instead of Raspbian

    • use openrc instead of systemd

  • Turn a Pi into a general PXE/Netboot device

    • portable to fix other systems

  • Use a Pi for USB booting

    • ability to modify USB files for booting

References

Credit to wltjr for his presentation and lessons on implementing and contrubution to the development of this project!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment