Making a Raspbian-based AMMP edge image to run on Raspberry Pi

_make_ammp_raspbian_img.md

Create Raspbian SD card with image from https://www.raspberrypi.org/downloads/raspbian/

Create file named ssh in boot partition, so that ssh is enabled on boot.

Log into Raspberry Pi (user pi, password raspberry).

Set strong password on pi user. (passwd) (optional) Copy SSH key

ssh-copy-id -i id_ed25519_ammpedge pi@[raspberry_IP]

Check /etc/fstab and add noatime boot flag for root partition if not already there (it is in Buster 2019-09-26 image).

Disable journaling (run as root) - see https://waal70blog.wordpress.com/2017/12/16/disable-journaling-on-sd-card-raspberry-pi/

echo u > /proc/sysrq-trigger 
echo s > /proc/sysrq-trigger 
tune2fs -O ^has_journal /dev/mmcblk0p2 
e2fsck -fy /dev/mmcblk0p2 
echo s > /proc/sysrq-trigger 
echo b > /proc/sysrq-trigger

Note that the last command will reboot the device.

Run the script add_raspbian_testing_repo.sh in order to add the testing repo to Raspbian:

wget https://gist.githubusercontent.com/svet-b/6d2f11b59e5fec4416e65f20bce49cb1/raw/bc9dc7224c509bf7037dc38119720e2d448c888f/add_raspbian_testing_repo.sh
chmod a+x add_raspbian_testing_repo.sh
sudo ./add_raspbian_testing_repo.sh

Update available packages.

sudo apt update && sudo apt upgrade -y

Remove rsyslog and swap manager to disable log persistence and swapping to disk. (Potentially also other unneeded packages?)

sudo apt purge rsyslog dphys-swapfile -y

To ensure that DHCP works even if network cable is not plugged in at boot time, and that time is synced over NTP, and that we can scan networks:

sudo apt install -y netplug chrony nmap

Chrony doesn't start properly unless we disable systemd-timesyncd (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=883241)

sudo systemctl disable systemd-timesyncd

Optional: install static wifi hotspot and configuration utility:

sudo apt install git -y
git clone https://github.com/svet-b/rpi-wifi-config.git
cd rpi-wifi-config
./install.sh
cd ..
rm -rf rpi-wifi-config
sudo apt purge git

After the above, you may want to edit /etc/hostapd/hostapd.conf to change the default access point SSID and passphrase.

Set device configuration

sudo raspi-config

• Select 5 Interfacing Options > P5 I2C > Yes
• Select 4 Localisation Options > I4 Change Wi-fi Country > Relevant country

Also potentially:

• Disable waiting for network on boot (3->B2->No)
• Set the hostname (2->N1->ammp-edge)
• Set the timezone to UTC (4->I2->None of the above->UTC)
• Disable serial console but enable serial port (5->P6->No, Yes)

sudo apt-get install i2c-tools -y
sudo systemctl disable fake-hwclock
cd ~
wget http://sferalabs.cc/files/strato/rtc-install
chmod 755 rtc-install
sudo ./rtc-install
sudo reboot

Edit /boot/config.txt and add

# Disable Bluetooth
dtoverlay=pi3-disable-bt

Also run

sudo systemctl disable hciuart

Disable the ttyAMA0 console service:

sudo systemctl disable serial-getty@ttyAMA0.service

And remove serial console output from the kernel boot command; edit /boot/cmdline.txt and remove the part console=serial0,115200. (NOTE: this is probably taken care of through the RPi config utility)

Install latest version of snapd (>2.40)

sudo apt install snapd/testing -y

If you get a dependency error when doing the above you may need to run sudo apt install libseccomp2/testing explicitly.

Edit /etc/ld.so.preload and comment out line:

/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so

Install snaps (if all goes well, snapd and core18 will also be installed as dependencies.

sudo snap install ammp-edge
sudo snap install ammp-connect
sudo snap connect ammp-edge:system-observe
sudo snap connect ammp-edge:hardware-observe
sudo snap connect ammp-edge:log-observe
sudo snap connect ammp-edge:network-observe
sudo snap connect ammp-edge:network-control 

Optional - for built-in Wifi AP

sudo snap install ammp-wifi-ap
sudo snap connect ammp-wifi-ap:network-control
sudo snap connect ammp-wifi-ap:network-manager
sudo snap connect ammp-wifi-ap:firewall-control

Optional - for specific project config

sudo snap-install ammp-provisioning-xyz
sudo snap install ammp-influxdb
sudo snap install ammp-grafana

Verify that services are running and that SSH tunnel is up.

Remove unnecessary packages, if any. Use apt list --installed to get a list. Though the Raspbian Lite image doesn't appear to contain anything eye-poppingly unnecessary.

sudo apt-get autoremove -y
sudo apt-get autoclean -y
sudo apt-get clean -y

Also remove old versions of snaps (in case you've refreshed some while setting up:

snap list --all | while read snapname ver rev trk pub notes; do if [[ $notes = *disabled* ]]; then sudo snap remove "$snapname" --revision="$rev"; fi; done

Shut down Raspberry Pi. Grab SD card and make an image of it. Mount it.

# Linux:
sudo dd if=/dev/sdb of=rpi.img bs=8M
# Mac:
sudo dd if=/dev/rdisk2 of=rpi.img bs=8m
# On Mac do the following in order to allow R/W mounting from Linux VM
sudo chmod 666 rpi.img

# Then mount the image:
sudo losetup -Pfv rpi.img
losetup -j rpi.img
# Note the device name associated with the image, e.g. loop8
mkdir rpiroot
sudo mount /dev/loop8p2 ./rpiroot
cd rpiroot

The following are run from within the rpi directory.

Remove snap configuration (which contains node ID) and SSH tunnel log file.

sudo rm var/snap/ammp-edge/common/*
sudo rm var/snap/ammp-connect/common/*

Remove SSH host keys - this way fresh keys and an individual fingerprint will be generated for each host using the image.

sudo rm etc/ssh/ssh_host_*

We run dpkg-reconfigure openssh-server on first boot in order to regenerate these.

Clean home directory and temporary folders

sudo rm -rf home/pi/*
sudo rm home/pi/.bash_history
sudo rm -rf tmp/* tmp/.*
sudo rm -rf var/tmp/* var/tmp/.*

If desired, remove logs from var/log:

sudo rm -rf var/log/*

Now we can unmount and shrink the filesystem. We can use a modified version of https://github.com/Drewsif/PiShrink/blob/master/pishrink.sh (in this repo), which also includes the SSH host key creation.

cd ..
# Replace with actual loop device where partition was mounted
sudo umount /dev/loop8p2
wget https://gist.github.com/svet-b/6d2f11b59e5fec4416e65f20bce49cb1/raw/2c58bbf5b185ed5e6e11eb725d182829a074bd2a/pishrink.sh
chmod a+x pishrink.sh
cp rpi.img rpi-original.img
sudo ./pishrink.sh rpi.img

Done. The resulting rpi.img file should be around 2GB and can be compressed to ~800MB.

add_raspbian_testing_repo.sh

#!/bin/bash -e

# From https://serverfault.com/a/382101

cat > /etc/apt/preferences.d/stable.pref << EOF 
# 500 <= P < 990: causes a version to be installed unless there is a
# version available belonging to the target release or the installed
# version is more recent

Package: *
Pin: release a=stable
Pin-Priority: 900
EOF

cat > /etc/apt/preferences.d/testing.pref << EOF 
# 100 <= P < 500: causes a version to be installed unless there is a
# version available belonging to some other distribution or the installed
# version is more recent

Package: *
Pin: release a=testing
Pin-Priority: 400
EOF

cp /etc/apt/sources.list /etc/apt/sources.list.d/stable.list
mv /etc/apt/sources.list /etc/apt/sources.list.orig

cat > /etc/apt/sources.list.d/testing.list << EOF
deb http://raspbian.raspberrypi.org/raspbian/ bullseye main contrib non-free rpi
EOF

clean_image.sh

#!/bin/bash -e

IMG_FILE=rpi.img
REMOTE_IMG_DIR=tmp
LOCAL_IMG_DIR=~
MOUNT_DIR=rpiroot

sudo mount -t vboxsf svet /mnt/svet
echo 'Mounted home directory'
echo 'Copying image to VM'

cp /mnt/svet/"$REMOTE_IMG_DIR"/"$IMG_FILE" "$LOCAL_IMG_DIR"/
echo 'Done'
echo

if [[ $(losetup -j "$LOCAL_IMG_DIR"/"$IMG_FILE") ]]; then
    LOOP_DEV=$(losetup -j "$LOCAL_IMG_DIR"/"$IMG_FILE" | head -n 1 | cut -d: -f1)
    echo "Image already available as device $LOOP_DEV"
else
    echo 'Creating loop device'
    sudo losetup -Pfv $IMG_FILE
    LOOP_DEV=$(losetup -j "$LOCAL_IMG_DIR"/"$IMG_FILE" | head -n 1 | cut -d: -f1)
    echo "Image available as device $LOOP_DEV"
fi

mkdir -p "$LOCAL_IMG_DIR"/$MOUNT_DIR

# Note that we're mounting the second (root) partition of the image
sudo mount "$LOOP_DEV"p2 $LOCAL_IMG_DIR/$MOUNT_DIR
cd $LOCAL_IMG_DIR/$MOUNT_DIR
echo "Mounted image as $LOCAL_IMG_DIR/$MOUNT_DIR"
echo 'Doing clean-up on image...'
echo '(safe to ignore some errors here)'
echo
echo 'Removing snap configuration and SSH tunnel log file.'

sudo rm var/snap/ammp-edge/common/*
sudo rm var/snap/ammp-connect/common/*

echo
echo 'Removing SSH host keys (will be regenerated on first boot)'

sudo rm etc/ssh/ssh_host_*

echo
echo 'Cleaning home directory and temporary folders'

sudo rm -rf home/pi/*
sudo rm home/pi/.bash_history
sudo rm -rf tmp/* tmp/.*
sudo rm -rf var/tmp/* var/tmp/.*

echo
echo 'Removing logs from var/log'

sudo rm -rf var/log/*

echo
echo 'Done with clean-up'
echo

cd ..
echo 'Sleeping 3 seconds and unmounting'
sleep 3
sudo umount "$LOOP_DEV"p2
echo 'Unmounted image'
echo 
echo 'Zeroing out free blocks on device with zerofree'
sudo zerofree "$LOOP_DEV"p2
echo 'Done'

sudo losetup -d "$LOOP_DEV"
echo "Removed loop device $LOOP_DEV"

echo
echo 'Downloading pishrink.sh'
wget -O pishrink.sh -q https://gist.github.com/svet-b/6d2f11b59e5fec4416e65f20bce49cb1/raw/2c58bbf5b185ed5e6e11eb725d182829a074bd2a/pishrink.sh
chmod a+x pishrink.sh
echo
echo 'Sleeping 15 seconds'
sleep 15
# The following error appears to be common at this stage. Not sure what is necessary to prevent it?
# losetup: rpi.img: failed to set up loop device: Resource temporarily unavailable
# losetup: : failed to use device: No such device
echo
echo 'Running pishrink.sh on image'
sudo ./pishrink.sh "$LOCAL_IMG_DIR"/"$IMG_FILE"

echo
echo 'Copying image back out of VM'

sudo cp "$LOCAL_IMG_DIR"/"$IMG_FILE" /mnt/svet/"$REMOTE_IMG_DIR"/new_"$IMG_FILE"
echo "Copied to $REMOTE_IMG_DIR/new_$IMG_FILE"
echo
echo 'DONE'

make_image.sh

#!/bin/bash -e

export IMG_DIR=~/tmp

sudo dd if=/dev/rdisk2 of="$IMG_DIR"/rpi.img bs=8m
sudo chmod 666 "$IMG_DIR"/rpi.img

ssh svet@ubuntu18 'bash -s' < clean_image.sh

pishrink.sh

#!/bin/bash

function cleanup() {
  if losetup $loopback &>/dev/null; then
	losetup -d "$loopback"
  fi
}

usage() { echo "Usage: $0 [-s] imagefile.img [newimagefile.img]"; exit -1; }

should_skip_autoexpand=false

while getopts ":s" opt; do
  case "${opt}" in
    s) should_skip_autoexpand=true ;;
    *) usage ;;
  esac
done
shift $((OPTIND-1))

#Args
img="$1"

#Usage checks
if [[ -z "$img" ]]; then
  usage
fi
if [[ ! -f "$img" ]]; then
  echo "ERROR: $img is not a file..."
  exit -2
fi
if (( EUID != 0 )); then
  echo "ERROR: You need to be running as root."
  exit -3
fi

#Check that what we need is installed
for command in parted losetup tune2fs md5sum e2fsck resize2fs; do
  which $command 2>&1 >/dev/null
  if (( $? != 0 )); then
    echo "ERROR: $command is not installed."
    exit -4
  fi
done

#Copy to new file if requested
if [ -n "$2" ]; then
  echo "Copying $1 to $2..."
  cp --reflink=auto --sparse=always "$1" "$2"
  if (( $? != 0 )); then
    echo "ERROR: Could not copy file..."
    exit -5
  fi
  old_owner=$(stat -c %u:%g "$1")
  chown $old_owner "$2"
  img="$2"
fi

# cleanup at script exit
trap cleanup ERR EXIT

#Gather info
beforesize=$(ls -lh "$img" | cut -d ' ' -f 5)
parted_output=$(parted -ms "$img" unit B print | tail -n 1)
partnum=$(echo "$parted_output" | cut -d ':' -f 1)
partstart=$(echo "$parted_output" | cut -d ':' -f 2 | tr -d 'B')
loopback=$(losetup -f --show -o $partstart "$img")
tune2fs_output=$(tune2fs -l "$loopback")
currentsize=$(echo "$tune2fs_output" | grep '^Block count:' | tr -d ' ' | cut -d ':' -f 2)
blocksize=$(echo "$tune2fs_output" | grep '^Block size:' | tr -d ' ' | cut -d ':' -f 2)

#Check if we should make pi expand rootfs on next boot
if [ "$should_skip_autoexpand" = false ]; then
  #Make pi expand rootfs on next boot
  mountdir=$(mktemp -d)
  mount "$loopback" "$mountdir"

  if [ $(md5sum "$mountdir/etc/rc.local" | cut -d ' ' -f 1) != "a25c1764c142f022c73245a00c5d1c28" ]; then
    echo "Creating new /etc/rc.local"
    mv "$mountdir/etc/rc.local" "$mountdir/etc/rc.local.bak"
    #####Do not touch the following lines#####
cat <<\EOF1 > "$mountdir/etc/rc.local"
#!/bin/bash
do_expand_rootfs() {
  ROOT_PART=$(mount | sed -n 's|^/dev/\(.*\) on / .*|\1|p')

  PART_NUM=${ROOT_PART#mmcblk0p}
  if [ "$PART_NUM" = "$ROOT_PART" ]; then
    echo "$ROOT_PART is not an SD card. Don't know how to expand"
    return 0
  fi

  # Get the starting offset of the root partition
  PART_START=$(parted /dev/mmcblk0 -ms unit s p | grep "^${PART_NUM}" | cut -f 2 -d: | sed 's/[^0-9]//g')
  [ "$PART_START" ] || return 1
  # Return value will likely be error for fdisk as it fails to reload the
  # partition table because the root fs is mounted
  fdisk /dev/mmcblk0 <<EOF
p
d
$PART_NUM
n
p
$PART_NUM
$PART_START

p
w
EOF

cat <<EOF > /etc/rc.local &&
#!/bin/sh
echo "Expanding /dev/$ROOT_PART"
resize2fs /dev/$ROOT_PART
rm -f /etc/rc.local; cp -f /etc/rc.local.bak /etc/rc.local; /etc/rc.local

EOF
reboot
exit
}
raspi_config_expand() {
/usr/bin/env raspi-config --expand-rootfs
if [[ $? != 0 ]]; then
  return -1
else
  rm -f /etc/rc.local; cp -f /etc/rc.local.bak /etc/rc.local; /etc/rc.local
  reboot
  exit
fi
}
echo "Generating SSH host keys"
dpkg-reconfigure openssh-server
echo "Attempting to use native raspi-config expand-rootfs"
raspi_config_expand
echo "WARNING: Using backup expand..."
sleep 5
do_expand_rootfs
echo "ERROR: Expanding failed..."
sleep 5
rm -f /etc/rc.local; cp -f /etc/rc.local.bak /etc/rc.local; /etc/rc.local
exit 0
EOF1
    #####End no touch zone#####
    chmod +x "$mountdir/etc/rc.local"
  fi
  umount "$mountdir"
else
  echo "Skipping autoexpanding process..."
fi

#Make sure filesystem is ok
e2fsck -p -f "$loopback"
minsize=$(resize2fs -P "$loopback" | cut -d ':' -f 2 | tr -d ' ')
if [[ $currentsize -eq $minsize ]]; then
  echo "ERROR: Image already shrunk to smallest size"
  exit -6
fi

#Add some free space to the end of the filesystem
extra_space=$(($currentsize - $minsize))
for space in 5000 1000 100; do
  if [[ $extra_space -gt $space ]]; then
    minsize=$(($minsize + $space))
    break
  fi
done

#Shrink filesystem
resize2fs -p "$loopback" $minsize
if [[ $? != 0 ]]; then
  echo "ERROR: resize2fs failed..."
  mount "$loopback" "$mountdir"
  mv "$mountdir/etc/rc.local.bak" "$mountdir/etc/rc.local"
  umount "$mountdir"
  losetup -d "$loopback"
  exit -7
fi
sleep 1

#Shrink partition
partnewsize=$(($minsize * $blocksize))
newpartend=$(($partstart + $partnewsize))
parted -s -a minimal "$img" rm $partnum >/dev/null
parted -s "$img" unit B mkpart primary $partstart $newpartend >/dev/null

#Truncate the file
endresult=$(parted -ms "$img" unit B print free | tail -1 | cut -d ':' -f 2 | tr -d 'B')
truncate -s $endresult "$img"
aftersize=$(ls -lh "$img" | cut -d ' ' -f 5)

echo "Shrunk $img from $beforesize to $aftersize"