This gist builds on the excellent work done by Graham Garner, Andreas Spiess, and Peter Scargill to make it easy for Makers to install IOTstack on a Raspberry Pi.
One concern with any out-of-the-box Raspberry Pi is the limited lifespan of its SD card. You'd be pretty annoyed if you got your IOTstack up and running, only to find that you were one of the unlucky ones whose SD card wore out in less than six months!
SSDs, on the other hand, while not immune to the same kinds of failure, tend to have projected life expectancies in the hundreds of years.
The problem is that, while a Raspberry Pi 3B+ can boot and run from an external USB drive, the Raspberry Pi 4B can't yet do this. Given its two USB-3 ports, that's a real pity.
The Internet contains a huge amount of how-to on getting Raspberry Pi's to talk to external storage devices but I was unable to find a step-by-step guide to take you from bare-metal to IOTstack running on an SSD attached to a USB-3 port of a Raspberry Pi 4B. This gist is an attempt to remedy that.
Although this gist assumes installing IOTstack is the primary task goal, it is not essential. If you simply want to get Raspbian running on an SSD, just skip the bits that talk about IOTstack and Docker.
Perform a bare-metal installation of Raspbian on a Raspberry Pi 4B such that the system:
-
Boots from an SD card.
-
Runs from a larger-capacity SSD attached to a USB-3 port.
-
Advertises itself using the multicast-DNS name "iot-hub.local"
-
Supports headless operation via SSH and VNC.
-
Supports SQLite3.
-
Has a base install of IOTstack and Docker with:
- Mosquitto
- Node-Red
- InfluxDB
- Grafana
- Portainer
Caveat: I am using an iMac running macOS Mojave 10.14 as my support platform. Some applications and Terminal commands are specific to that platform. You may need to adapt this gist if you are using MS Windows or Linux as your support platform.
Open a browser page at www.raspberrypi.org/downloads/raspbian/ and download an appropriate image ("Raspbian Buster with desktop"). This arrives as something like:
~/Downloads/2020-02-05-raspbian-buster.zip
Calculate the digital signature of the downloaded file and compare it with the signature published on the site to assure no tampering or corruption. This assumes OpenSSL is installed (hint: "brew install openssl").
$ openssl dgst -sha256 ~/Downloads/2020-02-05-raspbian-buster.zip
Insert the SD card in the Mac. Or, more precisely, insert the Raspberry Pi's microSD card into an SD card adapter and insert the adapter into your Mac. Use BalenaEtcher to write 2020-02-05-raspbian-buster.zip to the SD card. macOS will prompt for your administrator password. BalenaEtcher creates two partitions on the SD card:
- boot is mountable on the Mac
- the second partition (ext4) is not mountable on the Mac.
Depending on BalenaEtcher's user preferences, the boot partition may be unmounted automatically at the end of the etching process. If it is, just pull out the SD card adapter, re-insert it, and wait for the boot partition to appear on the desktop.
In macOS Terminal, instruct Raspbian to permit access via SSH.
$ touch /Volumes/boot/ssh
Next, set up your WiFi credentials. Throughout these instructions I use the VI text editor. Feel free to use the text editor of your choice.
$ vi /Volumes/boot/wpa_supplicant.conf
Copy the text below to the clipboard.
country=«CC»
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
ssid="«SSID»"
psk="«PSK»"
}
Switch back to the Terminal session, and paste the text into VI. Replace the embedded «delimited» fields with values appropriate to your situation:
- «CC» = a two-character upper-case country code, eg "AU"
- «SSID» = WiFi Network Name
- «PSK» = Password for WiFi network
Save and quit. Confirm that the WiFi credentials are in place.
$ cat /Volumes/boot/wpa_supplicant.conf
Eject the boot volume by dragging to the trash. Although this leaves some macOS Spotlight support files on the volume, it does not seem to worry Raspbian.
Remove the SD card from the Mac.
Insert the SD card into the Raspberry Pi. Apply power. Once it comes up and connects to WiFi it will advertise itself with the multicast-DNS name "raspberrypi.local". You can test for its presence with:
$ ping raspberrypi.local
Among other things, executing "ping" will give you the IP address assigned to the Raspberry Pi by DHCP. You can then find the MAC address of the WiFi interface via:
$ arp -a
and, in turn, use the MAC address to tell your DHCP server to assign a static IP address. While you can rely on a multicast-DNS name, you normally want a predictable IP address for an IoT server. The alternative is to configure the Raspberry Pi with a static IP address internally but a static assignment in a DHCP server is usually simpler and better for long-term maintenance.
Make sure SSH forgets any previous fingerprints for "raspberrypi.local". This may or may not return an error (either way, ignore the result).
$ ssh-keygen -R raspberrypi.local
Connect to the Raspberry Pi via SSH.
$ ssh pi@raspberrypi.local
Respond "yes" to fingerprint prompts from SSH and supply the Raspberry Pi's default password ("raspberry") when requested.
If you plan to set up a DHCP static assignment for the Ethernet port, you can identify its MAC address using:
$ ifconfig eth0
Basic configuration of the Raspberry Pi is via "raspi-config".
$ sudo raspi-config
Work through the following options:
1 Change User Password
⎣ «new password»
2 Network Options
⎣ N1 Hostname
⎣ «host name» eg "iot-hub"
4 Localisation Options
⎣ I2 Change Timezone
⎣ «set country and city»
⎣ I4 Change Wi-Fi Country
⎣ «two-character upper-case country code» eg "AU"
5 Interfacing Options
⎣ P3 VNC
⎣ enable VNC server
6 Advanced Options
⎣ A5 Resolution
⎣ (DMT Mode 82 1920x1080 60Hz 16:9)
8 Update
Finish (hint: [TAB], then right-arrow)
The wisdom of always changing the host name will make sense, come the day you get another Raspberry Pi. Having two devices on your network using the same multicast-DNS name ("raspberrypi.local") is a recipe for a mess.
Reboot (ignore any error referring to "raspberrypi.local").
$ sudo reboot
Use the macOS Terminal to re-connect with the Raspberry Pi under its new name (which, from this point on, is assumed to be "iot-hub"). The first two lines below clear any previous fingerprints from the SSH known hosts file.
$ ssh-keygen -R raspberrypi.local
$ ssh-keygen -R iot-hub.local
$ ssh pi@iot-hub.local
Again, respond "yes" to fingerprint prompts from SSH but this time supply the new password you just set in "raspi-config".
Worst case, in the event of problems, is the need to start over from BalenaEtcher.
Bring the system fully up-to-date (the first line is probably redundant but there is no harm in repeating the step).
$ sudo apt update
$ sudo apt dist-upgrade
Depending on the Raspbian image you chose as your starting point, some of the tools you are likely to need later may not be installed. The table below summarises what you should do next:
| Package | Test | Expected Path | Command |
|---|---|---|---|
| GIT | which git | /usr/bin/git | sudo apt install git |
| Access Control | which setfacl | /usr/bin/setfacl | sudo apt install acl |
| SQLite3 | which sqlite3 | /usr/bin/sqlite3 | sudo apt install sqlite3 |
| DNS Utilities | which dig | /usr/bin/dig | sudo apt install dnsutils |
Using GIT as an example, check whether "git" is available:
$ which git
The answer will EITHER be the expected path "/usr/bin/git" OR a null result. IF the answer is a null result, then execute the command in the fourth column:
sudo apt install git
Repeat those steps for the remaining rows in the table.
Generate a password for VNC.
$ sudo vncpasswd -print
You will be prompted for a password and to verify it. Use the same password as set in "raspi-config" (or you'll go mad). You will wind up with three lines like this:
Password:
Verify:
Password=«password hash»
The value in the «password hash» field above will be needed later. Copy the following lines to the clipboard.
Encryption=PreferOn
Authentication=VncAuth
Password=«password hash»
Create the VNC configuration file.
$ sudo vi /etc/vnc/config.d/common.custom
Paste the three lines copied to the clipboard in the previous step into VI and then replace the «password hash» with the value returned by "vncpassword" in the preceding step. Save and exit.
Confirm that the changes have been applied.
$ cat /etc/vnc/config.d/common.custom
Restart the VNC server (which was set running by the earlier "raspi_config").
$ sudo systemctl restart vncserver-x11-serviced
It should now be possible to connect to the Raspberry Pi from the Mac via VNC. In the macOS Finder, press Command+K and use this URL:
vnc://pi@iot-hub.local
Follow all GUI prompts to the bitter end. If prompted for another password change, use the same password as in previous steps (or you'll go mad). The system will suggest restarting. Accept that suggestion and close the VNC window.
The next few steps assume that:
- the SSD you want to use for Raspbian has been connected to your Mac and formatted via Disk Utility as a Master Boot Record FAT volume with the name "EXTSSD". Pre-formatting of the SSD is not essential but having known characteristics makes it easy to identify the SSD when it is first attached to the Raspberry Pi.
- The drive has been ejected from the Mac.
Connect the SSD to a USB-3 port on the Raspberry Pi.
In the macOS Terminal window, reconnect to the Raspberry Pi via SSH.
$ ssh pi@iot-hub.local
On the Raspberry Pi, locate the available block devices.
$ lsblk
The output should look similar to this:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 1 14.6G 0 disk
└─sda1 8:1 1 14.6G 0 part /media/pi/EXTSSD
mmcblk0 179:0 0 14.6G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 14.3G 0 part /
Each entry in the "NAME" column implies a "/dev/" prefix. We are interested in:
- the external SSD physical disk: /dev/sda,
- the external SSD logical volume: /dev/sda1, and
- the current root volume /dev/mmcblk0p2 on the SD card.
Notice how the external SSD logical volume has an entry in the "MOUNTPOINT" column. This means the volume is mounted. Unmount it.
$ sudo umount /dev/sda1
Prepare the external SSD physical disk for use by Raspbian. The "⏎" glyph is used throughout this documentation to mean "press the Return key".
$ sudo fdisk /dev/sda
d
n
p
1
⏎
⏎
(respond Y if asked to remove a prior signature)
w
Construct an ext4 type file system on the external SSD logical volume.
$ sudo mkfs -t ext4 /dev/sda1
Clone the root file system from the SD onto the SSD (refer back to the "lsblk" output as the source of the /dev/mmcblk0p2 and /dev/sda1 parameter values).
$ sudo dd if=/dev/mmcblk0p2 of=/dev/sda1 bs=64K conv=noerror,sync
Create a mount point for the SSD on the SD card.
$ sudo mkdir /ssd
Mount the cloned volume.
$ sudo mount /dev/sda1 /ssd
Edit the fstab for the running system (ie the SD) so that it knows about and will auto-mount the SSD at /ssd
$ sudo vi /etc/fstab
The final result should look something like this (the critical change is adding the 4th line).
proc /proc proc defaults 0 0
PARTUUID=d9b3f436-01 /boot vfat defaults 0 2
PARTUUID=d9b3f436-02 / ext4 defaults,noatime 0 1
/dev/sda1 /ssd ext4 defaults 0 2
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that
Make a mount-point for the SD card in the (non-running) SSD system.
$ sudo mkdir /ssd/sd
Edit the fstab for the new system (ie the clone on the SSD) so that it knows about and will auto-mount the SD at /sd
$ sudo vi /ssd/etc/fstab
The key changes are:
- 3rd line:
- change the mount point from "/" to "/sd". Remember, this is editing the fstab on the SSD which is not yet the running system.
- remove the ",noatime" mount option
- change the value of the right-most parameter (fsck order) from "1" to "2"
- Add the 4th line, "as is".
The final result should look something like the following:
proc /proc proc defaults 0 0
PARTUUID=d9b3f436-01 /boot vfat defaults 0 2
PARTUUID=d9b3f436-02 /sd ext4 defaults 0 2
/dev/sda1 / ext4 defaults,noatime 0 1
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that
What we have just done is to say:
- When the Raspberry Pi runs from the SD card, the SSD will be auto-mounted at /ssd
- When the Raspberry Pi runs from the SSD, the SD card will be auto-mounted at /sd
The Raspberry Pi will always boot from the /boot volume on the SD card but will then run Raspbian from whichever volume we choose in the next step.
Change into the boot partition.
$ cd /boot
At the moment, cmdline.txt is instructing the Raspberry Pi to run Raspbian from the SD. Make a backup of cmdline.txt which preserves that.
$ sudo cp cmdline.txt cmdline.run-from-sd-card.txt
Edit cmdline.txt to tell the Raspberry Pi to run Raspbian from the SSD.
$ sudo vi cmdline.txt
Replace the right hand side of "root=" with "/dev/sda1" so that the final result looks something like this:
console=serial0,115200 console=tty1 root=/dev/sda1 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles
Make a copy of that newly-edited file to preserve the instruction to run Raspbian from the SSD.
$ sudo cp cmdline.txt cmdline.run-from-ssd.txt
The presence of these two files both documents what you have done and makes it easy to swap back and forth between the two run methods by executing one of the following:
$ sudo cp cmdline.run-from-ssd.txt cmdline.txt
$ sudo cp cmdline.run-from-sd-card.txt cmdline.txt
Right now, the "run from SSD" variant is ready to take effect so let's do that.
$ sudo reboot
Once the Raspberry Pi comes back up, use the macOS Terminal to re-connect.
$ ssh pi@iot-hub.local
Because we cloned from a 16GB SD card, the current file system is limited to that size. You can prove that to yourself.
$ df -h
This line shows a 15GB partition (the rest of the space is in the boot partition):
/dev/root 15G 3.0G 11G 22% /
Tell the system to use all of the partition on the SSD.
$ sudo resize2fs /dev/sda1
Confirm that the expansion has taken effect.
$ df -h
This line shows a much larger partition:
/dev/root 441G 3.0G 420G 1% /
Reboot (prophylactic).
$ sudo reboot
Complete instructions for installing IOTstack on the Raspberry Pi are available at sensorsiot.github.io/IOTstack/. This is a summary.
Previously, the instructions were at github.com/gcgarner/IOTstack
Connect to the Raspberry Pi from macOS Terminal.
$ ssh pi@iot-hub.local
Clone the IOTstack GIT repository.
git clone https://github.com/SensorsIot/IOTstack.git ~/IOTstack
Previously, the URL was "https://github.com/gcgarner/IOTstack.git"
Install Docker.
$ cd ~/IOTstack
$ ./menu.sh
Choose "Install Docker"
Installing Docker involves a reboot of the Raspberry Pi so reconnect from macOS Terminal.
$ ssh pi@iot-hub.local
Back on the Raspberry Pi, select the Docker containers you want to install.
$ cd ~/IOTstack
$ ./menu.sh
Choose "Build Stack"
This is the basic set you should probably include:
- Portainer
- Node-Red
- InfluxDB
- Grafana and
- Eclipse-Mosquitto
To change what is installed, navigate with the arrow keys and press the space-bar to opt in or out. For example, to add Pi-Hole:
[x] Pi-Hole
When you are ready, press [TAB] to select "<Ok>" and press ⏎.
A list of Node-RED nodes is shown. Select each one you require by navigating with the arrow keys and pressing the space-bar. Example:
[x] node-red-node-sqlite
Note: Installing "node-red-node-sqlite" produces a massive number of compiler warnings during the subsequent "docker-compose up -d" step. While they seem alarming, these warnings can be ignored.
To complete the process press [TAB] to select "<Ok>" and press ⏎ to generate the YAML file.
Instruct Docker to download, install and activate all selected container services.
$ docker-compose up -d
Have a cup of coffee. Once the process completes, confirm that your chosen containers are running.
$ docker ps
Periodically, do this:
$ sudo apt update
$ sudo apt upgrade
$ sudo reboot
This will also update Docker if need be. If you suspect that a package you need is not being updated because of a dependency issue, you can consider replacing "sudo apt upgrade" with:
$ sudo apt full-upgrade
Bring the project up-to-date:
$ cd ~/IOTstack
$ git status
Review any changes flagged by git and take appropriate action (eg making backup copies). Then:
$ git pull origin master
If any changes applied by the pull appear to affect your stack (eg a template for one of your containerised processes has been updated), re-run the menu and follow your nose:
$ ./menu.sh
To update everything EXCEPT Node-Red:
$ cd ~/IOTstack
$ docker-compose pull
$ docker-compose up -d
To update everything INCLUDING Node-Red:
$ cd ~/IOTstack
$ docker-compose down
$ docker rmi "iotstack_nodered" "nodered/node-red"
$ docker-compose pull
$ docker-compose up --build -d
Remember that a rebuild of Node-Red takes a long time if SQLite needs to be recompiled and chucks up a lot of compiler warnings and even seems to stall. Be patient!
After any update, check if any extraneous images have been left behind:
$ docker images
Remove extraneous images using their "Image ID", as in:
$ docker rmi 0c9df6f99685
As well as VNC access, it is useful to be able to mount the Raspberry Pi's file system on the Mac desktop. The Raspberry Pi is ready to rock-'n-roll but some work needs to be done on the Mac. This assumes macOS Mojave 10.14. Your mileage may vary on earlier or later versions.
First, install SSHFS if it is not there already.
$ brew update
$ brew install sshfs
Establish a mount point on the macOS system. The mount point can be anywhere. For example:
$ mkdir ~/pi
Mount the remote volume. Note the trailing colon at the end of the domain name, followed by a space. Always be very careful to get the host name (in this case "iot-hub.local") exactly right. Any mistake can hang this command, hang the Finder (spinning beachball) and result in the need for a forced power-off/power-on. If you happen to have another Terminal window open, you can get away with a "killall sshfs" but that still leaves the mount point in an indeterminate state and a restart is the only way to recover. Not very Mac-like, I know.
$ sshfs pi@iot-hub.local: ~/pi
The remote volume will mount on your desktop. You can copy files into and out of the Raspberry Pi's volume. You can also access the remote volume from the macOS Terminal like this:
$ cd ~/pi
When you are finished, drag the volume to the trash to eject it.