Skip to content

Instantly share code, notes, and snippets.

@Paraphraser

Paraphraser/999058924152033470.md

Last active December 24, 2022 03:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Paraphraser/1ae57538b88662404c84ca6bddfc7008 to your computer and use it in GitHub Desktop.
Save Paraphraser/1ae57538b88662404c84ca6bddfc7008 to your computer and use it in GitHub Desktop.
Download ZIP
PiBuilder + Windows: Discord question
Raw
999058924152033470.md

PiBuilder + Windows: Discord question

Question 1

I tried to build a new installation using PiBuilder. I am a Windows user and connect with a terminal emulator (Putty) to the Pi. So far I was not successful. I tried to follow the description. But Windows only mounts the boot partition. The rest of the SD card is not visible to the OS. So I struggle with the step of configuring PiBuilder and also the many steps just to connect to a Pi using SSH.

Answer

macOS is the same as Windows - it can only mount the boot partition. That's why I put all the files on the boot partition. It's the easiest way of getting everything onto the Pi.

the bits of PiBuilder we do on Windows

It's a long time since I used Windows so I'm probably going to make a few mistakes here but please try to see past those to what I'm trying to demonstrate.

The very first time you use PiBuilder on Windows, you would Start»Run»cmd and do something like this:

cd Desktop
git clone https://github.com/Paraphraser/PiBuilder.git
cd PiBuilder
git checkout -b custom
cd boot

I assume there are GUI tools for cloning GitHub repositories. By all means use those rather than the command line. The only reason I used commands here is so I can focus on this working directory:

C:\Documents and Settings\Andreas\Desktop\PiBuilder\boot

If you run the dir command in that directory, you should see ssh (file) and scripts (directory).

It's a good idea to open the following file in a text editor:

C:\Documents and Settings\Andreas\Desktop\PiBuilder\boot\scripts\support\pibuilder\options.sh

and at least consider setting LOCALCC and LOCALTZ to their correct values.

You could try setting LOCALE_LANG to whatever you normally use but I have never actually tested anything other than the default of en_GB.UTF-8 and en_US.UTF-8 so it might be better to defer that and set it later using raspi-config.

PREFER_64BIT_KERNEL only has any effect if the image you choose is capable of running in 32-bit mode. The default will give you a full 32-bit system. I'd recommend starting from full 64-bit Bullseye, which ignores this option.

VM_SWAP is something you should think about if you're building a Pi that boots from SD. The values are explained in the file.

Once you've made your edits, save the file (taking care to ensure Windows doesn't add CR+LF line-endings).

What you do next depends on the approach you want to take. If you are going to use Balena Etcher, you need to create the userconf.txt and wpa_supplicant.conf files by hand and place them in the boot directory (emphasis: we are still on Windows so I mean the boot directory, not the boot partition).

See also what you can do on another Pi below.

If you are going to use Raspberry Pi Imager, you can either create the userconf.txt and wpa_supplicant.conf files by hand or launch Raspberry Pi Imager and do the same work in the GUI.

WiFi is, of course, optional on Pis that have Ethernet, but mandatory on Pis that don't (eg ZeroW2).

For credentials, I recommend "pi" and a strong password but it's entirely up to you. If you use Imager, I also recommend leaving the hostname as "raspberrypi".

Now, you attach either an SD card or an SSD to your PC. You run Raspberry Pi Imager or Balena Etcher to transfer the image (Bullseye 64-bit recommended) to the SD/SSD.

At the end of the imaging process, you want to be in the position where the boot partition of the SD/SSD is mounted as some drive letter. I'm going to assume X:.

What you need to do is copy the contents of the boot directory (ssh, scripts, plus userconf.txt and wpa_supplicant.conf if those exist) over to X:\. It is the equivalent of:

xcopy "C:\Documents and Settings\Andreas\Desktop\PiBuilder\boot\*" X:\ /E/H

And, yes, you can do this from Windows Explorer by selecting the contents of the "boot" directory and copying them onto the X: drive.

Then you eject the media, put it into the Pi, apply power, and wait for it to boot up.

the bits we do on the Pi - via SSH from Windows

I work exclusively via SSH because it's the lowest common denominator that should work in all situations.

The very first time you connect to a new Pi via SSH, you should do it like this:

  • on Windows:

     $ ssh-keygen -R raspberrypi.local
 $ ssh -4 pi@raspberrypi.local

    I have never used putty so I have no idea what the equivalent commands are.

    The first command is protective and gets rid of any "fingerprints" from another Pi.

    The second command will ask whether you want to trust the host (yes) and then prompt for the password.

  • on the Pi:

     $ /boot/scripts/01_setup.sh «hostname»

    where «hostname» is the name you actually want to use for the Pi. It should be different to "raspberrypi" (eg "iot-hub" - letters, digits and hyphens).

    When that script ends, it reboots the Pi.

  • on Windows:

     $ ssh-keygen -R raspberrypi.local
 $ ssh-keygen -R «hostname».local
 $ ssh -4 pi@«hostname».local

    The first command is "cleaning up as we go". The second is protective in case you had an older Pi of the same hostname (or are rebuilding an existing Pi and re-using the hostname).

  • on the Pi:

     $ /boot/scripts/02_setup.sh

    That script is very quick and also finishes in a reboot.

  • on Windows:

     $ ssh pi@«hostname».local

    There's no need to use ssh-keygen because things are as we need them to be. I've also omitted the -4 option because, at this point, IPv6 has been turned off on the Pi so it defaults to IPv4.

  • on the Pi:

     $ /boot/scripts/03_setup.sh

    That ends with a logout. You login again and run:

     $ /boot/scripts/04_setup.sh

    That ends with a reboot.

  • on Windows:

     $ ssh pi@«hostname».local

  • on the Pi:

     $ /boot/scripts/05_setup.sh

    That ends with a logout.

At this point, you have a working Pi with IOTstack cloned and all dependencies satisfied. You can either run the menu and choose containers, or restore a backup, and then bring up your stack.

what about using the console instead of SSH?

Although I have never tried it, you could probably do all this with a keyboard and screen attached to the Pi. The first time you boot the Pi, I think it comes up with the Desktop GUI. Running the 01_setup.sh script changes to "B1" so, after that, it will come up in the Console.

Question 2

I wonder if it would be possible to start with a new Pi SD card (produced according your description) and boot the Pi with this image. Then SSH into the Pi and download your repo. Finally run the different scripts. Would this work and what are the downsides? At least it would be much faster and less cumbersome

Answer

I'll start with why I use the boot partition and why I copy onto it rather than cloning with git:

  1. Because it's FAT, it's the one thing that is in common with the Pi, macOS and Windows. It's guaranteed to work without needing any extra software to mount the ext4 partition.

  2. I copy the contents of the PiBuilder "boot" directory onto the boot partition, rather than a git clone, so that you get a snapshot which is divorced from git. I think of this as recording "how this Pi was actually built".

    A snap-frozen copy of PiBuilder on /boot goes hand in hand with the baseline copies that the scripts make of cmdline.txt and config.txt, plus the /etc directory on the root partition.

    Armed with that basic information, it should always be possible to figure out whether "something" that is happening on the Pi:

    • came that way from the Raspberry Pi Foundation; or
    • was done by PiBuilder; or
    • had occurred since.

    I think of this as "auditability". It helps answer "what changed?" questions.

So, let's move to whether it would work if you did it like this:

  1. Use Balena Etcher or Raspberry Pi Imager to transfer the image to SD/SSD.

  2. Either use the Raspberry Pi Imager GUI to set up credentials and WiFi or create the ssh, userconf.txt and wpa_supplicant.conf files by hand and put them on the boot partition.

  3. Boot the Pi and connect to it somehow.

  4. On the Pi, run:

    $ git clone https://github.com/Paraphraser/PiBuilder.git ~/PiBuilder

  5. On the Pi, edit the content of:

    ~/PiBuilder/boot/scripts/support/pibuilder/options.sh

  6. On the Pi, run:

    $ ~/PiBuilder/boot/scripts/01_setup.sh «hostname»

  7. Re-connect then run the other scripts, in order, using the same path prefix.

I think that would work. I have never tested it but I can't think of any reason why it would not work.

Originally, everything in PiBuilder had hard-coded reliance on /boot but I've changed it so it can run from anywhere. I made it location-agnostic so that it was easier to run 04_setup.sh as a way of reinstalling docker and docker-compose. The 04 script definitely works from anywhere and I think all the other scripts will too but I've never tested a full build.

Now, aside from the "why I use the boot partition" discussed earlier, why don't I clone PiBuilder onto the Pi and run everything from there?

There are a few reasons but the most important ones are:

  1. It doesn't really solve the problem of what you do when you have multiple Raspberry Pis.

    The basic idea of PiBuilder is that you have a clone of the PiBuilder repo on your support machine (PC, Mac or another Linux box) where you have created a "custom" branch. When you edit "options.sh" you can add and commit that to your local branch. Then it's ready for the next Pi you want to build.

    So, suppose you have a Pi4 running from SSD plus a PiZeroW2 running from SD. The Pi4 wants swap enabled, the Zero doesn't. That's where the idea of options.sh@mypi4 and options.sh@myzero come in, so you can customise for those two machines.

    Then you decide that one machine needs static IP addresses while the others can get by with DHCP. You customise PiBuilder for the exception case - the machine that needs the static IP addresses.

    Or you decide the 03 script should be adding more apt packages than it is. You can customise the epilog script for the 03 script (or, because it's your clone, just add it to the 03 script).

    The basic discipline I follow is:

    1. I identify a need for a customisation on a particular Pi.
    2. I try alternatives and settle on a solution which I test.
    3. I update my working copy of PiBuilder on my support host and commit the changes with a decent message which will help me to remember what I was doing if I ever need to come back to it.

    If I ever need to rebuild a Pi, I can be confident that everything needed to get it back the way it was is in PiBuilder on my support host. Most of the time I just image the media, do the equivalent of the xcopy command above, move the media to the Pi, boot it up, and run the scripts in order. I don't even need to think about options.sh or other customisations because all that work is already saved.

  2. It doesn't really solve the initial chicken-and-egg problem.

    Let's assume I have solved the first problem with a local clone of PiBuilder on my support host. To use that local clone, I would need to get it onto the Pi. Something like this would do the trick:

    $ git clone -b custom ssh://mymac.mydomain.com//Volumes/Configurator/PiBuilder ~/PiBuilder

    But you'll note that relies on SSH so I would first need to have SSH working. The same would apply to SMB. Or I would need to set things up so that my local clone was accessible via http or https. That's a lot of work and a lot of things to explain to people who are starting out on their PiBuilder journey.

    Copying to /boot is something that is going to work without anything else being needed. No chicken-and-egg.

Having said all that, it always comes down to it being your Raspberry Pi, so you can do it whatever way works best for you.

What you can do on another Pi

If you already have a Pi running Raspbian (I don't think it matters if it's Buster or Bullseye), you can use PiBuilder to help you create userconf.txt and wpa_supplicant.conf. Here's how:

  1. Clone PiBuilder onto the Pi (or bring it up-to-date if it's already there):

    $ cd
$ git clone https://github.com/Paraphraser/PiBuilder.git ~/PiBuilder

  2. Create the user credentials - one optional argument is the username, defaults to "pi"

    $ ~/PiBuilder/boot/scripts/helpers/create_user_credentials.sh
Enter password for user pi: 
Re-enter password: 
userconf.txt created - needs to be copied to /boot

$ cat userconf.txt 
pi:$6$ajpCN43TI3mmdhv0$nDK47HQG82Q5wRDojIRq0lZk42NbGGQkuzrc06PhfGJVjB9.HB3122Jac89Ax74DYdu30rEM2pItF16l0o7K1/

  3. Create WiFi setup - the arguments are your SSID and country code:

    $ ~/PiBuilder/boot/scripts/helpers/create_wifi_credentials.sh "My Home WiFi" "AU"
/usr/bin/wpa_passphrase
Enter password for WiFi SSID My Home WiFi: 
Re-enter password: 
wpa_supplicant.conf created - needs to be copied to /boot

$ cat wpa_supplicant.conf 
country=AU
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

# below is output from wpa_passphrase "«SSID»" "«PSK»"
network={
	ssid="My Home WiFi"
	psk=89f1c01f95d8efa1533ceebe8a0918c2b2a11bc75f533fd4872577925205a85e
}

  4. Get both of those output files onto your support host, putting them in:

    C:\Documents and Settings\Andreas\Desktop\PiBuilder\boot

You may be able to run create_user_credentials.sh on a non-Pi but whether the result will work on the Pi depends on the version of OPENSSL. It's safer to do it on a Pi.

The create_wifi_credentials.sh script depends on wpa_passphrase so it will only work where that tool is available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment