RMS image build

gistfile1.txt

Install QEMU support for the target image platform -
For binary support only -

apt-get install qemu-user-static

For full system emulation - where supported -

apt-get install qemu-system

Note: Qemu provides full support for some platforms e.g. RPi2/3 but not for RPi4/5 however binary support is sufficient for 
our purposes.

Later we will need some Container tools to enable us to log into the image, install these -

sudo apt install systemd-container


By default release images are shrunk to minimise free space on the image and thus reduce the download size.
To give some usefull free space we can increase the partition size and then expand the root filesystem to add this extra space - 
If you were burning to pysical media the raspi-config Expand filesystem performs the same task, however it expands the partition
 to use all available free space - this is quite likely *not* what you want on your parent host...

A .img file is just a block based file container for the partition tables -

so first we need to add some space to the container
then expand the root partition
then grow the root filesystem to encompass the new space

First lets take a look at the latest RPi4 Bullseye RMS image and see how much is free-

our uncompressed image is in our home directory

test@rms-test:~$ ls
RMS_RPi4Bullseye_image_20231213.img
test@rms-test:~$ 

First we set the image up on the next free loopdevice  -the -f finds next free loopdev and the -P tells it to search the image 
for partition tables and the --show displays the loop device it found-

test@rms-test:~$ sudo losetup -fP --show RMS_RPi4Bullseye_image_20231213.img 
[sudo] password for test: 
/dev/loop6

So /dev/loop6p2 should be our root partition -/dev/loop6p1 is our boot partition which we don't need, lets create a staging dir to mount it on then we can inspect it -
mkdir temp


..and mount partition 2 to it - 

sudo mount /dev/loop6/p2 temp

now check size and freespace -

df -h temp
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop6p2     13G   12G  392M  97% /home/test/temp

As suspected the partition has been shrunk and there's only 392M free, so if we say want to tweak the image and run a short test 
capture we need to add some space, so lets add 10GB -

First we need to unount it and remove it from the loopdev -

sudo umount temp
sudo losetup -D

Then append some space on the end -

test@rms-test:~$ sudo dd if=/dev/zero bs=10M count=1024 >>RMS_RPi4Bullseye_image_20231213.img 
1024+0 records in
1024+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 29.8849 s, 359 MB/s   # <-- its an ssd so pretty quick

Now just as a check, take a look at the image file -

test@rms-test:~$ fdisk -l RMS_RPi4Bullseye_image_20231213.img 
Disk RMS_RPi4Bullseye_image_20231213.img: 22.75 GiB, 24422978048 bytes, 47701129 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc5c7b7be

Device                               Boot  Start      End  Sectors  Size Id Type
RMS_RPi4Bullseye_image_20231213.img1        8192   532479   524288  256M  c W95 FAT32 (LBA)
RMS_RPi4Bullseye_image_20231213.img2      532480 26729608 26197129 12.5G 83 Linux


So the partitions occupy less than 13GB but the image is now 22.75 GiB, lets expand the 2nd partition to occupy that space -

test@rms-test:~$ sudo growpart RMS_RPi4Bullseye_image_20231213.img 2
CHANGED: partition=2 start=532480 old: size=26197129 end=26729609 new: size=47168616 end=47701096

First present it the loopdev -
test@rms-test:~$ sudo losetup -fP --show RMS_RPi4Bullseye_image_20231213.img 
/dev/loop6


Now expand the filesystem into that new space -
We have to force a filesystem integrity check before resizing it -

test@rms-test:~$ sudo e2fsck -f /dev/loop6p2
e2fsck 1.46.5 (30-Dec-2021)
Pass 1: Checking inodes, blocks, and sizes
Inode 1070 extent tree (at level 1) could be narrower.  Optimize<y>? yes
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

rootfs: ***** FILE SYSTEM WAS MODIFIED *****
rootfs: 306308/812800 files (0.4% non-contiguous), 3032331/3274641 blocks


Finally resize the filesystem -

test@rms-test:~$ sudo resize2fs /dev/loop6p2
resize2fs 1.46.5 (30-Dec-2021)
Resizing the filesystem on /dev/loop6p2 to 5896077 (4k) blocks.
The filesystem on /dev/loop6p2 is now 5896077 (4k) blocks long.


Then mount it and check free -

test@rms-test:~$ sudo mount /dev/loop6p2 temp

test@rms-test:~$ df -h temp
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop6p2     23G   12G  9.9G  54% /home/test/temp

-we now have 9.9G free :-)

The root filesystem is now mounted on temp and looks like a regular root '/' filesystem -

test@rms-test:~$ ls temp
bin  boot  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

NOTE: you msy be tempted to add/edit files directly prior to booting the container, this will likely end in tears since the security model will detect this and you may find that from within the container even root cannot manipulate them.


Now because the QEMU emulator libraries we loaded earlier do not support RPi4 or 5 hardware we can't natively boot the system 
where ordinarily init owns the operating process tree, instead we can only boot into single user mode aka runlevel=1

So to do this -

test@rms-test:~$ sudo systemd-nspawn -D temp
Spawning container temp on /home/test/temp.
Press ^] three times within 1s to kill container.
root@temp:~# uname -m
aarch64

We have a shell session as user root and as you can see from the output of uname -m the system architecture is aarch64 and not that of the host - x86_64 

But remember this is a pre-built RMS image - lets logout and launch it again as user rms -

test@rms-test:~$ sudo systemd-nspawn -D temp -u rms
Spawning container temp on /home/test/temp.
Press ^] three times within 1s to kill container.
(vRMS) rms@temp:~/source/RMS $ 

We are logged in as user RMS, we have executed user rms's .bashrc and our default location is 
/home/rms/source/RMS

and we have activated the vRMS virtualenv

So we can pip install, add/remove packages and even run non-gui apps like StartCapture and everything will work.

Moreover if we quit the session and unmount the image, any changes we made are persisted to the image on-disk..

Lets try -1st edit .config and assign a valid camera IP to the device capture string, we don't need to change anything else
- we'll end up with data captures in ~/RMS/ArchivedFiles/XX0001.... directories, save the changes.

from the cmd prompt lets capture 6 minutes of camera output -


python -m RMS.StartCapture -d 0.1     


Log output -

2024/04/27 13:52:35-INFO-StartCapture-line:767 - Program start
2024/04/27 13:52:35-INFO-StartCapture-line:768 - Station code: XX0001
2024/04/27 13:52:35-INFO-StartCapture-line:782 - Program version: 20231213_130459, 46dfa1f662a7db2f82c38d4d3ddffa556ca3b74a
2024/04/27 13:52:35-INFO-StartCapture-line:810 - Freeing up disk space...
2024/04/27 13:52:35-INFO-DeleteOldObservations-line:241 - Need 5.12 GB for next night
2024/04/27 13:52:35-INFO-StartCapture-line:829 - Running for 0.1 hours...
2024/04/27 13:52:35-INFO-StartCapture-line:239 - Data directory: /home/rms/RMS_data/CapturedFiles/XX0001_20240427_135235_516310
2024/04/27 13:52:35-INFO-DownloadPlatepar-line:25 - Checking for new platepar on the server...
2024/04/27 13:52:35-DEBUG-DownloadPlatepar-line:27 - Can't contact the server: RSA private key file not found.
2024/04/27 13:52:35-INFO-Reprocess-line:87 - No platepar file found!
2024/04/27 13:52:35-INFO-StartCapture-line:258 - Initializing frame buffers...
2024/04/27 13:52:35-INFO-StartCapture-line:282 - Initializing frame buffers done!
2024/04/27 13:52:35-INFO-QueuedPool-line:204 - Loaded 0 backed up results...
2024/04/27 13:52:35-INFO-QueuedPool-line:204 - Using 1 cores
2024/04/27 13:52:35-INFO-StartCapture-line:96 - Press Ctrl+C to stop capturing...
2024/04/27 13:52:36-INFO-BufferedCapture-line:148 - Camera IP ping successful!
2024/04/27 13:52:36-INFO-BufferedCapture-line:163 - Initializing the video device...
2024/04/27 13:52:36-INFO-BufferedCapture-line:164 - Device: rtsp://192.168.0.150:554/user=admin&password=&channel=1&stream=0.sdp
2024/04/27 13:52:38-INFO-BufferedCapture-line:219 - Video device opened!
2024/04/27 13:52:38-INFO-BufferedCapture-line:331 - Grabbing a new block of 256 frames...
2024/04/27 13:52:47-INFO-BufferedCapture-line:474 - New block of raw frames available for compression with starting time: 1714225958.731498
2024/04/27 13:52:47-INFO-BufferedCapture-line:331 - Grabbing a new block of 256 frames...
2024/04/27 13:52:47-DEBUG-Compression-line:274 - Compressing frame block with start time at: 1714225958.731498
2024/04/27 13:52:53-DEBUG-Compression-line:295 - Compression time: 6.167 s
2024/04/27 13:52:55-DEBUG-Compression-line:302 - Saving time: 1.932 s
2024/04/27 13:52:55-INFO-DetectionTools-line:61 - Loaded mask: mask.bmp
....
....
....
2024/04/27 13:59:02-INFO-Reprocess-line:378 - Generating a timelapse...
2024/04/27 13:59:22-INFO-Reprocess-line:511 - Archiving detections to /home/rms/RMS_data/ArchivedFiles/XX0001_20240427_135235_516310
2024/04/27 13:59:22-INFO-ArchiveDetections-line:171 - Generating thumbnails...
2024/04/27 13:59:23-INFO-ArchiveDetections-line:192 - Generating a stack of detections...
2024/04/27 13:59:29-INFO-ArchiveDetections-line:210 - Captured stack saved to: /home/rms/RMS_data/CapturedFiles/XX0001_20240427_135235_516310/XX0001_20240427_135235_516310_captured_stack.jpg
2024/04/27 13:59:29-INFO-ArchiveDetections-line:237 - Detected stack could not be saved!
2024/04/27 13:59:29-DEBUG-shutil-line:1039 - changing into '/home/rms/RMS_data/ArchivedFiles/XX0001_20240427_135235_516310'
2024/04/27 13:59:29-INFO-shutil-line:899 - Creating tar archive
2024/04/27 13:59:30-DEBUG-shutil-line:1067 - changing back to '/home/rms/source/RMS'

So everything looks good.
NOTE: whilst it is old hardware - Intel i3 circa 2013 the compression times look poor @~6s per frame
this is an artefact of the emulation not being efficient, the save times to an SSD whilst not stellar are 
respectable @2secs so not terrible..!
the resultant ff_intervals shows numerous dropped frames - an artefact of the slwness of the emulation on what is a pretty old and lo-spec host.

Now one might think - what's the point?

I have a newer Intel i5 Gen-11 Nuc I use as a HTPC hanging off the back of my TV, I installed the two packages at the head 
of this posting, scp'd the RPi image to it and ran a capture again after running RMS_Update so utilising the new more accurate 
frame timing code -

Compression time: 3.091 s
Saving time: 0.679 s

The resultant ff_intervals image was flat - no dropped frames, Median 2.48s +/-0.000s etc

Press ] 3 times in a second and that kills the container, unmount the image, losetup -D to remove the loopback ...and you have a portable RMS image.