Raspberry Pi as a service

RaspberryPiAsAService.md

Raspberry Pi Zero as a Service

Problem setup

A friend of mine was spending about an hour a day performing manual data entry for his daily appointments. His administrative assistant would book appointments using commercial (and rather expensive) software. The software did not provide a straightforward method of exporting the appointment information; however, it did allow the admin assistant to print the appointments as a PDF document. My friend would then read paper-printed PDF documents and the fields into an Excel spreadsheet. This method was error-prone and tedious, and took him considerable time (since he wasn't a fast keyboard user to begin with).

My goal was to automate his workflow, and to produce a .csv file that he could easily incorporate into his database.

Constraints

Due to the sensitive nature of the system, there were numerous constraints:

• No new software could be installed on his Windows computer
• Creating an intranet server (e.g. REST service) was prohibited (prohibited by the IT department)
• Transferring the data out to an external server was prohibited
• The system had to be simple enough for his administrative assistant to use it. The admin had limited technical experience: for the device to be useful, it had to "just work"
• Any data created / modified could not be saved for confidentiality reasons

Problem solution

My solution was to create a "Raspberry Pi as a Service." The idea was to setup a Raspberry Pi as a USB On-The-Go ethernet (i.e. g_ether) device which would provide a portable web server. The portable server could then run a website that could take the PDF document, extract the necessary information, and then convert it into a .csv. The data would be uploaded to RAM.

As an additional step, I needed a system that did not look complicated. A solution that seemed hard to use would be rapidly abandoned. I wanted the solution to look elegant, with no chance of losing wires.

Solution details

1. Raspbian was flashed using Etcher. My development machine was a Ubuntu 18.04 XPS 13 laptop
2. I created a USB On-The-Go g_ether device using Raspbian as follows:
   1. add: dtoverlay=dwc2 on a new line below config.txt
   2. add: modules-load=dwc2,g_ether after rootwait in cmdline.txt
   3. add: an empty ssh file to /boot
   4. Configure: /etc/network/interfaces to have a static IP for usb0
3. After starting the device, I could SSH into it by calling ssh pi@raspberrypi.local on my host computer
4. After configuring the wireless internet on the Pi, I ran sudo apt-get install dnsmasq. The Pi was then setup to be a DHCP server. After rebooting the Pi, my host computer recognized the Pi as an ethernet device and was assigned an IP
5. I then installed the back-end by running:
   1. sudo apt-get install poppler-utils (for pdftotext)
   2. sudo apt-get install python-pip
   3. pip install flask and pip install flask-dropzone
6. I then modified the complete-redirect example of the flask-dropzone project to save the file to the /tmp folder. This ensured that unplugging the Pi would destroy any uploaded data
7. The Flask website was configured to launch at boot. I found that easiest solution was by using 'crontab -e' to run @reboot
8. The administrative assistant simply had to open a browser and clicking on a bookmark to access a pre-defined IP/port: https://192.168.10.1:4200. The PDF document was dragged into the site. Once uploaded, I ran pdftotext -layout to convert the PDF to a text file, and the user was then prompted to save the .csv to their computer.
9. I wrote a small awk program to convert the output of pdftotext to a .csv file

To solve my aesthetics issue, the Raspberry Pi was then connected to a MakerFocus Pi Zero USB-A Addon Board V1.1. This not only made the device look super cool, but eliminated the need for any wires. The elegant thing about this device is that it doesn't require any soldering!

Technical blunders

1. The original idea for this project was to use the g_mass_storage device rather than the g_ether device. The idea would be to have the assistant drag-and-drop the PDF document to the device, and for a CSV file to automagically appear. After a few hours of troubleshooting, I identified two problems. First, when the host computer copied data to the Pi mass storage device, Raspbian would not see the data unless the mass storage partition was re-mounted. I wasn't able to come up with an elegant solution short of constantly mounting and re-mounting the device. Second, if the host and Raspbian computers both mounted the mass storage device, the file invariably got corrupted. The g_ether approach was a compromise, and in my opinion is less elegant, but it seemed to work well.
2. I learned to save time by pre-configuring the wpa_supplicant.conf file on the host machine, prior to the first boot. But if I specified a static IP using /etc/network/interfaces/ for the iface usb0 device, I found that I also needed to specify the iface wlan0 device in order for the wireless card to work. For security reasons, I disabled the wlan0 device to run as startup (by not typing auto wlan0)
3. I tried several methods to get the flask server to run on startup. My first approach was to edit /etc/rc.local. My next approach was to create a systemd service that launched on startup once the usb0 device was loaded. Both of these failed. It turned out the issue was that the command to open the flask server failed when run as root, and only worked when run as the pi user. I suspect this has to do with how I called pip during the installation process. The easiest solution was to run crontab as the pi user