How to record data from an STWIN.box (STEVAL-STWINBX1) on a Raspberry Pi via USB and convert the resulting .dat files to Pandas df or numpy arrays.

convert_dat.py

'''
# README FIRST #

###Goal###
On a Raspberry Pi, read .dat files created by STWIN.box from disk and get numpy or pandas objects to work with. 

### Requirements ###
- Raspberry Pi
- ST-Link Debugger
- STWIN.box + USB-C Cable


### Guide ###

1. On your computer: Flash [this firmware](https://github.com/STMicroelectronics/fp-sns-datalog2/blob/bb0a1d7560b5c459908b112e091f42f284670259/Projects/STM32U585AI-STWIN.box/Applications/DATALOG2) to your STWIN.box with an ST-Link.

2. On Raspberry Pi: To stream data from the STWIN.box connected to your Raspberry Pi, follow [this guide](https://github.com/STMicroelectronics/fp-sns-datalog2/blob/bb0a1d7560b5c459908b112e091f42f284670259/Utilities/cli_example/README_Linux.md) to setup and run the recording executable.

3a) on bookworm 64bit, install libusb-1.0-0:armhf, libc6:armhf and libstdc++6:armhf as the compiled cli_example binary needs ld-linux-armhf.so.3
3b) On Raspberry Pi: Record some data with the ´cli_example´ tool. The ´cli_example´ executable will create a folder with a bunch of ´.dat´ files containing high resolution raw sensor data. The next steps will show how to convert the data to a dataframe.

4. On Raspberry Pi: To convert the recorded data, first install depencies: sudo apt-get install python3-dev libatlas-base-dev libopenjp2-7 libopenblas-dev

5. On Raspberry Pi: Install the ST HSDataLog libraries:

wget https://github.com/STMicroelectronics/fp-sns-datalog2/raw/bb0a1d7560b5c459908b112e091f42f284670259/Utilities/HSDPython_SDK/st_pnpl/dist/st_pnpl-2.1.0-py3-none-any.whl
pip install st_pnpl-2.1.0-py3-none-any.whl

wget https://github.com/STMicroelectronics/fp-sns-datalog2/raw/bb0a1d7560b5c459908b112e091f42f284670259/Utilities/HSDPython_SDK/st_hsdatalog/dist/noGUI/st_hsdatalog-3.0.1-py3-none-any.whl
pip install st_hsdatalog-3.0.1-py3-none-any.whl


6. On Raspberry Pi: Copy and run this script to see if conversion works. Modify it to your needs. On RP CM4, a 5 second recording (1.8M if gzipped) will take about 3.4s (numpy) or 9s (pandas) to convert all .dat files. For a 5min recording, it takes 20s (numpy).

(venv) sam@aikit:~ $ time python3 convert_dat.py 20240111_08_31_50/ all
2024-01-11 09:31:49,124 - HSDatalogApp.st_pnpl.DTDL.device_template_manager - INFO - dtmi found in locally in base supported models
2024-01-11 09:31:49,124 - HSDatalogApp.st_pnpl.DTDL.device_template_manager - INFO - dtmi: dtmi/appconfig/steval_stwinbx1/FP_SNS_DATALOG2_Datalog2-4.json
stts22h_temp, shape: (800, 1)
iis2mdc_mag, shape: (400, 3)
iis2iclx_acc, shape: (4000, 2)
ism330dhcx_acc, shape: (34000, 3)
imp23absu_mic, shape: (934000, 1) <--- this is an 80khz mic
iis2dlpc_acc, shape: (7000, 3)
ism330dhcx_gyro, shape: (34000, 3)
iis3dwb_acc, shape: (130000, 3) <--- this is an ultra wide band accelerometer
imp34dt05_mic, shape: (231000, 1)
ilps22qs_press, shape: (600, 1)

real	0m3.370s
user	0m3.917s
sys	0m0.257s

'''

import os
import sys
import st_hsdatalog.HSD_utils.converters as converters
from st_hsdatalog.HSD.HSDatalog import HSDatalog

def get_hsd(acq_folder='.'):
    hsd_factory = HSDatalog()
    hsd= hsd_factory.create_hsd(acq_folder)
    return hsd

def dat_to_df(hsd, sensor_name):
    '''
    Converts a dat file to a pandas dataframe.
    @param acq_folder: path to the acquisition folder
    @param sensor_name: name of the sensor to convert (see filenames in the acquisition folder)
    '''

    sensor = HSDatalog.get_sensor(hsd, sensor_name)
    #data_time = HSDatalog.get_dataframe(hsd, sensor)
    data_time = HSDatalog.get_data_and_timestamps(hsd, sensor)

    return sensor, data_time


if __name__ == '__main__':
    # get the path to the acquisition folder
    acq_folder = sys.argv[1]
    sensor_name = sys.argv[2]

    hsd = get_hsd(acq_folder)

    if sensor_name == 'all':
        #loop over all .dat files in folder
        for filename in os.listdir(acq_folder):
            if filename.endswith('.dat'):
                sensor_name = filename.split('.')[0]
                sensor, data_time = dat_to_df(hsd, sensor_name)
                print("%s, shape: %s"% (sensor_name, data_time[0][0].shape))
    else:
        sensor, data_time = dat_to_df(hsd, sensor_name)
        print(data_time)