This is for people who use Linux or Unix environments, and are OK with using shell programs.
Hopefully-helpful ways to manipulate data using Linux command line programs, the aim is to save a lot of time parsing things in python or R or matlab if we don't need to.
Copy, paste and modify commands from here into your shell window. Usually things here assume the bash shell. Lines starting with a hash are comments, don't copy those!
If this is all foreign to you, but you like the idea of using this stuff, the Software Carpentry introduction to the Unix shell is a great start: https://swcarpentry.github.io/shell-novice/
Using the awk text processor to convert KPH TRACK data downloaded from the ship or NIRD into timestamp, lat, lon. Based on KPH track .log files from JC2-2 in 2021.
note: this boldly assumes that ship logged data format is consistent. It may be necessary to clip different parts of strings out at some point for past / future voyages.
#put all the log files into one long file. avoid using .log as an extension
# NB this isn't guaranteed to sort things in time order!
cat *.log > allthelogs.txt
#start a file with some column headers
echo "timestamp,lat,lon" > timepos.csv
#read the long log file, and use the gnu awk text processor to do magic.
# the first line turns decimal minutes and seconds into decimal degrees
# the second line assembles an ISO timestamp from time and date columns, and assembles dd.dddddd positions
# the final line adds the results to the file we just made with column headers
awk -F, '{OFS=","}{londegs = substr($14,4) / 60}{latdegs = substr($13,3) / 60} \
{print substr($3,7) "-" substr($3,4,2) "-" substr($3,0,2) "T" substr($4,0,2) ":" substr($4,3,2) ":00Z", $13=substr($13,0,2) "." substr(latdegs,3), $14=substr($14,0,3) "." substr(londegs,3)}' \
allthelogs.csv >> timepos.csv