Fetch Luno trade history and save as CSV for charting purposes

fetchlunohistory.sh

#!/bin/bash
#
# About
#
# fetchlunohistory.sh
#   get date in epoch time of last trade in csv
#   get trades from luno from this time
#   reverse order
#   append to big log of all trades
#
# Log
#
# 2017-12-10  initiating script
# 2017-12-11  changing to use api trades
# 2019-07-25  fixing fields extracted by awk
#
# Setup notes
# 1. install jq, dateutils.dconv, tac, join, curl, awk, csvtool
# 2. setup directory including subdirectory 'history', edit script to match
# 3. setup history.csv with contents
#    1365895484895,600.000000000000,0.010000000000
# 4. add script to cron as follows, to run every minute
#    * * * * * /path/to/fetchlunohistory.sh >/dev/null 2>&1
#
# Luno API, replace [UNIXTIME] with epoch milliseconds
# https://api.mybitx.com/api/1/trades?pair=XBTZAR&since=[UNIXTIME]
#
# good starting point to initialise history file
# 1365895484895,600.000000000000,0.010000000000
#
# --------------------
#
# set to 1 to output debug info, else 0 to be silent
debug="0"

# set to 1 to step through script, else 0 to ignore
step="0"
if [ "$step" -eq "1" ]; then
set -x
trap read debug
fi

# binary locations
csvtoolbin=$(which csvtool)
tailbin=$(which tail)
awkbin=$(which awk)
curlbin=$(which curl)
reversetool=$(which tac)
jointool=$(which join)
unixdate=$(which dateutils.dconv)
jqbin=$(which jq)
trbin=$(which tr)

# primary history file
bigfile="/home/ubuntu/lunocsv/history.csv"

# the following is the result of several manual iterations
# and can probably be simplified into fewer steps and files
#
# set variables for tmp files in history directory
tmpcsv="/home/ubuntu/lunocsv/history/tmpcsv.csv"
tmpdrop="/home/ubuntu/lunocsv/history/tmpdrop.csv"
tmpdate="/home/ubuntu/lunocsv/history/tmpdate.txt"
tmpdateunix="/home/ubuntu/lunocsv/history/tmpdateunix.txt"
tmpprice="/home/ubuntu/lunocsv/history/tmpprice.txt"
tmpvolume="/home/ubuntu/lunocsv/history/tmpvolume.txt"
tmpformat="/home/ubuntu/lunocsv/history/tmpformat.csv"
tmpcombine="/home/ubuntu/lunocsv/history/tmpcombine.csv"
tmpreverse="/home/ubuntu/lunocsv/history/tmpreverse.csv"

# get first column of last line from luno trades log file (unixdate)
read fromtime <<<$("$tailbin" -1 "$bigfile" | "$awkbin" -F, '{print $1}')

# set query string to include this value
url="https://api.mybitx.com/api/1/trades?pair=XBTZAR&since=$fromtime"

# download to tmp csv file from specified date
"$curlbin" -s "$url" | "$jqbin" -r -c '.trades |(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv' | "$awkbin" -F, '{OFS=",";print $4,$2,$5}' | "$trbin" -d '"' > "$tmpcsv"

# drop the first row
"$csvtoolbin" drop 1 "$tmpcsv" > "$tmpdrop"

# split out the dates column
"$csvtoolbin" col 1 "$tmpdrop" > "$tmpdateunix"

# convert prices to 12 decimal places in single column file
cat "$tmpdrop" | "$awkbin" -F "\"*,\"*" '{print $2}' | "$awkbin" '{printf "%.12f\n", $1}' > "$tmpprice"

# convert volume to 12 decimal places in single column file
cat "$tmpdrop" | "$awkbin" -F "\"*,\"*" '{print $3}' | "$awkbin" '{printf "%.12f\n", $1}' > "$tmpvolume"

# combine all temp files into new csv
"$csvtoolbin" paste "$tmpdateunix" "$tmpprice" "$tmpvolume" > "$tmpcombine"

# reverse order of tmp file by date column
"$reversetool" "$tmpcombine" > "$tmpreverse"

# append to luno trades log file
cat "$tmpreverse" >> "$bigfile"

# DEBUG
if [ "$debug" == "1" ]; then
  echo "fromtime: $fromtime"
  echo "url:      $url"
fi

script doesn't currently log if trade is a buy or sell, or the sequence field.

Adapting the following guide might be a useful way to expand beyond bash & csv:

• https://medium.com/coinograph/storing-and-processing-billions-of-cryptocurrency-market-data-using-influxdb-f9f670b50bbd

There's also this project

• https://github.com/zkrige/luno-fetcher

This might be useful

Is there a tool like "jq" for CSV files?

• MLR on GitHub
• MLR manpage
• MLR for CSV manipulation