Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
hacker news firebase download
# generate file containing all URIs to execute to retrieve data from hacker news firebase API
# api docs:
echo generating file hn-uri.txt
echo file will contain $URICOUNT lines
# get current maxid from
echo max id $MAXID
echo min id $MINID
# for 10MM rows, this took 10 minutes on my machine and produced a 560MB file
for ((x=$MINID; x<$MAXID; x++))
# append a uri for the current id to the file
echo "$x.json" >> hn-uri.txt
if ! ((x % 10000)); then
echo current id $x
# note: on my macbook, for 10MM uris, this took just over 30 hours to complete
cat hn-uri.txt | xargs -P 100 -n 100 wget --quiet
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment