Skip to content

Instantly share code, notes, and snippets.

@DustinMorado
Last active May 8, 2020 09:40
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save DustinMorado/42e10ef1df4f60397ec2dd40c52a852b to your computer and use it in GitHub Desktop.
Save DustinMorado/42e10ef1df4f60397ec2dd40c52a852b to your computer and use it in GitHub Desktop.
#!/bin/bash
if [[ ! -d 3DEM ]]
then
mkdir 3DEM
fi
cd 3DEM
curl -s http://mail.ncmir.ucsd.edu/pipermail/3dem/ |\
grep -o "2[0-9]\+-[A-Z][a-z]\+\.txt\.gz" |\
while read input_url
do
output_txt=$(basename ${input_url} .gz |\
sed -e 's/January/01/' \
-e 's/February/02/' \
-e 's/March/03/' \
-e 's/April/04/' \
-e 's/May/05/' \
-e 's/June/06/' \
-e 's/July/07/' \
-e 's/August/08/' \
-e 's/September/09/' \
-e 's/October/10/' \
-e 's/November/11/' \
-e 's/December/12/')
curl -s http://mail.ncmir.ucsd.edu/pipermail/3dem/${input_url} |\
gunzip > ${output_txt}
done
find . -name '????-??.txt' | sort -t- -n -k1,2 | xargs cat > 3DEM.txt
find . -name '????-??.txt' -delete
cd ..
@Guillawme
Copy link

Super useful, thank you! Would it be difficult to adapt to make it work with CCPEM too?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment