Last active
December 5, 2020 04:30
-
-
Save bvandgrift/14e71b1de6dc9c397e68336146f710f7 to your computer and use it in GitHub Desktop.
cheap idempotent file fetching
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env bash | |
# given a single case id (e.g., 19820427023459I) will check to see if | |
# it exists in the output directory, otherwise fetch it -- | |
# it won't store the file if there's a problem with the request | |
# usage: ./fix-missing.sh <case_id> | |
if [ `ls out/${1}.html 2> /dev/null` ]; then | |
exit | |
else | |
curl -f -s -o out/${1}.html \ | |
"https://www.asias.faa.gov/apex/f?p=100:18:::NO::AP_BRIEF_RPT_VAR:${1}" | |
fi | |
# BEST USED when chained together with a file of IDs and xargs in parallel: | |
# cat id_list.txt | xargs -I{} -n 1 -P 32 ./fix-missing.sh {} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment