Skip to content

Instantly share code, notes, and snippets.

@viniroger
Last active October 18, 2019 13:12
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save viniroger/416987465851f49c9983baef7e9ed19f to your computer and use it in GitHub Desktop.
Save viniroger/416987465851f49c9983baef7e9ed19f to your computer and use it in GitHub Desktop.
Create URLs and download data from multiples links automatically
#!/bin/bash
# Download files from network
places=('BRB' 'PTR' 'SMS')
years=($(seq 2012 2018))
months=({01..12})
mkdir -p data_amb
for place in "${places[@]}"; do
echo $place
for year in "${years[@]}"; do
echo $year
ye=${year:2:4}
for month in "${months[@]}"; do
# Create URL
base='http://PUT_YOUR_BASE_LINK_HERE/'
data_link=$base$place'/'$year'/'$place$ye$month'ED.7z'
# Test URL
wget -q --spider $data_link
status=`echo $?`
echo "Status" $data_link ":" $status
# Download file if exists
if [ "$status" -eq "0" ]; then
wget -P data_amb $data_link
fi
done
done
done
# Decompress files - sudo apt install p7zip-full
cd data_amb
for file in $(ls *.7z); do
7z x $file
done
# Remove compressed files
rm *.7z
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment