Skip to content

Instantly share code, notes, and snippets.

@softberries

softberries/prepare-data.sh

Created October 23, 2017 12:08
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save softberries/2d675073eeaac4e04884e41acef66487 to your computer and use it in GitHub Desktop.
Save softberries/2d675073eeaac4e04884e41acef66487 to your computer and use it in GitHub Desktop.
Download ZIP
prep data bash script
Raw
prepare-data.sh
#!/usr/bin/env bash
S3_DESTINATION_FOLDER="grajo001log/users"
SITES_FILE=sites.txt
IFS=$'\n'
for site in `cat $SITES_FILE`
do
echo "$site"
wget https://archive.org/download/stackexchange/$site.7z
7z x $site.7z -otemp
aws s3 cp temp/Users.xml s3://$S3_DESTINATION_FOLDER/$site.xml
rm $site.7z
rm -Rf temp
done
echo 'Finished!'
exit 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment