Skip to content

Instantly share code, notes, and snippets.

@EscVector
Created August 11, 2022 03:25
Show Gist options
  • Save EscVector/d942e265b68ea5978b9b7a9e736eb5a1 to your computer and use it in GitHub Desktop.
Save EscVector/d942e265b68ea5978b9b7a9e736eb5a1 to your computer and use it in GitHub Desktop.
import requests
import wget
import os
BaseFolder ='https://s3-us-gov-west-1.amazonaws.com/cg-d4b776d0-d898-4153-90c8-8336f86bdfec/masters/'
datalist = ['arson','asr','cargo-theft','hate-crime','human-trafficking','pe','nibrs','reta','shr','supp']
try:
if not os.path.exists('./data/'):
os.mkdir('./data/')
for f in datalist:
for i in range(1985,2023,1):
url = BaseFolder+f+'/'+f+'-'+str(i)+'.zip'
print(url)
try:
response = wget.download(url,'./data/' + url.split('/')[6])
except:
print('ERR: '+ url)
except:
print("Program Error")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment