Skip to content

Instantly share code, notes, and snippets.

@yigitozgumus
Created November 29, 2020 20:15
Show Gist options
  • Save yigitozgumus/da0665a3cb2f180ee85649ceb6d5c738 to your computer and use it in GitHub Desktop.
Save yigitozgumus/da0665a3cb2f180ee85649ceb6d5c738 to your computer and use it in GitHub Desktop.
Simple function for downloading small datasets
def fetch(url):
import requests, gzip, os, hashlib, numpy, sys
fp = os.path.join("/tmp", hashlib.md5(url.encode('utf-8')).hexdigest())
if os.path.isfile(fp):
with open(fp, "rb") as f:
data = f.read()
else:
with open(fp, "wb") as f:
data = requests.get(url).content
f.write(data)
return numpy.frombuffer(gzip.decompress(data), dtype=np.uint8).copy()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment