Skip to content

Instantly share code, notes, and snippets.

@jbaiter
Created December 25, 2016 22:19
Show Gist options
  • Save jbaiter/8b80338a63345a0d6e18cd44d3d8e886 to your computer and use it in GitHub Desktop.
Save jbaiter/8b80338a63345a0d6e18cd44d3d8e886 to your computer and use it in GitHub Desktop.
threaded version
def fetch_from_remote(url):
return requests.get(url).content
def content_generator(urls):
with concurrent.futures.ThreadPoolExectur(max_workers=4) as pool:
futs = [pool.submit(fetch_from_remote, url) for url in urls]
for fut in concurrent.futures.as_completed(futs):
yield fut.result()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment