Skip to content

Instantly share code, notes, and snippets.

@avneesh91
Created January 12, 2020 13:04
Show Gist options
  • Save avneesh91/7bc2abf3a4353945496060259355ddd3 to your computer and use it in GitHub Desktop.
Save avneesh91/7bc2abf3a4353945496060259355ddd3 to your computer and use it in GitHub Desktop.
With_futures
import requests
from concurrent.futures import ThreadPoolExecutor
url_list = ['https://docs.python.org/3/library/concurrent.futures.html', \
'https://technokeeda.com', \
'http://home.pipeline.com/~hbaker1/Futures.html']
def get_futures_get(urls):
results = []
currs = ThreadPoolExecutor(max_workers=5)
for url in urls:
curr_future_result = currs.submit(worker_func, url, results)
# you can also use
# curr_future_result.result() to actually get
# the result from the future object. This however
# is a blocking call and should be done only after the
# line written below. For the purpose of the example I
# updating the result in the list I passed along with the
# function.
currs.shutdown(wait=True)
return results
def worker_func(url, result_list):
response = requests.get(url)
# note: This line is not thread safe, I am doing this
# only for demonstration purposes. Do not write your
# production code like this.
result_list.append([response, response.status_code])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment