Skip to content

Instantly share code, notes, and snippets.

@venkatsgithub1
Last active May 26, 2018 06:49
Show Gist options
  • Save venkatsgithub1/84b18d3ffa00e7ac0ae9ed69e6a0468a to your computer and use it in GitHub Desktop.
Save venkatsgithub1/84b18d3ffa00e7ac0ae9ed69e6a0468a to your computer and use it in GitHub Desktop.
A small test on requests with and without multiprocessing.
import requests
import time
url = 'https://xkcd.com/'
url_data = []
start = time.time()
for i in range(1821, 1831):
url_data.append(requests.get(url + str(i)))
print("time taken:", (time.time() - start))
"""
Machine Information:
Sockets: 1
Core(s) per socket: 2
CPU(s): 4
******************************
Output:
time taken: 14.421966791152954
******************************
"""
import time
from multiprocessing import Pool
def retrieve_data(integer):
import requests
url = 'https://xkcd.com/'
data = requests.get(url + str(integer))
return data
if __name__ == "__main__":
start = time.time()
pool = Pool()
result = pool.map(retrieve_data, range(1821, 1831))
pool.close()
pool.join()
print("time taken:", (time.time() - start))
"""
Machine Information:
Sockets: 1
Core(s) per socket: 2
CPU(s): 4
******************************
Output:
time taken: 4.337875127792358
******************************
"""
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment