Skip to content

Instantly share code, notes, and snippets.

@hoffrocket
Created August 28, 2012 00:29
Show Gist options
  • Save hoffrocket/3493802 to your computer and use it in GitHub Desktop.
Save hoffrocket/3493802 to your computer and use it in GitHub Desktop.
Python parallel http requests using multiprocessing
#!/usr/bin/env python
from multiprocessing import Process, Pool
import time
import urllib2
def millis():
return int(round(time.time() * 1000))
def http_get(url):
start_time = millis()
result = {"url": url, "data": urllib2.urlopen(url, timeout=5).read()[:100]}
print url + " took " + str(millis() - start_time) + " ms"
return result
urls = ['http://www.google.com/', 'https://foursquare.com/', 'http://www.yahoo.com/', 'http://www.bing.com/', "https://www.yelp.com/"]
pool = Pool(processes=5)
start_time = millis()
results = pool.map(http_get, urls)
print "\nTotal took " + str(millis() - start_time) + " ms\n"
for result in results:
print result
@danielpizarro
Copy link

why is this "tread safe"?

@stefansjs
Copy link

Because of the GIL. You don't have to worry about observability side effects in python (at the cost of performance, but then again you're using python)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment