Skip to content

Instantly share code, notes, and snippets.

@andriisoldatenko
Last active June 5, 2017 12:42
Show Gist options
  • Save andriisoldatenko/64e83f884ec7ce432495 to your computer and use it in GitHub Desktop.
Save andriisoldatenko/64e83f884ec7ce432495 to your computer and use it in GitHub Desktop.
Parallelism in one line
import urllib2
from multiprocessing.dummy import Pool as ThreadPool
urls = [
'http://www.python.org',
'http://www.python.org/about/',
'http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html',
'http://www.python.org/doc/',
'http://www.python.org/download/',
'http://www.python.org/getit/',
'http://www.python.org/community/',
'https://wiki.python.org/moin/',
'http://planet.python.org/',
'https://wiki.python.org/moin/LocalUserGroups',
'http://www.python.org/psf/',
'http://docs.python.org/devguide/',
'http://www.python.org/community/awards/'
# etc..
]
# Make the Pool of workers
pool = ThreadPool(4)
# Open the urls in their own threads
# and return the results
results = pool.map(urllib2.urlopen, urls)
#close the pool and wait for the work to finish
pool.close()
pool.join()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment