Skip to content

Instantly share code, notes, and snippets.

@daleobrien
Last active August 29, 2015 14:01
Show Gist options
  • Save daleobrien/9c085b68174bc4d98d8f to your computer and use it in GitHub Desktop.
Save daleobrien/9c085b68174bc4d98d8f to your computer and use it in GitHub Desktop.
Multi threaded map
from multiprocessing import Process, Queue
def mp_map(pfunc, iterable, debug=False):
q = Queue(2)
for i, u in enumerate(iterable):
t = Process(target=lambda q, i, u: q.put([i,
pfunc(u)]), args=(q, i, u))
t.start()
results = list()
while len(results) < len(iterable):
res = q.get()
results.append(res)
if debug:
print 'worker %d completed' % res[0]
return [x for (_, x) in sorted(results)]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment