Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Python multiprocessing hello world. Split a list and process sublists in different jobs
import multiprocessing
# split a list into evenly sized chunks
def chunks(l, n):
return [l[i:i+n] for i in range(0, len(l), n)]
def do_job(job_id, data_slice):
for item in data_slice:
print "job", job_id, item
def dispatch_jobs(data, job_number):
total = len(data)
chunk_size = total / job_number
slice = chunks(data, chunk_size)
jobs = []
for i, s in enumerate(slice):
j = multiprocessing.Process(target=do_job, args=(i, s))
jobs.append(j)
for j in jobs:
j.start()
if __name__ == "__main__":
data = ['a', 'b', 'c', 'd']
dispatch_jobs(data, 2)
@manugarri

This comment has been minimized.

Copy link

@manugarri manugarri commented Mar 3, 2014

very elegant. thanks for sharing!

@JasonArnotClark

This comment has been minimized.

Copy link

@JasonArnotClark JasonArnotClark commented Nov 11, 2016

As someone just learning to program, this finally made multiprocessing make sense. Thanks for posting.

@TheReal1604

This comment has been minimized.

Copy link

@TheReal1604 TheReal1604 commented Aug 10, 2018

That is a nice example, thank you!

@itouchup

This comment has been minimized.

Copy link

@itouchup itouchup commented Sep 5, 2018

Exactly what I needed to see. Thank you.

@dvandonkelaar

This comment has been minimized.

Copy link

@dvandonkelaar dvandonkelaar commented Dec 10, 2018

Good example for the beginner programming.

For python 3.7, the section
slice = chunks(data, chunk_size)

needs to be changed to:
slice = chunks(data, int(chunk_size))

To avoid:
TypeError: 'float' object cannot be interpreted as an integer

@ratmcu

This comment has been minimized.

Copy link

@ratmcu ratmcu commented Aug 8, 2019

Good example for the beginner programming.

For python 3.7, the section
slice = chunks(data, chunk_size)

needs to be changed to:
slice = chunks(data, int(chunk_size))

To avoid:
TypeError: 'float' object cannot be interpreted as an integer

or
chunk_size = total // job_number

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment