A pattern for using concurrent.futures.ThreadPoolExecutor()
to execute a series of "worker" functions in parallel, processing a queue of assigned "work".
How it works:
worker_init()
creates:- Queues for "work" and "results".
- A
threading.Event()
to denote "no more work in queue". - A
futures.ThreadPoolExecutor()
. - At this point, the pool of
worker()
functions are ready to accept work tasks from the work queue.
- A set of dummy tasks are added onto the
work
queue. Once all tasks added,finished.set()
denotes "no more tasks". - The
worker()
functions in the pool willwork.pop()
tasks off the queue, process the task and push a faux result onto theresult
queue. - A call to
pool.shutdown()
will pause until all workers have finished up. - Finally, dummy results are displayed from the worker pool.