Skip to content

Instantly share code, notes, and snippets.

@root-11
Created January 8, 2014 09:25
Show Gist options
  • Save root-11/8314018 to your computer and use it in GitHub Desktop.
Save root-11/8314018 to your computer and use it in GitHub Desktop.
From Linkedin Python post "Python parallelism in one line - A Better Model for Day to Day Threading Tasks" Does anyone know of similar "compact" examples of handling parallelism for the case where the workers may produce tasks that are put into the FIFO queue (maybe with `while` instead of map) ?
# requires python 3.2.3 or higher
import concurrent.futures
def f(x):
print ("working on task: %s" %{x})
if x-1 > 0 :
return x-1 # I want this task back in the queue.
def main():
Queue = [x for x in range(4)]
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
while len(Queue)>0:
task = Queue.pop(0)
future = executor.submit(f, task)
result = future.result()
if result is not None:
Queue.append(result) # here the f may return a task is to be added to the queue.
print ("processing completed.")
if __name__ == "__main__":
main()
# expected output:
# "working on task: 1"
# "working on task: 2" (returns 1 to the Queue)
# "working on task: 3" (returns 2 to the Queue)
# "working on task: 4" (returns 3 to the Queue)
# "working on task: 1"
# "working on task: 2" (returns 1 to the Queue)
# "working on task: 3" (returns 2 to the Queue)
# "working on task: 1"
# "working on task: 2" (returns 1 to the Queue)
# "working on task: 1"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment