python-parallel-execution.md

About

A list of links on parallel execution of processes / threads within Python

Parallel Execution in Python

• https://stackoverflow.com/questions/16071866/non-blocking-subprocess-call

• https://stackoverflow.com/questions/375427/a-non-blocking-read-on-a-subprocess-pipe-in-python/4896288#4896288

• https://stackoverflow.com/questions/17190221/subprocess-popen-cloning-stdout-and-stderr-both-to-terminal-and-variables/25960956#25960956

• https://github.com/mortoray/shelljob/blob/master/src/shelljob/proc.py non async

• https://code.activestate.com/recipes/440554-module-to-allow-asynchronous-subprocess-use-on-win/

• https://github.com/nmusatti/nxpy/blob/master/libs/nonblocking_subprocess/nxpy/core/nonblocking_subprocess/nonblocking_subprocess.py

• https://www.nurmatova.com/subprocesses-and-multithreading.html

• https://docs.python.org/3/library/threading.html#condition-objects

• https://stackoverflow.com/questions/2715847/read-streaming-input-from-subprocess-communicate/17698359#17698359

If you want a non-blocking approach, don't use process.communicate(). If you set the subprocess.Popen() argument stdout to PIPE, you can read from process.stdout and check if the process still runs using process.poll().

Some Gists with Code

• https://gist.github.com/mckaydavis/e96c1637d02bcf8a78e7
• https://gist.github.com/kirpit/1306188/ab800151c9128db3b763bb9f9ec19fda0df3a843

Applied so far

• https://pymotw.com/3/threading/
• https://pymotw.com/3/multiprocessing/communication.html#process-pools
• https://stackoverflow.com/questions/2332765/lock-mutex-semaphore-whats-the-difference/45567101#45567101

Async

• https://stackoverflow.com/questions/63544501/python-asyncio-how-to-have-a-producer-consumer-pattern-linked-to-a-watchdog-pro
• https://stackoverflow.com/questions/28492103/how-to-combine-python-asyncio-with-threads
• https://gist.github.com/mivade/f4cb26c282d421a62e8b9a341c7c65f6
• https://stackoverflow.com/questions/54525836/where-do-i-catch-the-keyboardinterrupt-exception-in-this-async-setup
• https://stackoverflow.com/questions/40571786/asyncio-coroutine-vs-async-def
• https://docs.python.org/3/library/asyncio-subprocess.html