I recently came across a situation where I wanted to capture the output of a subprocess started by a Python script,
but also let it print to the terminal normally. An example of where this may be useful is with something like curl
,
where progress is output to stderr
(with the -o
option). In an interactive program, you may want to show the user
that progress information, but also capture it for parsing in your script. By default,
subprocess.run
does not capture
any output, but the subprocess does print to the terminal. Passing stdout=subprocess.PIPE, stderr=subprocess.STDOUT
to subprocess.run
captures the output but does not let the subprocess print. So you don't see any output until
the subprocess has completed. Redirecting sys.stdout
or sys.stderr
doesn't work because it only replaces the
Python script's stdout
or stderr
, it doesn't have an effect on the subprocess'.
The only way to accomplish this seems to be to start the subprocess with the non-blocking subprocess.Popen
, poll
for available output, and both print it and accumulate it in a variable. The code shown here requires the
selectors
module, which is only available in Python 3.4+.
This was super helpful. With inspiration from this gist, I ended up writing a function that could do something similar while keeping stdout/stderr separate, shown here. I also had some problems trying to run it in a script remotely over SSH, and the post also discusses how I worked around those issues, in case anyone is trying to do that.