Writing to a python subprocess pipe

subprocess_pipe.md

Here's a few things I tried to write output to a python subprocess pipe.

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.communicate('Line number %d.\n' % x)

This seemed like the most obvious solution but it fails miserably. It seems that the first call to communicate also closes the pipe and the second loop raises an exception.

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)

This is expressly stated to be a bad idea in the docs, but it works - sort of. I get some weird behavior. There's no call to p.wait() (which communicate does by default) so anything after the loop runs before the subproccess (less in this case) is closed. Adding a call to wait after the loop causes even weirder behavior.

from subprocess import Popen, PIPE

out = []
p = Popen('less', stdin=PIPE)
for x in xrange(100):
    out.append('Line number %d.' % x)
p.communicate('\n'.join(out))

This works. We only have one call to communicate and that calls wait properly. Unfortunately, we have to create the entire output in memory before writing any of it out. We can do better:

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)
p.stdin.close()
p.wait()

The key it to close stdin (flush and send EOF) before calling wait. This is actually what communicate does internally minus all the stdout and stderr stuff I don't need. If I wanted to force the buffer to remain empty, I suppose I could do p.stdin.flush() on each loop, but why? Note that there probably should be some error checking on write (like there is in the source of communicate. Perhaps something like:

import errno

...

try:
    p.stdin.write(input)
except IOError as e:
    if e.errno != errno.EPIPE and e.errno != errno.EINVAL:
        raise

Note that errno.EPIPE is "Broken pipe" and errno.EINVAL is "Invalid argument".

So the final code looks like this:

from subprocess import Popen, PIPE
import errno

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    line = 'Line number %d.\n' % x
    try:
        p.stdin.write(line)
    except IOError as e:
        if e.errno == errno.EPIPE or e.errno == errno.EINVAL:
            # Stop loop on "Invalid pipe" or "Invalid argument".
            # No sense in continuing with broken pipe.
            break
        else:
            # Raise any other error.
            raise

p.stdin.close()
p.wait()

print 'All done!' # This should always be printed below any output written to less.

Thanks for sharing that! Very useful.

Anyone communicating with subrocesses over any text or terminal based protocol should consider using the pexpect library:

https://github.com/pexpect/pexpect
http://pexpect.readthedocs.org/en/latest/

pip install pexpect

Works at a higher level, supports bidirectional communication, scans for keywords, commands or messages using regular expressions, handles timeout, and more.

I'd be interested to know how much data you can write to stdin before you block forever (since you've filled the buffer and are blocking on the write, and the process you're writing to hasn't gotten EOF so it doesn't exit).

@rthille My bet is that less will consume the stream fast enough, but very good point.

It's cool, but how the consumer side looks? The process, which is started by Popen, if it's some other Python script

Thanks a lot for sharing. Can you also share (if there is a way) how to print the output without closing the input.write.

subprocess.run() makes things simpler in Python 3.7 if you already have the entire input you want to send.

In [1]: import subprocess 
   ...: mystr = ''' 
   ...: one 
   ...: two 
   ...: three''' 
   ...:  
   ...: cp = subprocess.run('sort', input=mystr, text=True, capture_output=True) 
   ...: print(cp.stdout)                                                                                                                                                                                           

one
three
two

Relevant quotes from the docs of subprocess.run():

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None)

The input argument is passed to Popen.communicate() and thus to the subprocess’s stdin. If used it must be a byte sequence, or a string if encoding or errors is specified or text is true. When used, the internal Popen object is automatically created with stdin=PIPE, and the stdin argument may not be used as well.

Changed in version 3.7: Added the text parameter, as a more understandable alias of universal_newlines. Added the capture_output parameter.

Hi, does anyone knows if "Broken pipe" error pops out when writing the 46th line, how to fix that?
Thank you very much!

Hi,

I have started a process using Popen . The created active process will be waiting for the input to be passed . I'm trying to send the input using stdin.write('help'). But the input is not reaching to the active process. In linux using proc i'm able to send the inputs but in windows since we dont have i'm stuck.

Can you please help me in sending the inputs to the process.

one_sil_controller = Popen(command, shell=False , stdin=PIPE, stdout=PIPE, stderr=STDOUT, bufsize=1)
one_sil_controller.stdin.write('help')

Thanks & Regards,
Shivaraj M M

Hi. I think you need to add tailing \n to your 'help'.

hey so I don't understand your code writes to an inputstream? I am trying to write to an inputstream using java to set initial text to the console or idea console. https://stackoverflow.com/questions/65145471/write-to-ide-console-and-read-the-text-later-with-scanner . Or is it an outputstream? I don't know enough python to tell.

Another option is to use p.poll after p.stdin.close()

p.stdin.close()
while p.returncode is None:
    p.poll()

Thank you , this helped me get through

已收到，谢谢。