Writing to a python subprocess pipe

subprocess_pipe.md

Here's a few things I tried to write output to a python subprocess pipe.

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.communicate('Line number %d.\n' % x)

This seemed like the most obvious solution but it fails miserably. It seems that the first call to communicate also closes the pipe and the second loop raises an exception.

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)

This is expressly stated to be a bad idea in the docs, but it works - sort of. I get some weird behavior. There's no call to p.wait() (which communicate does by default) so anything after the loop runs before the subproccess (less in this case) is closed. Adding a call to wait after the loop causes even weirder behavior.

from subprocess import Popen, PIPE

out = []
p = Popen('less', stdin=PIPE)
for x in xrange(100):
    out.append('Line number %d.' % x)
p.communicate('\n'.join(out))

This works. We only have one call to communicate and that calls wait properly. Unfortunately, we have to create the entire output in memory before writing any of it out. We can do better:

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)
p.stdin.close()
p.wait()

The key it to close stdin (flush and send EOF) before calling wait. This is actually what communicate does internally minus all the stdout and stderr stuff I don't need. If I wanted to force the buffer to remain empty, I suppose I could do p.stdin.flush() on each loop, but why? Note that there probably should be some error checking on write (like there is in the source of communicate. Perhaps something like:

import errno

...

try:
    p.stdin.write(input)
except IOError as e:
    if e.errno != errno.EPIPE and e.errno != errno.EINVAL:
        raise

Note that errno.EPIPE is "Broken pipe" and errno.EINVAL is "Invalid argument".

So the final code looks like this:

from subprocess import Popen, PIPE
import errno

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    line = 'Line number %d.\n' % x
    try:
        p.stdin.write(line)
    except IOError as e:
        if e.errno == errno.EPIPE or e.errno == errno.EINVAL:
            # Stop loop on "Invalid pipe" or "Invalid argument".
            # No sense in continuing with broken pipe.
            break
        else:
            # Raise any other error.
            raise

p.stdin.close()
p.wait()

print 'All done!' # This should always be printed below any output written to less.