Skip to content

Instantly share code, notes, and snippets.

@waylan
Created April 10, 2012 19:12
Show Gist options
  • Save waylan/2353749 to your computer and use it in GitHub Desktop.
Save waylan/2353749 to your computer and use it in GitHub Desktop.
Writing to a python subprocess pipe

Here's a few things I tried to write output to a python subprocess pipe.

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.communicate('Line number %d.\n' % x)

This seemed like the most obvious solution but it fails miserably. It seems that the first call to communicate also closes the pipe and the second loop raises an exception.

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)

This is expressly stated to be a bad idea in the docs, but it works - sort of. I get some weird behavior. There's no call to p.wait() (which communicate does by default) so anything after the loop runs before the subproccess (less in this case) is closed. Adding a call to wait after the loop causes even weirder behavior.

from subprocess import Popen, PIPE

out = []
p = Popen('less', stdin=PIPE)
for x in xrange(100):
    out.append('Line number %d.' % x)
p.communicate('\n'.join(out))

This works. We only have one call to communicate and that calls wait properly. Unfortunately, we have to create the entire output in memory before writing any of it out. We can do better:

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)
p.stdin.close()
p.wait()

The key it to close stdin (flush and send EOF) before calling wait. This is actually what communicate does internally minus all the stdout and stderr stuff I don't need. If I wanted to force the buffer to remain empty, I suppose I could do p.stdin.flush() on each loop, but why? Note that there probably should be some error checking on write (like there is in the source of communicate. Perhaps something like:

import errno

...

try:
    p.stdin.write(input)
except IOError as e:
    if e.errno != errno.EPIPE and e.errno != errno.EINVAL:
        raise

Note that errno.EPIPE is "Broken pipe" and errno.EINVAL is "Invalid argument".

So the final code looks like this:

from subprocess import Popen, PIPE
import errno

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    line = 'Line number %d.\n' % x
    try:
        p.stdin.write(line)
    except IOError as e:
        if e.errno == errno.EPIPE or e.errno == errno.EINVAL:
            # Stop loop on "Invalid pipe" or "Invalid argument".
            # No sense in continuing with broken pipe.
            break
        else:
            # Raise any other error.
            raise

p.stdin.close()
p.wait()

print 'All done!' # This should always be printed below any output written to less.
@tarek-kerbedj
Copy link

Thank you , this helped me get through

@brainfo
Copy link

brainfo commented Jul 31, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment