Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Writing to a python subprocess pipe

Here's a few things I tried to write output to a python subprocess pipe.

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.communicate('Line number %d.\n' % x)

This seemed like the most obvious solution but it fails miserably. It seems that the first call to communicate also closes the pipe and the second loop raises an exception.

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)

This is expressly stated to be a bad idea in the docs, but it works - sort of. I get some weird behavior. There's no call to p.wait() (which communicate does by default) so anything after the loop runs before the subproccess (less in this case) is closed. Adding a call to wait after the loop causes even weirder behavior.

from subprocess import Popen, PIPE

out = []
p = Popen('less', stdin=PIPE)
for x in xrange(100):
    out.append('Line number %d.' % x)
p.communicate('\n'.join(out))

This works. We only have one call to communicate and that calls wait properly. Unfortunately, we have to create the entire output in memory before writing any of it out. We can do better:

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)
p.stdin.close()
p.wait()

The key it to close stdin (flush and send EOF) before calling wait. This is actually what communicate does internally minus all the stdout and stderr stuff I don't need. If I wanted to force the buffer to remain empty, I suppose I could do p.stdin.flush() on each loop, but why? Note that there probably should be some error checking on write (like there is in the source of communicate. Perhaps something like:

import errno

...

try:
    p.stdin.write(input)
except IOError as e:
    if e.errno != errno.EPIPE and e.errno != errno.EINVAL:
        raise

Note that errno.EPIPE is "Broken pipe" and errno.EINVAL is "Invalid argument".

So the final code looks like this:

from subprocess import Popen, PIPE
import errno

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    line = 'Line number %d.\n' % x
    try:
        p.stdin.write(line)
    except IOError as e:
        if e.errno == errno.EPIPE or e.errno == errno.EINVAL:
            # Stop loop on "Invalid pipe" or "Invalid argument".
            # No sense in continuing with broken pipe.
            break
        else:
            # Raise any other error.
            raise

p.stdin.close()
p.wait()

print 'All done!' # This should always be printed below any output written to less.
@lzubiaur

This comment has been minimized.

Copy link

@lzubiaur lzubiaur commented Feb 18, 2014

Thanks for sharing that! Very useful.

@ghost

This comment has been minimized.

Copy link

@ghost ghost commented Jul 11, 2014

Anyone communicating with subrocesses over any text or terminal based protocol should consider using the pexpect library:

https://github.com/pexpect/pexpect
http://pexpect.readthedocs.org/en/latest/

pip install pexpect

Works at a higher level, supports bidirectional communication, scans for keywords, commands or messages using regular expressions, handles timeout, and more.

@rthille

This comment has been minimized.

Copy link

@rthille rthille commented Oct 6, 2015

I'd be interested to know how much data you can write to stdin before you block forever (since you've filled the buffer and are blocking on the write, and the process you're writing to hasn't gotten EOF so it doesn't exit).

@MaharishiCanada

This comment has been minimized.

Copy link

@MaharishiCanada MaharishiCanada commented Apr 25, 2017

@rthille My bet is that less will consume the stream fast enough, but very good point.

@yuchdev

This comment has been minimized.

Copy link

@yuchdev yuchdev commented Feb 2, 2018

It's cool, but how the consumer side looks? The process, which is started by Popen, if it's some other Python script

@ckesanapalli

This comment has been minimized.

Copy link

@ckesanapalli ckesanapalli commented Feb 17, 2019

Thanks a lot for sharing. Can you also share (if there is a way) how to print the output without closing the input.write.

@shivdhar

This comment has been minimized.

Copy link

@shivdhar shivdhar commented Feb 21, 2019

subprocess.run() makes things simpler in Python 3.7 if you already have the entire input you want to send.

In [1]: import subprocess 
   ...: mystr = ''' 
   ...: one 
   ...: two 
   ...: three''' 
   ...:  
   ...: cp = subprocess.run('sort', input=mystr, text=True, capture_output=True) 
   ...: print(cp.stdout)                                                                                                                                                                                           

one
three
two

Relevant quotes from the docs of subprocess.run():

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None)

The input argument is passed to Popen.communicate() and thus to the subprocess’s stdin. If used it must be a byte sequence, or a string if encoding or errors is specified or text is true. When used, the internal Popen object is automatically created with stdin=PIPE, and the stdin argument may not be used as well.

Changed in version 3.7: Added the text parameter, as a more understandable alias of universal_newlines. Added the capture_output parameter.

@brainfo

This comment has been minimized.

Copy link

@brainfo brainfo commented Dec 4, 2019

Hi, does anyone knows if "Broken pipe" error pops out when writing the 46th line, how to fix that?
Thank you very much!

@shivarajshivu

This comment has been minimized.

Copy link

@shivarajshivu shivarajshivu commented Sep 1, 2020

Hi,

I have started a process using Popen . The created active process will be waiting for the input to be passed . I'm trying to send the input using stdin.write('help'). But the input is not reaching to the active process. In linux using proc i'm able to send the inputs but in windows since we dont have i'm stuck.

Can you please help me in sending the inputs to the process.

one_sil_controller = Popen(command, shell=False , stdin=PIPE, stdout=PIPE, stderr=STDOUT, bufsize=1)
one_sil_controller.stdin.write('help')

Thanks & Regards,
Shivaraj M M

@razmikarm

This comment has been minimized.

Copy link

@razmikarm razmikarm commented Oct 16, 2020

Hi. I think you need to add tailing \n to your 'help'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.