Skip to content

Instantly share code, notes, and snippets.

@waylan
Created April 10, 2012 19:12
Show Gist options
  • Save waylan/2353749 to your computer and use it in GitHub Desktop.
Save waylan/2353749 to your computer and use it in GitHub Desktop.
Writing to a python subprocess pipe

Here's a few things I tried to write output to a python subprocess pipe.

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.communicate('Line number %d.\n' % x)

This seemed like the most obvious solution but it fails miserably. It seems that the first call to communicate also closes the pipe and the second loop raises an exception.

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)

This is expressly stated to be a bad idea in the docs, but it works - sort of. I get some weird behavior. There's no call to p.wait() (which communicate does by default) so anything after the loop runs before the subproccess (less in this case) is closed. Adding a call to wait after the loop causes even weirder behavior.

from subprocess import Popen, PIPE

out = []
p = Popen('less', stdin=PIPE)
for x in xrange(100):
    out.append('Line number %d.' % x)
p.communicate('\n'.join(out))

This works. We only have one call to communicate and that calls wait properly. Unfortunately, we have to create the entire output in memory before writing any of it out. We can do better:

from subprocess import Popen, PIPE

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    p.stdin.write('Line number %d.\n' % x)
p.stdin.close()
p.wait()

The key it to close stdin (flush and send EOF) before calling wait. This is actually what communicate does internally minus all the stdout and stderr stuff I don't need. If I wanted to force the buffer to remain empty, I suppose I could do p.stdin.flush() on each loop, but why? Note that there probably should be some error checking on write (like there is in the source of communicate. Perhaps something like:

import errno

...

try:
    p.stdin.write(input)
except IOError as e:
    if e.errno != errno.EPIPE and e.errno != errno.EINVAL:
        raise

Note that errno.EPIPE is "Broken pipe" and errno.EINVAL is "Invalid argument".

So the final code looks like this:

from subprocess import Popen, PIPE
import errno

p = Popen('less', stdin=PIPE)
for x in xrange(100):
    line = 'Line number %d.\n' % x
    try:
        p.stdin.write(line)
    except IOError as e:
        if e.errno == errno.EPIPE or e.errno == errno.EINVAL:
            # Stop loop on "Invalid pipe" or "Invalid argument".
            # No sense in continuing with broken pipe.
            break
        else:
            # Raise any other error.
            raise

p.stdin.close()
p.wait()

print 'All done!' # This should always be printed below any output written to less.
@shivdhar
Copy link

shivdhar commented Feb 21, 2019

subprocess.run() makes things simpler in Python 3.7 if you already have the entire input you want to send.

In [1]: import subprocess 
   ...: mystr = ''' 
   ...: one 
   ...: two 
   ...: three''' 
   ...:  
   ...: cp = subprocess.run('sort', input=mystr, text=True, capture_output=True) 
   ...: print(cp.stdout)                                                                                                                                                                                           

one
three
two

Relevant quotes from the docs of subprocess.run():

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, capture_output=False, shell=False, cwd=None, timeout=None, check=False, encoding=None, errors=None, text=None, env=None, universal_newlines=None)

The input argument is passed to Popen.communicate() and thus to the subprocess’s stdin. If used it must be a byte sequence, or a string if encoding or errors is specified or text is true. When used, the internal Popen object is automatically created with stdin=PIPE, and the stdin argument may not be used as well.

Changed in version 3.7: Added the text parameter, as a more understandable alias of universal_newlines. Added the capture_output parameter.

@brainfo
Copy link

brainfo commented Dec 4, 2019

Hi, does anyone knows if "Broken pipe" error pops out when writing the 46th line, how to fix that?
Thank you very much!

@shivarajshivu
Copy link

Hi,

I have started a process using Popen . The created active process will be waiting for the input to be passed . I'm trying to send the input using stdin.write('help'). But the input is not reaching to the active process. In linux using proc i'm able to send the inputs but in windows since we dont have i'm stuck.

Can you please help me in sending the inputs to the process.

one_sil_controller = Popen(command, shell=False , stdin=PIPE, stdout=PIPE, stderr=STDOUT, bufsize=1)
one_sil_controller.stdin.write('help')

Thanks & Regards,
Shivaraj M M

@razmikarm
Copy link

Hi. I think you need to add tailing \n to your 'help'.

@jredfox
Copy link

jredfox commented Dec 4, 2020

hey so I don't understand your code writes to an inputstream? I am trying to write to an inputstream using java to set initial text to the console or idea console. https://stackoverflow.com/questions/65145471/write-to-ide-console-and-read-the-text-later-with-scanner . Or is it an outputstream? I don't know enough python to tell.

@kirk86
Copy link

kirk86 commented Dec 4, 2021

Another option is to use p.poll after p.stdin.close()

p.stdin.close()
while p.returncode is None:
    p.poll()

@tarek-kerbedj
Copy link

Thank you , this helped me get through

@brainfo
Copy link

brainfo commented Jul 31, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment