Skip to content

Instantly share code, notes, and snippets.

@sebclaeys
Created September 21, 2011 13:56
Show Gist options
  • Save sebclaeys/1232088 to your computer and use it in GitHub Desktop.
Save sebclaeys/1232088 to your computer and use it in GitHub Desktop.
Python non-blocking read with subprocess.Popen
import fcntl
import os
from subprocess import *
def non_block_read(output):
fd = output.fileno()
fl = fcntl.fcntl(fd, fcntl.F_GETFL)
fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
try:
return output.read()
except:
return ""
############
# Use case #
############
sb = Popen("echo test; sleep 10000", shell=True, stdout=PIPE)
sb.kill()
sb.poll() # return -9
#sb.stdout.read() # Will block and will block forever cause nothing will come out since the job is done
non_block_read(sb.stdout) # will return '' instead of hanging for ever
Copy link

ghost commented Oct 4, 2013

Thanks. This works in python 3.x as well, unlike twisted. It's also not as verbose.

@pcborenstein
Copy link

Why does a function that sets u non-blocking try to return the first line from the handle? And why is this read in a try block? Something isn't right here...

@generatz
Copy link

Note that fcntl isn't available on Windows.

@sevketarisu
Copy link

Thank you you saved my life :)

@lttzzlll
Copy link

Note that fcntl isn't available on Windows. =_= ...

@bmr-cymru
Copy link

Just look at the code: the shell=True is only there so that the example (which is in shell notation) can work:

sb = Popen("echo test; sleep 10000", shell=True, stdout=PIPE)

If you don't use shell syntax, or rely on any other shell features, then you don't need shell=True.

@solotim
Copy link

solotim commented Oct 19, 2021

This is much simpler:

p = Popen("echo test; sleep 10000", shell=True, stdout=PIPE)
for x in iter(p.stdout.readline, b''):
    print(x)

@makermelissa
Copy link

Thank you so much. I've been trying just about everything to get this working with a read -p "Type something" command inside a script (which outputs to stderr without a newline making readline() not work) and this is the only thing that has worked properly.

@alercelik
Copy link

This is much simpler:

p = Popen("echo test; sleep 10000", shell=True, stdout=PIPE)
for x in iter(p.stdout.readline, b''):
    print(x)

p.stdout.readline() blocks. That is why this gist is about non_blocking_read

@alercelik
Copy link

Readline version (strips '\n')

def non_block_readline(output) -> str:
    fd = output.fileno()
    fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
    try:
        return output.readline().strip("\n")
    except:
        return ""

@MitchiLaser
Copy link

Since Python 3.5 os.set_blocking can do exactly the same thing:

import os
from subprocess import *

sb = Popen("echo test; sleep 10000", shell=True, stdout=PIPE)
os.set_blocking(sb.stdout.fileno(), False)  # That's what you are looking for
sb.kill()
sb.poll() # return -9
sb.stdout.read()  # This is not going to block. When the pipe is empty it returns an empty string.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment