Getting realtime output using Python Subprocess

README.md

Getting realtime output using Python Subprocess

A convenience module for shelling out with realtime output

Credit: Largely taken from https://www.endpoint.com/blog/2015/01/28/getting-realtime-output-using-python

The Problem

When I launch a long running unix process within a python script, it waits until the process is finished, and only then do I get the complete output of my program. This is annoying if I’m running a process that takes a while to finish. And I want to capture the output and display it in the nice manner with clear formatting.

subprocess.popen

To run a process and read all of its output, set the stdout value to PIPE and call communicate().

import subprocess
process = subprocess.Popen(['echo', '"Hello stdout"'], stdout=subprocess.PIPE)
stdout = process.communicate()[0]
print('STDOUT:{}'.format(stdout))

The above script will wait for the process to complete and then it will display the output.

run_command.py

# -*- coding: utf-8 -*-
"""
A convenience module for shelling out with realtime output

includes: 
- subprocess - Works with additional processes.
- shlex - Lexical analysis of shell-style syntaxes.
"""

import subprocess
import shlex


def run(command):
    process = Popen(command, stdout=PIPE, shell=True)
    while True:
        line = process.stdout.readline().rstrip()
        if not line:
            break
        yield line.decode('utf-8')

def run_command(command):
    process = Popen(shlex.split(command), stdout=PIPE)
    while True:
        output = process.stdout.readline().rstrip().decode('utf-8')
        if output == '' and process.poll() is not None:
            break
        if output:
            print(output.strip())
    rc = process.poll()
    return rc

if __name__ == "__main__":
    for path in run("ping -c 5 google.com"):
        print path