A simple and general solution for calling OS commands from Python
With Python, calling an OS command in a simple way has long been a pain in the neck, with multiple solutions continuously asked,
answered, counter-answered and sometimes deprecated (
os.system which was phased out,
The key is to realize that there are, really, four ways of calling an OS command from a high level language:
- as a command: you want to print the output as it comes
- as a function: you want no printed output, but a result in the form of a string
- as a function with side effect: you want to execute the command, watch what it does, and then analyse the output.
- as an ongoing process: you want to get every returned line as soon as it comes and do something with it.
The clever idea, as suggested by Zaiste, is to make a generator function that returns the result of the execution of the OS command, line by line.
This takes care of your fourth case, and implicitly of your first three.
Getting the result of the command in one go after is executed is done very simply: by applying
list to the result of the generator.
Subprocessreturns byte strings. The solution was to use the
str(x, ENCODING)solution, with
ENCODING = 'UTF-8'.
import sys, subprocess, shlex class Popen2(subprocess.Popen): "Context manager for Python2" def __enter__(self): return self def __exit__(self, type, value, traceback): if self.stdout: self.stdout.close() if self.stderr: self.stderr.close() if self.stdin: self.stdin.close() # Wait for the process to terminate, to avoid zombies. self.wait() def os_command(command, print_output=True, shell=False): """ Run an OS command (utility function) and returns a generator with each line of the stdout. In case of error, the sterr is forwarded through the exception. For the arguments, see run_os_command. If you are not sur between os_command and run_os_command, then the second is likely for you. """ ENCODING = 'UTF-8' if isinstance(command, str): # if a string, split into a list: command = shlex.split(command) # we need a proper context manager for Python 2: if sys.version_info < (3,2): Popen = Popen2 else: Popen = subprocess.Popen # Process: with Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=shell) as process: while True: line = process.stdout.readline().rstrip() if not line: # check error: process.poll() errno = process.returncode if errno: # get the error message: stderr_msg = str(process.stderr.read(), ENCODING) errmsg = "Call of '%s' failed with error %s\n%s" % \ (command, errno, stderr_msg) raise OSError(errno, errmsg) break # Python 3: converto to unicode: if sys.version_info > (3,0): line = str(line, ENCODING) if print_output: print(line) yield line def run_os_command(command, print_output=True, shell=False): """ Execute a command, printing as you go (unless you want to suppress it) Arguments: ---------- command: eithr a string, a list containing each element of the command e.g. ['ls', '-l'] print_output: print the results as the command executes (default: True) shell: call the shell; this activates globbing, etc. (default: False, as this is safer) Returns: -------- A string containing the stdout """ r = list(os_command(command, print_output=print_output, shell=shell)) return "\n".join(r) def os_get(command, shell=False): """ Execute a command as a function Arguments: ---------- command: a list containing each element of the command e.g. ['ls', '-l'] shell: call the shell; this activates globbing, etc. (default: False) Returns: -------- A string containing the output """ return run_os_command(command, print_output=False, shell=shell)
Case 1: Command
Case 2: Function (a string)
r = os_get(['ls'])
Which is really:
r = run_os_command(['ls'], print_output=False)
Case 3: Function with side effect (also printing)
r = run_os_command(['ls'])
Case 4: Get a generator and do something with it
for line in os_command(['ping', '-c100', 'www.google.com'], print_output=False): print("Look at what just happened:", line)
By default, it will print the lines as it goes, if you want to suppress that and do your own print, you have to set the
print_output flag to False.
The error management was difficult to get right.
Capturing exit codes
For a call of a legitimate command, but with incorrect parameters, an
OSError is raised (the solution found here was to "poll" the process before closing the generator and check the return code).
For a missing command on the OS, or incorrect command:
- On Python 3, it raises a
FileNotFoundError(this is automatically raised by
- On Python 2, this raises an
Context manager for subprocess.Popen
Popen, as of Python 3.2 requires a context, in order to make sure all files are closed (it has to be called with
with Popen(...) as process instead of
process=Popen(...), otherwise an warning is raised.
For compatibility with previous versions, a context manager Popen2 had to be added (see Stack Overflow: Exception handling when using Python's Subprocess Popen).