Skip to content

Instantly share code, notes, and snippets.

Last active December 7, 2022 01:09
  • Star 7 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
Star You must be signed in to star a gist
What would you like to do?
Stream data asynchronously through a subprocess in Python
Problem: provide two-way communication with a subprocess in Python.
See also:
import asyncio
import sys
import typing as T
async def enqueue(values: T.Iterable[bytes], stream: asyncio.StreamWriter):
for line in values:
# Yield to the asyncio loop
await stream.drain()
# Once we've exhausted values, we need to close the async stream to signal to
# the subprocess that it can exit
async def dequeue(stream: asyncio.StreamReader, callback: T.Callable[[bytes], None]):
while True:
line = await stream.readline()
if not line:
async def main():
proc = await asyncio.subprocess.create_subprocess_exec(
'sed', r's/o/o\n/g',
await asyncio.wait([
enqueue(sys.stdin.buffer, proc.stdin),
dequeue(proc.stdout, sys.stdout.buffer.write),
# I'm not completely sure the call to `communicate` is necessary
(stdout_data, stderr_data) = await proc.communicate()
await proc.wait()
if __name__ == '__main__':
loop = asyncio.get_event_loop()
rc = loop.run_until_complete(main())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment