Skip to content

Instantly share code, notes, and snippets.

@tbnorth
Created March 31, 2022 21:04
Show Gist options
  • Save tbnorth/5a5d0c2b33651428aa71f1d05fc9fa8f to your computer and use it in GitHub Desktop.
Save tbnorth/5a5d0c2b33651428aa71f1d05fc9fa8f to your computer and use it in GitHub Desktop.
"""Three ways to read data from a stream.
Suggested input:
one
three
END
one
three
END
one
three
Ctrl-D # unix terminal, ends input
Ctrl-Z # Windows terminal, ends input
The first two approaches can read unlimited entries without memory constraints, as
indicated by the responses appearing immediately after each input.
The third approach requires all the data fit in memory, as indicated by the absence of a
response until the data stream is ended with Ctrl-D / Ctrl-Z.
"""
import itertools
import sys
import time
def proc_stream(in_):
while "END" not in (text := next(in_).strip()):
length = len(text)
yield f"{length}: {text}"
def proc_data(in_):
answers = [f"{len(text)}: {text.strip()}" for text in in_]
return answers
def main():
# the Pythonic way, using a generator
for answer in proc_stream(sys.stdin):
timestamp = time.asctime()
print(f"{timestamp}: {answer}")
# the fancy schmancy ;-) functional programming way, approximately
for answer in map(
lambda x: f"{len(x)}: {x}",
itertools.takewhile(lambda x: "END" not in x, sys.stdin),
):
timestamp = time.asctime()
print(f"{timestamp}: {answer}")
# calling a function that reads all data into memory
for answer in proc_data(sys.stdin):
timestamp = time.asctime()
print(f"{timestamp}: {answer}")
if __name__ == "__main__":
main()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment