Skip to content

Instantly share code, notes, and snippets.

@BryanCutler

BryanCutler/tf_arrow_blog_pt9.py

Last active August 5, 2019 17:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save BryanCutler/b58eea7112be62582cc99f432b6f9118 to your computer and use it in GitHub Desktop.
Save BryanCutler/b58eea7112be62582cc99f432b6f9118 to your computer and use it in GitHub Desktop.
Download ZIP
TensorFlow Arrow Blog Part 9 - Read and Process Directory
Raw
tf_arrow_blog_pt9.py
def read_and_process_dir(directory):
"""Read a directory of CSV files and yield processed Arrow batches."""
for f in os.listdir(directory):
if f.endswith(".csv"):
filename = os.path.join(directory, f)
for batch in read_and_process(filename):
yield batch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment