Skip to content

Instantly share code, notes, and snippets.

@frankShih
frankShih / dataflow_json.py
Last active May 25, 2022 13:18
using GCP dataflow to read text file as JSON & split it into files
#!/usr/bin/env python3
from __future__ import absolute_import
import argparse
import json
# from google.cloud import storage
from datetime import datetime
import configparser
import setuptools
'''
If you created a requirements.txt file and your project spans multiple files,
you can get rid of the requirements.txt file and instead, add all packages
contained in requirements.txt to the install_requires field of the setup call.
'''
REQUIRED_PACKAGES = [
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.