Skip to content

Instantly share code, notes, and snippets.

@lauralorenz
Last active September 10, 2021 08:23
Show Gist options
  • Save lauralorenz/bb60c1019f291811807b3f5b7b4281ce to your computer and use it in GitHub Desktop.
Save lauralorenz/bb60c1019f291811807b3f5b7b4281ce to your computer and use it in GitHub Desktop.
from prefect import Flow, task, Parameter
@task
def get_s3_keys(bucket: str):
# get a list_of_keys from your s3 bucket with boto3
return list_of_keys
@task
def process_s3_key(key: str):
# do something to each of the files
with Flow("process_bucket") as flow:
bucket = Parameter('bucket')
s3_keys = get_s3_keys(bucket)
process_s3_key.map(s3_keys)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment