Skip to content

Instantly share code, notes, and snippets.

@dyao-sovrn
dyao-sovrn / combineS3Files.py
Created January 13, 2018 18:25 — forked from jasonrdsouza/combineS3Files.py
Python script to efficiently concatenate S3 files
'''
This script performs efficient concatenation of files stored in S3. Given a
folder, output location, and optional suffix, all files with the given suffix
will be concatenated into one file stored in the output location.
Concatenation is performed within S3 when possible, falling back to local
operations when necessary.
Run `python combineS3Files.py -h` for more info.
'''
@dyao-sovrn
dyao-sovrn / combineS3Files.py
Created January 13, 2018 18:25 — forked from jasonrdsouza/combineS3Files.py
Python script to efficiently concatenate S3 files
'''
This script performs efficient concatenation of files stored in S3. Given a
folder, output location, and optional suffix, all files with the given suffix
will be concatenated into one file stored in the output location.
Concatenation is performed within S3 when possible, falling back to local
operations when necessary.
Run `python combineS3Files.py -h` for more info.
'''