Skip to content

Instantly share code, notes, and snippets.

View aarsanjani's full-sized avatar
💭
I may be slow to respond.

Ali Arsanjani aarsanjani

💭
I may be slow to respond.
View GitHub Profile
@aarsanjani
aarsanjani / s3_file_combine.py
Created December 8, 2019 19:14 — forked from bwicklund/s3_file_combine.py
S3 file Concatenation/Combination. S3 Spark file merge.
import argparse
import boto3
import os
import threading
from fnmatch import fnmatch
# S3 multi-part upload parts must be larger than 5mb
MIN_S3_SIZE = 6000000
LOG_LEVEL = 'INFO'