Skip to content

Instantly share code, notes, and snippets.

@anteverse
Last active October 31, 2021 09:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save anteverse/100c4599698e3d1e18f2a717ad62021e to your computer and use it in GitHub Desktop.
Save anteverse/100c4599698e3d1e18f2a717ad62021e to your computer and use it in GitHub Desktop.
Put uploads in threads
zip_obj = s3_resource.Object("path/to/key.zip")
buffer = BytesIO(zip_obj.get()["Body"].read())
with ZipFile(buffer) as zipfile:
with TransferManager(
config=transfer_config,
client=boto3.client("s3"),
) as transfer_manager:
def _upload(file_name: str):
dest_key = build_destination_key(file_name)
transfer_manager.upload(
fileobj=zipfile.open(file_name),
key=dest_key,
bucket="my_bucket"
)
with ThreadPool(
processes=transfer_manager.config.max_submission_concurrency
) as pool:
pool.imap_unordered(
func=_upload,
iterable=zipfile.namelist(),
chunksize=chunk_size,
)
pool.join()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment