Skip to content

Instantly share code, notes, and snippets.

@hasnainv
hasnainv / s3_multipart_upload.py
Last active August 27, 2015 14:55 — forked from fabiant7t/s3_multipart_upload.py
Parallel S3 multipart upload with retries
import logging
import math
import mimetypes
from multiprocessing import Pool
import os
from boto.s3.connection import S3Connection
from filechunkio import FileChunkIO
@hasnainv
hasnainv / s3_multipart_upload.py
Last active November 14, 2016 18:06 — forked from chrishamant/s3_multipart_upload.py
Example of Parallelized Multipart upload using boto
#!/usr/bin/env python
"""Split large file into multiple pieces for upload to S3.
S3 only supports 5Gb files for uploading directly, so for larger CloudBioLinux
box images we need to use boto's multipart file support.
This parallelizes the task over available cores using multiprocessing.
Usage:
s3_multipart_upload.py <file_to_transfer> <bucket_name> [<s3_key_name>]
wget http://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz
tar -zxvf inotify-tools-3.14.tar.gz
cd inotify-tools-3.14
./configure
make
make install
updatedb