This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import logging | |
import math | |
import mimetypes | |
from multiprocessing import Pool | |
import os | |
from boto.s3.connection import S3Connection | |
from filechunkio import FileChunkIO | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
"""Split large file into multiple pieces for upload to S3. | |
S3 only supports 5Gb files for uploading directly, so for larger CloudBioLinux | |
box images we need to use boto's multipart file support. | |
This parallelizes the task over available cores using multiprocessing. | |
Usage: | |
s3_multipart_upload.py <file_to_transfer> <bucket_name> [<s3_key_name>] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wget http://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz | |
tar -zxvf inotify-tools-3.14.tar.gz | |
cd inotify-tools-3.14 | |
./configure | |
make | |
make install | |
updatedb |