Skip to content

Instantly share code, notes, and snippets.

@cjhanks
Last active August 29, 2015 13:57
Show Gist options
  • Save cjhanks/9655566 to your computer and use it in GitHub Desktop.
Save cjhanks/9655566 to your computer and use it in GitHub Desktop.
mpu-copy
def safe_mp_copy(target_b, target_k, source_b, source_k):
"""
:target_b: boto.s3.Bucket
Destination bucket
:target_k: str
Destination object key
:source_b: boto.s3.Bucket
Source bucket
:source_k: str
Source object key
"""
Chunk = int(4.5 * (1024 ** 3))
mpu = target_b.initiate_multipart_upload(target_k)
max_size = connect_s3().get_bucket(source_b).lookup(source_k).size
size = 0
for index in itertools.count(start = 1):
step = (max_size - size) if (size + Chunk > max_size) else Chunk
mpu.copy_part_from_key(source_b
, source_k
, index
, start = size
, end = (size + step - 1))
size += step
if size == max_size:
break
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment