Skip to content

Instantly share code, notes, and snippets.

@baldowl
Created February 18, 2011 07:21
Show Gist options
  • Save baldowl/833374 to your computer and use it in GitHub Desktop.
Save baldowl/833374 to your computer and use it in GitHub Desktop.
Bad script to play with S3's multipart uploads and Fog
#!/usr/bin/env ruby
require 'rubygems'
require 'fog'
bucket_name = 'my_bucket'
object_name = 'big.zip'
# Part size must be between 5MB and 5GB.
# (http://docs.amazonwebservices.com/AmazonS3/latest/dev/qfacts.html)
# Let's say we've splitted the original file with split(1).
parts = Dir.glob('big.zip.*').sort
storage = Fog::Storage.new(:provider => 'AWS')
puts 'Pending multipart uploads'
response = storage.list_multipart_uploads bucket_name
puts response.inspect, "\n\n"
puts 'Initiating multipart uploads'
response = storage.initiate_multipart_upload bucket_name, object_name
upload_id = response.body['UploadId']
puts "Upload ID: #{upload_id}"
# We could retrive the list of already uploaded parts via list_parts(), but
# it's better to do our homeworks and skip a request to AWS.
part_ids = []
parts.each_with_index do |part, position|
part_number = (position + 1).to_s
puts "Uploading #{part}"
File.open part do |part_file|
response = storage.upload_part bucket_name, object_name, upload_id,
part_number, part_file
part_ids << response.headers['ETag']
end
end
puts "Parts' ETags: #{part_ids.inspect}", "\n\n"
puts 'Pending multipart uploads'
response = storage.list_multipart_uploads bucket_name
puts response.inspect, "\n\n"
puts 'Completing multipart upload'
response = storage.complete_multipart_upload bucket_name, object_name,
upload_id, part_ids
puts response.inspect, "\n\n"
puts 'Pending multipart uploads'
response = storage.list_multipart_uploads bucket_name
puts response.inspect, "\n\n"
puts 'Checking the uploaded object'
response = storage.directories.get(bucket_name).files.get(object_name)
puts response.inspect, "\n\n"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment