Skip to content

Instantly share code, notes, and snippets.

@mziwisky
Created February 28, 2017 01:02
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mziwisky/4e9e71c1bdb6c70145acc785057aa545 to your computer and use it in GitHub Desktop.
Save mziwisky/4e9e71c1bdb6c70145acc785057aa545 to your computer and use it in GitHub Desktop.
aws-sdk Aws::S3::Object#copy_from vs #copy_to
irb(main):001:0> require_relative 'test'
AWS SDK version: 2.7.16
== TEST: test_copy_west_from_east
IT'S GOOD
== TEST: test_copy_east_from_west
IT'S GOOD
# ^ copy_from works fine regardless of direction
== TEST: test_copy_east_to_west
S3 client configured for "us-east-1" but the bucket "mziwisky-west-test" is in "us-west-2"; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts
IT'S GOOD
# ^ that first copy_to from us-east-1 to us-west-2 spit out a warning, but still worked by following the redirect
== TEST: test_copy_east_to_west
IT'S GOOD
# ^ running the copy_to from us-east-1 to us-west-2 a second time works w/o warning due to the bucket-to-region cache that the gem builds internally
== TEST: test_copy_west_to_east
Aws::S3::Errors::PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-core-2.7.16/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-core-2.7.16/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-core-2.7.16/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-core-2.7.16/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-core-2.7.16/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-core-2.7.16/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-core-2.7.16/lib/seahorse/client/plugins/response_target.rb:21:in `call'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-core-2.7.16/lib/seahorse/client/request.rb:70:in `send_request'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-core-2.7.16/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-resources-2.7.16/lib/aws-sdk-resources/services/s3/object_copier.rb:33:in `copy_object'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-resources-2.7.16/lib/aws-sdk-resources/services/s3/object_copier.rb:19:in `copy_to'
from /Users/mziwisky/.gem/ruby/2.1.7/gems/aws-sdk-resources-2.7.16/lib/aws-sdk-resources/services/s3/object.rb:103:in `copy_to'
from /Users/mziwisky/test.rb:63:in `test_copy_west_to_east'
from /Users/mziwisky/test.rb:88:in `block in <top (required)>'
from /Users/mziwisky/test.rb:86:in `each'
from /Users/mziwisky/test.rb:86:in `<top (required)>'
from (irb):1:in `require_relative'
from (irb):1
from /Users/mziwisky/.rubies/ruby-2.1.7/bin/irb:11:in `<main>'
# ^ copy_to from us_west_2 to us_east_1 explodes
require 'aws-sdk'
def get_buckets
# clients get creds from env
east_client = Aws::S3::Client.new(region: 'us-east-1')
west_client = Aws::S3::Client.new(region: 'us-west-2')
east_res = Aws::S3::Resource.new(client: east_client)
west_res = Aws::S3::Resource.new(client: west_client)
east_bucket = east_res.bucket('mziwisky-east-test')
west_bucket = west_res.bucket('mziwisky-west-test')
east_bucket.create rescue Aws::S3::Errors::BucketAlreadyOwnedByYou
west_bucket.create rescue Aws::S3::Errors::BucketAlreadyOwnedByYou
[east_bucket, west_bucket]
end
def setup
east_bucket, west_bucket = get_buckets
east_object = east_bucket.object('east-thing')
west_object = west_bucket.object('west-thing')
[east_object, west_object]
end
def populate(object)
object.put(body: 'foobar')
end
def cleanup
get_buckets.each do |bucket|
bucket.delete! rescue Aws::S3::Errors::NoSuchBucket
end
end
def assert_equal(o1, o2)
good = o1.get.body.string == o2.get.body.string
puts "IT'S #{good ? 'GOOD' : 'A FAILURE'}"
end
def test_copy_east_to_west
east_object, west_object = setup
populate(east_object)
east_object.copy_to(west_object)
assert_equal(east_object, west_object)
ensure
cleanup
end
def test_copy_west_from_east
east_object, west_object = setup
populate(east_object)
west_object.copy_from(east_object)
assert_equal(east_object, west_object)
ensure
cleanup
end
def test_copy_west_to_east
east_object, west_object = setup
populate(west_object)
west_object.copy_to(east_object)
assert_equal(east_object, west_object)
ensure
cleanup
end
def test_copy_east_from_west
east_object, west_object = setup
populate(west_object)
east_object.copy_from(west_object)
assert_equal(east_object, west_object)
ensure
cleanup
end
puts "AWS SDK version: #{Aws::VERSION}"
%i[
test_copy_west_from_east
test_copy_east_from_west
test_copy_east_to_west
test_copy_east_to_west
test_copy_west_to_east
].each do |test|
puts "== TEST: #{test}"
send test
puts
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment