Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
S3 File Resource for Chef
# Source accepts the protocol s3:// with the host as the bucket
# access_key_id and secret_access_key are just that
s3_file "/var/bulk/the_file.tar.gz" do
source "s3://your.bucket/the_file.tar.gz"
access_key_id your_key
secret_access_key your_secret
owner "root"
group "root"
mode 0644
end
#
# Author:: Christopher Peplin (<peplin@bueda.com>)
# Copyright:: Copyright (c) 2010 Bueda, Inc.
# License:: Apache License, Version 2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
class Chef
class Provider
class S3File < Chef::Provider::RemoteFile
def action_create
Chef::Log.debug("Checking #{@new_resource} for changes")
if current_resource_matches_target_checksum?
Chef::Log.debug("File #{@new_resource} checksum matches target checksum (#{@new_resource.checksum}), not updating")
else
Chef::Log.debug("File #{@current_resource} checksum didn't match target checksum (#{@new_resource.checksum}), updating")
fetch_from_s3(@new_resource.source) do |raw_file|
if matches_current_checksum?(raw_file)
Chef::Log.debug "#{@new_resource}: Target and Source checksums are the same, taking no action"
else
backup_new_resource
Chef::Log.debug "copying remote file from origin #{raw_file.path} to destination #{@new_resource.path}"
FileUtils.cp raw_file.path, @new_resource.path
@new_resource.updated = true
end
end
end
enforce_ownership_and_permissions
@new_resource.updated
end
def fetch_from_s3(source)
begin
protocol, bucket, name = URI.split(source).compact
name = name[1..-1]
AWS::S3::Base.establish_connection!(
:access_key_id => @new_resource.access_key_id,
:secret_access_key => @new_resource.secret_access_key
)
obj = AWS::S3::S3Object.find name, bucket
Chef::Log.debug("Downloading #{name} from S3 bucket #{bucket}")
file = Tempfile.new("chef-s3-file")
file.write obj.value
Chef::Log.debug("File #{name} is #{file.size} bytes on disk")
begin
yield file
ensure
file.close
end
rescue URI::InvalidURIError
Chef::Log.warn("Expected an S3 URL but found #{source}")
nil
end
end
end
end
end
class Chef
class Resource
class S3File < Chef::Resource::RemoteFile
def initialize(name, run_context=nil)
super
@resource_name = :s3_file
end
def provider
Chef::Provider::S3File
end
def access_key_id(args=nil)
set_or_return(
:access_key_id,
args,
:kind_of => String
)
end
def secret_access_key(args=nil)
set_or_return(
:secret_access_key,
args,
:kind_of => String
)
end
end
end
end
@cparedes

This comment has been minimized.

Copy link

@cparedes cparedes commented Jan 12, 2011

I've ran into an issue with the library, mostly with URI.parse - if the bucket name does not contain a dot in the FQDN portion of the URI, then URI.parse throws a URI::InvalidURIError exception, even if having an FQDN without a dot in S3 is completely valid.

I've used the following bits of code for uri, name, and bucket:

uri = URI.split(source).compact
name = uri[2][1..-1]
bucket = uri[1]

@peplin

This comment has been minimized.

Copy link
Owner Author

@peplin peplin commented Jan 12, 2011

Thanks for the report - I'm happy to see someone else finds this useful. I'm not able to reproduce the bug you describe - is this an example of what returns a URI::InvalidURIError for you?

>> URI.parse("s3://bueda/test.zip").path[1..-1]
=> "test.zip"
@cparedes

This comment has been minimized.

Copy link

@cparedes cparedes commented Jan 12, 2011

Hey there,

Try the following:

>> URI.parse("s3://bueda_foobar/test.zip")
URI::InvalidURIError: the scheme s3 does not accept registry part: bueda_foobar (or bad hostname?)
    from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/uri/generic.rb:195:in `initialize'
    from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/uri/common.rb:492:in `new'
    from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/uri/common.rb:492:in `parse'
    from (irb):4

Seems it's underscores that are causing issues...

@cparedes

This comment has been minimized.

Copy link

@cparedes cparedes commented Jan 12, 2011

Also, wanted to say that it was indeed very helpful! Thanks so much for coding this up, Christopher!

@peplin

This comment has been minimized.

Copy link
Owner Author

@peplin peplin commented Jan 12, 2011

Thanks - I guess using URI is a bit of a stretch since bucket names don't have to be valid FQDNs. I've updated this to use the URI.split() method as you do.

@jwarchol

This comment has been minimized.

Copy link

@jwarchol jwarchol commented Aug 16, 2011

How do you actually use this? Sorry, I'm new to Chef and trying to understand this as I build up my stack. Would these files go into a cookbook of their own? Could you give a quick example of how you're using them?

@peplin

This comment has been minimized.

Copy link
Owner Author

@peplin peplin commented Aug 16, 2011

Sure, put s3_file.rb in the libraries/ folder of any cookbook (create it if it doesn't exist) and it should be automatically imported.

Alternatively, make a standalone s3 cookbook with the file in s3/libraries/ and in other cookbooks, just call include_recipe "s3" before using it.

@casualjim

This comment has been minimized.

Copy link

@casualjim casualjim commented Oct 5, 2011

hi I added support for all regions and removed the need for the aws-s3 gem

https://gist.github.com/1264036

@poobury

This comment has been minimized.

Copy link

@poobury poobury commented Jan 9, 2012

Hi,

I've added to the excellent work of peplin and casualjim (thanks), to add checking for XML error responses from Amazon and to add dual support for S3 and other protocols provided by the out-of-the-box RemoteFile (e.g. http(s)).

https://gist.github.com/1351043

Thanks for the work, folks!

@tlfbrito

This comment has been minimized.

Copy link

@tlfbrito tlfbrito commented Mar 28, 2013

I'm getting an NameError: uninitialized constant Chef::Provider::S3File::AWS.
What I'm doing wrong?

@DavidAllison

This comment has been minimized.

Copy link

@DavidAllison DavidAllison commented Apr 1, 2013

@TiagoBrito @poobury @casualjim @peplin I've created an update to @peplin's original gist using AWS-SDK at https://gist.github.com/DavidAllison/5288249

It includes minor changes that make it work with the newer AWS-SDK gem (the current version as of this writing). Peplin's older gem won't work with the newer AWS-SDK gem since the API is no longer the same.

@poobury and @casualjim, I really like where you guys went with this, but unfortunately I wasn't able to easily make either or your recipes work with AWS's temporary S3 credential granting based on IAM policies (whereas the AWS-SDK gem from Amazon handles this quite nicely). So I ended up modernizing @peplin's original work.

@gitfool

This comment has been minimized.

Copy link

@gitfool gitfool commented Dec 2, 2013

These all have the same problem - to be generally useful, the temporary file needs to be written in binary mode! e.g. for the above:

file = Tempfile.new("chef-s3-file")
file.binmode
file.write obj.value
@pl1ght

This comment has been minimized.

Copy link

@pl1ght pl1ght commented Feb 21, 2014

Getting a "no resource or method named s3_file for Chef::recipe "default" when we try to use this. ANy ideas?

@kszarek

This comment has been minimized.

Copy link

@kszarek kszarek commented Mar 4, 2014

Have you tried to add include_recipe "s3_file" into your cookbook recipe?

@mxshrestha

This comment has been minimized.

Copy link

@mxshrestha mxshrestha commented Mar 21, 2014

i get "no resource or method named s3_file for Chef::recipe "default" no matter what i try

@karthicktkm

This comment has been minimized.

Copy link

@karthicktkm karthicktkm commented Aug 19, 2015

Hi,

I want how to write (or) the S3 file.

Thanks
Karthi

@slowenthal

This comment has been minimized.

Copy link

@slowenthal slowenthal commented Feb 21, 2016

Can this resource copy a directory of files to a local directory as well as just a single file?

@raylobosco

This comment has been minimized.

Copy link

@raylobosco raylobosco commented Aug 25, 2016

Hello. I am getting the following... any help appreciated:

19: class Chef
20: class Provider
21>> class S3File < Chef::Provider::RemoteFile
22: def action_create
23: Chef::Log.debug("Checking #{@new_resource} for changes")
24:
25: if current_resource_matches_target_checksum?
26: Chef::Log.debug("File #{@new_resource} checksum matches target checksum (#{@new_resource.checksum}), not updating")
27: else
28: Chef::Log.debug("File #{@current_resource} checksum didn't match target checksum (#{@new_resource.checksum}), updating")
29: fetch_from_s3(@new_resource.source) do |raw_file|
30: if matches_current_checksum?(raw_file)

[2016-08-25T09:50:43-04:00] ERROR: Running exception handlers
[2016-08-25T09:50:43-04:00] ERROR: Exception handlers complete
[2016-08-25T09:50:43-04:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated
[2016-08-25T09:50:43-04:00] ERROR: uninitialized constant #Class:0x00000003144990::Chef::Provider::RemoteFile
[2016-08-25T09:50:43-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

@lkjangir

This comment has been minimized.

Copy link

@lkjangir lkjangir commented Jul 11, 2017

Hi,
this doesn't work for me when I don't pass access keys, actually the instance already has an IAM role assigned and has enough permission on bucket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment