Skip to content

Instantly share code, notes, and snippets.

@doooby
Last active August 29, 2015 14:13
Show Gist options
  • Save doooby/2cb24eb2ea1180755c3c to your computer and use it in GitHub Desktop.
Save doooby/2cb24eb2ea1180755c3c to your computer and use it in GitHub Desktop.
Direct (POST) uploading to AWS S3 using aws-sdk gem

Direct uploading to AWS S3

Since there are new regions of Amazon S3 services that accepts only Signature v4 (e.g. frankfurt) you cannot use aws-sdk-v1 gem's #presigned_url functionality. And newer aws-sdk gem (beeing stil in preview for 2.x) doesn't have a functionality to do direct uploading either. Here's my inplementation of that functionality, build on top of aws-sdk v2.0.16.pre and jquery (it's using some javascript stuff that older browsers won't support though).

For some basic information see this Heroku's Direct to S3 Image Uploads in Rails article https://devcenter.heroku.com/articles/direct-to-s3-image-uploads-in-rails . A "how to" by Amazaon itself in Authenticating Requests in Browser-Based Uploads Using POST (AWS Signature Version 4) http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-UsingHTTPPOST.html (notice beelow are subsections of details, like how to create signing policy).

An exapmle for simultaneous multiple images browser-based uploading

So, I build it the way, that only things that server needs to know after files has been uploaded to s3, are that in what 'folder' it was saved and the count of them (given that their keys are 0000, 0001 ... 9999 complementing with the folder the full path in the bucket). In this example I wanted from user additionaly only a caption of the image gallery that's being uploaded. This is the form needed:

<div class="new_gallery">
  <input type="file" name="files_input" id="files_input" multiple="multiple">
  <form id="save_gallery_form" action="/images/add_photo_gallery" accept-charset="UTF-8" method="post">
    <input type="hidden" name="images_count" id="images_count">
    <input type="hidden" name="folder" id="folder">
    <input type="text" name="label" id="label" placeholder="caption of gallery">
  </form>  
  <a id="button_upload" href="#">Start uploading</a>
  <br>
  <div id="files_list">
    <!-- this is what each file will produce after selecting something -->
    <!--<div class="file_field"><span class="title">IMG_4780.JPG</span><span class="key">0000</span></div>-->
    <!--<div class="file_field"><span class="title">IMG_4781.JPG</span><span class="key">0001</span></div>-->
  </div>
</div>

also add this js on the page: (fit these events to your needs)

$(function(){
    var post_data = <%= Aws::FotocubeS3.post_data_for_images.to_json.html_safe %>; // a rails .erb way to hand over that data
    DirectToS3uploader(post_data, {
      start: function () {
        writeStatus('Uploading files ...');
      },
      upload_fail: function () {
        writeStatus('Uploading failed!');
      },
      to_save: function (data) {
        writeStatus('Saving gallery ...');
        $('#folder').val(data['folder']);
        $('#images_count').val(data['files'].length);
        $('#save_gallery_form').submit();
      },
      preconditions: function () {
        if ($('#label').val()) return true;
        writeStatus('Caption of gallery must be filled.');
      }
    });
  });

Implementation explanation

I'll skip that javascript part, there's nothing important to explain that the previous example did not covered. It's just using one AJAX call and displays very simple progress bar per file, while POST-ing it to your S3 bucket.

The most important thing is the policy, that AWS requires. The DirectPostHelper class will generate the basics for you using #generate_policy (producing a DirectPostHelper::PostPolicy instance). You can specify duration of this policy - if no argument given, it will set up expiration time for 2.hours. You only need add your specific restrictions; follow above mentioned Amazon docs to check out what there is to restrict. Afterwards the method #generate_params on your DirectPostHelper instance produces form params that are essential to make a valid POST request to S3.

Now, Given that MY_PROJECT_BUCKET is a Aws::S3::Bucket class that is given to DirectPostHelper's constructor, this is how I create policy in the example - the custom Aws::MyProjectS3#post_data_for_images method:

post_helper = DirectPostHelper.new MY_PROJECT_BUCKET
# base of the key under which the file will be saved in S3 - aka folder in this case
folder = "images/#{Time.zone.now.utc.strftime '%Y%m%d%H%M%S'}/"
# additional custom params sent to S3 
# 'acl' => 'public-read' means everybody can see that image
custom_params = {'content-type' => 'image/jpeg', 'acl' => 'public-read'}

policy = post_helper.generate_policy # hand over expiration time (by default it's 2.hours)
# the policy must include every additional parameter (here using exact match)
custom_params.each_pair{|k, v| policy.exact_condition k, v}
# this is how to set that the policy acceps only key that begins with given folder
policy.fit_in_condition 'starts-with', '$key', folder
# this is how to set up the range of image size
policy.fit_in_condition 'content-length-range', 1, 10485760

{ # this is what is returned and what my DirectToS3uploader javascript needs
    url: bucket.url,
    folder: folder,
    params: post_helper.generate_params(policy, custom_params)
}
function DirectToS3uploader (s3_data, events) {
var files_input = $('#files_input');
var button_upload = $('#button_upload');
var files_list = $('#files_list');
if (!files_input[0]) throw "Files input not present on the page!";
if (!button_upload[0]) throw "Upload button not present on the page!";
if (!files_list[0]) throw "Files list div not present on the page!";
var files = [];
function each_file (clbk) {
var i;
for (i=0; i<files.length; i+=1) clbk(files[i]);
}
function lock() {
files_input.attr('disabled', true);
files_input.off('change', on_files_selected);
button_upload.off('click', on_form_submit);
}
function unlock() {
files_input.attr('disabled', false);
files_input.on('change', on_files_selected);
button_upload.on('click', on_form_submit);
}
function fire_event(action, data) {
if (typeof events[action]==='function') return events[action](data);
}
function on_files_selected() {
var files_to_upload, file, i;
files_to_upload = files_input[0].files;
files.length = 0;
files_list.html('');
for (i=0; i<files_to_upload.length; i+=1){
file = FileToUpload(('0000'+i).substr(-4,4), files_to_upload[i]);
files.push(file);
file.addToList();
}
}
function on_form_submit() {
if (files.length===0) return;
if (events['preconditions'] && !fire_event('preconditions')) return;
lock();
fire_event('start');
var done_count = 0;
function one_completed(file) {
if (!file.uploaded) {
each_file(function (file) { file.abort(); });
unlock();
fire_event('upload_fail');
}
else {
done_count += 1;
if (done_count==files.length) {
fire_event('to_save', {folder: s3_data['folder'], files: files});
unlock();
}
}
}
each_file(function (file) { file.upload(one_completed); });
}
function FileToUpload(key, file){
var file_klass, div_tag, xhr;
div_tag = $('<div class="file_field"><span class="title">'+file.name+
'</span><span class="key">'+key+'</span></div>');
file_klass = {
key: key,
uploaded: false,
addToList: function () {
files_list.append(div_tag);
},
upload: function (on_complete) {
var form_data, progress_bar, progress, p;
if (file_klass.uploaded) return;
form_data = new FormData();
for (p in s3_data['params']) {
if (s3_data['params'].hasOwnProperty(p)) {
form_data.append(p, s3_data['params'][p]);
}
}
form_data.append('key', s3_data['folder']+key);
form_data.append('file', file);
progress_bar = $('<span class="progress_bar"><span class="loading" style="width: 0">'+
'&nbsp;</span></span>');
progress = progress_bar.children();
div_tag.append(progress_bar);
xhr = $.ajax({
url: s3_data['url'],
type: 'POST',
data: form_data,
dataType: 'XML',
processData: false,
contentType: false,
cache: false,
error: function (data) {
progress.html(data.statusText);
progress.css('width', '100%');
progress.removeClass('loading');
progress.addClass('fail');
console.log([file_klass, data]);
},
success: function () {
progress.html('done');
progress.removeClass('loading');
progress.addClass('done');
file_klass.uploaded = true;
},
xhr: function() {
var myXhr = $.ajaxSettings.xhr(), progress_int;
if(myXhr.upload){
myXhr.upload.addEventListener('progress',function (data) {
progress_int = parseInt(data.loaded / data.total * 100);
progress.css('width', progress_int+'%');
}, false);
}
return myXhr;
},
complete: function () {
on_complete(file_klass);
}
});
},
abort: function () {
if (xhr) xhr.abort();
}
};
return file_klass;
}
unlock();
}
module Aws
class MyProjectS3
def self.post_data_for_images
post_helper = DirectPostHelper.new MY_PROJECT_BUCKET
folder = "images/#{Time.zone.now.utc.strftime '%Y%m%d%H%M%S'}/"
custom_params = {'content-type' => 'image/jpeg', 'acl' => 'public-read'}
policy = post_helper.generate_policy
custom_params.each_pair{|k, v| policy.exact_condition k, v}
policy.fit_in_condition 'starts-with', '$key', folder
policy.fit_in_condition 'content-length-range', 1, 10485760
{
url: bucket.url,
folder: folder,
params: post_helper.generate_params(policy, custom_params)
}
end
class DirectPostHelper
def initialize(bucket)
@bucket = bucket
@time_iso8601 = Time.zone.now.utc.strftime '%Y%m%dT%H%M%SZ'
end
def generate_policy(expire_in=2.hours)
policy = PostPolicy.new expire_in
policy.exact_condition 'x-amz-algorithm', 'AWS4-HMAC-SHA256'
policy.exact_condition 'x-amz-date', @time_iso8601
policy.exact_condition 'x-amz-credential', credential
policy.exact_condition 'bucket', @bucket.name
policy
end
def generate_params(policy, params={})
params.merge! 'x-amz-signature' => signature(policy),
'policy' => policy.encoded,
'x-amz-algorithm' => 'AWS4-HMAC-SHA256',
'x-amz-date' => @time_iso8601,
'x-amz-credential' => credential
end
private
def credential
[
@bucket.client.config.credentials.access_key_id,
@time_iso8601[0,8],
@bucket.client.config.region,
's3',
'aws4_request'
] * '/'
end
def signature(policy)
k_date = hmac("AWS4#{@bucket.client.config.credentials.secret_access_key}", @time_iso8601[0,8])
k_region = hmac(k_date, @bucket.client.config.region)
k_service = hmac(k_region, 's3')
k_credentials = hmac(k_service, 'aws4_request')
hexhmac(k_credentials, policy.encoded)
end
def hmac(key, value)
OpenSSL::HMAC.digest(OpenSSL::Digest.new('sha256'), key, value)
end
def hexhmac(key, value)
OpenSSL::HMAC.hexdigest(OpenSSL::Digest.new('sha256'), key, value)
end
class PostPolicy
def initialize(expire_in)
@expire_time = Time.zone.now.utc + expire_in
@conditions = []
end
def exact_condition(value_name, arg)
@conditions << {value_name => arg}
end
def fit_in_condition(value_name, arg1, arg2)
@conditions << [value_name, arg1, arg2]
end
def encoded
hash = {
'expiration' => @expire_time.strftime('%Y-%m-%dT%H:%M:%S.%LZ'),
'conditions' => @conditions
}
Base64.encode64(hash.to_json).gsub("\n","")
end
end
end
end
end
#files_list
.file_field
span
display: inline-block
.title
width: 13em
.key
width: 4em
.progress_bar
width: 15em
border: 1px outset gray
.loading
background-color: #aac4ff
.done
background-color: #b1ffa2
text-align: center
.fail
background-color: #ff9185
text-align: center
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment