Uploading Images to Amazon S3 using Paperclip Gem

amazon_paperclip

-- Workflow - from http://blog.littleblimp.com/post/53942611764/direct-uploads-to-s3-with-rails-paperclip-and

Our app will work as follows:

User uploads their file directly to a temporary directory on S3
A form callback posts the temporary file URL to our app
Our app creates a new Document object, sets some initial data from the temporary S3 file, then queues a background process to move the temporary file to the location that Paperclip expects it to be and to process thumbnails if required
Show users a message if they visit a file page while its still being processed


1. Create a bucket on S3. It is best to create separate buckets for all environments (development, production)
2. Properties tab of bucket - click “Edit CORS configuration” and paste
      <CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
          <CORSRule>
              <AllowedOrigin>*</AllowedOrigin>
              <AllowedMethod>POST</AllowedMethod>
              <MaxAgeSeconds>3000</MaxAgeSeconds>
              <AllowedHeader>*</AllowedHeader>
          </CORSRule>
      </CORSConfiguration>
3. Create a new file -> config/aws.yml:
    defaults: &defaults
      access_key_id: "devkey"
      secret_access_key: "devsecret"
    development:
      <<: *defaults
      bucket: "myapp-development"
    test:
      <<: *defaults
      bucket: "myapp-test"
    production:
      access_key_id: <%=ENV["AWS_ACCESS_KEY_ID"]%>
      secret_access_key: <%=ENV["AWS_SECRET_ACCESS_KEY"]%>
      bucket: "myapp"
4. Create a new file -> config/initializers/aws.rb:
  require 'aws-sdk'
  # Rails.configuration.aws is used by AWS, Paperclip, and S3DirectUpload
  Rails.configuration.aws = YAML.load(ERB.new(File.read("#{Rails.root}/config/aws.yml")).result)[Rails.env].symbolize_keys!
  AWS.config(logger: Rails.logger)
  AWS.config(Rails.configuration.aws)
   
5. Create a new file -> config/initializers/paperclip.rb
  Paperclip::Attachment.default_options.merge!(
    url:                  ':s3_domain_url',
    path:                 ':class/:attachment/:id/:style/:filename',
    storage:              :s3,
    s3_credentials:       Rails.configuration.aws,
    s3_permissions:       :private,
    s3_protocol:          'https'
  )
   
6. Create a new file -> config/initializers/s3_direct_upload.rb
  S3DirectUpload.config do |c|
    c.access_key_id     = Rails.configuration.aws[:access_key_id]
    c.secret_access_key = Rails.configuration.aws[:secret_access_key]
    c.bucket            = Rails.configuration.aws[:bucket]
    c.region            = "s3"
  end

7. application.js -> Add:
  //= require s3_direct_upload

8. Create a new file -> assets/javascripts/documents.js:
  $(function() {
    $('#s3_uploader').S3Uploader(
      { 
        remove_completed_progress_bar: false,
        progress_bar_target: $('#uploads_container')
      }
    );
    $('#s3_uploader').bind('s3_upload_failed', function(e, content) {
      return alert(content.filename + ' failed to upload');
    });
  });

9. Create a new file -> views/documents/index.html.erb:
  <%= s3_uploader_form callback_url: documents_url,
    id: "s3_uploader",
  	callback_param: "document[direct_upload_url]",
  	expiration: 24.hours.from_now.utc.iso8601,
  	max_file_size: 100.megabytes do %>
  	<%= file_field_tag :file, multiple: true %>
  <% end %>
  <div id="uploads_container"></div>
  <script id="template-upload" type="text/x-tmpl">
  <div id="upload_{%=o.unique_id%}" class="upload">
    <h5>{%=o.name%}</h5>
    <div class="progress progress-striped active"><div class="bar" style="width: 0%"></div></div>
  </div>
  </script>

10. Create a new file -> views/documents/create.js.erb
  <% if @document.persisted? %>
    $('#upload_<%=params[:unique_id]%>').hide();
  <% else %>
    $('#upload_<%=params[:unique_id]%> div.progress').removeClass('active progress-striped').addClass('progress-danger');
  <% end %>

11. rails g model document ... ->
  class CreateDocuments < ActiveRecord::Migration
    def change
      create_table :documents do |t|
        t.integer :user_id, null: false
        t.string :direct_upload_url, null: false
        t.attachment :upload
        t.boolean :processed, default: false, null: false
        t.timestamps
      end
      add_index :documents, :user_id
      add_index :documents, :processed
    end
  end

12. models/document.rb:
  class Document < ActiveRecord::Base
 
    # Environment-specific direct upload url verifier screens for malicious posted upload locations.
    DIRECT_UPLOAD_URL_FORMAT = %r{\Ahttps:\/\/s3\.amazonaws\.com\/myapp#{!Rails.env.production? ? "\\-#{Rails.env}" : ''}\/(?<path>uploads\/.+\/(?<filename>.+))\z}.freeze
    
    belongs_to :user
    has_attached_file :upload
   
    validates :direct_upload_url, presence: true, format: { with: DIRECT_UPLOAD_URL_FORMAT }
      
    before_create :set_upload_attributes
    after_create :queue_processing
    
    attr_accessible :direct_upload_url
    
    # Store an unescaped version of the escaped URL that Amazon returns from direct upload.
    def direct_upload_url=(escaped_url)
      write_attribute(:direct_upload_url, (CGI.unescape(escaped_url) rescue nil))
    end
    
    # Determines if file requires post-processing (image resizing, etc)
    def post_process_required?
      %r{^(image|(x-)?application)/(bmp|gif|jpeg|jpg|pjpeg|png|x-png)$}.match(upload_content_type).present?
    end
    
    # Final upload processing step
    def self.transfer_and_cleanup(id)
      document = Document.find(id)
      direct_upload_url_data = DIRECT_UPLOAD_URL_FORMAT.match(document.direct_upload_url)
      s3 = AWS::S3.new
      
      if document.post_process_required?
        document.upload = URI.parse(URI.escape(document.direct_upload_url))
      else
        paperclip_file_path = "documents/uploads/#{id}/original/#{direct_upload_url_data[:filename]}"
        s3.buckets[Rails.configuration.aws[:bucket]].objects[paperclip_file_path].copy_from(direct_upload_url_data[:path])
      end
   
      document.processed = true
      document.save
      
      s3.buckets[Rails.configuration.aws[:bucket]].objects[direct_upload_url_data[:path]].delete
    end
        
    protected
    
    # Set attachment attributes from the direct upload
    # @note Retry logic handles S3 "eventual consistency" lag.
    def set_upload_attributes
      tries ||= 5
      direct_upload_url_data = DIRECT_UPLOAD_URL_FORMAT.match(direct_upload_url)
      s3 = AWS::S3.new
      direct_upload_head = s3.buckets[Rails.configuration.aws[:bucket]].objects[direct_upload_url_data[:path]].head
   
      self.upload_file_name     = direct_upload_url_data[:filename]
      self.upload_file_size     = direct_upload_head.content_length
      self.upload_content_type  = direct_upload_head.content_type
      self.upload_updated_at    = direct_upload_head.last_modified
    rescue AWS::S3::Errors::NoSuchKey => e
      tries -= 1
      if tries > 0
        sleep(3)
        retry
      else
        false
      end
    end
    
    # Queue file processing
    def queue_processing
      Document.delay.transfer_and_cleanup(id)
    end
   
  end

Thanks @kwaters12. This is very helpful. Do you have any tips on what to do once transfer_and_cleanup finishes? Can that loop back to the controller and fire an event? Or change some page state variable? Right now the image gets uploaded, the page refreshes and then image processing/resizing happens silently some several seconds later with no user feedback.