Massive file upload using JQueryUpload + Nginx + Rails + Sidekiq

Massive file upload using JQueryUpload + Nginx + Rails + Sidekiq.md

Massive file upload using JQueryUpload + Nginx + Rails + Sidekiq

The goal: Upload several files (virtually big files) to a Rails application without compromising the user experience.

Architecture

jQuery File Upload + Nginx + Rails (Carrierwave) + Sidekiq

1. jQuery File Upload (http://blueimp.github.io/jQuery-File-Upload/)

2. Nginx

3. Rails (Carrierwave)

Add an asset_upload_controller.rb with a base upload action:

class AssetsUploadController < ApplicationController     
  def upload
  end
end

Let's now modify routes.rb to route each request for assets_upload/upload to our controller:

post "assets_upload/upload" => 'assets_upload#upload'

Once the controller and the route have been created, we can create a factory that will instantiate a new asset processor given the content type.

module AssetProcessors
  class Factory

    SUPPORTED_CONTENT_TYPES = %w{ audio/mp3 video/mp4 }
    PROCESSORS = {
      'audio/mp3'  => AudioAssetProcessor,
      'video/mp4'  => VideoAssetProcessor
    }

    def self.supports?(content_type)
      SUPPORTED_CONTENT_TYPES.include?(content_type)
    end

    def self.create(content_type, file_name, tmp_path)
      processor = PROCESSORS.fetch(content_type, NullAssetProcessor)
      processor.new(file_name, tmp_path)
    end

  end
end

Where VideoAssetProcessor is:

module AssetProcessors
  class VideoAssetProcessor < AssetProcessor

    def process
      resource = File.open(src_path)
      video = VideoAsset.create(resource: resource)
    end

  end
end


class VideoUploader < CarrierWave::Uploader::Base
  
  storage :file

  def store_dir
    upload_dir_root = Rails.env.test? ? "test" : "uploads"
    "#{upload_dir_root}/#{model.class.to_s.underscore}/video/#{model.id}"
  end

  def extension_white_list
    %w( mp4 )
  end
end

4. Sidekiq

What is Sidekiq? Sidekiq is a background processing framework for Ruby (in alternative you can pick Delayed::Job or Resque). Sidekiq is easily to be integrated into your Rails application and was build with a particular aim at performances.

In order to integrate Sidekiq into our Rails application, we have to add it to our Gemfile:

gem 'sidekiq'

And install the new dependency:

bundle install

Once Sidekiq is integrated, we can start implementing the worker that will carry out the job in background. The first step to take in this sense consists in creating the directory workers in our Rails app directory. In such folder we will create the file upload_worker.rb with the actual implementation of the job.

class UploadWorker
  include Sidekiq::Worker

  def perform(content_type, file_name, tmp_path)
    processor = AssetProcessors::Factory.create(content_type, file_name, tmp_path)
    processor.execute
  end
end

Where AssetProcessors::Factory will instantiate a new asset processor according to file content type calling execute on it; so for example for a file with content type mp4 VideoUploader will be called.

Let's now update assets_upload_controller.rb so that it will use our Sidekiq worker.

class AssetsUploadController < ApplicationController

  def upload
    assets_params.each do |params|
      UploadWorker.perform_async(params[:content_type], params[:file_name], params[:tmp_path])
    end
  end

  private
 
  def assets_params
    params.require(:assets)
  end

end

Run Sidekiq:

bundle exec sidekiq