Skip to content

Instantly share code, notes, and snippets.

@janko janko/
Last active Nov 18, 2018

What would you like to do?
Plans for the new derivate_endpoint Shrine plugin

Derivate Endpoint

Derivate blocks are executed in context of the uploader instance:

class ImageUploader < Shrine
  plugin :derivate_endpoint

  derivate :thumbnail do |file, context|
    width, height = context[:params][:size].split("x")
      .resize_to_limit!(with, height)

Mounts in the same way as download_endpoint:

class ImageUploader < Shrine
  plugin :derivate_endpoint, prefix: "images/derivates"
Rails.application.routes.draw do
  mount ImageUploader.derivate_endpoint => "images/derivates"

URL is generated with Shrine::UploadedFile#derivate_url, which returns prefix + base64-json-encoded original file data + additional parameters (see urlsafe_serialization plugin):

photo.image.derivate_url(:thumbnail, params: { size: "500x500" })
#=> "/images/derivates/thumbnail/<base64-json-encoded-data>"


Because more metadata to serialize equals longer URL, user opts in for metadata they will need during processing (by default no metadata is included):

plugin :derivate_endpoint, metadata: %w[mime_type]

derivate :thumbnail do |file, context|
  context[:uploaded_file].metadata #=> { "mime_type" => "image/jpeg" }
  # ...
photo.image.derivate_url(:thumbnail) # longer URL, because it also includes "mime_type" metadata

It's possible to set it per derivate:

plugin :derivate_endpoint, metadata: -> (name) do
  metadata  = %w[mime_type]
  metadata += %w[width height] if name == :thumbnail

derivate :thumbnail do |file, context|
  context[:uploaded_file].metadata #=> { "mime_type" => "image/jpeg", "width" => 150, "height" => "100" }
  # ...

Cache Busting

Derivates are intended to be cached indefinitely (Cache-Control: max-age=<1-year>). Since updating the processing block won't result in different URL (unlike with Active Storage or Dragonfly), we need to manually cache bust it:

# bump version for each derivate
plugin :derivate_endpoint, version: 1

We can also configure it per derivate:

# bump version for each derivate, and then bump again for 500x500 thumbnail
plugin :derivate_endpoint, version: -> (name, options) do
  version  = 1
  version += 1 if name == :thumbnail && options[:size] == [500, 500]

Caching to storage

We can have processed derivates be uploaded to storage. Then on next request the already processed derivate from storage is served, so processing is skipped (like Active Storage). This is useful because CDNs don't seem to keep the files for very long, see refile/refile#229.

plugin :derivate_endpoint, cache: true
plugin :derivate_endpoint, cache: -> (context) do
  context[:name] == :size_800

The default upload location would be /path/to/<original-id>/<derivate>.<ext>. It can also be configured:

plugin :derivate_endpoint, cache_location: -> (context) do
  original_id = context[:uploaded_file].id
  directory   = original_id.sub(/\.\w+$/, "")
  extension   = File.extname(context[:derivate].path)

  [original_id, context[:name], extension].join("/")

It would be possible to specify the cache storage:

plugin :derivate_endpoint, cache: true, cache_storage: :derivates_store
plugin :derivate_endpoint, cache: true, cache_storage: -> (context) do
  context[:derivate].to_s.start_with?("size_") ? :thumbnail_store : :other_store

By default the derivate will always be streamed through the endpoint; on first request the result of processing will be streamed, on subsequent the uploaded file would be streamed with rack_response plugin.

But if we wanted to avoid streaming the file through our app, we could choose to instead redirect to the uploaded derivate (like Active Storage does):

plugin :derivate_endpoint, cache: true, cache_redirect: -> (context) do

If the user is able to generate non-expiring URLs, then maybe it would be more performant to set "301 Moved Permanently" status (default would be "302 Found", aka temporary redirect), because then perhaps CNDs would remember that and next time request the redirect URL directly, avoiding the unnecessary roundtrip to the derivate endpoint.

plugin :derivate_endpoint,
  cache: true,
  cache_redirect: -> (context) { context[:uploaded_file].url },
  cache_redirect_status: 301

Dynamic parameters

Sometimes you might want to process the derivates differently depending on some information that's known at the time of generating the derivate URL. For example, you might want to do more intensive processing for paid customers.

Therefore Shrine::UploadedFile#derivate_url should support passing additional parameters (like we've already seen with :size):

photo.image.derivate_url(:thumbnail, params: { foo: "bar" })
derivate :thumbnail do |file, context|
  context[:params][:foo] #=> "bar"
  # ...


Content-Disposition response header would by default be composed of derivate name + original id (and would use the upcoming content_disposition gem):

Content-Disposition: inline; filename="size_500-sd00kl9ad8guadf9ds.jpg"

The filename can be changed:

plugin :derivate_endpoint, filename: -> (context) do
  context[:params][:filename] ||

The disposition can also be changed globally:

plugin :derivate_endpoint, disposition: "attachment"
plugin :derivate_endpoint, disposition: -> (context) do
  context[:name].to_s.start_with?("size_") ? "inline" : "attachment"

or per URL:

photo.image.derivate_url(:thumbnail, disposition: "attachment")
# or probably
photo.image.derivate_url(:thumbnail, force_download: true)


By default it would be set using Rack::Mime, but the user could change it:

plugin :derivate_endpoint, content_type: -> (context) do

Maybe also even per URL:

photo.image.derivate_url(:thumbnail, content_type: "text/plain")

Original downloading

By default the original file is automatically downloaded to disk and passed to the derivate block as the first argument.

derivate :thumbnail do |file, context|
  file # result of Shrine::UploadedFile#download
  context[:uploaded_file] # original Shrine::UploadedFile
  # ...

But the user might want to stream the uploaded file to the processor, or use a remote processing service like For that reason we allow user to disable automatic original download:

plugin :derivate_endpoint, download: false

derivate :thumbnail do |context|
  context[:uploaded_file].open do |io|
    # ... streaming ...
  # or"<USERNAME>/500x500/#{context[:uploaded_file.url]}")

It could be configured per derivate:

plugin :derivate_endpoint, download: -> (context) do
  [:size_500, :size_300].include?(context[:derivate])

Custom endpoint implementation

We should expose methods to the user which allow them to easily build their own implementation of the derivate endpoint. For example, since most Rails authenticatication gems don't support authenticating Rack application, they might want to call it from their controller so that they can add authentication.

One option for them would be to retain the URL format and use Shrine.derivate_response:

class ImageUploader < Shrine
  plugin :derivate_endpoint, prefix: "images/derivates"
# config/routes.rb
Rails.application.routes.draw do
  get "images/derivates/*rest" => "derivates#download"
class DerivatesController < ApplicationController
  def download
    set_rack_response ImageUploader.derivate_response(env) # infers everything from env


  def set_rack_response((status, headers, body))
    self.status = status
    self.response_body = body
# still generates the same URL, but now it points to our controller
photo.image.derivate_url(:thumbnail, size: [500, 500])

They can also implement their own URL format and use Shrine::UploadedFile#derivate_response:

Rails.application.routes.draw do
  resources :photos do
    member do
      get "thumbnail"
class PhotoController < ApplicationController
  def derivate
    photo = Photo.find(params[:id])
    image = photo.image
    set_rack_response image.derivate_response(:thumbnail, size: params.values_at(:width, :height))


  def set_rack_response((status, headers, body))
    self.status = status
    self.response_body = body
photo_thumbnail_url(photo, width: 500, height: 500) #=> "/photos/123/thumbnail?width=500&height=500

URL host

By default the URLs are relative, because Shrine has no knowledge of the current host:

#=> /images/derivates/thumbnail/...

The user could set the host either globally:

plugin :derivate_url, host: ""

Or per URL:

photo.image.derivate_url(:thumbnail, host: request.base_url)

Signing URLs

Derivate URLs should be signed to prevent tampering. Allowing the user to theoretically generate their own URLs would be dangerous, because an attacker could DoS with generating many URLs with differernt versions. Or they can pass custom processing parameters and potentially avoid a paywall.

We can use the SHA256 digest of the URL path with HMAC, like Active Storage, Refile, and Dragonfly all use.

class ImageUploader < Shrine
  plugin :derivate_endpoint, secret_key: "my secret key"
#=> "/images/derivates/thumbnail/<signed-SHA256-digest>--<base64-json-encoded-data>"

Then in the endpoint itself we would verify the signature, and abort if it doesn't match. The :secret_key should probably be a mandatory option.

URL expiration

I guess it would be good to allow creating expiring URLs.

plugin :derivate_endpoint, expires_in: 3600
photo.image.derivate_url(:thumbnail) # uses the 3600 default expiration
photo.image.derivate_url(:thumbnail, expires_in: 7200)


Nginx can be configured to serve files on disk. We can pass the file from our controller to Nginx by including the Rack::Sendfile middleware and returning a file response body that responds to #to_path and returns the path to the file (this functionality is provided by Rack::File).

# on Rails (config/application.rb)
config.action_dispatch.x_sendfile_header = "X-Sendfile" # Apache and lighttpd
# or
config.action_dispatch.x_sendfile_header = "X-Accel-Redirect" # Nginx
# on Rack-based web fraweworks:
use Rack::Sendfile, "X-Sendfile" # Apache and lighttpd
# or
use Rack::Sendfile, "X-Accel-Redirect" # Nginx
plugin :derivate_endpoint, sendfile: true

This isn't the default behaviour because in this case we cannot delete the Tempfile after we've served it, which might not suit some people.

This should also work with :cache option. By default when derivate cached to storage is served, it's streamed through the app (rack_response plugin). We could add a :sendfile option to Shrine::UploadedFile#to_rack_response to download the file to disk and serve it via Rack::File.

uploaded_file.to_rack_response(sendfile: true) # this would be called by derivate_endpoint

Integration with derivates plugin

This endpoint could nicely integrate with the upcoming derivates plugin, so that it allows the user to generate the direct URL to the derivate if it has already been processed on upload in a background job, otherwise to generate URL to the derivate endpoint.

class ImageUploader < Shrine
  plugin :processing
  plugin :versions
  plugin :derivate_endpoint

  process(store) do |io, context|
    # ...
    { size_300: size_300, size_800: size_800 }
  derivate :size_500 do |file, context|
    # ...
photo.image_url(:size_300) #=> ""
photo.image_url(:size_500) #=> "/images/derivates/size_500/..."

One can also generate dynamic derivates from existing ones, beacuse #derivate_url is defined on any Shrine::UploadedFile object.

# generates derivate_endpoint URL where :size_500 derivate is the source file

I don't want to experiment with saving uploaded derivates to a DB record at this point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.