Note: many issues listed below are not specific to Rails or to Ruby. They can also exist in other languages or web frameworks presenting similar challeneges. It is important to note these challenges, because they can impede the workflow and best practices taught and used by Rails practitioners.
There are many different versions of Ruby in use. This is very unique, in that, in the Python world, you have many versions, but almost everyone will be using the last 2.X or the last 3.X. In the Java world, most use cases are covered by the last two major releases of either Java released by Oracle or on OpenJDK.
Ruby applications can be based on 1.8, 1.9, 2.0 and all intermediary versions all the way to 2.3. This includes "upstream" Ruby but also REE (an "Enterprise Edition" set of patches, for older versions of Ruby), and a number of "Ruby on X" ports: JRuby (on top of the JVM), IronRuby (on top of .NET), Maglev, Rubinius, RubyMotion, and more.
Earlier versions are used in production because upgrading presentes challenges such as critical gems are not backward-compatible. Debugging tools like pry, rubocop, byebug... do work on many versions, but not consistently on all Ruby verisons.
As a result, most Ruby programmers (and people deploying Ruby apps) rely on tools like rvm or rbenv to install a specific version of Ruby. These tools enable developers to switch between different versions of Ruby, and between different sets of gems (when different applications have conflicting requirements).
Ruby programmers expect to be able to request a specific version of Ruby on demand. This is not currently available on Docker Hub and developers will have to create Dockerfiles that are specific to development environment requirements.
Docker Hub currently has Ruby 2.1, 2.2, 2.3; which means the images on Docker Hub can support newer applications. However, migrating older applications to a container will require custom builds.
Ruby is avalable on Docker Hub
Using RVM poses other challenges. Specifically, RVM works with a combination of custom PATH and shell functions. After installing RVM in a Dockerfile (e.g. curl get.rvm.io | bash); to use RVM, it must be run in a login shell. However, the shell started by Docker (when executing a "RUN" build step) is not a login shell.
One workaround is to wrap all commands with "bash -l -c", e.g.:
RUN bundle install
Becomes
RUN bash -l -c "bundle install"
Another workaround is to symlink /bin/sh to /usr/local/rvm/bin/rvm-shell. If you do that, you also need to change /bin/which so that it uses #!/bin/dash instead of #!/bin/bash (on many Linux systems, /bin/which is implemented as a shell script; and rvm-shell depends on /bin/which; so you get an infinite loop). Albeit clunky, that workaround is the one that was the most satisfying (less side effects, less potential for mistake if you forget to wrap RUN commands properly, less risk to use system Ruby my mistake...)
Bundler is a very popular tool in the Ruby world. It is "an exit from dependency hell, and ensures that the gems needed are present in development, staging, and production."
Bundler makes sure that a Ruby app can access the gems listed in the Gemfile, and only those gems. This is different from Python's pip: with pip, you can install a bunch of dependencies from a requirements.txt file, then manually install a few extras. With bundler, everything has to be in the Gemfile. This is advantage when you want to avoid "leaking" dependencies; i.e. being unaware that your project does, in fact, use some gems that are not declared in your Gemfile. However, it means that in a Docker environment, you have to use a separate Gemfile.tip file where you add extra gems during development.
# Example from: https://github.com/cpuguy83/docker-rails-dev-demo/blob/master/Dockerfile
FROM ruby:2.2
RUN apt-get update && apt-get install -y sqlite3 libsqlite3-dev openssl libssl-dev libyaml-dev libreadline-dev libxml2-dev libxslt1-dev
WORKDIR /opt/myapp
ENV RAILS_ENV production
# Add Gemfile stuff first as a build optimization
# This way the `bundle install` is only run when either Gemfile or Gemfile.lock is changed
# This is because `bundle install` can take a long time
# Without this optimization `bundle install` would run if _any_ file is changed within the project, no bueno
ADD Gemfile /opt/myapp/
ADD Gemfile.lock /opt/myapp/
RUN bundle install
# This will now install anything in Gemfile.tip
# This way you can add new gems without rebuilding _everything_ to add 1 gem
# Anything that was already installed from the main Gemfile will be re-used
ADD Gemfile.tip /opt/myapp/
RUN bundle install
# Add rake and its dependencies
ADD config /opt/myapp/config
ADD Rakefile /opt/myapp/
# Add everything else
# Any change to any file after this point (if not in .dockerignore) will cause the build cache to be busted here
# This includes changes to the Dockerfile itself
# Goal here is to do as little as possible after this entry
ADD . /opt/myapp
ENV PATH /opt/myapp/bin:$PATH
ENTRYPOINT ["/opt/myapp/bin/start.rb"]
CMD ["server"]
Bundler becomes particularly problematic when switching back and forth between different branches. Each new branch with a different Gemfile (even if 90% of the versions therein are the same) will trigger a full build (which can easily amount to half an hour for an average Rails project, with a decent computer and internet access; and can be multiple hours for a big project and/or a more modest hardware setup or internet access). The work around to Bundler running on each build is to copy the Gemfile and Gemfile.lock to a temporary directory and run Bundler from there. If neither file is changed during subsequent builds, the ADD instructions will be cached. Below is a working example:
FROM ubuntu:12.10
MAINTAINER brian@morearty.org
# Install dependencies.
RUN apt-get update
RUN apt-get install -y curl git build-essential ruby1.9.3 libsqlite3-dev
RUN gem install rubygems-update --no-ri --no-rdoc
RUN update_rubygems
RUN gem install bundler sinatra --no-ri --no-rdoc
# Copy the Gemfile and Gemfile.lock into the image.
# Temporarily set the working directory to where they are.
WORKDIR /tmp
ADD railsapp/Gemfile Gemfile
ADD railsapp/Gemfile.lock Gemfile.lock
RUN bundle install
# Everything up to here was cached. This includes
# the bundle install, unless the Gemfiles changed.
# Now copy the app into the image.
ADD railsapp /opt/railsapp
# Set the final working dir to the Rails app's location.
WORKDIR /opt/railsapp
# Set up a default runtime command
CMD rails server thin
The previous issue becomes totally unbearable when doing a git bisect, or trying to find which specific revision introduced a specific bug or broke a specific test. Each modification (even minor) of the Gemfile triggers a 30-minute full rebuild (instead of taking a couple of minutes tops).
In the case of with slow internet connections, it is possible to run a local gem server. Rubygems.org provides a guide for running a local gem server. The caveat to a local gem server is that the Dockerfile becomes then less portable, since it relies on the gem server to be up. Note that even with a mirror, a lot of gems take a long time to install, because they build native extensions.
In practice, the divide and conquer strategy appears to be successful; i.e. spin up cloud instances, split the commits in ranges, and assign a range to each instance, and do a simple "docker build" each time. Howerver, in a test of ~100 commits (with more than half of the time spent installing dependencies vs. actually running tests) this took over 6 hours
Mounting a volume across builds would help, since bundler could use this as a gem cache. (This also has been asked by people building Python apps, in particular using SciPy and NumPy, which are notoriously slow to build.) People generally understand the argument that build-time volumes break reproducibility of builds, but it still leaves them short of a good solution to achieve fast builds.
One of the tenets of Docker is to use the same container images in dev, testing, preprod, and prod. In Rails, there is the notion of environments. A typical Rails project will have dev, production, test, staging. (Not always all of them, sometimes more; but this is typical.)
Setting the RAILS_ENV environment variable switches between environments. Each environment has:
- a different set of gems (you don't have the debugging tools in production),
- a different way to handle static assets (you don't precompile JS, CSS, etc. in dev),
- a different database (at least a different server, but sometimes a totally different engine),
- a bunch of things that behave differently (e.g. in test a bunch of things are replaced by mocks),
- etc.
This architecture does not fit within the Docker model for the reasons listed below.
There isn't a clear solution for testing environments. It is common practice to load extra gems in the testing environment (e.g. capybara, to simulate a browser environment). The traditional Docker approach would be to have two images, production and testing, the latter extending the former. This could be implemented with a Gemfile (used for all environments) and Gemfile.testing, and Dockerfile (used for production) and Dockerfile.testing (which would be identical to Dockerfile, except for a couple of lines handling Gemfile.testing); but even if that worked fine, it would require a significant change in the workflow for people accustomed to traditional Gemfile handling (in which statements delineate clearly which gem sets are used in which environments).
Assets include javascript, CSS, images... These resources often need to be transformed before being served to the user:
- javascript can be "minified";
- CSS can be compiled (using e.g. SASS or LESS);
- images can be resized or converted.
In dev mode, this is done on-the-fly by the application server (a file can be edited and immediately reload in your browser, without having to recompile assets or restart the application server).
In production, however, assets are pre-compiled, and served by a static server. Sometimes, pre-compiled assets are pushed to a CDN instead of being served directly.
Assets are meant to be heavily cached by the clients. To enable that, asset URLs are seeded, e.g. "/assets/main.css" becomes "/assets/main-eb665b0c5b3ae55b0765b570d225700e.css", and when the file is modified, the name changes too (to force a cache miss from the clients). This means:
- that at any given time, a client might still have (in cache) an older version of a CSS, JS, or other asset, itself referencing another asset that hasn't been loaded yet (if the asset is loaded on demand, e.g. a hover image); therefore, old versions of assets should be kept forever (or at least long enough to avoid breaking too many users when rolling out a new version);
- that a manifest is generated after the pre-compilation, containing the mapping giving the seeded path for each asset. The manifest is required by the application server to produce correct URLs pointing to the assets.
The assets pipeline has an extra challenge: when pre-compiling assets, the whole Rails framework is brought up, including the database connection (even if it's not strictly necessary 99% of the time). This means that the database server must be available to pre-compile the assets. This makes it extra difficult to run the assets pipeline from a Dockerfile.
This requires a radical change in the workflow. Instead of handling dev and production differently, it makes more sense to have a single environment (it is more the "Docker way"), and always compute assets on the fly. To avoid overloading the app server with assets compilation, a caching web server can be used. Some people use Squid or Varnish for this, but in this specific scenario, since expiration doesn't need to be handled, NGINX can do the job very well.
Rails, like a few other complex frameworks, makes a clear distinction between the application server and the web server. The web server handles static files (and often deals with redirections, logging, conditionals...) and hands off everything else to the app server, which is the actual Ruby part. The web server is (almost always) necessary, because it appears to be unfeasible to achieve high performance static file serving with a pure Ruby solution. The most frequently used web servers are Apache or NGINX, while the popular app servers include Thin, Unicorn, and Mongrel.
Sometimes the app server is a module in the web server. While the latter seems simpler from an architectural point of view, it implies a more complex configuration for the web server, since the subtleties of "which Ruby interpreter to use" and "which gem set to activate" now propagate to the web server. Furthermore, app server and web server have to share some information about the static assets (see previous paragraph).
There are at minimum four possibilities for app server and web server containerization, each with their pros and cons:
- app server and web server in separate containers, assets pre-compiled at build time: this achieves clean separation, and efficient handling of assets; but the assets have to be built in the app server, then extracted to be copied to the web server before building the web server image; moreover, the database has to be available at build time;
- app server and web server in separate containers, assets pre-compiled at run time, placed in a volume: this achieves clean separation; there is a transient period (just after deployment) where assets have yet to be built, and the service is not fully operational; so zero-downtime deployment requires extra steps; since the assets are built at run-time, database availability is not an issue;
- app server and web server in separate processes but in the same container: this considerably simplifies asset handling (though database availability remains an issue), but it requires using a process manager in the container, since we're running two independent processes in the same container;
- app server and web server in the same process (and in the same container): this obviates the question of whether to put them in separate containers or not, but it is much less flexible, since it reduces the choice of app servers down to Passenger and mod_ruby. Also, configuration becomes a bit trickier (but admittedly, this should be a one-time cost).
In short, there is no "good" solution here: while the single process model is simpler, some apps rely on the specific behavior of e.g. Unicorn, while others leverage Thin's async event model, etc.
Rails has a sophisticated database migration mechanism, helping operators to ensure that the database schema is exactly like it should be, i.e. in sync with ActiveRecord models as they are declared in the application code. This is not a problem per se and Rails even comes with a few functions to run migrations in a rather safe way, i.e. "bring up the database schema to what it should be, no matter what."
However, this mechanism hasn't been designed with automation in mind. The typical workflow when deploying a Rails app is to execute Capistrano from a central machine. Capistrano will distribute the new version of the code, pre-compile assets, execute database migrations, and so on. But in a container-based environment, there is no central machine. If migrations are executed by the app container, in a scaled environment, you end up with multiple migrations running at the same time. This requires some extra rules.
In addition there is an overall reliance on the database for any rake task. It is common practice that virtually all rake tasks (like assets pre-compilations) would bring up the database driver, even when it's not necessary at all. This adds an extra constraint on helper scripts and various tools that people write to maintain and operate their code.
This can be alleviated by the use of the (undocumented) rake task "db:abort_if_pending_migrations", which assesses whether the database is ready.
The general idea is to use an entrypoint that will:
- wait for the database server to be up and accepting connections;
- (if using a separate database for the app) check if the application database exists, and create it otherwise;
- check if migrations are pending (this will also check if the schema has been created), and run migrations otherwise;
- if the previous step failed, try again (this will protect against concurrent migration execution by multiple frontends).
This is probably not optimal (especially from a concurrent execution point of view) but it does the job acceptably.
As indicated above, many things (including asset pre-compilation) require database access, even though 99% of the time, it is unnecessary. Heroku veterans told me that their solution had been to mock the database presence when running specific things (like assets pre-compilation) to let those steps run without the database.
I'd like to thank all the people who helped me to machete my way around the Ruby and Rails jungles; in particular:
- AJ Bowen from Voteraide
- Laura Frank from Codeship,
- Noah Zoschke from Convox.
For those who stumble upon this Gist, it was the WIP for what became the official document: https://github.com/docker/labs/tree/master/developer-tools/ruby.
Kudos to Sophia for writing it!