Heroku uses buildpacks to compile your application into a slug that is used across dynos for scaling horizontally quickly. A slug is a tar.gz archive of your app’s repository with certain pre-deploy features baked into the filesystem. Since everything to run your application is included in the archive, scaling becomes a simple matter of transferring it to a dyno, unpacking, and running the appropriate process. This is how Heroku achieves scaling-by-moving-a-slider.
For example, the Ruby buildpack will:
- install ruby locally
- install the jvm/jruby (if you’re using it)
- install/run bundler and install your gems to Rails.root/vendor
- create your database.yml (which ends up reading from your app’s environment variables)
- install nodejs binaries (if you’re using them)
- precompile your assets (which has its own issues)
Here is what happens when you deploy to Heroku.
- You push code to your remote Heroku repository. This is something like
- Heroku’s git server notices and assigns the push to be handled by a special dyno used only for compiling slugs. A key thing to notice is that its build environment is the same as the dynos runtime environment, so you can compile binaries that are distributable between your dynos.
- Looks at its api for your "repo url". If it doesn't have one yet, it makes a new one for your app.
- Downloads your repo’s tar.gz to the dyno from S3 and unpacks it. This includes your git repository hosted on Heroku.
- Runs the pre-receive hook on your repo. This is a Heroku-supplied script that sets up your build environment and PATH. (You can see this script using the repo plugin, listed below).
slug-compilerchecks out your app's code to a temporary directory, something like /tmp/build_xxxx
slug-compilerlooks at your app's environment config variables for BUILDPACK_URL. (Set using heroku config:set)
- If it finds it, it’ll try to clone it to the dyno and run its
- If it doesn’t find it, it runs through the Heroku standard buildpacks, running
bin/detecton them until one answers positively that it can build on that codebase. (Exits status 0)
- Assuming there is a buildpack found that says it can compile, it then runs the buildpack's
bin/compilescript with two arguments: $1
build_dirthe directory of your app's code checkout; $2
cache_dirthe "cache directory", a directory that will persist between successful slug compilations.
bin/compilecreates/changes/deletes files in
cache_dir, and exits successfully (status 0).
bin/releaseis called with
build_dir, expecting some YAML output describing default process types and addons that should be installed.
- slug-compiler packages up the
build_dirinto an internally-accessible slug to distribute amongst your dynos. It also packages your whole app repo, including your cache directory, and uploads it to S3 to be used the next time you deploy.
- Internal Heroku mechanisms push the new slug to all of your dynos and restart the processes on them. Or it spins up new dynos with your code and replaces your old dynos. Since we don't work at Heroku, we can only speculate as to which it is. My bet's on the latter.
Misc notes and tips
To get access to your app’s repo as hosted with Heroku, the heroku-repo plugin lets you download it, mess with it, and re-upload it. This was invaluable for a cache-leak bug we discovered with Heroku support which was making deploys take 2 minutes longer each deploy we did. You can also run git-gc on your repo to trim its size and potentially make your deploys go faster.
You can precompile binaries into your repo using a tool called Vulcan.
If you are compiling libraries to use for native extension based Ruby gems using your own buildpack, you need to have a way to copy a
.bundle/config file into your
build_dir before the Heroku ruby buildpack gets a hold of it, if you're using e.g. heroku-buildpack-multi. Unfortunately there's no current way to separate production and development environments in terms of
.bundle/config. We have one in our repo called
In the current version of heroku-buildpack-ruby, it also stores the
.bundle/config in the
cache_dir and uses that one each deploy (and then never updates it), so you’ll need to clear the cache if you change the file. In our own deployment, we clear it every deploy.
Speaking of which, your .bundle/config needs to looks something like this.
Build directives in .bundle/config have the format BUNDLE_BUILD__(gem name uppercase): --with-whatever-include=/app/libprefix/usr/include --with-whatever-lib=/app/libprefix/usr/lib
You'll notice that there are references to
/app, however, on the build server they don't have an
/app directory yet. In our compile step, we do something like this:
rm -f $CACHE_DIR/.bundle cp -v -R $BUILD_DIR/.heroku_bundle $BUILD_DIR/.bundle sed -i "s,/app,$BUILD_DIR,g" $BUILD_DIR/.bundle/config
/app with the current
build_dir, where Bundler can expect your compiled library.
In Heroku production dynos though, your app lives in