gistfile1.txt

---
layout: post
title: "Gem Activation and You, Part 2: Bundler and Bin Stubs"
date: 2013-05-15 03:03
comments: true
categories: ruby
---

Most of this article will expect some basic familiar with Bundler,
and [Gem Specifications, Activation, and Dependencies](http://erik.hollensbe.org/2013/05/11/gem-activation-and-you/).

There's been a bit of discussion in the past about
[Bundler](https://gembundler.com), when it should be used, when it should not
be used. [Yehuda Katz](http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/)
has written extensive commentary on the topic, and you should read that.

I'm going to boil it down, and will elaborate implicitly later:

* If you have a standalone executable for others to use, Bundler can make
  dependency problems less obvious by hiding dependencies which are valid
  according to your gemspec, but broken. You should at least ensure it's
  operating without bundler before releasing it.
* If you have a project directory, use Bundler extensively, and set everything
  you use in the `Gemfile`. Use `bundle exec` extensively.
  * Consequently, if you're developing a gem, that is your project directory.
    Just don't check in the `Gemfile.lock`, but still use `bundle exec` like
    it's going out of style. Routinely `bundle update` or set arbitrary
    hard dependencies in your `Gemfile` to ensure your gem plays well with
    others.

## Why all this?

Because even though we've been using [Semantic Versioning](http://semver.org/)
long before Tom Preston-Werner wrote his treatise on the subject, you still
have to play ball with a lot of people. A lot of people don't use Semantic
Versioning.

Ivory Towers are for people who never get dirty; ignore the real world at your
own peril. Bundler is a tool for assisting you with dealing with the real
world. Just like you have things like the CGI specification, and HTTP, Rails is
there to assist by putting XSS and CSRF protection -- things you need for
modern web programming. RubyGems is the basics, and Bundler is the cherry on
the top to assist with real world application problems.

That said, using bundler liberally can hide certain classes of problems, or
empower you to discover them.

## How does Bundler work?

Let's start with a quick note on what Gem Requirements are first. So, a Gem
Requirement is a specification of a version, such as `>= 0`, which always means
the latest version, or `~> 1.2.3`, which means anything `>= 1.2.3` but also
anything `<= 1.3.0`. Gem Requirements have a
[few operators which have basic code mappings](https://github.com/rubygems/rubygems/blob/master/lib/rubygems/requirement.rb#L19).
You should read them.

How Bundler works, in a nutshell: For a given `Gemfile`, Bundler will use the
latest version of everything that fits the default Gem Requirement (the default
requirement being `>= 0`), and given any conflicts, slowly reduces the value of
each Gem's version until it violates the `Gemfile's` Requirement or the
Specification's Requirements. Presuming it's able to solve the formula, it
spits out a `Gemfile.lock` which contains what conclusion it came to. If not,
it tells you where the conflict lies.

### Let's see this in action

As mentioned in the previous article, both the `chef` and `vagrant` `1.0.x` do
not play nicely together on a dependency level. However, if you're willing to
accept `chef` `10.18.2` instead of the latest hotness, `11.4.4`, you can use it
with `vagrant` `1.0`.

Here's an example Gemfile to play with:

{% codeblock Gemfile.rb %}
gem 'chef'
gem 'vagrant', '= 1.0.7'
{% endcodeblock %}

Put that in a directory and run `bundle`. You should see something like this:

{% codeblock %}
Using archive-tar-minitar (0.5.2) 
Using bunny (0.7.9) 
Using erubis (2.7.0) 
Using highline (1.6.18) 
Using json (1.5.4) 
Using mixlib-log (1.6.0) 
Using mixlib-authentication (1.3.0) 
Using mixlib-cli (1.3.0) 
Using mixlib-config (1.1.2) 
Using mixlib-shellout (1.1.0) 
Using moneta (0.6.0) 
Using net-ssh (2.2.2) 
Using net-ssh-gateway (1.1.0) 
Using net-ssh-multi (1.1) 
Using ipaddress (0.8.0) 
Using systemu (2.5.2) 
Using yajl-ruby (1.1.0) 
Using ohai (6.16.0) 
Using mime-types (1.23) 
Using rest-client (1.6.7) 
Using polyglot (0.3.3) 
Using treetop (1.4.12) 
Using uuidtools (2.1.4) 
Using chef (10.18.2) 
Using ffi (1.8.1) 
Using childprocess (0.3.9) 
Using i18n (0.6.4) 
Using log4r (1.1.10) 
Using net-scp (1.0.4) 
Using vagrant (1.0.7) 
Using bundler (1.3.5) 
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
{% endcodeblock %}

Notice how we've specified the latest version of `chef`, but we got `10.18.2`
instead? This is because of the `net-ssh` dependencies they share -- `10.18.2`
depends on a version of `net-ssh` that `vagrant` is ok with, so Bundler, to
solve the formula, rolls our chef back.

Change your Gemfile to look like this:

{% codeblock Gemfile.rb %}
gem 'chef', '~> 11.0'
gem 'vagrant', '= 1.0.7'
{% endcodeblock %}

This sets `chef` to have a minimum version of `11.0` but not as high as `12.0`.
Run `bundle update`. You will see this:

{% codeblock %}
Resolving dependencies...
Bundler could not find compatible versions for gem "net-ssh":
  In Gemfile:
    chef (~> 11.0) ruby depends on
      net-ssh (~> 2.6) ruby

    vagrant (= 1.0.7) ruby depends on
      net-ssh (2.2.2)
{% endcodeblock %}

Voila! We have a constraint violation on `net-ssh` -- `chef` depends on `2.6`
or better, and `vagrant` just isn't going to let that happen. If you read the
first article, you'll notice this is the same constraint violation we saw
before.

## Bin Stubs, or how those command-line tools get run.

Now that we understand how Bundler works, let's have fun with tools like `rake`
or `thor` or `gist`. These are tools you commonly would run *outside* of a
bundled environment, but still have consequences within the RubyGems system.

This is because they correspond to *activated gems*, and what gems are
activated largely depends on what gets installed. The scripts you actually run
are called "bin stubs", or little scripts that look a lot like this (this one's for `rake`):

{% codeblock rake.binstub.rb %}

[15] erikh@speyside ~/tmp% cat `which rake`
#!/usr/bin/env ruby
#
# This file was generated by RubyGems.
#
# The application 'rake' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require 'rubygems'

version = ">= 0"

if ARGV.first
  str = ARGV.first
  str = str.dup.force_encoding("BINARY") if str.respond_to? :force_encoding
  if str =~ /\A_(.*)_\z/
    version = $1
    ARGV.shift
  end
end

gem 'rake', version
load Gem.bin_path('rake', 'rake', version)
{% endcodeblock %}

There's that `gem` call again! If you notice, it's parsing the version out from
`_version_`, and activating that version or the latest version if omitted
(because `version` will be nil). This is just like the example from the first
article.

This means if you have rake `0.9.6` and rake `10.0.0`, by default, `rake`
`10.0.0` will be run. However, if you do this:

{% codeblock %}
rake _0.9.6_ my_target
{% endcodeblock %}

`0.9.6` will be run instead. The point is, the script is there to facilitate
this, and gem activation in general. The importance of these notions will be
important for our next part...

## Why `bundle exec` is really really really really important for your bundled projects

Type `bundle gem foo` -- this will create a project skeleton for a gem called
`foo`. It will generate in the `foo` directory a few files, including a
`Gemfile`, a `Rakefile`, and a `foo.gemspec`.

Let's add something to that `Rakefile`. How about this at the end?

{% codeblock Rakefile.rb %}
require 'json'
p JSON::VERSION
{% endcodeblock %}

And this to the `foo.gemspec` in the right spot:

{% codeblock Rakefile.rb %}
spec.add_dependency 'json', '= 1.5.4'
{% endcodeblock %}

Then `gem install json` to get the latest version, then `bundle install`.

If we type the command to get the list of tasks, `rake -T`, we should see something like
this:

{% codeblock Rakefile.rb %}
"1.7.7"
rake build    # Build foo-0.0.1.gem into the pkg directory.
rake install  # Build and install foo-0.0.1.gem into system gems.
rake release  # Create tag v0.0.1 and build and push foo-0.0.1.gem to Rubygems
{% endcodeblock %}

What? We just told bundler to use 1.5.4! Bundler never got considered here. The
tool, `bundle exec` was created to ensure that all activations happen under the
watchful eye of bundler.

Type `bundle exec rake -T` and see how this changes:

{% codeblock Rakefile.rb %}
"1.5.6"
rake build    # Build foo-0.0.1.gem into the pkg directory.
rake install  # Build and install foo-0.0.1.gem into system gems.
rake release  # Create tag v0.0.1 and build and push foo-0.0.1.gem to Rubygems
{% endcodeblock %}

Now, if there were conflicting gems on your machine that you would require, or
just want to make sure you have the right version, running *without* `bundle
exec` ensures that's possible. This is a great thing for one-off commandline
tools, but not so great for applications, or projects in general. If you
develop commandline tools, you should test with and without bundler to ensure
the behavior in the presence of other dependencies is desired.

### RubyGems 2.0 can use Gemfiles

Bundler can solve a whole host of constraint problems, but RubyGems 2.0 now
considers Gemfiles as well; this actually made the above example a lot harder
to do that it has been before as `bundle exec` is not nearly as necessary
anymore. Still, to be on the safe side, you should use it for now.

## Conclusion

Bundler, Bin Stubs and RubyGems all work together to create a smart system at
the cost of a little cognitive dissonance -- the expectation that there should
be one source of truth is honored, but it is evaluated amongst many truths in
relationship to its own. When you don't care, it's great. When you do, you have
this article to help you figure out what to do. :)

Stay tuned for Part 3 where we discuss packaging RubyGems with other packaging
systems.