Developing with dependent local gem repositories

gistfile1.md

When developing an application where you have split out shared functionality to multiple dependent gem repositories can get cumbersome when you

• need to edit the Gemfile in order to swap local :path sources with :git or just plain Rubygems sources,
• then forget to bundle update,
• wonder why your git repository is dirty and it's just the modified Gemfile and Gemfile.lock,
• accidentally commit the Gemfile.lock with local :path sources bundled etc. etc.

So what about this strategy:

A. Create a file .Gemfile.local containing:

base = '..' # might need to tweak this according to your directory layout
instance_eval File.read(File.expand_path('Gemfile'))

B. Change your actual Gemfile to something like this:

base ||= 'git://github.com/travis-ci'
type = base[0, 2] == '..' ? :path : :git

gem 'foo', type => "#{base}/foo"
gem 'bar', type => "#{base}/bar"

C. Add the .Gemfile* to your .gitignore file so that other developers can create their own .Gemfile.local and tweak the base path to their directory layout.

D. Switch between both Gemfiles using export BUNDLE_GEMFILE=.Gemfile.local and export BUNDLE_GEMFILE=Gemfile

Voila. It is acceptably easy to switch between using local vs remote gem sources with just a few lines of code. Switching to local/remote gem sources won't dirty the git repository.

Obviously, if you've changed your Gemfile then you still need to remember to switch to remote gem sources and bundle update once you're done and want to commit/push changes (where you don't gitignore the Gemfile.lock).

What do you think?

@svenfuchs I'd go for D. I think it's the least painful, and you should be able to commit your Gemfile.lock and Gemfile.local.lock without hitting any major issues.

@myabc wut? D? You mean as in http://en.wikipedia.org/wiki/D_(programming_language) ?

I've done added this to my bash profile to support this workflow:

function local_gemfile_on()
{
  export BUNDLE_GEMFILE='Gemfile.local'; echo "Gemfile.local is being used…"
}

function local_gemfile_off()
{
  unset BUNDLE_GEMFILE; echo "Gemfile is being used…"
}

And came up with this Rake task to keep a Gemfile.local up-to-date with your main Gemfile: https://gist.github.com/2016633

Both solutions are still "quick and dirty", but work for me. Any suggestions on optimization welcome (particularly since my shell skills leave a lot to be desired).

@svenfuchs Yes, I was suggesting abandoning Ruby completely. How about this: http://www.dwebframework.org/

Nice, thank you!

I'd prefer .Gemfile.local (i.e. a dot file) so that it doesn't clutter my dir listing so much. Also it kinda hints at the fact that this is "my" config and gitignored (even though I don't use that consistently, obviously)

@myabc Ugh ... GO AWAY!! ;)

@svenfuchs Yeah, I have the same problem. I set ENV["BUNDLE_PATH"] = 1 and check it in the Gemfile, so it would be a mix of B and D. I think you can additionally add a pre-commit git hook that aborts if BUNDLE_PATH is set to 1.

In any case, maybe we could change Bundler to support both :path and :git options at once? In such cases, Bundler would consider the source of the git repository to be the given path if it exists, otherwise it would proceed as usual.

@wycats @indirect @hone wdyt?

@indirect was saying on the Ruby Rogues that they know this is a pain point and where thinking of adding some sort of "priority source" feature to bundler. The idea being that if your gem can be found in the "priority source" (i.e. a directory or something) then the version that's there will be used in preference to other sources.

There are some issues and pull requests for bundler related to this topic. Like this one. Hopefully, we'll get some built-in solution for that in the future. For now, on our work project, we check env variable in the Gemfile to switch between gem from github and from local directory.

Recently I spotted another nice solution in travis-worker Gemfile. Should work great for some cases, but I wish I could extract this method into some other gem. As it's used right in the Gemfile it makes some problems. Is there any way to extend Gemfile DSL without using forked version of bundler (that is obviously not a solution)?

@josevalim, idea with pre-commit hook is pretty cool, thanks.

wow, thanks for the feedback, people.

@josevalim having both options combined with a env variable could work. but then again devs often times have their local sources in different places which is hard to solve with having everything in one Gemfile and committed.

I love the pre-commit hook idea!

@jonleighton same, the priority source feature would need to allow to set different paths for different devs (and ideally per gem, too, which is one drawback of my solution above ... all the repositories need to be subdirectories of one dir).

@KL-7 yeah, the solution in travis-worker requires to add/delete symlinks in order to switch sources.

Thinking about it I guess if Bundler supported some sort of plugins (like Rubygems does) we'd be good to go. Everybody could monkeypatch away and experiment with different concepts before anything gets added to Bundler itself.

After talking to Yehuda and Andre, I have started an implementation for supporting this feature in bundler: rubygems/bundler#1779 Note: the patch above is functional in case you want to try it out but keep in mind that `bundle config rack.local ~/path` is a *nix user config. This means setting `bundle config rack.local ~/path` will apply to all repositories that have rack as a git dependency. Also, the only way to remove a config is by manually editing ~/.bundle/config.

@josevalim Awesome work, dude! That'll come in really handy. But I was wondering if Bundler should limit a local override option to git-only dependencies. The problem of gems in development remains for rubygems-dependencies. I opened an issue, mainly for discussion at first: rubygems/bundler#1802.

Thanks, glad that you enjoyed. I have commented on the issue you have opened for bundler.