Skip to content

Instantly share code, notes, and snippets.

@cprice404

cprice404/lein-parent-notes.md

Last active October 13, 2016 16:36
Show Gist options
  • Save cprice404/652988f4be73a54aaadb9472cd9e10be to your computer and use it in GitHub Desktop.
Save cprice404/652988f4be73a54aaadb9472cd9e10be to your computer and use it in GitHub Desktop.
Download ZIP
lein-parent notes
Raw
lein-parent-notes.md

lein-parent

Problem Statement

We've had an ongoing issue around the complexity of managing version numbers for Clojure libraries across all of our Clojure projects. We have well over 20 different Clojure repos now, and each of them can potentially specify its own version of every dependency that it consumes (including transitive dependencies). Lein's :pedantic? feature helps us ensure that we're using well-known, consistent versions of any given dependency within the context of an individual Clojure project, but it doesn't give us any guarantees of consistency across the projects, or in the final uberjar artifacts that we produce and package.

This is painful in terms of daily maintenance of the repos; any time you bump the version of one dependency for a project, there's a high likelihood that you'll end up needing to play the "lein whack-a-mole" game to resolve the versions of the other libraries that it brings in transitively. This led us to put together this best practices document for managing dependency conflicts, and has resulted in most of our projects having a commented section in the project.clj file that says something like "transitive dependency conflicts", and lists out explicit version numbers of transitive dependencies, even though the projects themselves don't directly consume those artifacts. As these lists grow over time, the number of conflicts we run into increases, making the whole problem snowball.

This also means that the CI pipelines for any given project may or may not be using the same versions of various clojure libraries as the final uberjar artifacts will include, which makes our test coverage less complete.

This also makes it really difficult to maintain a repo such as pe-aio, which has to deal with the version conflicts for every version of every transitive dependency of every single one of our projects.

How do we improve this situation?

We're not going to completely solve this problem overnight, but there are some obvious goals that should help reduce the pain and maintenance costs around this:

  • Consolidate our list of dependency version numbers into a single place that can be referenced by all the other projects
  • Eliminate the need for any "transitive dependency conflict resolution" sections in the project.clj files of all of the projects
  • Drastically reduce the number of explicit version numbers listed in each project's project.clj file, even for their first-class dependencies
  • Work toward a state where our biggest test pipelines are running with a more consistent, well-known set of versions of the various clojure libraries.

OK, but how do we do that?

The approach we've been working on involves the use of a new feature in leiningen 2.7.1 called :managed-dependencies, along with a lein plugin called lein-parent.

lein-parent allows you to specify a "parent project", and inherit some project properties from it:

:parent-project {:coords [puppetlabs/clj-parent "0.1.4"]
                 :inherit [:managed-dependencies]}

:managed-dependencies allows you to specify a list of dependencies and their versions without actually realizing them. In other words, it allows you to say "hey, I'm not sure I'm going to have a dependency on clj-time, but if I do, it should be version 0.12.0". Then, in the regular :dependencies section of your project, you can simply omit the version number for clj-time, and it will be inherited from the :managed-dependencies section of your parent project.

Do you have an example?

Sure! We recently rolled out these changes to the OSS puppetserver repo, and several of its upstreams (specifically: kitchensink, trapperkeeper, and trapperkeeper-webserver-jetty9).

Here's the new "Puppet Clojure Parent Project" (released to clojars):

https://github.com/puppetlabs/clj-parent/blob/master/project.clj

Here are some (merged) PRs that illustrate the changes to some other repos, to consume the parent project:

Will all of this work in my dev environment?

Yes, mostly. :) You need lein 2.7.1, and there may be a few quirks in various editors. Specifically:

  • vim/fireplace: sounds like it works fine?
  • emacs: it should work fine, though there is a known issue with the clojure-emacs/refactor-nrepl plugin. It sounds like it should be easy to fix so hopefully they'll have it fixed soon.
  • IntelliJ/Cursive: You need Cursive 1.4.0-eap3 or newer. Should work fine in that version.

So, should I go update all of my repos right this second?

Meh. These changes are pretty new and if you're faint of heart you might want to let the OSS Puppet Server and upstream projects get a little more mileage before you dive in. That said, if you're willing to try it out now, there aren't any known issues that should prevent it and any feedback would be greatly appreciated.

If you find yourself dealing with the version whack-a-mole frequently, then you might be able to save yourself some work in the long run by updating things now.

Overall it's probably a fine strategy to just update the various project.clj files opportunistically when you find yourself in them for other reasons.

Aren't I just going to have the same problem w/rt needing to update the parent version in all of my projects all the time?

Well, yes and no. Several of us discussed this question offline, and this is what we came up with as a plan for getting started. We can certainly evolve this over time as we get a feel for what works well and what doesn't. And this is all definitely up for discussion if you have any ideas or input.

  • We should do a new release of the parent project every time we bump a version in it.
  • We should send an e-mail to the dev list every time we do a parent release.
  • You don't need to update the parent version in every project every time it gets released. Maintainers of each of the four PE JVM processes (puppetserver, puppetdb, console, orchestrator) should decide when they want to bump to a new parent version, and should do it pretty much simultaneously on all of their main constituent services (e.g., for puppetserver, we'd bump the parent version in lockstep on puppetserver, file sync, and code manager, plus the ezbake composite packaging repo).
  • It's probably not a huge deal if some of the non-leaf projects (trapperkeeper-jetty9, etc.) just have their parent updated opportunistically. It's really the repos where we have the most comprehensive and useful test suites that we want to be a bit more diligent about.
  • Even though updating the parent is going to be a bit of hassle, it should be significantly less work than updating versions of individual components one at a time, and hopefully the subsequent whack-a-mole game goes away almost completely.

After we try this out for a while and see how it goes, there are a few other ideas we batted around, but ultimately decided that they'd be too much work or too much risk to try right out of the gate:

  • Use SNAPSHOT versions for the parent artifact. This would allow us to ensure that all projects got the newest changes to the parent project relatively shortly after it was modified. However, it would require some tight coordination of when to do a final release of the parent project prior to any OSS/PE release, and would come with all of the usual issues that make SNAPSHOTs so much fun to work with (non-deterministic builds, etc.)
  • Write some tooling that knows how to go find and update the parent versions in all of the relevant repos. This probably wouldn't be all that hard, and definitely seems automatable... but it's not yet obvious that it wouldn't cause other problems, or that it would be worth the effort. So it seems like an option to keep in our back pocket for now.

Where is the parent project artifact deployed?

Clojars.

Hmmm. So what about PE artifacts, then?

After discussion, we decided to just list pe artifacts directly in the same parent project, and publish it to clojars. Some points discussed:

  • OSS projects need to have a parent that is available on clojars or else people outside of PL (or not on the VPN) wouldn't be able to build and run them.
  • Maintaining two separate parent projects (one for OSS, one for PE) is possible but seemed like unnecessary extra work.
  • Since the parent project doesn't try to realize any of the dependencies - it only lists them - it doesn't hurt anything for there to be PE artifacts listed in the OSS parent project.
  • If we include the PE artifacts in the parent project, there's no IP being exposed to the world beyond the PE artifact names and version numbers. Seems harmless.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment