Explicit dependency management in sbt

sbt-dependency-management-guide.md

Some of these practices might be based on wrong assumptions and I'm not aware of it, so I would appreciate any feedback.

1. avoiding some dependency conflicts:

   • install sbt-explicit-dependencies globally in your ~/.sbt/{0.13,1.0}/plugins/plugins.sbt
   • run undeclaredCompileDependencies and make the obvious missing dependencies explicit by adding them to libraryDependencies of each sub-project
   • (optionally) run unusedCompileDependencies and remove some obvious unused libraries. This has false positives, so ; reload; Test/compile after each change and ultimately run all tests to see that it didn't break anything
   • (optionally) add undeclaredCompileDependenciesTest to the CI pipeline, so that it will fail if you have some undeclared dependencies

2. keeping dependencies up to date and resolving conflicts:

   • install sbt-updates globally in your ~/.sbt/{0.13,1.0}/plugins/plugins.sbt
   • run dependencyUpdates and bump all non-major versions. Major versions updates should be done one by one with care and love and testing.
   • include all explicit libraryDependencies in dependencyOverrides to force their versions. This is supposed to have the same effect as applying force() on all libraryDependencies, but isn't ivy-specific. The point is to prevent conflict manager choosing automatically some version required by a transitive dependency instead of the one you wrote explicitly. This is done in project/Dependencies.scala
   • it might become difficult to maintain versions in libraryDependencies and dependencyOverrides in sync, so a common practice is to define values for versions of each dependency, put them in project/Versions.scala and use throughout build.sbt. This is also convenient for libraries that are split in multiple artifacts which have to have the same version.

3. using sbt-assembly:

   • try running assembly for each project (starting from the independent ones) and see if there are any merge conflicts
   • if there are two different libraries that contain conflicting class files (same path, different content), use shading to rename one of them:
     • don't use .inAll because it will rename classes in both of the libraries and it will be the same situation. Instead use .inLibrary or .inProject
     • you might need to use .inAll for something else, e.g. you have a predefined runtime classpath with some outdated libraries and want to use newer versions, but avoid conflicts with that external classpath
   • avoid using exclude or excludeDependencies because you may throw away some library which is needed by one of the transitive dependencies and it will fail in runtime with MethodNotFoundException or something like that
   • avoid overriding merge strategy on class files (using first/last/discard strategies), because it's the same as excluding some classes. Use merge strategy overrides only for some trivial conflicts or non-class files, e.g. to merge two .properties files with concat or filterDistinctLines strategy

4. resolving more conflicts:

   • run evicted for each project and inspect the list of automatically resolved conflicts
   • try to minimize the number of lines marked as [warn], those are conflicts with potentially binary incompatible versions
   • if some of the libraries introducing the conflict are yours (company-owned), go and update its dependencies to solve the conflict. Unfortunately, it's more often the other way around: your libraries are more up to date than some external ones that you don't have access to. Anyway, consider updating those external libraries or contribute to them if they are opensource.
   • use sbt-dependency-graph installed globally to untangle the dependencies and understand the origin of the conflicts. The whatDependsOn task is very useful for that.

5. Rinse and repeat. I numbered these steps because IMO it's better to do them in this order, but after each step it might be useful to go through the previous steps again.

Some useful links:

• "What are your tips/tricks/tools for dealing with dependency hell?"

  • https://github.com/cb372/sbt-explicit-dependencies
  • good point (human factor):

    Do a better job of articulating WHY certain dependencies are used rather than dumping everything in common and then wondering why SBT is so complex.

• advices from Eugene Yokota:

  • avoid -SNAPSHOT outside of local testing
  • avoid version range
  • evicted for eviction report
  • sbt -no-colors update > update.txt
  • jrudolph/sbt-dependency-graph
  • rtimush/sbt-updates

• semantics of dependency resolvers by Eugene Yokota

  the exact rules governing how the user-specified constraints are interpreted vary from one dependency resolver to another. I call these rules the semantics of the dependency resolvers.

• sbt docs:

  • library management
  • dependency management flow
  • strict conflict manager
    • I decided not to use it, because it works only in the cases when you have a limited number of dependencies, most of them are explicit and controlled by you (i.e. minimum external dependencies, or just NIH syndrome). I can expand my arguments here if needed.

This is an outstanding guide.