Explicit dependency management in sbt

readme.md

Some of these practices might be based on wrong assumptions and I'm not aware of it, so I would appreciate any feedback.

1. avoiding some dependency conflicts:

   • add sbt-explicit-dependencies to the project/plugins.sbt
   • run undeclaredCompileDependencies and add any missing explicit dependencies to libraryDependencies of each sub-project
   • (optionally) run unusedCompileDependencies and remove some obvious unused libraries. This has false positives, so ; reload; Test/compile after each change and ultimately run all tests to see that it didn't break anything
   • (optionally) add undeclaredCompileDependenciesTest to the CI pipeline, so that it will fail if you have some undeclared dependencies

2. keeping dependencies up to date and resolving conflicts:

   • install sbt-updates globally in your ~/.sbt/{0.13,1.0}/plugins/plugins.sbt
   • run dependencyUpdates and bump all non-major versions. Major versions updates should be done one by one with care and love and testing.
   • include all explicit libraryDependencies in dependencyOverrides to force their versions. This is supposed to have the same effect as applying _ force() on all libraryDependencies, but isn't ivy-specific. The point is to prevent conflict manager choosing automatically some version required by a transitive dependency instead of the one you wrote explicitly. This is done in project/Dependencies.scala

3. using sbt-assembly:

   • try running assembly for each project (starting from the independent ones) and see if there are any merge conflicts
   • if there are two different libraries that contain conflicting class files (same path, different content), use shading to rename one of them:
     • don't use .inAll because it will rename classes in both of the libraries and it will be the same situation. Instead use inLibrary or inProject
   • avoid using exclude or excludeDependencies because you may throw away some library which is needed by one of the transitive dependencies and it will fail in runtime with MethodNotFoundException or something like that
   • avoid overriding merge strategy on class files (using first/last/discard strategies), because it's the same as excluding some classes. Use merge strategy overrides only for some trivial conflicts or non-class files, e.g. to merge two .properties files with concat or filterDistinctLines strategy

4. resolving more conflicts:

   • run evicted for each project and inspect the list of automatically resolved conflicts
   • try to minimize the number of lines marked as [warn], those are conflicts with potentially binary incompatible versions
   • if some of the libraries introducing the conflict are yours (company-owned), go and update its dependencies to solve the conflict. Unfortunately, it's more often another way around: your libraries are more up to date than some external ones that you don't have access to. Anyway, consider updating those external libraries.
   • use sbt-dependency-graph installed globally to untangle the dependencies and understand the origin of the conflicts. The whatDependsOn task is very useful for that.

5. Rinse and repeat. I numbered these steps because IMO it's better to do them in this order, but after each step it might be useful to go through the previous steps again.

Some useful links:

• "What are your tips/tricks/tools for dealing with dependency hell?"

  • https://github.com/cb372/sbt-explicit-dependencies
  • good point (human factor):

    Do a better job of articulating WHY certain dependencies are used rather than dumping everything in common and then wondering why SBT is so complex.

• advices from Eugene Yokota:

  • avoid -SNAPSHOT outside of local testing
  • avoid version range
  • evicted for eviction report
  • sbt -no-colors update > update.txt
  • jrudolph/sbt-dependency-graph
  • rtimush/sbt-updates

• sbt docs:

  • library management
  • dependency management flow
  • strict conflict manager
    • I decided not to use it, because it works only in the cases when you have a limited number of dependencies, most of them are explicit and controlled by you (i.e. minimum external dependencies, or just NIH syndrome). I can expand my arguments here if needed.