Skip to content

Instantly share code, notes, and snippets.

@laughedelic
Last active May 14, 2024 16:55
Show Gist options
  • Save laughedelic/f16beb155ac2d258c1d06d7823d5ffc4 to your computer and use it in GitHub Desktop.
Save laughedelic/f16beb155ac2d258c1d06d7823d5ffc4 to your computer and use it in GitHub Desktop.
Explicit dependency management in sbt

Some of these practices might be based on wrong assumptions and I'm not aware of it, so I would appreciate any feedback.

  1. avoiding some dependency conflicts:

    • install sbt-explicit-dependencies globally in your ~/.sbt/{0.13,1.0}/plugins/plugins.sbt
    • run undeclaredCompileDependencies and make the obvious missing dependencies explicit by adding them to libraryDependencies of each sub-project
    • (optionally) run unusedCompileDependencies and remove some obvious unused libraries. This has false positives, so ; reload; Test/compile after each change and ultimately run all tests to see that it didn't break anything
    • (optionally) add undeclaredCompileDependenciesTest to the CI pipeline, so that it will fail if you have some undeclared dependencies
  2. keeping dependencies up to date and resolving conflicts:

    • install sbt-updates globally in your ~/.sbt/{0.13,1.0}/plugins/plugins.sbt
    • run dependencyUpdates and bump all non-major versions. Major versions updates should be done one by one with care and love and testing.
    • include all explicit libraryDependencies in dependencyOverrides to force their versions. This is supposed to have the same effect as applying force() on all libraryDependencies, but isn't ivy-specific. The point is to prevent conflict manager choosing automatically some version required by a transitive dependency instead of the one you wrote explicitly. This is done in project/Dependencies.scala
    • it might become difficult to maintain versions in libraryDependencies and dependencyOverrides in sync, so a common practice is to define values for versions of each dependency, put them in project/Versions.scala and use throughout build.sbt. This is also convenient for libraries that are split in multiple artifacts which have to have the same version.
  3. using sbt-assembly:

    • try running assembly for each project (starting from the independent ones) and see if there are any merge conflicts
    • if there are two different libraries that contain conflicting class files (same path, different content), use shading to rename one of them:
      • don't use .inAll because it will rename classes in both of the libraries and it will be the same situation. Instead use .inLibrary or .inProject
      • you might need to use .inAll for something else, e.g. you have a predefined runtime classpath with some outdated libraries and want to use newer versions, but avoid conflicts with that external classpath
    • avoid using exclude or excludeDependencies because you may throw away some library which is needed by one of the transitive dependencies and it will fail in runtime with MethodNotFoundException or something like that
    • avoid overriding merge strategy on class files (using first/last/discard strategies), because it's the same as excluding some classes. Use merge strategy overrides only for some trivial conflicts or non-class files, e.g. to merge two .properties files with concat or filterDistinctLines strategy
  4. resolving more conflicts:

    • run evicted for each project and inspect the list of automatically resolved conflicts
    • try to minimize the number of lines marked as [warn], those are conflicts with potentially binary incompatible versions
    • if some of the libraries introducing the conflict are yours (company-owned), go and update its dependencies to solve the conflict. Unfortunately, it's more often the other way around: your libraries are more up to date than some external ones that you don't have access to. Anyway, consider updating those external libraries or contribute to them if they are opensource.
    • use sbt-dependency-graph installed globally to untangle the dependencies and understand the origin of the conflicts. The whatDependsOn task is very useful for that.
  5. Rinse and repeat. I numbered these steps because IMO it's better to do them in this order, but after each step it might be useful to go through the previous steps again.

Some useful links:

@michaelahlers
Copy link

This is an outstanding guide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment