These are some notes on my process when I review dependency updates for both JavaScript and Python projects at Hypothesis.
Before adding a dependency to a project, consider the impact on future maintenance. If what the dependency does for the project can be implemented with only a few lines of code, or can be implemented in an alternative way using dependencies the project already has, it may be a better choice to do that and avoid the dependency.
- Vet that the update is safe to merge. See "Vetting an update" below
- Merge the update
- After merging a batch of dependency update PRs, wait for QA build to complete, then do a quick manual test of core functionality. For the client app, I typically visit https://hypothes.is/docs/help and test creating, editing and removing an annotation. For the lms app I test creating and viewing a QA assignment.
- Deploy the update to production
Vetting an update involves checking several aspects of the change:
- Does the update cause any regressions in our application or require any changes to our code?
- Did anything change in this update which might otherwise affect the application or that developers should be aware of?
- Do I trust this update not to introduce any security/privacy/performance issues through malice or carelessness?
The primary check here is whether our automated test suite passed. For most dependencies, if our test suite passes then that provides adequate safety to merge the update.
If the tests did not pass, inspect the build logs to determine whether the issue is due to a change in the package or a flakey failure. If the failure looks flakey (eg. unrelated test failed), try triggering a rebuild. In Jenkins this can be done by going to the "Jenkins => {Project Name} => {Branch name}" section in the navbar and clicking "Build now" on the left.
Flakey tests are a significant cause of pain during development, especially as a codebase grows. If you see a test flake, it is often a good idea to raise awareness of the issue in an appropriate #eng- channel.
- Most of our dependencies use semantic versioning. Semver major updates (
2.x.x=>3.x.x) usually means that there will be a breaking change for at least some users and require closer inspection than minor or patch updates. - Inspect the changelog for the update, which Dependabot includes in the PR. See if there are any references to breaking changes which may affect us.
Code from third-party dependencies runs with the same privileges as any of our own code, and has the same access to user data. Therefore it is important to have a reasonable degree of confidence in the trustworthiness of the update. The process of vetting an update only needs to be done once if the same update is submitted by Dependabot to many projects.
Reviewing every line of every change is generally not practical. Therefore I take a few things into account:
- If the package is maintained by a trusted organization (eg. Facebook, Sentry, Babel) we may rely on their processes to ensure that all changes have been reviewed by trusted developers and that only trusted maintainers have published updates to the package
- If this is a widely used package (eg.
@babel/core, autoprefixer) maintained by a team of developers then we may rely on the development team and wider community to vet changes. - If this is a package maintained by a well-known developer, I may eyeball the diff quickly to check that nothing obviously malicious has been included. Dependabot includes a link to the diff on the PR.
- If this is a package maintained by a less well-known developer, I may eyeball the diff a little more closely to check that it seems sane
Dependabot will highlight when a package is published by a different author than the one who published previous updates. If a dependency update mentions a change of maintainer, this should be a cue to do a closer check than normal.
"Our Software Dependency Problem". Russell Cox (Go language maintainer): https://research.swtch.com/deps
- A post written by one of the Go maintainers to raise awareness of the risks that come with modern dependency management
Hypothesis staff can also find some useful notes from Sean here: https://stackoverflow.com/c/hypothesis/a/182/3