The current Go module design and implementation targets small Go projects.
Those projects consume raw unchanged third party projects, and rely blindly on the QA done by those other projects. Their only needs are to download those projects, check they’ve not been tampered with (via notaries), regularly check for updates.
third party code
↓
direct or proxied
download,
without changes
(upstream integrity checks required)
↓
use in project without checks
(blind third party trust)
Big integration projects, however, can not afford this direct pipeline.
Big integration projects can be:
- Linux distributions
- huge software projects like Kubernetes (as noted by @thockin)
There is too much third party code in play. Every member of the team can not be expected to master all the third party parts. All the third party projects may not be up to the project QA standards. Expecting all of them to deal timely with issues all the time (and thus being able to use them unchanged) is unrealistic. Continuous internet downloads make CI/CD prohibitively expensive, unreliable and unreproducible.
Therefore any big integration project will have to give up on direct internet third party code use. It will work from a curated baseline of third party code. This baseline will receive small last-mile fixes to deal with the problems encountered while integrating a large mass of third-party code, and mask the lag of getting all those fixes accepted by the original third party projects. This baseline will be broken up in individual components to allow different teams to take ownership the curation of different parts of the third party codebase. And, because all this is a huge amount of work, the organisation managing the project will want to share it with its other projects.
third party code
↓
direct or proxied
download
without changes
(upstream integrity checks required)
↓
curating
– checking,
– fixing,
– removing unneeded/broken/dead parts
(upstream integrity checks voided)
↓
curated baseline
↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓
use in project A … use in project N
(no other third party code or internet download checks permitted)
The aim of this report series is to make Go module tooling integration-friendly, helping big integration projects to constitute and manage their baselines, and consume the result in all their sub-projects.
For historical reasons baselines have been created in GOPATH/vendor mode so far. But, there is no reason for this mode to persist. All the code management wins produced by modules in the direct download case, are also desirable in the integration/baseline case.
Therefore, there is a tooling need to convert the existing GOPATH/vendor baselines of third party code into baselines of Go modules.
This document gives the general context for a a series of reports, focused at making Go modules work in integrator workflows. It’s being filled at the request of @mdempsky in golang-dev. It is informed both by the results of this golang-dev discussion, and by trying to make things work with Go module tooling as it exists today (experiment¹).
- use several goproxy sources simultaneously
- go mod split, propose loop-breaking module splits
- go mod buildrequires, list the build requirements of a set of unpacked modules
- go mod requires, list the direct requirements for using a module
- go mod index, reindex a goproxy directory
- go mod pack, pack sources as module files
- per-goproxy disabling of any notary check
- go mod discover, discover a set of unpacked modules within a directory
- go mod tidy a specific module descriptor
- go mod build, build all the binaries provided by a module
- go mod test, execute all the unit tests of a module
- go mod foreign, list the foreign content of a module
- provide foreign content extension points
- go list has too many (more than zero) side effects. The module auto-download breakage, however, is not limited to
go list - cmd/go/internal/{modfile, semver}: factor out of internal. The factoring out needs, however, are not limited to just modfile and semver
¹ The experimental code is probably horrible. That's not the point. Most of its functions are generic and should have been provided by generic Go tooling in the first place.