-
-
Save abadger/1f14caded92117fbd3036842c875701c to your computer and use it in GitHub Desktop.
| What is going into ACD | |
| ====================== | |
| * ansible-base | |
| * community.general | |
| * Ansible maintained collections | |
| * Various vendor maintained collections | |
| * Other community maintained collections | |
| Several Scenarions | |
| ================== | |
| Note: I'm assuming ansible-base will have its own release process and upload its own tarball no matter what collections or ACD do | |
| ACD Lightweight | |
| --------------- | |
| "A Snaphost of Various Upstreams" | |
| * ACD made a time-based release. | |
| * Downstreams (Linux distros, mainly) | |
| * package ansible-base. | |
| * Downstreams also package the collections in ACD, likely having separate binary rpms to install each collection whether or not they use the single ACD tarball to build a single srpm or multiple srpms with individual upstreams | |
| * Other upstreams make new releases. | |
| * ACD makes a new release that pulls in the changes in the other upstreams | |
| * The new release contains the latest versions of the other upstreams | |
| * Other upstreams are in charge of their own release cycles, deprecation periods, etc. | |
| ACD heavyweight process | |
| ----------------------- | |
| "ACD is just Ansible by another name" | |
| * ansible-base makes a new major release | |
| * ACD asks the other upstreams to test their release with the new ansible-base release | |
| * ACD produces some number of pre-releases on a schedule | |
| * ACD coordinates a go/no-go with the other upstreams on a schedule | |
| * ACD makes a new major release. | |
| * Downstreams make new packages (Note that depending on the downstream, they may have started earlier by updating the ansible-base package or they may have waited until ACD is released to do so) | |
| * ansible-base makes a new minor release | |
| * ACD coordinates with other upstreams to pull in their latest backwards compatible release | |
| * This could be implemented as: | |
| (a) Rely on semantic versioning and automatically pull in the latest compatible | |
| (b) Explicit information from other upstreams. Default to shipping the version shipped with the last release | |
| * ACD makes a new minor release | |
| * [More minor releases] | |
| *** There are a multitude of scenarios between these two *** | |
| How the ACD package gets made | |
| ============================= | |
| Dual Package Scenario | |
| --------------------- | |
| * ansible-base is its own package, created by the core team | |
| * ACD constructs a python namespace package that will install into site-packages/ansible_collections with all of the dependent collections. | |
| * grab sources for dependent collections | |
| * place them into an ansible_collections directory with the appropriate filesystem structure | |
| * Write a setup.py file that installs these on disk as a namespace package. | |
| * setup.py has a runtime dependency on the ansible-base package | |
| * Create the sdist of this | |
| * Upload the sdist to pypi | |
| Single Package Scenario | |
| ----------------------- | |
| * ACD grabs the ansible-base package and bundles it. | |
| * ACD constructs the site-packages/ansible_collections portion the same as the dual package scenario | |
| [Note: I don't believe this will work out well. Advantage: gives ACD control to change ansible-base if needed. Disadvantage: pretty much everything else. More work, more complex, repeated work that's already been done] | |
| Separate Packaging Scenario, ACD does all the work | |
| -------------------------------------------------- | |
| * ansible-base is its own package, created by the core team | |
| * Packages of other upstreams | |
| * Grab the sources for all dependent collections | |
| * Package each of those so they create an ansible_collections namespace and then place their collections inside of that | |
| * Create a setup.py that has a runtime dep on the ansible-base package | |
| * Create the sdist of this | |
| * Upload the sdist to pypi | |
| * ACD package | |
| * ACD constructs the namespace package that will install into site-packages. | |
| * ACD constructs a setup.py for itself. | |
| * Adds runtime dependencies on ansible-base and the dependent collections which we've just packaged | |
| * Open question: How loose are the versions of the deps? | |
| * Create the sdist of this | |
| * Upload the sdist to pypi | |
| Separate Packaging Scenario, upstreams do the work | |
| -------------------------------------------------- | |
| * ansible-base is its own package, created by the core team | |
| * Other upstreams use tooling we write to package and upload their collections to pypi | |
| * (See steps from Separate Packaging, ACD does the work scenario to see what the tooling will need to do) | |
| * Other upstreams tell us what versions to include in the next ACD release. | |
| * ACD package | |
| * ACD constructs the namespace package that will install into site-packages. | |
| * ACD constructs a setup.py for itself. | |
| * Adds runtime dependencies on ansible-base and the dependent collections using versions that other upstreams provided to us | |
| * Create the sdist of this | |
| * Upload the sdist to pypi |
Dylan has confirmed that ansible/ansible will be packaged as ansible-base on (at least PyPi)
ACD heavyweight process
I don't think that's maintainable, even in the short term, especially since this is all going to be a new process (with a different group of people maintaining it). I think the starting point should be the general idea of a 'lightweight' process.
One of my main concerns as someone helping maintain some migrated modules is that we would like to start working on improving those modules' architecture (and soon), but we are also fearful that any potentially-breaking changes we make (following semver for our collection releases) could be pulled into 2.10 and make the ansible 2.9 to acd ansible 2.10 upgrade more risky for some users.
It would be nice to be able to say for ACD, 'pin 2.10 to the collection 0.9.x release series' for Kubernetes collection... but at that point, someone would have to then maintain the list of mappings for versions, and I'm not even sure how we'd document/communicate/track all of that from 2.10 to 2.11 (e.g. what if K8s collection goes from 0.9.x to 1.x to 2.x in the time between 2.10 and 2.11... that could be a very bad experience for ACD users—but on the flip side, the 1.x series might not even work with newer versions of Kubernetes if the 2.x changes had major rework to work with newer versions of K8s!
It gets complicated fast, which is why maybe saying "ACD will take the stable release version of each of these collections" is the best option.
And then, IMO, pushing people towards using ansible-base instead of ACD (that's what I'll be doing), and requiring collections explicitly as-needed.
I'd like to explain a bit how OpenStack handles picking what versions of all the deps to include in its test environments and releases. OpenStack does it all with python and pip, but Ansible could do something similar.
There is a global "constraints" file. Every dependency of OpenStack goes in there. The constraints file lists the latest version of a dependency that has been tested as part of the whole distribution. When an integration test starts, pip is fed two files. One file is all the requirements of a given package, like say OpenStack Nova. Additionally it's fed the constraints file. The Requirements file can be open ended, could even list requirements without versions. But the constraints file will put a cap on how new of a thing could be installed. The net result is an installation with all the tested versions in it.
Periodically there will be attempts at raising constraints. If a particular package, say Nova, now requires something newer than what's in constraints, a change request to the constraints file goes out and full integration tests are ran with a new higher constraint, potentially merged. This is how newer versions of dependencies are brought into the system.
I would think that ACD can work something like this. ACD has a listing of collections, and a upper boundary on what version of each collection it brings in. This is what's used at release time to create the bundle, and then tagged/branched for point releases thereafter. During development periods, as collections create newer releases, they can propose a bump to their version in the ACD listing. This would in turn trigger all the integration tests and sign off from $WHOMEVER. It should be fairly easy to bump a collection listing during development, and steadily more difficult if proposed as a point release for a stable version of ACD.
The mechanism used shouldn't prevent end users from overriding the constraint and installing a newer version of a collection, should they take direct action to do so. This can allow users of a collection to get access to newer versions of things between releases of ACD, knowing that they are forging their own path. The next time they perform an ACD install, they should be made aware if they already have a collection version installed that goes beyond the constraint of upstream ACD.
How any of this would work in the conext of Ansible and the installer tools, ¯\(ツ)/¯ , but I thought I'd throw this out as food for thought.
I think it would be good if ansible-galaxy collection install would point out that the newly installed/upgraded collection(s) hide other installed copies of this collection(s). Otherwise users can easily forget that a collection contained in ACD they manually upgraded once now forever stays at that version, even though they installed newer verisons of ACD (because the newer version coming with ACD is hidden).
Example from the Angular CLI (which prints this at every invocation though, which I think is too much): Your global Angular CLI version (9.0.1) is greater than your local version (8.3.23). The local Angular CLI version is used. / To disable this warning use "ng config -g cli.warnings.versionMismatch false".
For everyone not simply installing ACD and nothing else manually, the approach lined out by @geerlingguy probably makes more sense as it avoids this problem.
I am +1 "Separate Packaging Scenario, upstreams do the work" and "ACD Lightweight"
What @omgjlk wrote seems to be a pragmatic approach.
The mechanism used shouldn't prevent end users from overriding the constraint and installing a newer version of a collection, should they take direct action to do so. This can allow users of a collection to get access to newer versions of things between releases of ACD, knowing that they are forging their own path. The next time they perform an ACD install, they should be made aware if they already have a collection version installed that goes beyond the constraint of upstream ACD.
Fully agree, users are (read: should be) able to go their own path with collections and use a specific version.
I see ansible being a critical key part in many customers' infrastructures. They want to have control. In practice, users may use a ACD in a previous version and update the collections just to "test" (read: do their homework) if everything still works as expected.
Short; ACD should use latest version of a collection. Expect users to leave path and guide them.
I'm + 1 to the following:
- ACD Lightweight
- Not single package scenario.
And maybe not "Separate Packaging Scenario, upstreams do the work."
Other upstreams tell us what versions to include in the next ACD release.
If they don't or stop doing the work, what do we need to include? (if i understandupstreamsright in this context) - Latest version of collections by default. At least, because when we change ansible-base in the devel, i suppose that it triggers tests in collections of also the latest state (not 100% sure).
Plus good documentation how to prepare for upgrade / replace the collection to a desired version if something unexpected happened.
Nice write up, thank you.
I wonder if putting some details around the user experience under "ACD Lightweight" & "ACD Heavyweight" (ie early on in the proposal) would help people follow the proposal easier.