light vs heavyweight ACD

gistfile1.txt

What is going into ACD
======================

* ansible-base
* community.general
* Ansible maintained collections
* Various vendor maintained collections
* Other community maintained collections


Several Scenarions
==================

Note: I'm assuming ansible-base will have its own release process and upload its own tarball no matter what collections or ACD do


ACD Lightweight
---------------

"A Snaphost of Various Upstreams"

* ACD made a time-based release.
* Downstreams (Linux distros, mainly)
  * package ansible-base.
  * Downstreams also package the collections in ACD, likely having separate binary rpms to install each collection whether or not they use the single ACD tarball to build a single srpm or multiple srpms with individual upstreams
* Other upstreams make new releases.
* ACD makes a new release that pulls in the changes in the other upstreams
  * The new release contains the latest versions of the other upstreams
    * Other upstreams are in charge of their own release cycles, deprecation periods, etc.


ACD heavyweight process
-----------------------

"ACD is just Ansible by another name"

* ansible-base makes a new major release
* ACD asks the other upstreams to test their release with the new ansible-base release
* ACD produces some number of pre-releases on a schedule
* ACD coordinates a go/no-go with the other upstreams on a schedule
* ACD makes a new major release.
* Downstreams make new packages (Note that depending on the downstream, they may have started earlier by updating the ansible-base package or they may have waited until ACD is released to do so)
* ansible-base makes a new minor release
* ACD coordinates with other upstreams to pull in their latest backwards compatible release
  * This could be implemented as:
    (a) Rely on semantic versioning and automatically pull in the latest compatible
    (b) Explicit information from other upstreams.  Default to shipping the version shipped with the last release
* ACD makes a new minor release
* [More minor releases]

*** There are a multitude of scenarios between these two ***


How the ACD package gets made
=============================


Dual Package Scenario
---------------------

* ansible-base is its own package, created by the core team
* ACD constructs a python namespace package that will install into site-packages/ansible_collections with all of the dependent collections.
  * grab sources for dependent collections
  * place them into an ansible_collections directory with the appropriate filesystem structure
  * Write a setup.py file that installs these on disk as a namespace package.
  * setup.py has a runtime dependency on the ansible-base package
  * Create the sdist of this
  * Upload the sdist to pypi

  
Single Package Scenario
-----------------------

* ACD grabs the ansible-base package and bundles it.
* ACD constructs the site-packages/ansible_collections portion the same as the dual package scenario

[Note: I don't believe this will work out well.  Advantage: gives ACD control to change ansible-base if needed.  Disadvantage: pretty much everything else.  More work, more complex, repeated work that's already been done]

  
Separate Packaging Scenario, ACD does all the work
--------------------------------------------------

* ansible-base is its own package, created by the core team
* Packages of other upstreams
  * Grab the sources for all dependent collections
  * Package each of those so they create an ansible_collections namespace and then place their collections inside of that
  * Create a setup.py that has a runtime dep on the ansible-base package
  * Create the sdist of this
  * Upload the sdist to pypi
* ACD package
  * ACD constructs the namespace package that will install into site-packages.
  * ACD constructs a setup.py for itself.
  * Adds runtime dependencies on ansible-base and the dependent collections which we've just packaged
  * Open question: How loose are the versions of the deps?
  * Create the sdist of this
  * Upload the sdist to pypi

  
Separate Packaging Scenario, upstreams do the work
--------------------------------------------------

* ansible-base is its own package, created by the core team
* Other upstreams use tooling we write to package and upload their collections to pypi
  * (See steps from Separate Packaging, ACD does the work scenario to see what the tooling will need to do)
* Other upstreams tell us what versions to include in the next ACD release.
* ACD package
  * ACD constructs the namespace package that will install into site-packages.
  * ACD constructs a setup.py for itself.
  * Adds runtime dependencies on ansible-base and the dependent collections using versions that other upstreams provided to us
  * Create the sdist of this
  * Upload the sdist to pypi

I think it would be good if ansible-galaxy collection install would point out that the newly installed/upgraded collection(s) hide other installed copies of this collection(s). Otherwise users can easily forget that a collection contained in ACD they manually upgraded once now forever stays at that version, even though they installed newer verisons of ACD (because the newer version coming with ACD is hidden).

Example from the Angular CLI (which prints this at every invocation though, which I think is too much): Your global Angular CLI version (9.0.1) is greater than your local version (8.3.23). The local Angular CLI version is used. / To disable this warning use "ng config -g cli.warnings.versionMismatch false".

For everyone not simply installing ACD and nothing else manually, the approach lined out by @geerlingguy probably makes more sense as it avoids this problem.

I am +1 "Separate Packaging Scenario, upstreams do the work" and "ACD Lightweight"

What @omgjlk wrote seems to be a pragmatic approach.

The mechanism used shouldn't prevent end users from overriding the constraint and installing a newer version of a collection, should they take direct action to do so. This can allow users of a collection to get access to newer versions of things between releases of ACD, knowing that they are forging their own path. The next time they perform an ACD install, they should be made aware if they already have a collection version installed that goes beyond the constraint of upstream ACD.

Fully agree, users are (read: should be) able to go their own path with collections and use a specific version.

I see ansible being a critical key part in many customers' infrastructures. They want to have control. In practice, users may use a ACD in a previous version and update the collections just to "test" (read: do their homework) if everything still works as expected.

Short; ACD should use latest version of a collection. Expect users to leave path and guide them.

I'm + 1 to the following:

1. ACD Lightweight
2. Not single package scenario.
   And maybe not "Separate Packaging Scenario, upstreams do the work."
   Other upstreams tell us what versions to include in the next ACD release.
   If they don't or stop doing the work, what do we need to include? (if i understand upstreams right in this context)
3. Latest version of collections by default. At least, because when we change ansible-base in the devel, i suppose that it triggers tests in collections of also the latest state (not 100% sure).
   Plus good documentation how to prepare for upgrade / replace the collection to a desired version if something unexpected happened.