build-dependencies.md

Dependencies in Reactive charms

Introduction

In the OpenStack team, and potentially elsewhere, there is a need to maintain stable as well as devel charms. With non-reactive charms, this is fairly simple.

However, with reactive charms, when a charm is rebuilt the dependencies are taken from the master branch of the various layers and interfaces, and the charm-helpers library, which is baked into layer-basic, is the latest available on PyPi.

The issue is serious: the OpenStack is unable to support stable/17.02 charms without completely manually rebuilding them based on downloading all of the layers and manually editing wheelhouse.txt files to ensure that the versions that were available at the time the stable was cut are the correct versions, along with any cherry picked patches.

The builds are starting to fail less than six months after the original freeze to stable, which means that patches cannot be cherry-picked back to stable charms.

Options

1. Abandon supporting stable charms.
2. Build version support into the dependency management system.

The OpenStack team doesn't feel it is possible to abandon maintaining stable charms as this will impact on our support offer. Therefore, it seems necessary to explore what options are available to support versioning in the build tools, and how this might be implemented.

Where are dependencies resolved in a charm build?

The dependencies in a built charm are:

• Python modules; defined in the layer's wheelhouse.txt
• layers and interfaces; defined in the layer.yaml

To take a typical example of an OpenStack API charm, the aodh charm (pronounced 'erhh'), has the following dependencies:

Charm layer:

layer.yaml:

includes: ['layer:openstack-api', 'interface:mongodb']
options:
  basic:
    use_venv: True
    include_system_packages: True
repo: https://github.com/openstack/charm-aodh
config:
  deletes:
    - verbose

no wheelhouse.txt

layer:openstack-api

layer.yaml

includes: ['layer:openstack-principle', 'interface:mysql-shared',
           'interface:rabbitmq', 'interface:keystone',
           'interface:hacluster', 'interface:openstack-ha']
repo: 'https://github.com/openstack/charm-layer-openstack-api'

no wheelhouse.txt

layer:openstack-principal

layer.yaml

includes: ['layer:openstack']
repo: 'https://github.com/openstack/charm-layer-openstack-principle'

no wheelhouse.txt

layer:openstack

layer.yaml

includes: ['layer:basic']
repo: 'https://github.com/openstack/charm-layer-openstack'

wheelhouse.txt

netifaces>=0.10.4
# charms_openstack - pull from github until first release
git+https://github.com/openstack/charms.openstack.git#egg=charms.openstack

Note: yes, the wheelhouse.txt does refer to the master branch of the github repo.

layer:basic

layer.yaml

defines:
  packages:
    type: array
    default: []
    description: Additional packages to be installed at time of bootstrap
  use_venv:
    type: boolean
    default: false
    description: >
      Install charm dependencies (wheelhouse) into a Python virtual environment
      to help avoid conflicts with other charms or libraries on the machine.
  include_system_packages:
    type: boolean
    default: false
    description: >
      If using a virtual environment, allow the venv to see Python packages
      installed at the system level.  This reduces isolation, but is necessary
      to use Python packages installed via apt-get.

wheelhouse.txt

pip>=7.0.0,<8.2.0
charmhelpers>=0.4.0,<1.0.0
charms.reactive>=0.1.0,<2.0.0

Interface files

Note that the interface files don't declare any dependencies, although they could, I believe, have a wheelhouse.txt.

Requirements

The charms for OpenStack require:

1. The ability to tag all of the dependencies at the 'versions' they were when the charm was 'cut' as a stable charm. e.g. stable/17.02
2. The ability to backport or cherry pick bug fixes back into both the charm layer (easy) and the dependencies (hard) so that a new 'stable/17.02' charm can be released with those bug fixes.

And this needs to work for layers/interfaces the OpenStack charms team doesn't control.

Some options to get the conversation going

There are a number of ways that this could be solved. Some of the ideas that have been raised at one point or another is:

• Write a 'freeze' tool, that tracks down ALL of the dependencies, builds a custom layers and interfaces directories, fix up the wheelhouse.txt to capture the current requirements. This would 'freeze' the charm by making it a really 'fat' charm layer with all the dependencies baked in. However, it would not be linked to the original repositories and thus would make it harder to cherry pick fixes into the charm.
• Add a feature to the layer.yaml file that will allow a version/tag to be added to specify what should be downloaded to the charm during build.
• Allow a dependencies.txt file to be submitted along with the charm build that would allow allow any layer's dependencies to be overridden. i.e. to override a particular charm-helpers version, or layer tag.

Our ideal solution

The ideal solution from the perspective of the OpenStack charms team is:

1. The dependencies can be discovered/set for Python modules and layers separately.
2. The dependencies are stored in a file in the charm layer and override any layers below the layer with the dependencies in it. This is to enable 'freezing' at a layer as well as at a top charm.
3. The dependency can be specified as a tag/branch for layers (or Python modules via git/bzr) so that bug fixes to stable dependencies get built into the charm.