A brain dump on how OpenStack template loaders work in charm-helpers.

openstack-template-loaders.txt

  **OpenStack Releases and template loading**

  When the object is instantiated, it is associated with a specific OS
  release. This dictates how the template loader will be constructed.

  The constructed loader attempts to load the template from several places
  in the following order:
  - from the most recent OS release-specific template dir (if one exists)
  - the base templates_dir
  - a template directory shipped in the charm with this helper file.

  For the example above, '/tmp/templates' contains the following structure::

  /tmp/templates/nova.conf
  /tmp/templates/api-paste.ini
  /tmp/templates/grizzly/api-paste.ini
  /tmp/templates/havana/api-paste.ini

  Since it was registered with the grizzly release, it first seraches
  the grizzly directory for nova.conf, then the templates dir.

  When writing api-paste.ini, it will find the template in the grizzly
  directory.

  If the object were created with folsom, it would fall back to the
  base templates dir for its api-paste.ini template.

  This system should help manage changes in config files through
  openstack releases, allowing charms to fall back to the most recently
  updated config template for a given release

  The haproxy.conf, since it is not shipped in the templates dir, will
  be loaded from the module directory's template directory, eg
  $CHARM/hooks/charmhelpers/contrib/openstack/templates. This allows
  us to ship common templates (haproxy, apache) with the helpers.

  **Context generators**

  Context generators are used to generate template contexts during hook
  execution. Doing so may require inspecting service relations, charm
  config, etc. When registered, a config file is associated with a list
  of generators. When a template is rendered and written, all context
  generates are called in a chain to generate the context dictionary
  passed to the jinja2 template. See context.py for more info.
  """

How it works with ChoiceLoader:

- a list of loaders is formed:
    - template directory loader specified in a charm is used first;
    - template directory from charm-helpers is used after that.
- the list is formed as follows:
    - charm-helpers' template directory loader is added first;
    - then individual loaders are added for each release at the beginning of the loader list:
    -   tmpl_dirs = [(rel, os.path.join(templates_dir, rel))
    -   for rel in six.itervalues(OPENSTACK_CODENAMES)]
    - ...
  loaders = [FileSystemLoader(templates_dir)] # this has a side effect of openstack dirs being inspected several times
    -   for rel, tmpl_dir in tmpl_dirs:
    -       if os.path.isdir(tmpl_dir):
          loaders.insert(0, FileSystemLoader(tmpl_dir))
    -       if rel == os_release:
          break
    - as inserts are done to the beginning of the list, older OpenStack releases will be of smaller priority:
        - OPENSTACK_CODENAMES = OrderedDict([
        -   ('2011.2', 'diablo'),
  ('2012.1', 'essex'),
...
  ('2017.2', 'pike'),
  ('2018.1', 'queens'),
        - as a result, if the newest release defines a template, it will be used as its FileSystemLoader is first in the list.
- all of the loaders are FileSystemLoaders;
    - FileSystemLoader does take subdirectories into account:
        - https://stackoverflow.com/a/9644828
        - https://github.com/pallets/jinja/blob/2.10/jinja2/loaders.py#L189-L202
- ChoiceLoader goes through the list - if one loader was not sufficient it will use the next one until it runs of loaders
- included templates are "templates" (if it's not obvious) and need to be loadable by loaders provided by the environment you use
    - http://jinja.pocoo.org/docs/2.10/api/#high-level-api
        - "The core component of Jinja is the Environment. It contains important shared variables like configuration, filters, tests, globals and others."
        - http://jinja.pocoo.org/docs/2.10/api/#jinja2.Environment.get_template
            - "get_template Load a template from the loader. If a loader is configured this method asks the loader for the template and returns a Template. If the parent parameter is not None, "
    - http://jinja.pocoo.org/docs/2.10/templates/#include
        - "Included templates have access to the variables of the active context by default. For more details about context behavior of imports and includes, see Import Context Behavior." (the default behavior may be changed)
    - therefore, sections available in charm-helpers are perfectly include-able in other templates as the last loader that handles charm-helper-specific templates will do the job.
- basename is taken from a template file so you cannot put files under a directory in a charm
    -       def render(self, config_file):
          if config_file not in self.templates:
               log('Config not registered: %s' % config_file, level=ERROR)
               raise OSConfigException
    -                ctxt = self.templates[config_file].context()
    -                _tmpl = os.path.basename(config_file)
    -                try:
               template = self._get_template(_tmpl)

jinja2 has a ChoiceLoader: "This loader works like the PrefixLoader just that no prefix is specified. If a template could not be found by one loader the next one is tried."
http://code.nabla.net/doc/jinja2/api/jinja2/loaders/jinja2.loaders.ChoiceLoader.html

class PrefixLoader(BaseLoader):
  """A loader that is passed a dict of loaders where each loader is bound to a prefix. The prefix is delimited from the template by a slash per default, which can be changed by setting the `delimiter` argument to something else::

get_loader returns:
https://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel/view/head:/charmhelpers/contrib/openstack/templating.py#L42
  :returns: jinja2.ChoiceLoader constructed with a list of
  jinja2.FilesystemLoaders, ordered in descending
  order by OpenStack release.
...
         # the bottom contains tempaltes_dir and possibly a common templates dir
         # shipped with the helper.
         loaders = [FileSystemLoader(templates_dir)]
         helper_templates = os.path.join(os.path.dirname(__file__), 'templates')
         if os.path.isdir(helper_templates):
             loaders.append(FileSystemLoader(helper_templates))

         for rel, tmpl_dir in tmpl_dirs:
             if os.path.isdir(tmpl_dir):
                 loaders.insert(0, FileSystemLoader(tmpl_dir))
             if rel == os_release:
                 break
         # demote this log to the lowest level; we don't really need to see these
              # lots in production even when debugging.
         log('Creating choice loader with dirs: %s' %
             [l.searchpath for l in loaders], level=TRACE)
         return ChoiceLoader(loaders)