Automatic checking of required vars -- New pattern to replace https://github.com/openstack/openstack-ansible-os_keystone/blob/16dafa4718a1b1f97e4d46a2359f7b2c4cf3a015/tasks/main.yml#L20

main.yml

- name: Load required variables
  include_vars:
    file: defaults/required.yml
    name: "{{ testing | bool | ternary(omit,'required') }}"

- fail:
    msg: "{{ item.key }} is not defined"
  when:
    - "vars[item.key] is not defined"
    - not testing
  with_dict: "{{ required | default({}) }}"

This will allow:

• Removal of vars in roles to express what's required
• Load automatically testing vars (cleanup of os_ overrides)
• Ability to load other roles required variables if needed
• Automatic documentation

Another simplification topic:

Integration vars, like the repo_server_port for example, aren't required for a role like keystone, while keystone needs to have knowledge of it, if deploying on production.

For that (if deploying to production), the repo server role should, after its run, export the vars that can be used for integration purposes. It must be IN the role (or at least on the target node itself), this way you have accurate variable resolution (or at least you follow the ansible one).

Multiple things can happen when deploying keystone:

• The deployer has overriden repo_server_port with a globally scoped variable (like user_*.yml). Then this var would still be scoped for the keystone.
• The deployer has used a group/host var for the repo_server_port, scoping the port change to certain repo hosts. Then keystone hosts would be unaware of the change. In that case, the required variable must be fetched from a source of truth.
• The deployer hasn't overriden it. Then the required var could be fetched from the defaults.

The last two cases can be simplied into one: whether or not you have locally overriden it, the state is uploaded in and fetched from a source of truth.

...

Now we need to know what/when/how to upload into the source of truth.

For the "what", we can list all the integration variables which will be consumed by other roles into vars/main.yml (for this example, in of our repo role) under a pre-determined key.

For example:

source_of_truth_facts:
  - repo_server_port

This list of vars should be as small as possible, as hitting the source of truth can be expensive, but it must be exhaustive.
Maybe some creative thinking is required to reduce the amount of vars to upload into the source of truth.

At the end of a successful deploy/rerun, those vars would be re-uploaded, server by server. Running at the end make sures no operation in the role would alter the variable.

Alternatively, to prevent a series of copies into the source of truth at the end of the run, a batch operation could run per server. (the how). Or a side system could push the data to the source of truth. The advantage of the side system would make sure that plays (and not only roles) would be able to push things as well. The problem of the side system is additional moving parts, and monitoring of their state/liveness.

Conversely, a series of facts that need to be known for the deploy should happen at the beginning of the role.
For example, in the keystone case, keystone would have to list which vars need fetching for its integration. These vars can be listed in a well known key, and will hit the source of truth, if it's not defined already.

I honestly do not understand anything in Another simplification topic.

Let me clarify that bit then.

The keystone role currently requires to know repo_server_port when deploying to production. This can be generalized to other examples: glance needs to know variable X (let's say "keystone_endpoint_url") representing the service catalog url of keystone for example.

Because of the first simplification we now know what should be defined as input of the role to make it work.
These inputs are, by default, wired into openstack-ansible group vars for example. They are reduced in size so that ansible runs fast.

These vars can be overriden by a deployer on a case by case basis using:

• user_*.yml
• group vars
• host vars

We cannot say in advance where the deployer will override things, and if he overrides things.
The role should not care if the defaults were overriden or not.

If an override happens on user_*.yml, that's easy: the override would apply to all the nodes. But that's not the case with group_vars/host_vars. Let's say someone overrides keystone endpoint with a keystone group var. The glance group won't get that variable change but still he should be aware of it.

Two ways to fix that:

• Document the (limited set of) inputs of each role and be rigorous to use those inputs only when doing integration work
• Automaticly wire variables.

The whole post above was for automatic wiring of variables following a play.

Let's continue the glance integration with keystone example, with a keystone group override of keystone endpoint url.
A role var in keystone could expose what would be its role' outputs. Let's say that variable listing the "integration points" (or outputs) would be named "source_of_truth_facts". That variable would be a list, and have an item "keystone_endpoint_url".

Benefits:

• Self documenting the outputs of the role can help the generation of an expected input list of other roles. ("Was the input of my role inside other roles's output?" Please note that question can be answered automatically by a lookup)
• Each list item "source_of_truth_facts" can be templated, uploaded to a source of truth, and used by other roles.

Example, let's say we have a source of truth etcd, or simply a flat file on the deploy node.
Inside the keystone playbook + role run, we know what the "keystone_endpoint_url" value is. We check the role's "source_of_truth_facts" variable, and we loop over its items. Each item is a variable itself that can be templated and uploaded to the source of truth. (For example, templating a static file on a node deploy node or other if the source of truth is a flat file).

Then other roles which would require the keystone_endpoint_url as input could be automatically wired by checking the templated result in the source of truth. So for example, reading a fact of the deploy node before running the role.

That brings a lot of flexibility, as the source of truth can be switched to something else.

Hope it helps