README.md

Coming from puppet I really got used to the hierarchical data structure hiera provides. It's easy to use, yet pretty powerful when used in combination with the module_data puppet module, which allows you to put your hiera-data in the module directory itself, while still being able to globally override stuff from the main hiera source.

The approach I present here allows you to do something similar with salt and it's pillar data by using Jinja2 only. No additional modules whatsoever are required for this to work.

Configuration data is being split up into four levels:

• state defaults
• filtered state defaults
• pillar data
• filtered pillar data

Each level overrides its predecessor, allowing you to easily set and override defaults for different envoronments, platforms, etc...

State defaults are set in settings.yml in the state directory. Common defaults are set under the config key. config is a simple python dict which may be of arbitrary depth and complexity.

The second key in settings.yml is lookup. lookup is a list of dicts of dicts. Elements of the list are always processed in FIFO order, thus also being a hierarchy of its own:

lookup:
  - [grain_A1]:
      [grain_A1_value_1]:
        [config_key_1]: value_1
        [config_key_2]: value_2
        ...
      [grain_A1_value_2]:
        ...
    [grain_An]:
      [grain_An_value_1]:
      ...
      
  - [grain_B1]:
      ...

See state_foo_settings.yml and pillar_foo_init.sls for an example. The resulting data would look like this:

  # On a non-Debian minion:
  
  settings:
    a: 1
    b: 99
    c:
      ca: 31
      cb: 32
    d: [ 5,6 ]
    
  # On a Debian minion:
  
  settings:
    a: 99
    b: 99
    c:
      ca: 31
    d: [ 5,6 ]

There are only four rules for merging:

• merging is always done in FIFO order
• dicts get recursively merged
• everything else gets overridden
• null values delete keys

So when you want to unset a value/remove a key, simply set it to null and it will be completely removed from the resulting dict.

Merged settings can be loaded via Jinja2 in a single line, similar to the map.jinja approach which is currently being used for salt formulas:

{%- from 'foo/settings.sls' import settings with context %}

{%- set some_var = settings.get('some_key', 'default_value') %}

To use this in your state, all you have to do is:

• provide a /srv/salt/YOUR_STATE/settings.yml containing your defaults
• copy contents from state_foo_settings.sls below into /srv/salt/YOUR_STATE/settings.sls
• copy contents from state_macro.sls below into /srv/salt/macro.sls
• load the settings in your state files as shown above

If you do not want to ship the macros apart from you module, macro.sls can also be merged with your settings.sls.

NOTICE:

Pillar data follows the same rules settings.yml does, except that you of course have to put all your data into a dict whose key is named after your module. Just like you do without this merging stuff.

pillar_foo_init.sls

# vim: set filetype=yaml expandtab tabstop=2 shiftwidth=2 textwidth=0:

foo:
  config:
    b: 99
    d: [ 5,6 ]
    
  lookup:
    - os_family:
        Debian:
          c:
            cb: null

state_foo_init.sls

# vim: set filetype=yaml expandtab tabstop=2 shiftwidth=2 textwidth=0:

{%- from 'foo/settings.sls' import settings with context %}

state_foo_settings.sls

# vim: set filetype=jinja expandtab tabstop=2 shiftwidth=2 textwidth=0:

{%- from 'macros.sls' import load_settings with context %}

{%- set settings = { } %}
{% do load_settings('foo', settings) %}

state_foo_settings.yml

# vim: set filetype=yaml expandtab tabstop=2 shiftwidth=2 textwidth=0:

config:
  a: 1
  b: 2
  c:
    ca: 31
    cb: 32
  d: [ 4,5 ]
  
lookup:
  - os_family:
      Debian:
        a: 99

state_macro.sls

# vim: set filetype=jinja expandtab tabstop=2 shiftwidth=2 textwidth=0:

{%- macro deep_merge(a, b): %}
  {%- for k,v in b.items(): %}
    {%- if ((v is not defined) or (v == None)): %}
      {%- do a.pop(k) %}
    {%- else: %}
      {%- if v is mapping: %}
        {%- if a[k] is not mapping: %}
          {%- do a.update({ k: { } }) %}
        {%- endif %}
        {%- do deep_merge(a[k], v) %}
      {%- else: %}
        {%- do a.update({ k: v }) %}
      {%- endif %}
    {% endif %}
  {%- endfor %}
{%- endmacro %}

{%- macro load_settings(name, settings): %}
  {%- import_yaml name + '/defaults.yml' as defaults %}
  {%- set p = salt['pillar.get']('{{name}}', { }) %}

  {# merge default config #}
  {%- do deep_merge(settings, defaults.get('config', { })) %}

  {# merge default lookups #}
  {%- for l in defaults.get('lookup', [ ]): %}
    {%- for k,v in l.items(): %}
      {%- set m = salt['grains.filter_by'](v, grain=k) %}
      {%- if m is mapping: %}
        {%- do deep_merge(settings, m) %}
      {%- endif %}
    {%- endfor %}
  {%- endfor %}

  {# merge pillar config #}
  {%- do deep_merge(settings, p.get('config', { })) %}

  {# merge pillar lookups #}
  {%- for k,v in p.get('lookup', { }).items(): %}
    {%- set m = salt['grains.filter_by'](v, grain=k) %}
    {%- if m is mapping: %}
      {%- do deep_merge(settings, m) %}
    {%- endif %}
  {%- endfor %}
{%- endmacro %}