collecting-and-merging-variables-in-3.10.x-and-3.12.x.org.org

[#A] IRC: Collecting and merging variables in 3.10.x and 3.12.x

variablesmatching_as_data() is not able to find variables defined within the same bundle in 3.10.3. One bundle must define the variables, then subsequently, from another bundle, variablesmatching_as_data() can be used to consolidate them (note this is not necessary on 3.12.x).

bundle agent cmerge from the stdlib can possibly be used as a workaround until getting to a newer version of the binaries where variablesmatching_as_data can be used.

Note, there is one significant difference between cmerge and variablesmatching_as_data. cmerge merges all variables into one data container, so if you have conflicting keys, in the variables being merged, then the last one merged will win. variablesmatching_as_data on the other-hand merges into a single JSON object, but the name of each variable is used as the key for that variables data, so cases where the same key are used in different variables that are collected together will not result in one key being over-written.

This policy illustrates the differences.

body file control
{
  # Things used from stdlib 
    # - bundle agent cmerge
    # - bundle edit_line insert_lines
  inputs => { "$(sys.libdir)/stdlib.cf" };
}
bundle agent init
{
  files:
    "/var/cfengine/billing/$(first.peers)/$(first.year)-$(first.month)"
      edit_line => insert_lines( '{ "conflict": "$(first.peers)", "unique-$(first.peers)": "value-$(first.peers)" }' ),
      create => "true";
}
bundle agent first
{
  vars:
    "peers" slist => { "one-zap", "two.paz" };
    "year" string => strftime( localtime, "%Y", now() );
    "month" string => strftime( localtime, "%m", now() );

  "cmap_peers[$(peers)]" string => canonify( $(peers) );

  "invoices_$(cmap_peers[$(peers)])"
    meta => { "invoice" },
    data => readdata("/var/cfengine/billing/$(peers)/$(year)-$(month)", "JSON"),
    if => fileexists( "/var/cfengine/billing/$(peers)/$(year)-$(month)" );

  "invoice_vars"
    slist => variablesmatching( ".*", "invoice" );

  "unified_invoices"
    data => variablesmatching_as_data( ".*",       # Variables with any name, in any bundle, in any namespace
                                       "invoice"); # Tagged with invoice
  "s_unified_invoices"
    #string => format( "%s", unified_invoices );
    string => string_mustache( "{{%-top-}}", unified_invoices );

  methods:
    "" usebundle => cmerge( "invoices", @(invoice_vars) );

}
bundle agent second
{
    vars:
      "later_unified_invoices"
        data => variablesmatching_as_data( ".*",       # Variables with any name, in any bundle, in any namespace
                                       "invoice"); # Tagged with invoice
      "s_later_unified_invoices"
      #string => format( "%s", unified_invoices );
        string => string_mustache( "{{%-top-}}", later_unified_invoices );


   reports:
    "CFEngine $(sys.cf_version)";
    "first.unified_invoices exists" if => isvariable( "first.unified_invoices" );
    "s_unified_invoices =  '$(first.unified_invoices)')";
    "second.s_unified_invoices =  '$(s_later_unified_invoices)')";
    "cmerge '$(cmerge.invoices_str)'";
    
}
bundle agent main
{
    methods:
      "first";
      "second"; 
}

[root@hub ~]# rm -rf /var/cfengine/billing/
[root@hub ~]# cf-agent -KIf ./t.cf -b init
    info: Using command line specified bundlesequence
    info: Created file '/var/cfengine/billing/one-zap/2019-07', mode 0600
    info: Edit file '/var/cfengine/billing/one-zap/2019-07'
    info: Created file '/var/cfengine/billing/two.paz/2019-07', mode 0600
    info: Edit file '/var/cfengine/billing/two.paz/2019-07'
[root@hub ~]# cf-agent -KIf ./t.cf 
R: CFEngine 3.10.3
R: first.unified_invoices exists
R: second.s_unified_invoices =  '{
  "default:first.invoices_one_zap": {
    "conflict": "one-zap",
    "unique-one-zap": "value-one-zap"
  },
  "default:first.invoices_two_paz": {
    "conflict": "two.paz",
    "unique-two.paz": "value-two.paz"
  }
}')
R: cmerge '{"conflict":"one-zap","unique-one-zap":"value-one-zap","unique-two.paz":"value-two.paz"}'