woe and anguish

config.sls

# vim: sts=2 ts=2 sw=2
{% from "vsftpd/map.jinja" import vsftpd with context %}

{# mjinks 2015/09/15: isn't this bass-ackwards? i've added include
for this file to the main vsftpd.sls and that works. so taking
this one out.

...checked, at least one other more elaborate formula (users) does
it this way. vindicated.

include:
  - vsftpd
#}

vsftpd_config:
  file.managed:
    - name: {{ vsftpd.vsftpd_config }}
    - source: {{ vsftpd.vsftpd_config_src }}
    - template: jinja
    - user: root
    - mode: 644
    - watch_in:
      - service: {{ vsftpd.service }}

currently giving 16 sets of this

$ sudo salt 'ftp*' state.sls vsftpd test=True
[...]
     Started: 14:28:28.431515
    Duration: 30.651 ms
     Changes:   
----------
          ID: vsftpd_user_conf_sullivro
    Function: file.managed
        Name: /etc/vsftpd/vsftpd_user_conf/sullivro
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1560, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/states/file.py", line 1423, in managed
                  **kwargs
                File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 3142, in check_managed_changes
                  **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/modules/file.py", line 2841, in get_managed
                  **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 121, in render_tmpl
                  output = render_str(tmplstr, context, tmplpath)
                File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 346, in render_jinja_tmpl
                  trace=tracestr)
              SaltRenderError: Jinja error: global name 'l_vsftpd_user_config' is not defined
              Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 306, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 894, in render
                  return self.environment.handle_exception(exc_info, True)
                File "<template>", line 39, in top-level template code
              NameError: global name 'l_vsftpd_user_config' is not defined
              
              ; line 39
              
              ---
              [...]
                {%- set value = config_dict.get(keyword, default) -%}
                {%- if value is string or value is number -%}
              {{ keyword }}={{ value }}
                {%- else -%}
              {%- for single_value in value -%}
              {{ keyword }}={{ single_value }}    <======================
              {% endfor -%}
                {%- endif -%}
              {%- endmacro -%}
              
              {#- macros for render option according to present -#}
              [...]
              ---

excerpt from the pillar we're trying to draw from

vsftpd_user_config:

  billingplatform:
    anonymous_enable: 'NO'
    chroot_list_enable: 'YES'
    chroot_list_file: '/etc/vsftpd/vsftpd.chroot_list'
    chroot_local_user: 'YES'
    connect_from_port_20: 'YES'
[...]
    user_config: 'YES'

  brighthouse:
    anonymous_enable: 'NO'
    chroot_list_enable: 'YES'
[...and so on]

init.sls

# vim: sts=2 ts=2 sw=2

{% from "vsftpd/map.jinja" import vsftpd with context %}

vsftpd:
  {% if vsftpd.server is defined %}
  pkg.installed:
    - name: {{ vsftpd.server }}
  {% endif %}
  service.running:
    - enable: True
    - name: {{ vsftpd.service }}
  {% if vsftpd.server is defined %}
    - require:
      - pkg: {{ vsftpd.server }}
  {% endif %}

include:
  - vsftpd.config
  - vsftpd.vsftpd_user_config

single states show success, but they lie

While trying to narrow this down, just learned that if I call one of the problem states individually, they come back 
"Result: True", but the file doesn't get changed. If I make delibarate edits to the file, salt doesn't notice.

If I *remove* the file, salt does notice, but all it does is touch it to a blank file, no contents get written.



$ sudo salt 'ftp*' state.single file.managed  /etc/vsftpd/vsftpd_user_conf/telarix
ftp01.chi5.prlss.int:
----------
          ID: /etc/vsftpd/vsftpd_user_conf/telarix
    Function: file.managed
      Result: True
     Comment: File /etc/vsftpd/vsftpd_user_conf/telarix exists with proper permissions. No changes made.
     Started: 16:31:10.215160
    Duration: 6.088 ms
     Changes:   

Summary
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1

vsftpd_user_conf-TMPL

#!yaml_jinja
#...or? #!jinja|yaml

{#
mjinks 2015/09/21:
this is a template for per-user configs in:
   /etc/vsftpd/vsftpd_user_config/[username]
it's derived from the template for the main vsftpd.conf file,
downloaded from public archive.

ISSUE: upstream, we wave the costructions:
   option_default_uncommented('fieldname', '/path/to/fieldname')
...and:
   option('fieldname', '/path/to/fieldname')

...which seem to imply the same thing: "don't comment [fieldname]".
but, they're not rendered that way. worse, their rendering is
inconsistent from the top of the template to the bottom, and for
the life of me i can't figure out what causes the change.

nonetheless (or maybe for that reason), i'm copying verbatim
all the macro settings from that template and using them again
here. at worst, it's consistent. at best, we'll gain some
insight into why it's all set up that way, and maybe fix the
bumps.  #}

{#- present in vsftpd_config and known in actual file options -#}
{%- set processed_options = [] -%}

{#- generic renderer used for vsftpd known options, -#}
{#- and unknown options -#}

{%- macro render_option(keyword, default, config_dict=vsftpd_user_config) -%}
  {%- set value = config_dict.get(keyword, default) -%}
  {%- if value is string or value is number -%}
{{ keyword }}={{ value }}
  {%- else -%}
{%- for single_value in value -%}
{{ keyword }}={{ single_value }}
{% endfor -%}
  {%- endif -%}
{%- endmacro -%}

{#- macros for render option according to present -#}
{%- macro option_impl(keyword, default, present) -%}
  {%- if present -%}
    {%- do processed_options.append(keyword) -%}
    {%- set prefix='' -%}
  {%- else -%}
    {%- set prefix='#' -%}
  {%- endif -%}
  {#- add prefix to keyword -#}
  {%- set keyword = prefix ~ keyword -%} 
{{ render_option(keyword, default) }}
{%- endmacro -%}

{#- macros for render option commented by default -#}
{%- macro option(keyword, default, present) -%}
{{ option_impl(keyword, default, keyword in vsftpd_user_config) }}
{%- endmacro -%}
{#- macros for render option uncommented by default -#}
{%- macro option_default_uncommented(keyword, default, present) -%}
{{ option_impl(keyword, default, True) }}
{%- endmacro -%}
{# END CRIB - mjinks #}
# THIS FILE IS MANAGED BY SALT. changes here will be lost when
# salt brings this host up to high state.

# aside: mjinks doesn't like "uncomment this to allow...". leads
# to confusion and omission. we say what we mean. so while upstream
# sometimes says "uncomment this to allow...", we uncomment
# everything, and set the corresponding 'NO' or 'YES' to get the
# behavior we want.
#
# because this is vsftpd, that can still lead to unexpected behavior,
# but there's only so much we can do...  

# mjinks 2015/09/22: for ease of edit this [file from] template
# removes a lot of the comments present in original. for reference
# see salt server or, if i'm being good, wiki.

# mjinks 2015/09/22: this is a per-user config file. isn't anonymous
# already past by the time we get to the point of reading this option?
{{ option_default_uncommented('anonymous_enable', 'NO') }}
# mjinks: as above: isn't this superfluous?
{{ option_default_uncommented('local_enable', 'YES') }}
#
# Uncomment this to enable any form of FTP write command.
# was:
# write_enable=YES
{{ option_default_uncommented('write_enable', 'YES') }}
# ...now continue below........
#
{{ option_default_uncommented('local_umask', '002') }}
{{ option_default_uncommented('dirmessage_enable', 'YES') }}
{{ option_default_uncommented('xferlog_enable', 'YES') }}
{{ option_default_uncommented('dual_log_enable', 'YES') }}
{{ option_default_uncommented('pasv_enable', 'YES') }}
{{ option_default_uncommented('pasv_promiscuous', 'YES') }}
{{ option_default_uncommented('pasv_min_port', '50000') }}
{{ option_default_uncommented('pasv_max_port', '55000') }}
{{ option_default_uncommented('connect_from_port_20', 'YES') }}
{{ option_default_uncommented('vsftpd_log_file', '/var/log/vsftpd.log') }}
{{ option_default_uncommented('xferlog_file', '/var/log/xferlog.log') }}
{{ option_default_uncommented('xferlog_std_format', 'NO') }}
{{ option_default_uncommented('nopriv_user', 'ftpsecure') }}
{{ option_default_uncommented('ftpd_banner', '"Peerless Network Download Server"') }}
{{ option_default_uncommented('chroot_list_enable', 'YES') }}
{{ option_default_uncommented('chroot_list_file', '/etc/vsftpd/vsftpd.chroot_list') }}
# typically custom changes start here
{{ option_default_uncommented('chroot_local_user', 'NO') }}
#allow_writeable_chroot=YES
{{ option_default_uncommented('hide_ids', 'YES') }}
{{ option_default_uncommented('tcp_wrappers', 'NO') }}
{{ option_default_uncommented('use_localtime', 'YES') }}
{{ option_default_uncommented('userlist_deny', 'NO') }}
{{ option_default_uncommented('userlist_enable', 'YES') }}
{{ option_default_uncommented('userlist_file', '/etc/vsftpd/vsftpd.user_list') }}
{{ option_default_uncommented('user_config_dir', '/etc/vsftpd/vsftpd_user_conf') }}

# vim:ft=yaml:

vsftpd_user_config.sls

#!yaml_jinja
#...or? #!jinja|yaml

{# this is cargo-culted from salt/users/init.sls, with trims and adjustmants
#}
{# at this writing, at command line, calls to the users pillar give back
   populated fields like we want, matching up with the ones in the template
   file. but high state doesn't.
#}

{# this adjusted from a post on google groups:
  https://groups.google.com/forum/#!msg/salt-users/Rd8sjNhblgQ/rBBHPHqeAwAJ
...which links to:
  https://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html#salt.states.file.managed
#}
{% from "vsftpd/map.jinja" import vsftpd with context %}

{% for name, conf in pillar.get('vsftpd_user_config', {}).iteritems() if conf.absent is not defined or not conf.absent %}
vsftpd_user_conf_{{ name }}:
  file.managed:
    - name: /etc/vsftpd/vsftpd_user_conf/{{ name }}
    - source: salt://vsftpd/files/vsftpd_user_conf-TMPL
    - template: jinja
    - context:
        vsftpd_config: {{ conf | yaml }}
    - user: root
    - mode: 644
    - watch_in:
      - service: {{ vsftpd.service }}
{% endfor %}


# vim: sts=2 ts=2 sw=2

weird: the service salts fine on its own

[the state also whines about the serice it manages...]

----------
          ID: vsftpd
    Function: service.running
      Result: False
     Comment: One or more requisite failed: vsftpd.vsftpd_user_config.vsftpd_user_conf_xcentrix, vsftpd.vsftpd_user_config.vsftpd_user_conf_CBFiles, vsftpd.vsftpd_user_config.vsftpd_user_conf_telarix, vsftpd.vsftpd_user_config.vsftpd_user_conf_comcast, vsftpd.vsftpd_user_config.vsftpd_user_conf_brighthouse, vsftpd.vsftpd_user_config.vsftpd_user_conf_deactivated-ATTFiles, vsftpd.vsftpd_user_config.vsftpd_user_conf_billingplatform, vsftpd.vsftpd_user_config.vsftpd_user_conf_intelepeer, vsftpd.vsftpd_user_config.vsftpd_user_conf_twtelecom, vsftpd.vsftpd_user_config.vsftpd_user_conf_sullivro, vsftpd.vsftpd_user_config.vsftpd_user_conf_uscc, vsftpd.vsftpd_user_config.vsftpd_user_conf_deltacom, vsftpd.vsftpd_user_config.vsftpd_user_conf_contact, vsftpd.vsftpd_user_config.vsftpd_user_conf_netnumber, vsftpd.vsftpd_user_config.vsftpd_user_conf_pricing, vsftpd.vsftpd_user_config.vsftpd_user_conf_vndrpaetec
     Started: 
    Duration: 
     Changes:   

[...but, when called on its own, the service checks out fine.]

$ sudo salt 'ftp*' state.single service.running vsftpd test=True 
ftp01.chi5.prlss.int:
----------
          ID: vsftpd
    Function: service.running
      Result: True
     Comment: The service vsftpd is already running
     Started: 14:37:00.748089
    Duration: 20.082 ms
     Changes: