*Org ORG Export*.txt

# Created 2016-05-01 Sun 16:06
#+TITLE: Download files if they exist on a remote host without promises not kept
#+AUTHOR: Nick Anderson
#+SETUPFILE: ~/src/org-html-themes/setup/theme-readtheorg.setup

This example shows downloading files from a server if they exist on the remote
server. The policy promises that a local directory should contain copies of the
files that match a list of regular expressions on a remote server. As noted in
[[https://docs.cfengine.com/docs/master/reference-promise-types-files.html#local-and-remote-searches][local and remote searches]] when =depth_search= is combined with =copy_from= qa
remote search is made after the search over local base-path objects. Since we
are promising to copy files opportunistically if they match a pattern this
allows the absence of a remote file to not result in a promise not kept.

** Policy Explanation

I like to use bodies from the standard library when possible, so I make sure
that's included.

#+BEGIN_SRC cfengine3
  body file control
  {
        inputs => { "/home/nickanderson/CFEngine/masterfiles/lib/stdlib.cf" };

  }
#+END_SRC

Next I define the list of servers to download the assets from. If the promise
fails it should be attempted on each subsequent server until it is successful. I
also define the list of files I want to download. In this case I am downloading
a policy file named for the unqualified hostname.

*Note:* Since I am writing a stand-alone policy I use =bundle agent main= since
it is conveniently the default bundle if no =bundlesequence= is defined. If I
were to integrate this into a larger policy set I would use a different bundle
name.

#+BEGIN_SRC cfengine3
  bundle agent main
  {
    vars:
        "servers" slist => { "localhost" };
        "assets"
          slist => { "apacryphon_v3.*" },
          comment => "Only files on the remote server matching this list of
                      regular expressions will be downloaded.";
#+END_SRC

=/var/cfengine/deploy/= should contain a copy of any files matching one of the
regular expressions defined for =assets= that exist on the first accessible
remote =servers=. I define classes based on the results of the =files= promise
to help illustrate the behavior.

*Note:* If you are going to be downloading policy or data files to be used with
your main policy I suggest placing them *outside* =/var/cfengine/inputs=. If you
have the =cfengine_inturnal_purge_policies= class defined so that local files
that do not exist on the remote host are purged. This synchronization behavior
helps to avoid potential duplicate definition of bundle errors when files are
renamed. It also keeps things tidy which I find helpful if I need to debug
something.

#+BEGIN_SRC cfengine3
    files:
        "/var/cfengine/deploy/."
          copy_from => overlay( "/var/cfengine/srv/app", @(servers) ),
          depth_search => recurse_with_base(1), # [1]
          file_select => by_name( @(assets) ), # [2], [3]
          classes => results("namespace", "app_assets"); # [3]

       # @[1] This should probably be set to 1 If you recurse deeper than a single
       # directory, please understand that file_select is meatching on the leaf
       # file name and not the directory ath it is in.

       # @[2] Passing in as a list may have efficiency gains as oposed to
       # iterating over the files at this level.

       # @[3] The result here is based on the scope of the file iteration. For
       # example, when passing a list to file select as opposed to iterating with
       # a scalar the resulting classes will be defined based on any repair, not
       # based on any specific file. Be careful to not introduce iteration in the
       # definition of this class unless you are already iterating for the file
       # select. You would only do this if you wanted classes that identified the
       # specific files

    vars:
        "c" slist => classesmatching( "app_assets.*" );
    reports:
        "Result Classes:";
        "$(const.t)$(c)";
  }

  body copy_from overlay( path, servers )
  # @brief Copy files from `path` on first accessible `servers`.
  {
          servers => { @(servers) };
          source => "$(path)";
          compare => "digest";
          copy_backup => "false";
          collapse_destination_dir => "false";
          verify => "true";
          type_check => "false"; # We trust the upstream file tree
                                 # if a file is changed to a directory
                                 # we expect it was intentional.

     # Defaults specified for completeness.
          preserve => "false";
          trustkey => "false";
          purge => "false";
          encrypt => "false";
          check_root => "false";
          stealth => "false";
  }
#+END_SRC

I define an access rule on hosts with the =asset_server= or =policy_server=
class granting open access to =/var/cfengine/srv=.

#+BEGIN_SRC cfengine3
  bundle server assets_share
  {
    access:
      asset_server|policy_server::
        "/var/cfengine/srv/"
          admit => { "0.0.0.0/32" };
  }
#+END_SRC

*Note:* =cf-serverd= must be started with this policy on a host defining one of
the noted classes before the copy will work. Here is an example of access
summary logs from =cf-serverd= verbose output showing the controls that apply to
the shared path.

#+CAPTION: Snippet from ~cf-serverd -Fv~
#+BEGIN_EXAMPLE
verbose:  === BEGIN summary of access promises ===
verbose: 	Path: /var/cfengine/srv/
verbose: 		admit_ips: 0.0.0.0/32
verbose: 		admit_ips: 192.168.42.0/24
verbose: 		admit_hostnames: $(sys.policy_hub)/16
#+END_EXAMPLE

*Note:* I believe the agent has some optimization's with regard to making
network connections to local resources. This is due to the lack of logging I
have observe red in =cf-serverd= =vebose= and =debug= level outputs when
connecting on =localhost= vs the logging I see from connecting to an IP bound to
an interface on the executing host.

** Demo

*** Setup Environment

First we need to have some content to serve.

#+BEGIN_SRC sh
  mkdir -p /var/cfengine/srv/app
  touch /var/cfengine/srv/app/apacryphon_v1.7z
  touch /var/cfengine/srv/app/apacryphon_v3.5.7z
  touch /var/cfengine/srv/app/apacryphon_v3.9.7z
#+END_SRC

#+BEGIN_SRC sh
  tree /var/cfengine/srv
#+END_SRC

#+RESULTS: 
#+BEGIN_EXAMPLE
/var/cfengine/srv
└── app
    ├── apacryphon_v1.7z
    ├── apacryphon_v3.5.7z
    └── apacryphon_v3.9.7z

1 directory, 3 files
#+END_EXAMPLE

And let's see what assets have already been deployed to =/var/cfenigne/deploy=.

#+BEGIN_SRC sh
  tree /var/cfengine/deploy
#+END_SRC

In fact we can see =/var/cfengine/deploy= doesn't even exist yet.

#+RESULTS: 
#+BEGIN_EXAMPLE
/var/cfengine/deploy [error opening dir]

0 directories, 0 files
#+END_EXAMPLE

*** See files get copied

Now let's run the policy and then check the contents of =/var/cfengine/deploy=.

#+BEGIN_SRC sh
  /var/cfengine/bin/cf-agent -KIf /tmp/download_files_if_exist.cf
#+END_SRC

#+RESULTS: 
#+BEGIN_EXAMPLE
    info: Copying from 'localhost:/var/cfengine/srv/app/apacryphon_v3.5.7z'
    info: Copying from 'localhost:/var/cfengine/srv/app/apacryphon_v3.9.7z'
R: Result Classes:
R: 	app_assets_reached
R: 	app_assets_kept
R: 	app_assets_repaired
#+END_EXAMPLE

#+BEGIN_SRC sh
  tree /var/cfengine/deploy
#+END_SRC

#+RESULTS: 
#+BEGIN_EXAMPLE
/var/cfengine/deploy
├── apacryphon_v3.5.7z
└── apacryphon_v3.9.7z

0 directories, 2 files
#+END_EXAMPLE

*Note:* If =cf-serverd= has not been started with the proper =access_rules= you
 may see errors. For example:

#+BEGIN_EXAMPLE
    info: Can't stat file '/var/cfengine/srv/app' on 'localhost' in files.copy_from promise
R: Result Classes:
R: 	app_assets_failed
R: 	app_assets_error
R: 	app_assets_reached
R: 	app_assets_not_kept
#+END_EXAMPLE

*** See that we don't copy files unnecessarily

If we run the policy again we find that no unnecessary repairs were made.

#+BEGIN_SRC sh
  /var/cfengine/bin/cf-agent -KIf /tmp/download_files_if_exist.cf
#+END_SRC

#+RESULTS: 
#+BEGIN_EXAMPLE
R: Result Classes:
R: 	app_assets_reached
R: 	app_assets_kept
#+END_EXAMPLE

*** See that we don't delete local files that have gone missing upstream

If we move the served file out of the way and run the policy we can see that no
repairs are made and no promises are broken.

#+BEGIN_SRC sh
  rm /var/cfengine/srv/app/apacryphon_v3.5.7z
#+END_SRC

#+BEGIN_SRC sh
  /var/cfengine/bin/cf-agent -KIf /tmp/download_files_if_exist.cf
#+END_SRC

#+RESULTS: 
#+BEGIN_EXAMPLE
R: Class defined as the result of promising the /tmp directory contained any matching files
R: 	personality_file_reached
R: 	personality_file_kept
#+END_EXAMPLE

*Note:* The previously download file was not purged, for that behavior you would
need a =copy_from= body with =purge= set to =true=.