puppet.md

Improving the Puppet deployment

The Puppet integration has been unchanged for some time. There are various improvements possible. In this document I intend to first describe the current situation and various things we can improve. The overall goal is to decouple components to allow for more flexible installation and support more deployment scenarios.

Current situation

There are four possible (optional) integration points. For maximum integration, these are currently all deployed by default in the installer.

Callbacks to Foreman

Two of them are deployed via our foreman::puppetmaster Puppet class. Both live in the same Foreman Puppet module and are configured via /etc/puppetlabs/puppet/foreman.yaml. They use client certificates to authenticate directly to the Foreman. On the Foreman side it's checked that the hostname on the certificate is a valid Foreman Proxy.

These are mostly contributed by the community.

The ENC.

Puppetserver allows using an External Node Classifier (ENC). The Foreman ENC implementation is pretty straight forward standalone Ruby script that's usually deployed to /etc/puppetlabs/puppet/node.rb on the Puppetserver. This integration is clean because there's a well defined boundry and interface between Puppet and our script.

In the typical deployment the Puppetserver calls node.rb $AGENT_HOSTNAME on every Puppet agent catalog compilation to get the ENC. Our script reads the facts from disk (as created by Puppetserver) and sends them to Foreman. This can result in creating a host in Foreman and ensures any Foreman parameters that depend on facts are correct.

An alternative deployment is as a watcher where it uses inotify to receive events about new host facts. This allows integration without being an ENC.

Reporting

The reporting script is deployed by dropping it in a Ruby library directory. It's the one we've used for many years and generally it's stable but it isn't very clean. If the Ruby version changes, it sometimes needs to be redeployed but in the Puppet AIO installation the path is stable.

It has built up some cruft and still has all the code to handle ancient Puppet reports since that was never removed. That means it'll likely still work with Puppet 0.25.

Foreman Proxy integration

On the Foreman Proxy side we have two features each with multiple providers.

Puppet

The Foreman Proxy Puppet feature is responsible for providing an API to Foreman with information about the Puppet environment.

First of all, there's the Run support. This was originally exposing the (now removed) puppet kick functionality. There are multiple providers, such as MCO, Salt, SSH, customrun that still work. On the Foreman side there's a global setting to show the Puppet run button, even if the particular Foreman Proxy doesn't support it.

The other side is listing Puppet environments and its classes. The first provider is the legacy implementation which loads the Puppet gem to parse classes. The current implementation queries the Puppetserver REST API.

Puppet CA

A Puppetserver can also be a CA. In Foreman this is exposed as a separate feature since Puppet agents can use a different CA server than the compile server.

The API it exposes to Foreman allows managing certificates. This means listing and revoking certificates as well as dealing with Certificate Signing Requests.

The classic provider, that is still the default, implements autosigning of CSRs. To do this, it modifies /etc/puppetlabs/puppet/autosign.conf and places the hostname of servers to provision. This introduces a race condition where an attacker can present a certificate before the actual server and receive a certificate. Foreman tried to handle this by making this window as small as possible, but there's a better solution.

The modern provider uses tokens to securely authenticate. While provisioning a server, Foreman creates a JWT which is signed. This is sent to the Foreman Proxy which can independently verify it's a valid token. It is also sent in the provision template so the agent can embed it in the CSR as a challengePassword. The Puppet CA server is configured to call back to the Foreman proxy which verifies or rejects the token. All this communication happens over TLS secured HTTP connections which means a shared filesystem is no longer required. It also replaces the trivial to spoof hostnames authentication.

Foreman

On the Foreman side a host can be assigned a Foreman Proxy as a Puppet server. For this the Foreman Proxy needs to expose the Puppet feature. The same can be done for a Puppet CA server. To clients the Foreman Proxy hostname is communicated as the Puppet (CA) hostname.

Moving forward

These implementations have certain drawbacks. First of all, they require the Foreman Proxy and Puppetserver to share a hostname and in the case of the classic signing provider a shared filesystem. This makes scaling by clustering harder. It also doesn't allow multi homing where a separate network is used for Foreman to Proxy communication. The user is also required to use our Puppet module (directly or via the installer) or manually extract it.

There are various steps we can take and many of them can be implemented independently.

Dropping Puppet 3 support

Support for Puppet 3 was dropped with 1.19 but the legacy Puppet implementation is still part of the Foreman Proxy codebase. It's time to drop this. This will allow us to drop the Puppet gem which greatly simplifies the testing matrix. It also removes a lot of code.

#26591

Dropping Puppet from the default Katello scenario

This is mostly unrelated to the other proposals.

Unlike the Foreman scenario, the Katello scenario doesn't depend on the Puppet CA for certificates. That means it's possible to not deploy it, saving a lot of resources. The advice would be to always deploy it on a Proxy.

Due to a technical limitation it's currently not possible to disable the module and have the correct defaults when enabled. It makes sense to remove this limitation, but it should be considered to fully drop Puppetserver support from the scenario.

Implementing multi homing

Using the new proxy registation protocol, we can expose the Puppet URLs. This allows Foreman to decouple from the Proxy's hostname and provision hosts with the correct Puppet URLs.

Decoupling the Puppetserver and Foreman Proxy hostnames removes one limitation of deploying them on separate machines.

#26164

TODO: change from registration PR to final URL

Moving the scripts out of puppet-foreman

While it's been discussed many times, but it never materialized. Alternative deployment scenarios in mind, it's time for a separate package. It could be a standalone git repository which is delivered as a tarball, RPM and DEB. The currently Puppet module will change from installing raw files to ensuring a package. The modern signing provider has a script that now lives in the smart-proxy git repository. This should be part of the same package.

This allows deploying outside of the installer context and move a lot of test code out of the puppet-foreman repository.

Moving to the modern Puppet CA provider by default

For many of the listed benefits, this provider is clearly superior. Adoption will be higher if it's the default.

Foreman already deploys the challengePassword in the CSR by default, even if it won't be used. That means it will involve making sure the users have the correct permissions to read various files and configuring Puppetserver to use the callback script.

Decoupling the Foreman Proxy filesystem from the Puppet filesystem removes one limitation of deploying them on separate machines.

Supporting external Puppetservers

Puppet Enterprise supports load balanced setups. Some people may want to deploy in containers. We should allow this. With the above we've removed the limitations that prevented to do this.

The challenges will be to make sure the Puppetservers have the correct configuration deployed and have the right credentials. Ideally it'd only talk to the Foreman Proxy and never to the Foreman itself.

Possible considerations are reading reports from other sources, such as PuppetDB.

short_term.md

End user experience

Current

To get a default installation, the following command is executed:

foreman-installer --scenario [katello|foreman-proxy-content]

Both include a Puppetserver, even though it's optional. To install without Puppetserver, the following command will achieve that.

foreman-installer --scenario [katello|foreman-proxy-content] --no-enable-puppet --foreman-proxy-puppet false --foreman-proxy-puppetca false --foreman-proxy-content-puppet false

It is also possible to only build a Puppet compile server:

# Rely on defaults
foreman-installer --scenario [katello|foreman-proxy-content] --puppet-server-ca false --foreman-proxy-puppetca false

# Explicit
foreman-installer --scenario [katello|foreman-proxy-content] --puppet-server true --foreman-proxy-puppet true --puppet-server-ca false --foreman-proxy-puppetca false --foreman-proxy-content-puppet true

In a lot of scenarios a Puppetserver is not desired by the user. If it is, it may make a lot more sense to deploy it on a separate machine via the foreman-proxy-content scenario.

Proposed

In the first phase we focus on removing the Puppetserver installation from the default, making it optional. This means the argumentless installer will achieve the same result as the previous long command.

To enable the Puppetserver, the goal is that the following command works:

foreman-installer --scenario [katello|foreman-proxy-content] --enable-puppet

By default this includes the CA server. To only build a compile server, the CA server can be disabled as follows:

# Passing in --puppet-server-ca implies --enable-puppet
foreman-installer --scenario [katello|foreman-proxy-content] --puppet-server-ca false

To do this, we need changes in Kafo to store more information in the answers file.

This assumes it's possible to leverage Hiera to automatically couple answers, but there has been no proof of concept to show this works in practice. It may be possible through hooks. Worst case we will need the user to be explicit by passing in more parameters:

foreman-installer --scenario [katello|foreman-proxy-content] --enable-puppet --foreman-proxy-puppet true --foreman-proxy-puppetca true --foreman-proxy-content-puppet true

Implementation

In building this, the initial focus will be the Kafo work to introduce an answers file version 2. This means we'll need to store the file format somewhere. A logical place to store this is the scenario file. A helper to migrate the content should be provided by Kafo to easily upgrade.

Answer file version 1 is as follows:

classname: false # don't include this class
classname: true # include and use the defaults
classname:
  param: value # include and override the default(s)

A side effect of this is that it's impossible to disable a class and store answers. Kafo will also store all defaults in their fully expanded form. This means we can't use Hiera features to do lookups and it's impossible to hide parameters even if the user shouldn't touch them.

The result is now that we have to store the following as defaults for a correct deployment:

puppet:
  server: true
  server_environments_owner: apache
  server_foreman_ssl_ca: /etc/pki/katello/puppet/puppet_client_ca.crt
  server_foreman_ssl_cert: /etc/pki/katello/puppet/puppet_client.crt
  server_foreman_ssl_key: /etc/pki/katello/puppet/puppet_client.key

A draft proposal for the new format would be:

classname:
  enabled: true # Other values: false, always
  parameters:
    param: value
  blacklist:
    - param
  whitelist:
    - param

Enabled now also accepts the option always which means there's no way to disable a class. This makes sense in the satellite scenario where certain plugins are always installed via the satellite RPM meta package. Generating the --[no-]enable-foreman-plugin-PLUGIN gives the illusion of control and only clutters the UI.

Parameters is optional.

The whitelist supersedes the blacklist. Both are optional. It is possible to store an answer for a blacklisted or non-whitelisted parameter and it is passed to application. It just doesn't show up in the UI. This allows setting a lookup answer.

The proposed default for Puppet would be:

puppet:
  enabled: false
  parameters:
    server: true
    server_environments_owner: apache
    server_foreman_ssl_ca: "%{lookup('certs::puppet::ssl_ca_cert')}"
    server_foreman_ssl_cert: "%{lookup('certs::puppet::client_cert')}"
    server_foreman_ssl_key: "%{lookup('certs::puppet::client_key')}"
  blacklist:
    - server_environments_owner
    - server_foreman_ssl_ca
    - server_foreman_ssl_cert
    - server_foreman_ssl_key

There may be other options we wish to hide, but it's wise to limit the scope for this project.

We should evaluate how many users do wish to change these values and if we're not breaking some deployment. To allow users to unblacklist options it's wise to keep generating the parser cache for these parameters it's as simple as removing them from the blacklist. This has a neglible overhead and saves developer time by not having to optimize an existing process.

Introducing answer coupling is more complicated because the answer is composed. It's whether the puppet class is enabled and the answer to server (or server_ca). We also wish to support remote Puppetservers reached over HTTP in which case the puppet module is disabled, but the answer is still true. The user does need to be able to answer. At this point I have no concrete proposal on how to solve this which is why the worst case is requiring additional parameters.

Lastly we should get rid of the foreman_proxy_content::puppet answer. The best implementation depends on the feasibility of a few things, but can be done.

Alternatives

Kafo already converts the internal answers file to a native Hiera data file:

classes:
  - puppet

puppet::server: true
puppet::server_ca: true
# Many others

It is also possible to store the answers in the native format. That does mean we need to describe the scenario somewhere else, such as which classes can be enabled. If we wish to implement the black-/whitelisting of parameters that needs to be stored somewhere else. That's why this is less desirable.