Documentation preview for new Puppet Enterprise deployment.

README.md

master

Server setup, hardening, and CHANGELOG documentation for Puppet Master, master.

Initialization

This section assumes:

• A fresh installation of CentOS Linux 7 (hostnamectl)
• A new user account with an SSH key installed (`~/.ssh/authorize_keys)

Securing SSH

See also https://wiki.centos.org/HowTos/Network/SecuringSSH

1. Login to verify user's SSH key is installed correctly
2. Rotate user's password if necessary
3. Ensure user's SSH directory permissions are properly set:

   chmod 0700 ~/.ssh; chmod 0600 ~/.ssh/authorized_keys
   

4. Disable password authentication (only after verifying user's SSH key is installed correctly):

   # Note:
   # Copies /etc/ssh/sshd_config to current working directory before performing replace with sed
   # To apply, change SSHD_CONFIG variable to /etc/ssh/sshd_config
   
   SSHD_CONFIG="./sshd_config"
   
   cp /etc/ssh/sshd_config .
   sed -i 's/^PasswordAuthentication yes$/PasswordAuthentication no/' "$SSHD_CONFIG"
   
   grep '^PasswordAuthentication' "$SSHD_CONFIG"
   

5. Consider using a non-standard port for SSH
6. Ensure service is restarted after making configuration changes:

   service sshd restart
   

Installing Puppet Enterprise

This section assumes a new license file has been obtained with a link to download the latest version of Puppet Enterprise.

1. Download the latest version of Puppet Enterprise for CentOS 7 and restrict file access:

   PE_DOWNLOAD_URL="https://s3.amazonaws.com/pe-builds/released/2017.2.1/puppet-enterprise-2017.2.1-el-7-x86_64.tar.gz"
   
   curl -O "$PE_DOWNLOAD_URL" && chmod 0600 *.tar.gz
   

2. Download the file's corresponding GPG signature:

   PE_SIGNATURE_URL="https://s3.amazonaws.com/pe-builds/released/2017.2.1/puppet-enterprise-2017.2.1-el-7-x86_64.tar.gz.asc"
   
   curl -O "$PE_SIGNATURE_URL" && chmod 0600 *.tar.gz.asc
   

3. To verify the file's release signature, import the vendor's public key signing key and perform the verification:

   curl https://downloads.puppetlabs.com/puppet-gpg-signing-key.pub | gpg --import
   

   Ensure the following returns an exact match of key's fingerprint: 6F6B 1550 9CF8 E59E 6E46 9F32 7F43 8280 EF8D 349F

   gpg --fingerprint 0x7F438280EF8D349F
   

   Note: The following should minimally indicate a good signature, but can also warn about lack of key certification if a valid trust path has not yet been established.

   gpg --verify puppet-enterprise-2017.2.1-el-7-x86_64.tar.gz.asc
   

4. Once the file has been successfully verified, it may be used to install Puppet Enterprise using an appropriate installation method

Installation Notes

Installation method used: Web-based (monolithic)

Note: Verify sufficient resources have been provisioned before running the installer (i.e. 8 GB RAM / 256 GB HDD)

Path:

• Temporarily grant current SSH user's IP exclusively access to port 3000:

   CURRENT_IP=$(echo "$SSH_CLIENT" | awk '{print $1}')
  
   firewall-cmd --zone=public --add-rich-rule='
   	rule family="ipv4"
   	source address="'"$CURRENT_IP"'"
   	port protocol="tcp" port="3000" accept'
  

  Note: Without the --permanent option (not recommended), this rule will be removed on next reboot or by issuing firewall-cmd --reload

• Installation method: [1] Guided install

• Follow instructions to browse to the Web Installer on port 3000 using HTTPS

Web Installer

Note: Replace example.com with your actual domain.

• The Puppet master component: Install on this server
• Puppet master FQDN: master.example.com
• Puppet master DNS aliases: puppet,master,master.example.com
• Database support: Install PostgreSQL on the Puppet master for me.
• Console 'admin' password: <new admin password>

When installation has finished, the final PE configuration file can be found at /etc/puppetlabs/enterprise/conf.d/pe.conf to a secure location (Caution: This file contains the console admin's credentials).

Next steps:

• Remove the temporary rich-rule for port 3000 by issuing firewall-cmd --reload
• Cleanup working directory: rm -rf ./puppet-enterprise-*
• Add a rich-rule (preferably temporary) to allow access to port 443 to authorized IPs/networks if and when console access is needed
• Review Configuring and tuning the console

From here, the Puppet Master itself will be added to the list of nodes and should be managed as such, including:

• Managing firewall exceptions for console administrators (port 443), new agent nodes (port 8140), etc.

Installing the License File

The license.key file should be copied to the appropriate path (/etc/puppetlabs/license.key) and minimally readable by pe-puppet:

chown pe-puppet:pe-puppet /etc/puppetlabs/license.key
chmod 0400 /etc/puppetlabs/license.key

Note: On the next Puppet run, the permissions on this file may be reverted to 0644 by Puppet_enterprise::License/File

Adding [existing] agent nodes

If an agent node was previously configured to use another master, it may require its ssldir to be removed to generate a new signing request against the new master:

rm -r $(puppet agent --configprint ssldir)
puppet agent -t

Port 8140 must also be open to agent nodes in order to connect:

./firewall-cmd-exception.sh <agent-ip> 8140

Workflow

Describes various procedures surrounding infrastructure management with Puppet Enterprise.

Aliasing sudo command with environment variable PATH

If $PATH is reset when using sudo, the current user's path may be "imported" using the following:

> sudo puppet --version
sudo: puppet: command not found

> sudo env "PATH=$PATH" puppet --version
4.10.1

An alias may also be created to provide a shortcut:

> alias s='sudo env "PATH=$PATH"'
> s puppet --version
4.10.1

Enabling package data collection

Consider enabling package data collection to report package inventory across your infrastructure.

Note: When enabling package data collection on the Puppet master node, the puppet_enterprise::profile::agent class may already be applied to the PE Infrastructure Agent classification group.

Installing Git

Declare the git package in the master's node manifest:

node 'node.example.com' {
	package { 'git': ensure => installed }
}

Consider using a personal access token when authenticating Git credentials.

Automation (Source Control)

Highly recommended: Consider using Code Manager to automate deployment of environments with continuous integration and support environment isolation.

Notes:

• Requires authentication token generated with puppet-access
• Automated triggering requires access to Code Manager on port 8170 from build agent, CI, etc. Internal build agent recommended to eliminate exposing Puppet server externally.

   ./firewall-cmd <build-agent-ip> 8170
  

• Do NOT use puppet module if Code Manager is in use (https://docs.puppet.com/pe/latest/cmgmt_puppetfile.html#about-modules)
• Ensure the server's private SSH key is set with the correct permissions:

   chown pe-puppet:pe-puppet /etc/puppetlabs/ssh/id-control_repo.rsa && chmod 0400 /etc/puppetlabs/ssh/id-control_repo.rsa
  

• As of 5/23/17, Code Manager does not support spaces a control repository's path well (i.e. "My Repository"). Consider removing spaces.
• With Code Manager configured, be sure to specify :control_branch and environment-specific content in each environment's Puppetfile
• Evaluate the specific needs of each of your environments and consider removing :default_branch if it is not suitable for your environments
• Pro tip: Commit an empty .env-{control-branch} file to the root of each branch in the control repository (where {control-branch} is replaced with the corresponding environment) to see which branch your IDE is operating on at all times.

Divided Infrastructure

While the environment-based workflow facilitates testing new code before pushing to production, it lacks in organizational structure where grouping nodes by departments or application domains may be a requirement. Not much is said about divided infrastructure either, except for the general recommendation of splitting teams into their own environments.

Consider the following layout for organizing environment branches in your control repository to support multiple application domains in multiple departments:

Department A
- Environments: Staging, Production
- Clusters: A, B

Department B
- Environments: Development, Staging, Production
- Applications: A, B

Department C
- Environments: Staging, Production
- Services: A, B

In source control, the following "flattened" environment branches would be created with restricted access to their respective departments:

Control Repository (Environment Branches)
- departmentA_clusterA_staging
- departmentA_clusterB_production

- departmentB_applicationA_development
- departmentB_applicationA_staging
- departmentB_applicationB_production

- departmentC_serviceA_staging
- departmentC_serviceB_production

Tips:

• When only a single control repository is available for all departments (i.e. single Puppet master), a possible workaround is to allow environment branches to be deployed automatically, making the control repository autonomous.
• For auditing and management purposes, separate department repositories may be required, configured to deploy only approved code to the autonomous control repository as the final step in its release definition.
• To offer up a template control branch, clone the puppetlabs/control-repo into a branch named !base and lock the branch. The exclamation mark prefix (!) prevents Code Manager from being deployed as having an invalid environment name and locking the branch prevents the template from being modified in source control.

backup-target.sh

#!/bin/sh
NAME="$1"
TARGET="$2"
OWNER="$3"

BACKUP="$NAME-$(date +'%Y%m%d_%H%M%S').tar.gz"

if [ -z "$NAME" ] || [ -z "$TARGET" ]; then
  echo "Usage: $0 <name> <target> [owner]"
  exit 1
fi

tar -czvf "$BACKUP" "$TARGET" && chmod 0400 "$BACKUP"

if [ ! -z "$OWNER" ]; then
  chown "$OWNER:$OWNER" "$BACKUP"
fi

codemanager-deploy.ps1

# Code Manager (Deploy All)
param([String]$token, [String]$server, [Int]$port = 8170)

$headers = @{ "Content-Type" = "application/json"; "X-Authentication" = $token }
$uri = [String]::Format('https://{0}:{1}/code-manager/v1/deploys', $server, $port)
$body = New-Object PsObject -Property @{'deploy-all'=$True} | ConvertTo-Json -Compress

# Ignore SSL errors
Add-Type "using System.Net; using System.Security.Cryptography.X509Certificates; public class TrustAllCertsPolicy : ICertificatePolicy { public bool CheckValidationResult(ServicePoint srvPoint, X509Certificate certificate, WebRequest request, int certificateProblem) { return true; } }"
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

Exit !((Invoke-WebRequest -Method POST -Headers $headers -Uri $uri -Body $body).StatusCode -eq 200)

codemanager-webhook.ps1

param([String]$uri)

add-type "using System.Net; using System.Security.Cryptography.X509Certificates; public class TrustAllCertsPolicy : ICertificatePolicy {public bool CheckValidationResult(ServicePoint srvPoint, X509Certificate certificate, WebRequest request, int certificateProblem) {return true; } }"
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

Exit !((Invoke-WebRequest -Uri $uri -Method POST).StatusCode -eq 200)

current-ip.sh

#!/bin/sh
echo "$SSH_CLIENT" | awk '{print $1}'

firewall-cmd-exception.sh

#!/bin/sh
# To add a temporary firewall exception for the current SSH user's IP to access port 443 using this script:
# ./firewall-cmd-exception.sh $(echo "$SSH_CLIENT" | awk '{print $1}') 443
#
SRC="$1"
PORT="$2"

if [ -z "$SRC" ] || [ -z "$PORT" ]; then
	echo "Usage: $0 <source> <port>"
	exit 1
fi

echo "Adding temporary firewall exception for $SRC on port $PORT"
firewall-cmd --zone=public --add-rich-rule='
	rule family="ipv4"
	source address="'"$SRC"'"
	port protocol="tcp" port="'"$PORT"'" accept'

generate-eyaml-keys.sh

#!/bin/sh
sudo mkdir -p /etc/puppetlabs/puppet/eyaml/ && sudo chown pe-puppet:pe-puppet /etc/puppetlabs/puppet/eyaml/ && sudo chmod 0500 /etc/puppetlabs/puppet/eyaml/
sudo eyaml createkeys && sudo chown pe-puppet:pe-puppet ./keys && sudo chmod 0400 ./keys/* && sudo mv ./keys/* /etc/puppetlabs/puppet/eyaml/ && sudo rmdir ./keys