Skip to content

Instantly share code, notes, and snippets.

@mattborja mattborja/ Secret forked from rdev5/
Created Jul 4, 2018

What would you like to do?

Web Server Cluster (Managed by Puppet)


  • Master: puppetmaster
  • Agents: web01, web02


Changeset: 20161019

  • Download and install Puppet agent for Windows
  • Run Puppet agent to create certificate signing requests and sign on Puppet server
  • Set runinterval in agent section of configuration (C:\ProgramData\PuppetLabs\puppet\etc\puppet.conf)
  • Ensure nodes are targeted in site.pp (i.e. node '', '')


Changeset: 20161019

Note: It is recommended to review and establish a trust path to one of the signatures on the release keys described by this document.

  • Add SSH key for authentication
  • Enable Puppet Collection 1 repository:
    • sudo rpm -Uvh
  • Import Puppet signing key for verifying Puppet packages:
  • Verify key's fingerprint:
    • gpg --list-key --fingerprint 7F438280EF8D349F
  • Import Puppet release key for verifying RPM packages:
  • Verify key's fingerprint:
    • gpg --list-key --fingerprint 0x1054B7A24BD6EC30
  • Import to RPM:
    • sudo rpm --import 1054b7a24bd6ec30.asc
  • Download and verify Puppet packages:
    • curl -O
    • curl -O
    • gpg --verify puppetserver-2.6.0.tar.gz.asc puppetserver-2.6.0.tar.gz
  • Download and verify RPM packages:
    • curl -O
    • curl -O
    • sudo rpm -vK *.rpm
  • Install Puppet server:
    • sudo yum install puppetserver
  • Review JVM tuning for Puppet server:
    • grep JAVA_ARGS /etc/sysconfig/puppetserver
  • Start Puppet server:
    • sudo service puppetserver start
  • Set server variable in [master] section of system config:
    • sudo ``which puppet`` config set server --section master
  • Open port 8140 to accept certificate signing requests from Puppet agents:
    • sudo firewall-cmd --zone=public --add-port=8140/tcp --permanent && sudo firewall-cmd --reload
  • Run Puppet agents and sign certificate requests on master
    • sudo ``which puppet`` cert list
    • sudo ``which puppet`` cert sign
    • sudo ``which puppet`` cert sign
  • Install Puppet approved puppet/iis module (requires Windows Server 2008 or higher):
    • puppet module install puppet-iis
    • Note: Dependencies in Puppet repository may not be in sync with project repo (i.e. puppet-windowsfeature) and may need to be installed manually:
      • Download latest puppet-windowsfeature
      • sudo ``which puppet`` module install puppet-windowsfeature.tar.gz
  • Create default main manifest file:
    • sudo touch /etc/puppetlabs/code/environments/production/manifests/site.pp

Changeset: 20161020

  • Install PuppetDB:
    • sudo ``which puppet`` resource package puppetdb ensure=latest
  • Install PostgreSQL:
    • sudo yum install postgresql
    • sudo postgresql-setup initdb
    • sudo service postgresql start
    • TODO: Configure PostgreSQL
  • Create PuppetDB user and database:
    • sudo -u postgres sh
    • createuser -DRSP puppetdb
    • createdb -E UTF8 -O puppetdb puppetdb
    • exit
  • Install RegExp-optimized index extension pg_trgm
    • sudo yum install postgresql-contrib
    • sudo -u postgres sh
    • `psql puppetdb -c 'create extension pg_trgm'
    • TODO: Finish setup
  • Import and cross-compare Foreman Release Signing Keys:
  • Download and verify Foreman RPM:
    • curl -O
    • sudo rpm -vK foreman-release.rpm
  • Enable EPEL and Foreman repositories if not already installed:
    • sudo rpm -ivh
  • Enable Foreman repositories:
    • sudo yum install
      • Or from local verified RPM: sudo yum localinstall foreman-release.rpm
    • Note: Packages may fail if optional RPMs are not enabled in RHEL7
      • sudo subscription-manager repos --list
      • sudo subscription-manager repos --enable rhel-7-server-optional-rpms
  • Install Foreman installer:
    • sudo yum install foreman-installer
  • Run Foreman installer:
    • sudo foreman-installer
    • Note: If any packages fail to install, try running them manually to debug (i.e. sudo yum install foreman-proxy)
  • Open port 443 to allow access to web console:
    • sudo firewall-cmd --zone=public --add-port=443/tcp --permanent && sudo firewall-cmd --reload
    • TODO: Review LDAP Authentication
  • Install NTP module into production to be managed by Foreman
    • sudo ``which puppet`` module install puppetlabs/ntp
  • In Foreman console, import modules:
    • Configure -> Classes -> Import from...
  • Follow instructions to Override Default NTP Pool
    • Set default servers parameter to type array with value: ["","","",""]
  • Apply NTP class to puppetmaster node and test:
    • Edit PuppetClasses for puppetmaster host, add ntp from list of available classes and click "Submit" to apply changes
    • sudo ``which puppet`` agent --test
  • In Foreman Console ("Web Cluster"), create Config Group Web Server and add Puppet classes:
    • iis
    • stdlib
  • Create Host Group Web Cluster:
    • Environment: production
    • Puppet CA: puppetmaster
    • Puppet Master: puppetmaster
    • Included Config Groups: Web Server
  • Assign hosts to Web Cluster host group

Node Configuration

  • Begin creating manifests for new IIS configuration:
    • Define nodes in /etc/puppetlabs/code/environments/production/manifests/site.pp
    • Define classes in /etc/puppetlabs/code/environments/production/manifests/www.pp
    • Import classes and add to Config Group (Web Server)
    • Configuration changes may now be made to nodes/classes (www.pp) and tested
      • Note: Some "configuration" not supported by existing modules may need to be bootstrapped with puppetlabs-powershell

Hiera Configuration Files

Changeset: 20161021

  • Install hiera-eyaml for supporting encrypted configuration values:
    • sudo gem install hiera-eyaml
    • sudo ``which puppetserver`` gem install hiera-eyaml
  • Build script to securely rotate eyaml keys:
    KEY_UID="$(date +'%Y%m%d_%H%M%S')"
    sudo mkdir -p "$KEYDIR"
    sudo chown -R puppet:root "$KEYDIR"
    sudo chmod -R 0500 "$KEYDIR"
    sudo `which eyaml` createkeys --pkcs7-private-key="$PRIVATE_KEY" --pkcs7-public-key="$PUBLIC_KEY"
    sudo chmod 0400 "$PRIVATE_KEY"
    sudo chmod 0400 "$PUBLIC_KEY"
    sudo ls -lhaR "$KEYDIR"
  • Build script for encrypting stdin with eyaml:
    sudo `which eyaml` encrypt --stdin --pkcs7-private-key="$PRIVATE_KEY" --pkcs7-public-key="$PUBLIC_KEY"
  • Generate keys and edit /etc/puppetlabs/puppet/hiera.yaml to support eyaml backend:
    • TODO: Finish


Note: No existing Couchbase modules for Puppet support Windows (manual installation required). Changeset: 20161024

  • Review tighten down file permissions on manifests as necessary.
    • sudo ls -lha /etc/puppetlabs/code/environment/production/manifests/*.pp
    • sudo chmod 0640 /etc/puppetlabs/code/environment/production/manifests/*.pp
  • Manually install and configure Couchbase cluster:
    • Separate data and index paths
    • Configure hostname
    • Enable services: Data, Index, Query
    • Data RAM quota: 2048
    • Index RAM quota: 512
    • Bucket type: Couchbase
    • Per Node RAM Quota: 2048
    • Cache Metadata: Value ejection
    • Enable replicas: 1
    • Set bucket disk I/O priority: Low (default)
    • Enable Flush
    • Enable software update notifications
    • Set administrator account credentials
    • Review Network Ports (Node-to-Node) for enabling communication between nodes.
    • Review Security Considerations for securing Couchbase cluster
    • Add additional nodes using Add Server from the Server Nodes tab (do not join from other nodes)

Node-to-node communications over IPsec with data encryption (Windows)

Changeset 20161025

  • Create Inbound Rule to allow communication on node-to-node ports:
    • Action: Allow the connection if it is secure (require the connections to be encrypted)
    • Authorized computers (Remote Computers)
      • Only allow connections from these computers (node1, node2, ... nodeN)
    • Profiles (Advanced): Domain, Private, Public
    • Local Ports: 8091, 8092, 11209-11210, 4369, 21100-21299
    • Remote Port: All
  • Create Connection Security Rule to support IPsec
    • Remote Computers
      • Endpoint 1: node1
      • Endpoint 2: node2, ... nodeN
    • Profile (Advanced): Domain, Private, Public
    • IPsec tunneling (leave disabled to use Transport mode)
    • Authentication
      • Require inbound and outbound
      • Method: Computer (Kerberos V5)
    • Protocols and Ports
      • Protocol type: TCP
      • Endpoint 1 port: All ports
      • Endpoint 2 port: All ports
  • Specify Data Protection Settings (Windows Firewall Properties -> IPsec Settings -> IPsec defaults)
    • Data protection: Advanced
      • Require encryption for all connection security rules that use these settings
      • Remove weak ciphers (i.e. 3DES)
      • Specify data integrity and encryption algorithm (i.e. AES-CBC 128 / SHA-1 for ESP protocol)
      • Require encryption for all connection security rules that use these settings
  • Copy firewall properties, inbound rules, and connection security rules to other nodes and verify connections
  • TODO: Enable communication over IPsec between Puppet master and agent nodes
  • TODO: Move changeset (Windows Firewall) into Puppet manifest

Install/Update Couchbase 4.1.0-5005 Community Edition (build-5005) to 4.5.1 Enterprise Edition

  • Download and verify Couchbase 4.5.1 Enterprise installer
  • Stop Couchbase service: net stop CouchbaseServer (must be run as Administrator)
  • Run Couchbase installer (automatic backup/upgrade)
  • Update firewall configuration as recommended
    • Local ports: 4369, 8091-8094, 9100-9105, 9998-9999, 11209-11211, 11214-11215, 18091-18093, 21100-21299
  • TODO: Review bucket configuration and configure Settings

File Sync

  • Add "Create File Sync Schedule Task" to Puppet manifest
  • Set any configuration parameters in hiera (i.e. common.yaml)
  • Create scheduled task on each node to sync from publish directory
    • ROBOCOPY \\path\to\root\ C:\Inetpub\wwwroot /MIR /E
    • Set credentials for running task (General -> Security options)


Manual installation

  • "Double-copy" SSL certificate (with private key) from existing server into certmgr.msc
  • Enumerate certificate thumbprint:
    • Import-Module WebAdministration; dir cert:\localmachine\my
  • Create binding:
    • Import-Module WebAdministration; New-WebBinding -Name "WebsiteName" -IP "*" -Port 443 -Protocol https
  • TODO: Import RSA keys onto server

Changeset 20161031

  • TODO: Add IIS Crypto to manifests for configuring SSL ciphers

Changeset 20161101

  • Secure Puppet code/ directories:
    • sudo chown -R puppet:puppet /etc/puppetlabs/code/
    • sudo find /etc/puppetlabs/code/ -type d | sudo xargs chmod 0700
    • sudo find /etc/puppetlabs/code/ -type f | sudo xargs chmod 0600
  • Setup fileserver for transfering files to agents:
    • sudo mkdir /etc/puppetlabs/puppet/installs/
    • sudo touch /etc/puppetlabs/puppet/fileserver.conf

Source Control

Changeset 20161103

  • Install dos2unix to convert line endings
    • sudo yum install dos2unix

Changeset 20161104

  • To enable puppetrun in Foreman:
    • Edit /etc/foreman/settings.yaml and enable puppetrun, setting puppet_conf appropriately
  • Stub empty entries for "unsupported" OS (i.e. 10.0) and install separately (PowerShell)
    • /etc/puppetlabs/code/environments/production/modules/iis/manifests/features/*.pp
  • Review and ensure commands are being executed properly (i.e. puppetlabs-powershell@1.0.6 does not execute on Windows 10.0)
    • sudo ``which puppet`` module upgrade puppetlabs-powershell (Caution: This may cause clients to stop responding if it hangs due to failed dependencies)
    • Note: ruby.exe can lock cache/lib causing cleanup tasks to fail in Puppet agent (i.e. Changeset 20161107
  • Install Couchbase manually and add Firewall exceptions
    • New-NetFirewallRule -DisplayName "Couchbase Server 4.5.1" -Direction Inbound -Protocol TCP -LocalPort 4369,8091,9100-9105,9998,9999,11209-11211,11214,11215,18091-18093,21100-21299 -RemoteAddress, -Action Allow
  • Setup server and add nodes from single node
  • TODO: Bootstrap IPsec

Changeset 20161108

  • DEBUG: memcached.exe (APPCRASH)

    • Couchbase 4.5.1 is broken under Windows 10.0 (Anniversary Update)
    • Couchbase 3.1.6 appears to be a stable replacement until further testing can be done with Couchbase 4.6.x (currently Developer Preview)
    • Solution: Install Couchbase separately onto dedicated Linux server (4.5.1 enterprise)
  • Setup Couchbase cluster (manual) and bi-directional cross-datacenter replication (XDCR)

    • For each cluster (region):
      • Install Couchbase on each cluster node
      • Build cluster using first node (i.e. Add Server)
      • Add firewall exceptions to allow networking between nodes and clusters ONLY
        • Example: $localPort = ("4369", "8091", "9100-9105", "9998", "9999", "11209-11211", "11214", "11215", "18091-18093", "21100-21299"); $remoteAddresses = ("...")
      • Add remote cluster in XDCR with Encryption Enabled using public key of remote cluster node (Settings -> Cluster).
        • Regenerate certificate if/as necessary
        • Create replications for each bucket to be replicated
        • Specify additional replication details under Advanced Settings:
          • XDCR Max Replications per Bucket: 32 (> 16)
          • XDCR workers per Replication: 4
          • XDCR Checkpoint Interval: 1800
          • XDCR Batch Count: 1024 (> 500)
          • XDCR Batch Size (kB): 2048
          • XDCR Failure Retry Interval: 20 (< 30)
          • XDCR Optimistic Replication Threshold: 256
  • Gradually add new nodes to cluster (one region at a time) and record differentials:

    • See Errata
  • TODO: Add to Puppet configuration (see REST API for Couchbase Console)


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.