Skip to content

Instantly share code, notes, and snippets.

@mattborja

mattborja/web00.md Secret

Forked from rdev5/web00.md
Created July 4, 2018 11:32
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mattborja/5de0c7a0640e3acf45ef82ee2ed9fc34 to your computer and use it in GitHub Desktop.
Save mattborja/5de0c7a0640e3acf45ef82ee2ed9fc34 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
web00.md

Web Server Cluster (Managed by Puppet)

Nodes:

  • Master: puppetmaster
  • Agents: web01, web02

web01,web02

Changeset: 20161019

  • Download and install Puppet agent for Windows
  • Run Puppet agent to create certificate signing requests and sign on Puppet server
  • Set runinterval in agent section of configuration (C:\ProgramData\PuppetLabs\puppet\etc\puppet.conf)
  • Ensure nodes are targeted in site.pp (i.e. node 'web01.example.com', 'web02.example.com')

puppetmaster

Changeset: 20161019

Note: It is recommended to review and establish a trust path to one of the signatures on the release keys described by this document.

  • Add SSH key for authentication
  • Enable Puppet Collection 1 repository:
    • sudo rpm -Uvh https://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
  • Import Puppet signing key for verifying Puppet packages:
  • Verify key's fingerprint:
    • gpg --list-key --fingerprint 7F438280EF8D349F
  • Import Puppet release key for verifying RPM packages:
  • Verify key's fingerprint:
    • gpg --list-key --fingerprint 0x1054B7A24BD6EC30
  • Import to RPM:
    • sudo rpm --import 1054b7a24bd6ec30.asc
  • Download and verify Puppet packages:
    • curl -O https://downloads.puppetlabs.com/puppet/puppetserver-2.6.0.tar.gz
    • curl -O https://downloads.puppetlabs.com/puppet/puppetserver-2.6.0.tar.gz.asc
    • gpg --verify puppetserver-2.6.0.tar.gz.asc puppetserver-2.6.0.tar.gz
  • Download and verify RPM packages:
    • curl -O https://yum.puppetlabs.com/puppetlabs-release-pc1-el-7.noarch.rpm
    • curl -O https://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm
    • sudo rpm -vK *.rpm
  • Install Puppet server:
    • sudo yum install puppetserver
  • Review JVM tuning for Puppet server:
    • grep JAVA_ARGS /etc/sysconfig/puppetserver
  • Start Puppet server:
    • sudo service puppetserver start
  • Set server variable in [master] section of system config:
    • sudo ``which puppet`` config set server puppetmaster.domain.com --section master
  • Open port 8140 to accept certificate signing requests from Puppet agents:
    • sudo firewall-cmd --zone=public --add-port=8140/tcp --permanent && sudo firewall-cmd --reload
  • Run Puppet agents and sign certificate requests on master
    • sudo ``which puppet`` cert list
    • sudo ``which puppet`` cert sign web01.domain.com
    • sudo ``which puppet`` cert sign web02.domain.com
  • Install Puppet approved puppet/iis module (requires Windows Server 2008 or higher):
    • puppet module install puppet-iis
    • Note: Dependencies in Puppet repository may not be in sync with project repo (i.e. puppet-windowsfeature) and may need to be installed manually:
      • Download latest puppet-windowsfeature
      • sudo ``which puppet`` module install puppet-windowsfeature.tar.gz
  • Create default main manifest file:
    • sudo touch /etc/puppetlabs/code/environments/production/manifests/site.pp

Changeset: 20161020

  • Install PuppetDB:
    • sudo ``which puppet`` resource package puppetdb ensure=latest
  • Install PostgreSQL:
    • sudo yum install postgresql
    • sudo postgresql-setup initdb
    • sudo service postgresql start
    • TODO: Configure PostgreSQL
  • Create PuppetDB user and database:
    • sudo -u postgres sh
    • createuser -DRSP puppetdb
    • createdb -E UTF8 -O puppetdb puppetdb
    • exit
  • Install RegExp-optimized index extension pg_trgm
    • sudo yum install postgresql-contrib
    • sudo -u postgres sh
    • `psql puppetdb -c 'create extension pg_trgm'
    • TODO: Finish setup
  • Import and cross-compare Foreman Release Signing Keys:
  • Download and verify Foreman RPM:
    • curl -O https://yum.theforeman.org/releases/1.13/el7/x86_64/foreman-release.rpm
    • sudo rpm -vK foreman-release.rpm
  • Enable EPEL and Foreman repositories if not already installed:
    • sudo rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
  • Enable Foreman repositories:
    • sudo yum install https://yum.theforeman.org/releases/1.13/el7/x86_64/foreman-release.rpm
      • Or from local verified RPM: sudo yum localinstall foreman-release.rpm
    • Note: Packages may fail if optional RPMs are not enabled in RHEL7
      • sudo subscription-manager repos --list
      • sudo subscription-manager repos --enable rhel-7-server-optional-rpms
  • Install Foreman installer:
    • sudo yum install foreman-installer
  • Run Foreman installer:
    • sudo foreman-installer
    • Note: If any packages fail to install, try running them manually to debug (i.e. sudo yum install foreman-proxy)
  • Open port 443 to allow access to web console:
    • sudo firewall-cmd --zone=public --add-port=443/tcp --permanent && sudo firewall-cmd --reload
    • TODO: Review LDAP Authentication
  • Install NTP module into production to be managed by Foreman
    • sudo ``which puppet`` module install puppetlabs/ntp
  • In Foreman console, import modules:
    • Configure -> Classes -> Import from...
  • Follow instructions to Override Default NTP Pool
    • Set default servers parameter to type array with value: ["0.us.pool.ntp.org","1.us.pool.ntp.org","2.us.pool.ntp.org","3.us.pool.ntp.org"]
  • Apply NTP class to puppetmaster node and test:
    • Edit PuppetClasses for puppetmaster host, add ntp from list of available classes and click "Submit" to apply changes
    • sudo ``which puppet`` agent --test
  • In Foreman Console ("Web Cluster"), create Config Group Web Server and add Puppet classes:
    • iis
    • stdlib
  • Create Host Group Web Cluster:
    • Environment: production
    • Puppet CA: puppetmaster
    • Puppet Master: puppetmaster
    • Included Config Groups: Web Server
  • Assign hosts to Web Cluster host group

Node Configuration

  • Begin creating manifests for new IIS configuration:
    • Define nodes in /etc/puppetlabs/code/environments/production/manifests/site.pp
    • Define classes in /etc/puppetlabs/code/environments/production/manifests/www.pp
    • Import classes and add to Config Group (Web Server)
    • Configuration changes may now be made to nodes/classes (www.pp) and tested
      • Note: Some "configuration" not supported by existing modules may need to be bootstrapped with puppetlabs-powershell

Hiera Configuration Files

Changeset: 20161021

  • Install hiera-eyaml for supporting encrypted configuration values:
    • sudo gem install hiera-eyaml
    • sudo ``which puppetserver`` gem install hiera-eyaml
  • Build script to securely rotate eyaml keys: 
    #!/bin/sh
KEYDIR="/path/to/keydir"
KEY_UID="$(date +'%Y%m%d_%H%M%S')"
PRIVATE_KEY="$KEYDIR/$KEY_UID"
PUBLIC_KEY="$KEYDIR/$KEY_UID.pub"

sudo mkdir -p "$KEYDIR"
sudo chown -R puppet:root "$KEYDIR"
sudo chmod -R 0500 "$KEYDIR"

sudo `which eyaml` createkeys --pkcs7-private-key="$PRIVATE_KEY" --pkcs7-public-key="$PUBLIC_KEY"

sudo chmod 0400 "$PRIVATE_KEY"
sudo chmod 0400 "$PUBLIC_KEY"
sudo ls -lhaR "$KEYDIR"
  • Build script for encrypting stdin with eyaml: 
    #!/bin/sh
PRIVATE_KEY="/path/to/private/key"
PUBLIC_KEY="/path/to/public/key"

sudo `which eyaml` encrypt --stdin --pkcs7-private-key="$PRIVATE_KEY" --pkcs7-public-key="$PUBLIC_KEY"
  • Generate keys and edit /etc/puppetlabs/puppet/hiera.yaml to support eyaml backend:
    • TODO: Finish

Couchbase

Note: No existing Couchbase modules for Puppet support Windows (manual installation required). Changeset: 20161024

  • Review tighten down file permissions on manifests as necessary.
    • sudo ls -lha /etc/puppetlabs/code/environment/production/manifests/*.pp
    • sudo chmod 0640 /etc/puppetlabs/code/environment/production/manifests/*.pp
  • Manually install and configure Couchbase cluster:
    • Separate data and index paths
    • Configure hostname
    • Enable services: Data, Index, Query
    • Data RAM quota: 2048
    • Index RAM quota: 512
    • Bucket type: Couchbase
    • Per Node RAM Quota: 2048
    • Cache Metadata: Value ejection
    • Enable replicas: 1
    • Set bucket disk I/O priority: Low (default)
    • Enable Flush
    • Enable software update notifications
    • Set administrator account credentials
    • Review Network Ports (Node-to-Node) for enabling communication between nodes.
    • Review Security Considerations for securing Couchbase cluster
    • Add additional nodes using Add Server from the Server Nodes tab (do not join from other nodes)

Node-to-node communications over IPsec with data encryption (Windows)

Changeset 20161025

  • Create Inbound Rule to allow communication on node-to-node ports:
    • Action: Allow the connection if it is secure (require the connections to be encrypted)
    • Authorized computers (Remote Computers)
      • Only allow connections from these computers (node1, node2, ... nodeN)
    • Profiles (Advanced): Domain, Private, Public
    • Local Ports: 8091, 8092, 11209-11210, 4369, 21100-21299
    • Remote Port: All
  • Create Connection Security Rule to support IPsec
    • Remote Computers
      • Endpoint 1: node1
      • Endpoint 2: node2, ... nodeN
    • Profile (Advanced): Domain, Private, Public
    • IPsec tunneling (leave disabled to use Transport mode)
    • Authentication
      • Require inbound and outbound
      • Method: Computer (Kerberos V5)
    • Protocols and Ports
      • Protocol type: TCP
      • Endpoint 1 port: All ports
      • Endpoint 2 port: All ports
  • Specify Data Protection Settings (Windows Firewall Properties -> IPsec Settings -> IPsec defaults)
    • Data protection: Advanced
      • Require encryption for all connection security rules that use these settings
      • Remove weak ciphers (i.e. 3DES)
      • Specify data integrity and encryption algorithm (i.e. AES-CBC 128 / SHA-1 for ESP protocol)
      • Require encryption for all connection security rules that use these settings
  • Copy firewall properties, inbound rules, and connection security rules to other nodes and verify connections
  • TODO: Enable communication over IPsec between Puppet master and agent nodes
  • TODO: Move changeset (Windows Firewall) into Puppet manifest

Install/Update Couchbase 4.1.0-5005 Community Edition (build-5005) to 4.5.1 Enterprise Edition

  • Download and verify Couchbase 4.5.1 Enterprise installer
  • Stop Couchbase service: net stop CouchbaseServer (must be run as Administrator)
  • Run Couchbase installer (automatic backup/upgrade)
  • Update firewall configuration as recommended
    • Local ports: 4369, 8091-8094, 9100-9105, 9998-9999, 11209-11211, 11214-11215, 18091-18093, 21100-21299
  • TODO: Review bucket configuration and configure Settings

File Sync

  • Add "Create File Sync Schedule Task" to Puppet manifest
  • Set any configuration parameters in hiera (i.e. common.yaml)
  • Create scheduled task on each node to sync from publish directory
    • ROBOCOPY \\path\to\root\ C:\Inetpub\wwwroot /MIR /E
    • Set credentials for running task (General -> Security options)

SSL

Manual installation

  • "Double-copy" SSL certificate (with private key) from existing server into certmgr.msc
  • Enumerate certificate thumbprint:
    • Import-Module WebAdministration; dir cert:\localmachine\my
  • Create binding:
    • Import-Module WebAdministration; New-WebBinding -Name "WebsiteName" -IP "*" -Port 443 -Protocol https
  • TODO: Import RSA keys onto server

Changeset 20161031

  • TODO: Add IIS Crypto to manifests for configuring SSL ciphers

Changeset 20161101

  • Secure Puppet code/ directories:
    • sudo chown -R puppet:puppet /etc/puppetlabs/code/
    • sudo find /etc/puppetlabs/code/ -type d | sudo xargs chmod 0700
    • sudo find /etc/puppetlabs/code/ -type f | sudo xargs chmod 0600
  • Setup fileserver for transfering files to agents:
    • sudo mkdir /etc/puppetlabs/puppet/installs/
    • sudo touch /etc/puppetlabs/puppet/fileserver.conf

Source Control

Changeset 20161103

  • Install dos2unix to convert line endings
    • sudo yum install dos2unix

Changeset 20161104

  • To enable puppetrun in Foreman:
    • Edit /etc/foreman/settings.yaml and enable puppetrun, setting puppet_conf appropriately
  • Stub empty entries for "unsupported" OS (i.e. 10.0) and install separately (PowerShell)
    • /etc/puppetlabs/code/environments/production/modules/iis/manifests/features/*.pp
  • Review and ensure commands are being executed properly (i.e. puppetlabs-powershell@1.0.6 does not execute on Windows 10.0)
    • sudo ``which puppet`` module upgrade puppetlabs-powershell (Caution: This may cause clients to stop responding if it hangs due to failed dependencies)
    • Note: ruby.exe can lock cache/lib causing cleanup tasks to fail in Puppet agent (i.e. invoke_command.ps) Changeset 20161107
  • Install Couchbase manually and add Firewall exceptions
    • New-NetFirewallRule -DisplayName "Couchbase Server 4.5.1" -Direction Inbound -Protocol TCP -LocalPort 4369,8091,9100-9105,9998,9999,11209-11211,11214,11215,18091-18093,21100-21299 -RemoteAddress 172.20.85.81-172.20.85.83,172.21.85.81-172.21.85.83 -Action Allow
  • Setup server and add nodes from single node
  • TODO: Bootstrap IPsec

Changeset 20161108

  • DEBUG: memcached.exe (APPCRASH)

    • Couchbase 4.5.1 is broken under Windows 10.0 (Anniversary Update)
    • Couchbase 3.1.6 appears to be a stable replacement until further testing can be done with Couchbase 4.6.x (currently Developer Preview)
    • Solution: Install Couchbase separately onto dedicated Linux server (4.5.1 enterprise)

  • Setup Couchbase cluster (manual) and bi-directional cross-datacenter replication (XDCR)

    • For each cluster (region):
      • Install Couchbase on each cluster node
      • Build cluster using first node (i.e. Add Server)
      • Add firewall exceptions to allow networking between nodes and clusters ONLY
        • Example: $localPort = ("4369", "8091", "9100-9105", "9998", "9999", "11209-11211", "11214", "11215", "18091-18093", "21100-21299"); $remoteAddresses = ("...")
      • Add remote cluster in XDCR with Encryption Enabled using public key of remote cluster node (Settings -> Cluster).
        • Regenerate certificate if/as necessary
        • Create replications for each bucket to be replicated
        • Specify additional replication details under Advanced Settings:
          • XDCR Max Replications per Bucket: 32 (> 16)
          • XDCR workers per Replication: 4
          • XDCR Checkpoint Interval: 1800
          • XDCR Batch Count: 1024 (> 500)
          • XDCR Batch Size (kB): 2048
          • XDCR Failure Retry Interval: 20 (< 30)
          • XDCR Optimistic Replication Threshold: 256

  • Gradually add new nodes to cluster (one region at a time) and record differentials:

    • See Errata

  • TODO: Add to Puppet configuration (see REST API for Couchbase Console)

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment