Instantly share code, notes, and snippets.

Embed
What would you like to do?

Snapt Installation (2018)

Last updated: 3/6/18

This installation guide applies to the recommended Snapt VM Image (OpenSUSE) from https://downloads.snapt.net/

Base Server Setup

  1. Login to the server (SSH) as snapt:snapt and change account password
  2. Change the default root account password (root:root)
  3. Copy SSH key to server
    • Append SSH key to /home/snapt/.ssh/authorized_keys
    • Apply hardened SSH configuration (see sshd_config.sample) and reload with service sshd reload
  4. Review /etc/crontab for Snapt cron job (/srv/www/htdocs/bin/cronjob.php) and replace root user with lighttpd. For more insight as to what Snapt's cron job is doing, consider installing the Snapt Cron plugin.
  5. Disable management connections on port 8080 via: Setup -> Configuration -> Snapt Configuration -> Set HTTP Access to Off.
    • Note: This will allow you to still connect to Snapt UI over HTTPS on port 8081 but will set the unencrypted HTTP 8080 port to bind to 127.0.0.1 only.

Snapt UI Setup (Checklist)

Important: If you encounter any bugs (i.e. changes not taking effect, etc.), please review the Bugs & Workarounds section for known workarounds.

To begin, login to the UI using https:// on port 8081

Before installing any plugins or performing routine maintenance, be sure to complete a preliminary checklist (Dashboard -> Status Check) in the following order:

  1. Time Sync (time.example.com)
  2. Email (From snapt@example.com; SMTP mail.example.com:25)
  3. Framework Version -> Run Framework Update (note the message in progress, "This may take several minutes, please be patient and do not use the UI.")
  4. Automatic Updates (Scheduler)
    • Tip: Consider enabling automatic updates at weekly intervals but leaving automatic Framework Updates (Versions) disabled.
  5. When using the Snapt Image, you can check for system patches (Setup -> Update Manager -> System Patches tab)
    • Tip: Consider installing all non-interactive patches. This will create a job ("Patches install") that will be processed in the background.
  6. If necessary, apply any plugin updates available (Setup -> Modules & Plugins -> Updates tab)

Note:

  • Not every UI will be automatically refreshed when an update has completed. Simply refresh the page to view the latest status.
  • It is NOT necessary for services under Programs to be running if they have not yet been configured.

Bugs & Workarounds (Snapt UI Checklist)

Framework Version Update

As of 9/20/17, the current Snapt VM Image is at Framework Version 8.7 (2017-05-02). Below identifies some workarounds for bugs which may be encountered.

No changes effected after running Framework Update
  • Inspect the console for Javascript errors. Specifically:

    update:172 Uncaught ReferenceError: doUpdate is not defined
      at HTMLButtonElement.onclick (update:172)
    

    If encountered, it may be due to delayed tab rendering causing elements to become orphaned from their event handlers:

    1. Refresh the UI and wait for Update Results to come through (bottom)
    2. Open the Manual Update tab. The Update Results section may still show "Waiting.." but can be safely ignored.
    3. Click Run Framework Update from this tab. As noted, this may take several minutes, please be patient and do not use the UI.
    4. Once finished, return to Status Check to verify Framework Version is now up to date.

System Patches

Checking for new patches returns 323 patches needed (132 security patches) as of 9/20/17 using the current Snapt VM Image. Attempting to install all non-interactive patches may not work the first time. Below are workarounds:

  • Retry checking for new patches and installing all non-interactive. This second run may take several minutes and hang the UI. Viewing running processes on the server, however, reveals zypper patch is in fact running and as such, the UI should be left alone until the operation has completed on the server:
    ps aux | grep zypper
    
  • Not every patch will be automatically installed (i.e. interactive patches). After non-interactive patches have been installed, the remaining interactive patches should be installed from the server as they may require input, permission to reboot, etc. Repeat the process of checking for new patches and installation until there are no patches remaining:
    sudo zypper patch
    sudo reboot
    

Automatic Updates ("Not Configured")

You may observe differences reported between the Status Check page and the Update Manager as to whether Automatic Updates are in fact configured. As of 9/20/17, the Status Check page should be preferred as it acknowledges Scheduler settings whereas the Update Manager UI appears to require "Include Versions" to also be enabled before it is indicated as "Configured."

Snapt UI Post-Setup (Configuration)

This section will describe numerous post-setup tasks including plugins. If there are any updates pending for Plugins, they should be completed via Setup -> Modules & Plugins -> Updates tab followed by a UI refresh.

Email Notifications

  • Verify Email notifications are enabled and enable if necessary (Setup -> Notifications -> Email). This activates notification settings over personal email settings tied to the user account.
  • Specify a distribution list for multiple recipients to receive notifications (i.e. To: snapt@example.com)

Snapt Redundancy V2

Important: There are two (2) Redundancy plugins: Snapt Redundancy and Snapt Redundancy V2. Only install Snapt Redundancy V2!

  • Install from Setup -> Modules & Plugins -> Add Plugins -> Misc tab
  • Once installed, refresh the Manage Plugins page and begin configuring Virtual IPs at Setup -> Redundancy V2 -> Virtual IP Management

Local Replication

  • Ensure slave nodes are fully up to date with framework, system patches, plugins, and configure Automatic updates
  • Install plugins to match master (i.e. Snapt Balancer, etc.)
  • Copy the Slave Key (Setup -> Redundancy V2 -> Local Replication) and and slave node to master

Caution: If you are configuring Snapt for redundancy and are using their VM image, be advised that each instance will likely have the same hostname. Consequently, synchronizing Redundancy will cause the slave node(s) to receive a "master" configuration and send out gratuitous ARP. To remedy this, ensure all Snapt nodes have their own unique hostname. Upon a successful Force Sync, the slave node should be reverted to Slave Mode, specifying a master node. Once this is done, simply Reload Redundancy and verify all VIPs have switched from ACTIVE to STANDBY status.

Please note the following items which will NOT be synchronized by Redundancy V2
  • Time Sync
  • Email
  • Service Status (Start/Stop)
  • Plugins (i.e. Snapt Cron)
  • /etc/lighttpd/lighttpd.conf (i.e. disabling HTTP 8080)
  • hostname
  • Performance Options (Setup -> Configuration)

Virtual IP Management

Note: Virtual IPs must be managed from this UI before they may be used by a Load Balancer Group.

Begin creating virtual IPs for load balancer groups and Start Redundnacy.

Note: You may get an error indicating "You cannot start redundancy without 2 servers and 1 VIP." However, notice that the VIP you enter still takes effect. This message should be interpreted as "Be sure to add redundant servers before attempting to start Redundancy."

To add redundant Snapt Servers (Setup -> Redundancy V2 -> Server Management)

  • Begin with the current Snapt node you're managing
  • Add a second Snapt node
  • If appropriate, set the current Snapt node you're managing as the master in Setup -> Redundancy V2 -> Settings -> Operation Mode
  • Finally, revisit Snapt Dashboard (Setup -> Redundancy V2) and click Start Redundancy

Snapt Balancer

On 9/20/17, it has been confirmed that at this point, Snapt Image does successfully bring up Virtual IPs whereas the bundle installer on CentOS fails. Root cause appears to be segmentation faults in keepalived and is being tracked at https://github.com/acassen/keepalived/issues/650.

  • Install from Setup -> Modules & Plugins -> Add Plugins -> Snaptins and ensure plugins are up to date
  • Consider enabling Auto-Reload HAProxy when changes are applied (HAProxy -> Configuration -> Snapt Settings)

Create your first load balancer

Note: As of 9/20/17, virtual IPs must be created before they may be used in Balancer group. For convenience this may be done in bulk via Setup -> Redundancy V2 -> Virtual IP Management -> Bulk Add tab (reload required)

The below may be used to create a HTTP load balancer useful for testing purposes:

  • Balancer -> Create a Load Balancer
  • Create a HTTP load balancer
  • Specify a new load balancer group name (i.e. www-test)
    • Note: Due to restrictions in the way Snapt stores certain metrics and the API (since 8/30/17, ticket #2650), load balancer group names must satisfy the regular expression: ^[A-Za-z0-9\-]+$ (i.e. dots in domain names are not allowed)
  • Select the virtual IP address created in Virtual IP Management from the drop down
  • Leave the default HTTP Port (80)
  • Add the IP(s) and respective port(s) for your machine or a test server running a demo web application
  • Click Add Wizard Group. When finished, you will be redirected to the Balancer Dashboard.

Note: Starting the load balancer will fail with the following message if you do not have at least 2 servers as aforementioned or Redundancy is not running.

[ALERT] 262/182729 (3378) : Starting proxy www-test: cannot bind socket [127.0.0.2:80]
A socket bind attempt failed. Either another program is listening on the port or you have duplicate group/frontend ports.

Redundant servers may be added via Setup -> Redundancy V2 -> Server Management and should consist only of Snapt nodes, not your actual load balanced servers.

With Redundancy and VIPs running, Balancer is now ready to be started (Balancer -> Balancer Dashboard -> Start Balancer).

Managing UDP Load Balancers

As of 9/21/17, Snapt Balancer may not be used to create UDP load balancers as it is built on HAProxy. However, Snapt Accelerator is based on NGINX which has recently added support for UDP load balancing via upstream groups.

Using the Snapt UI

To effectively "load balance" UDP services:

  1. Verify Snapt server is running at least NGINX version 1.9.13 (/usr/sbin/nginx -v)
  2. Ensure Snapt Accelerator has been installed (Modules & Plugins -> Add Plugins -> Snaptins)
  3. Define upstreams for each UDP port (Accelerator -> UDP Upstreams)
  4. Add backend servers for each upstream (Accelerator -> UDP Upstreams -> Servers button of active upstream -> Add server tab)
  5. Define virtual IPs for each upstream (Accelerator -> UDP Servers -> Add Server tab)
Bulk Editing (Direct)

A brief inspection of the NGINX configuration on a Snapt server reveals stream groups are written to /etc/nginx/udp_upstreams and /etc/nginx/udp_servers.

Below are examples of editing streams in bulk via heredocs (server access required):

# Define upstreams with backend servers (UDP Upstreams)
cat <<EOF> /etc/nginx/udp_upstreams/service_53.conf
upstream service_53 {
  server host01.example.com:53;
  server host02.example.com:53;
  server host03.example.com:53;
  server host04.example.com:53;
}
EOF

cat <<EOF> /etc/nginx/udp_upstreams/service_137.conf
upstream service_137 {
  server host01.example.com:137;
  server host02.example.com:137;
  server host03.example.com:137;
  server host04.example.com:137;
}
EOF

# Define upstream virtual IPs (UDP Servers)
sudo cat<<EOF> /etc/nginx/udp_servers/service_53.conf
server {
  listen 127.0.0.2:53 udp;
  proxy_pass service_53;
}
EOF

sudo cat<<EOF> /etc/nginx/udp_servers/service_137.conf
server {
  listen 127.0.0.2:137 udp;
  proxy_pass service_137;
}
EOF
Reload and test

When finished, Reload Accelerator and verify virtual IP binding on the server:

netstat -ln
udp        0      0 127.0.0.2:53       0.0.0.0:*
udp        0      0 127.0.0.2:137      0.0.0.0:*

Performance Tuning

A number of highly recommended performance tuning options are easily accessible via: Setup -> Configuration -> Performance Options.

As noted in the UI, it is advisable to enable all available options. A "Recommended Defaults" button is conveniently provided to do this. Be sure to reboot when finished.

Additional resources are available for verifying and further tuning your Snapt environment:

haproxy.cfg

global
  nbproc 2
  cpu-map 1 0
  cpu-map 2 1
  maxconn 2000000

/etc/security/limits.conf

* soft nofile 10000000
* hard nofile 10000000
root soft nofile 10000000
root hard nofile 10000000

/etc/sysctl.conf

Important: Reading up on net.ipv4.tcp_tw_reuse is highly recommended before enabling.

# File descriptors (these settings should be verified on reboot with ulimit -n)
fs.file-max = 10000000
fs.nr_open = 10000000

# TCP Buffer
net.ipv4.tcp_mem = 786432 1697152 1945728
net.ipv4.tcp_rmem = 4096 4096 16777216
net.ipv4.tcp_wmem = 4096 4096 16777216

# Mitigate Port Exhaustion
net.ipv4.ip_local_port_range = 1000 65535
net.ipv4.tcp_tw_reuse = 1

Snapt WAF

  • Install from Setup -> Modules & Plugins -> Add Plugins -> Snaptins and ensure plugins are up to date

Note: Snapt Accelerator plugin must also be installed and activated.

To add custom rulesets

  1. Create a new ruleset at WAF -> WAF Management -> Rulesets -> Create Ruleset tab
  2. Edit the new ruleset, reviewing all existing patterns and making adjustments as necessary. Pay especially close attention to Patterns and the Zones in which they will be analyzed (i.e. Header, Arguments, Body, URL)
  3. When ready, click Enable.

Please note: Only one (1) ruleset may be active at a time.

Load Balancer Migration (Barracuda 540)

Last updated: 3/6/18

This guide outlines steps for migrating services off of a Barracuda 540 appliance with over 200 servers across 88 services.

Taking Inventory

Important: It is highly recommended that you ensure HAProxy has been properly tuned and load tested before migrating production services.

Before getting started, look for and remove any services that are no longer in use. A "report" that can help identify which services are inactive can be fabricated by checking the Server Health page. Look for services which are disabled or have all of its backend servers offline; marked with a red X.

Next, collect the following reports (Basic -> Reports):

  • Services Summary
  • Server Summary

The following should also be noted to minimize disruption and for implementing their HAProxy equivalent:

  • Web Translations (i.e. server-less HTTP-to-HTTPS redirects)
  • Certificates
  • Virtual IPs
  • Content Rules (i.e. layer 7 redirection)
Port 93743
Protocol 2
HostKey /etc/ssh/ssh_host_rsa_key
SyslogFacility AUTHPRIV
LogLevel INFO
LoginGraceTime 60
PermitRootLogin no
MaxAuthTries 4
HostbasedAuthentication no
IgnoreRhosts yes
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
GSSAPIAuthentication no
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding no
PermitUserEnvironment no
ClientAliveInterval 300
ClientAliveCountMax 5
UseDNS no
Banner none
Subsystem sftp /usr/libexec/openssh/sftp-server
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment