Last updated: 3/6/18
This installation guide applies to the recommended Snapt VM Image (OpenSUSE) from https://downloads.snapt.net/
- Login to the server (SSH) as
snapt:snaptand change account password - Change the default
rootaccount password (root:root) - Copy SSH key to server
- Append SSH key to
/home/snapt/.ssh/authorized_keys - Apply hardened SSH configuration (see
sshd_config.sample) and reload withservice sshd reload
- Append SSH key to
- Review
/etc/crontabfor Snapt cron job (/srv/www/htdocs/bin/cronjob.php) and replacerootuser withlighttpd. For more insight as to what Snapt's cron job is doing, consider installing the Snapt Cron plugin. - Disable management connections on port 8080 via: Setup -> Configuration -> Snapt Configuration -> Set HTTP Access to Off.
- Note: This will allow you to still connect to Snapt UI over HTTPS on port 8081 but will set the unencrypted HTTP 8080 port to bind to 127.0.0.1 only.
Important: If you encounter any bugs (i.e. changes not taking effect, etc.), please review the Bugs & Workarounds section for known workarounds.
To begin, login to the UI using https:// on port 8081
Before installing any plugins or performing routine maintenance, be sure to complete a preliminary checklist (Dashboard -> Status Check) in the following order:
- Time Sync (time.example.com)
- Email (From snapt@example.com; SMTP mail.example.com:25)
- Framework Version -> Run Framework Update (note the message in progress, "This may take several minutes, please be patient and do not use the UI.")
- Automatic Updates (Scheduler)
- Tip: Consider enabling automatic updates at weekly intervals but leaving automatic Framework Updates (Versions) disabled.
- When using the Snapt Image, you can check for system patches (Setup -> Update Manager -> System Patches tab)
- Tip: Consider installing all non-interactive patches. This will create a job ("Patches install") that will be processed in the background.
- If necessary, apply any plugin updates available (Setup -> Modules & Plugins -> Updates tab)
Note:
- Not every UI will be automatically refreshed when an update has completed. Simply refresh the page to view the latest status.
- It is NOT necessary for services under Programs to be running if they have not yet been configured.
As of 9/20/17, the current Snapt VM Image is at Framework Version 8.7 (2017-05-02). Below identifies some workarounds for bugs which may be encountered.
-
Inspect the console for Javascript errors. Specifically:
update:172 Uncaught ReferenceError: doUpdate is not defined at HTMLButtonElement.onclick (update:172)If encountered, it may be due to delayed tab rendering causing elements to become orphaned from their event handlers:
- Refresh the UI and wait for Update Results to come through (bottom)
- Open the Manual Update tab. The Update Results section may still show "Waiting.." but can be safely ignored.
- Click Run Framework Update from this tab. As noted, this may take several minutes, please be patient and do not use the UI.
- Once finished, return to Status Check to verify Framework Version is now up to date.
Checking for new patches returns 323 patches needed (132 security patches) as of 9/20/17 using the current Snapt VM Image. Attempting to install all non-interactive patches may not work the first time. Below are workarounds:
- Retry checking for new patches and installing all non-interactive. This second run may take several minutes and hang the UI. Viewing running processes on the server, however, reveals
zypper patchis in fact running and as such, the UI should be left alone until the operation has completed on the server:ps aux | grep zypper - Not every patch will be automatically installed (i.e. interactive patches). After non-interactive patches have been installed, the remaining interactive patches should be installed from the server as they may require input, permission to reboot, etc. Repeat the process of checking for new patches and installation until there are no patches remaining:
sudo zypper patch sudo reboot
You may observe differences reported between the Status Check page and the Update Manager as to whether Automatic Updates are in fact configured. As of 9/20/17, the Status Check page should be preferred as it acknowledges Scheduler settings whereas the Update Manager UI appears to require "Include Versions" to also be enabled before it is indicated as "Configured."
This section will describe numerous post-setup tasks including plugins. If there are any updates pending for Plugins, they should be completed via Setup -> Modules & Plugins -> Updates tab followed by a UI refresh.
- Verify Email notifications are enabled and enable if necessary (Setup -> Notifications -> Email). This activates notification settings over personal email settings tied to the user account.
- Specify a distribution list for multiple recipients to receive notifications (i.e. To: snapt@example.com)
Important: There are two (2) Redundancy plugins: Snapt Redundancy and Snapt Redundancy V2. Only install Snapt Redundancy V2!
- Install from Setup -> Modules & Plugins -> Add Plugins -> Misc tab
- Once installed, refresh the Manage Plugins page and begin configuring Virtual IPs at Setup -> Redundancy V2 -> Virtual IP Management
- Ensure slave nodes are fully up to date with framework, system patches, plugins, and configure Automatic updates
- Install plugins to match master (i.e. Snapt Balancer, etc.)
- Copy the Slave Key (Setup -> Redundancy V2 -> Local Replication) and and slave node to master
Caution: If you are configuring Snapt for redundancy and are using their VM image, be advised that each instance will likely have the same hostname. Consequently, synchronizing Redundancy will cause the slave node(s) to receive a "master" configuration and send out gratuitous ARP. To remedy this, ensure all Snapt nodes have their own unique hostname. Upon a successful Force Sync, the slave node should be reverted to Slave Mode, specifying a master node. Once this is done, simply Reload Redundancy and verify all VIPs have switched from
ACTIVEtoSTANDBYstatus.
- Time Sync
- Service Status (Start/Stop)
- Plugins (i.e. Snapt Cron)
- /etc/lighttpd/lighttpd.conf (i.e. disabling HTTP 8080)
- hostname
- Performance Options (Setup -> Configuration)
Note: Virtual IPs must be managed from this UI before they may be used by a Load Balancer Group.
Begin creating virtual IPs for load balancer groups and Start Redundnacy.
Note: You may get an error indicating "You cannot start redundancy without 2 servers and 1 VIP." However, notice that the VIP you enter still takes effect. This message should be interpreted as "Be sure to add redundant servers before attempting to start Redundancy."
To add redundant Snapt Servers (Setup -> Redundancy V2 -> Server Management)
- Begin with the current Snapt node you're managing
- Add a second Snapt node
- If appropriate, set the current Snapt node you're managing as the master in Setup -> Redundancy V2 -> Settings -> Operation Mode
- Finally, revisit Snapt Dashboard (Setup -> Redundancy V2) and click Start Redundancy
On 9/20/17, it has been confirmed that at this point, Snapt Image does successfully bring up Virtual IPs whereas the bundle installer on CentOS fails. Root cause appears to be segmentation faults in
keepalivedand is being tracked at acassen/keepalived#650.
- Install from Setup -> Modules & Plugins -> Add Plugins -> Snaptins and ensure plugins are up to date
- Consider enabling Auto-Reload HAProxy when changes are applied (HAProxy -> Configuration -> Snapt Settings)
Note: As of 9/20/17, virtual IPs must be created before they may be used in Balancer group. For convenience this may be done in bulk via Setup -> Redundancy V2 -> Virtual IP Management -> Bulk Add tab (reload required)
The below may be used to create a HTTP load balancer useful for testing purposes:
- Balancer -> Create a Load Balancer
- Create a HTTP load balancer
- Specify a new load balancer group name (i.e. www-test)
- Note: Due to restrictions in the way Snapt stores certain metrics and the API (since 8/30/17, ticket #2650), load balancer group names must satisfy the regular expression:
^[A-Za-z0-9\-]+$(i.e. dots in domain names are not allowed)
- Note: Due to restrictions in the way Snapt stores certain metrics and the API (since 8/30/17, ticket #2650), load balancer group names must satisfy the regular expression:
- Select the virtual IP address created in Virtual IP Management from the drop down
- Leave the default HTTP Port (80)
- Add the IP(s) and respective port(s) for your machine or a test server running a demo web application
- Click Add Wizard Group. When finished, you will be redirected to the Balancer Dashboard.
Note: Starting the load balancer will fail with the following message if you do not have at least 2 servers as aforementioned or Redundancy is not running.
[ALERT] 262/182729 (3378) : Starting proxy www-test: cannot bind socket [127.0.0.2:80]
A socket bind attempt failed. Either another program is listening on the port or you have duplicate group/frontend ports.
Redundant servers may be added via Setup -> Redundancy V2 -> Server Management and should consist only of Snapt nodes, not your actual load balanced servers.
With Redundancy and VIPs running, Balancer is now ready to be started (Balancer -> Balancer Dashboard -> Start Balancer).
As of 9/21/17, Snapt Balancer may not be used to create UDP load balancers as it is built on HAProxy. However, Snapt Accelerator is based on NGINX which has recently added support for UDP load balancing via
upstreamgroups.
To effectively "load balance" UDP services:
- Verify Snapt server is running at least NGINX version 1.9.13 (
/usr/sbin/nginx -v) - Ensure Snapt Accelerator has been installed (Modules & Plugins -> Add Plugins -> Snaptins)
- Define upstreams for each UDP port (Accelerator -> UDP Upstreams)
- Add backend servers for each upstream (Accelerator -> UDP Upstreams -> Servers button of active upstream -> Add server tab)
- Define virtual IPs for each upstream (Accelerator -> UDP Servers -> Add Server tab)
A brief inspection of the NGINX configuration on a Snapt server reveals stream groups are written to
/etc/nginx/udp_upstreamsand/etc/nginx/udp_servers.
Below are examples of editing streams in bulk via heredocs (server access required):
# Define upstreams with backend servers (UDP Upstreams)
cat <<EOF> /etc/nginx/udp_upstreams/service_53.conf
upstream service_53 {
server host01.example.com:53;
server host02.example.com:53;
server host03.example.com:53;
server host04.example.com:53;
}
EOF
cat <<EOF> /etc/nginx/udp_upstreams/service_137.conf
upstream service_137 {
server host01.example.com:137;
server host02.example.com:137;
server host03.example.com:137;
server host04.example.com:137;
}
EOF
# Define upstream virtual IPs (UDP Servers)
sudo cat<<EOF> /etc/nginx/udp_servers/service_53.conf
server {
listen 127.0.0.2:53 udp;
proxy_pass service_53;
}
EOF
sudo cat<<EOF> /etc/nginx/udp_servers/service_137.conf
server {
listen 127.0.0.2:137 udp;
proxy_pass service_137;
}
EOF
When finished, Reload Accelerator and verify virtual IP binding on the server:
netstat -ln
udp 0 0 127.0.0.2:53 0.0.0.0:*
udp 0 0 127.0.0.2:137 0.0.0.0:*
A number of highly recommended performance tuning options are easily accessible via: Setup -> Configuration -> Performance Options.
As noted in the UI, it is advisable to enable all available options. A "Recommended Defaults" button is conveniently provided to do this. Be sure to reboot when finished.
Additional resources are available for verifying and further tuning your Snapt environment:
- HAProxy Sizing Recommendations
- Load Testing HAProxy (Part 1, Part 2, Part 3)
- Stack Exchange Case Study
- Tools for checking CPU information in Linux
- Using
ssfor monitoring network connections - Tuning HAProxy for 300K concurrent TCP socket connections
- Mapping HAProxy processes to CPU cores
- Multithreaded remote shell client for executing commands on multiple hosts in parallel
- Increase OS UDP Buffer to Improve Performance
- UDP Drops on Linux
- Tuning Your Linux Kernel and HAProxy Instance for High Loads
- Tuning NGINX
global
nbproc 2
cpu-map 1 0
cpu-map 2 1
maxconn 2000000
* soft nofile 10000000
* hard nofile 10000000
root soft nofile 10000000
root hard nofile 10000000
Important: Reading up on
net.ipv4.tcp_tw_reuseis highly recommended before enabling.
# File descriptors (these settings should be verified on reboot with ulimit -n)
fs.file-max = 10000000
fs.nr_open = 10000000
# TCP Buffer
net.ipv4.tcp_mem = 786432 1697152 1945728
net.ipv4.tcp_rmem = 4096 4096 16777216
net.ipv4.tcp_wmem = 4096 4096 16777216
# Mitigate Port Exhaustion
net.ipv4.ip_local_port_range = 1000 65535
net.ipv4.tcp_tw_reuse = 1
- Install from Setup -> Modules & Plugins -> Add Plugins -> Snaptins and ensure plugins are up to date
Note: Snapt Accelerator plugin must also be installed and activated.
- Create a new ruleset at WAF -> WAF Management -> Rulesets -> Create Ruleset tab
- Edit the new ruleset, reviewing all existing patterns and making adjustments as necessary. Pay especially close attention to Patterns and the Zones in which they will be analyzed (i.e. Header, Arguments, Body, URL)
- When ready, click Enable.
Please note: Only one (1) ruleset may be active at a time.