Linux LVS layer 4 load balancer general notes, pointers and recommendations

gistfile1.txt

=============
GENERAL INFO

Linux Virtual Server (LVS) is a high availability Layer4 load balancer that transparently for the users presents cluster
of real servers as one virtual server.  
Internal cluster architecture is hidden and only IP of virtual server (VIP) is available to the outside world.  
In general LVS is used to load balance TCP and UDP protocols and doesn't care what machines and operating systems are used 
as real servers. 
Properly configured LVS system is able to handle several virtual servers with dozens of real servers in a cluster, 
detect real server malfunction and remove it from cluster in seconds, change weight of real servers based on information 
that describe real servers load (it can be almost anything, ie: CPU load, amount of free memory, number of emails in queue, 
number of idle Apache servers, number of concurrent sessions to database etc.) 
and peform subsecond switch over in case of master LVS failure without breaking existing connection

LVS server consists of following software:
- LVS kernel code that holds table that describes virtual services (IP and port) and real servers associated with it 
so called RIP - real server IP and port). 
There can be many VIPs defined on single LVS machine. 
LVS kernel module supports 10 different load balancing algorithms 
(ie. round robin, weighted least connection, source hashing, destination hasing etc.). 
LVS has three methods of redirecting packets to real servers: NAT (most common), Direct Routing and Tunneling.
- User space application for manipulating kernel tables: ipvsadm (somehow similar to iptables)
- User space application for monitoring health of real servers and getting information about 
real servers load to change weight accordingly


LVS code is available in stock Linux kernel (2.6) and requires no patching.

=============
HA setup

To avoid SPOF you need to run LVS on two machines, one is master the other is slave.

Install keepalived (https://github.com/acassen/keepalived) on both boxes.
Keepalived is a deamon that uses VRRP protocol to monitor master failure and do
a fail-over transition from master to slave.
Test shows that it can switch from master to slave in about second.
It sends gratuitous ARP so the routers can update their tables and use MAC address of the slave.

Keepalived has healthcheck configuration for the real servers (including HTTP GET check that
checks the web server output against the specified digest) with weights, "sorry server" and can
send notification emails.

Keepalived can also change the weights of the real server dynamically.

NOTE: usually it's best to healh check real servers (and change their weight) based
on a high level test, for example, in case of an SMTP server it's good to have it's weight
adjusted based on the number of emails in the mail queue.
In case of the DB server it's a good combo to check cpu load and number of concurrent DB connections.

Health checks should be run pretty often (e.g every 20 seconds) so the load balancer
can adopt to a changing conditions on the real servers (e.g changing their weight, or removing them from 
the kernel LVS table)