Created
April 6, 2015 18:16
-
-
Save snehesht/ab1f9279bd61cebd5a81 to your computer and use it in GitHub Desktop.
proxy loadbalancing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Src : http://stackoverflow.com/questions/17590816/kernel-based-linux-data-relay-between-two-tcp-sockets | |
# With Nginx | |
TCP Connections load-balancing | |
# With HAProxy | |
Using Linux TCP Splicing with HAProxy | |
Willy Tarreau <w@1wt.eu> | |
- 2007/01/06 - | |
Alexandre Cassen has started a project called Linux Layer7 Switching (L7SW), | |
whose goal is to provide kernel services to help userland proxies achieving | |
very high performance. Right now, the project consists in a loadable kernel | |
module providing TCP Splicing under Linux. | |
TCP Splicing is a method by which a userland proxy can tell the kernel that | |
it considers it has no added value on the data part of a connection, and that | |
the kernel can perform the transfers it itself, thus relieving the proxy from | |
a potentially heavy job. There are two advantages to this method : | |
- it reduces the number of process wakeups | |
- it reduces the number of data copies between user-space and kernel buffers | |
This method is particularly suited to protocols in which data is sent till | |
the end of the session. This is the case for FTP data for instance, and it | |
is also the case for the BODY part of HTTP/1.0. | |
The great news is that haproxy has been designed from the beginning with a | |
clear distinction between the headers and the DATA phase, so it was a child's | |
game to add hooks to Alex's library in it | |
Be careful! Both versions are to be considered BETA software ! Run them on | |
your systems if you want, but do not complain if it crashes twice a day ! | |
Anyway, it seems stable on our test machines. | |
In order to use TCP Splicing on haproxy, you need : | |
- Linux Layer7 Switching code version 0.1.1 : [ http://linux-l7sw.sf.net/ ] | |
- Haproxy version 1.3.5 : [ http://haproxy.1wt.eu/download/1.3/src/ ] | |
Then, you must untar both packages in any location, let's assume you'll | |
be using /tmp. First extract l7sw and : | |
$ cd /tmp | |
$ tar zxf layer7switch-0.1.1.tar.gz | |
$ cd layer7switch-0.1.1 | |
L7SW currently only supports Linux kernel 2.6.19+. If you prefer to use it | |
on a more stable kernel, such as 2.6.16.X, you can apply this patch to the | |
L7SW directory : | |
[ http://haproxy.1wt.eu/download/patches/tcp_splice-0.1.1-linux-2.6.16.diff ] | |
$ patch -p1 -d kernel < tcp_splice-0.1.1-linux-2.6.16.diff | |
Alternatively, if you prefer to run it on 2.4.33+, you can apply this patch | |
to the L7SW directory : | |
[ http://haproxy.1wt.eu/download/patches/tcp_splice-0.1.1-linux-2.4.33.diff ] | |
$ patch -p1 -d kernel < tcp_splice-0.1.1-linux-2.4.33.diff | |
Then build the kernel module as described in the L7SW README. Basically, you | |
just have to do this once your tree has been patched : | |
$ cd kernel | |
$ make | |
You can either install the resulting module (tcp_splice) or load it now. During | |
early testing periods, it might be preferable to avoid installing anything and | |
just load it manually : | |
$ sudo insmod tcp_splice.*o | |
$ cd .. | |
Now that the module is loaded, you need to build the libtcpsplice library on | |
which haproxy currently relies : | |
$ cd userland/libtcpsplice | |
$ make | |
$ cd .. | |
For the adventurous, there's also a proof of concept in the userlan/switchd | |
directory, it may be useful if you encounter problems with haproxy for | |
instance. But it is not needed at all here. | |
OK, L7SW is ready. Now you have to extract haproxy and tell it to build using | |
libtcpsplice : | |
$ cd /tmp | |
$ tar zxf haproxy-1.3.5.tar.gz | |
$ cd haproxy-1.3.5 | |
$ make USE_TCPSPLICE=1 TCPSPLICEDIR=/tmp/layer7switch-0.1.1/userland/libtcpsplice | |
There are other options to make, which are hugely recommended, such as | |
CPU=, REGEX=, and above all, TARGET= so that you use the best syscalls and | |
functions for your system. Generally you will use TARGET=linux26, but 2.4 users | |
with an epoll-patched kernel will use TARGET=linux24e. This is very important | |
because failing to specify those options will disable important optimizations | |
which might hide the tcpsplice benefits ! Please consult the haproxy's README. | |
Now that you have haproxy built with support for tcpsplice, and that the module | |
is loaded, you have to write a config. There is an example in the 'examples' | |
directory. Basically, you just have to add the "option tcpsplice" keyword BOTH | |
in the frontend AND in the backend sections that you want to accelerate. | |
If the option is specified only in the frontend or in the backend, then no | |
acceleration will be used. It is designed this way to allow some front-back | |
combinations to use it without forcing others to use it. Of course, if you use | |
a single "listen" section, you just have to specify it once. | |
As of now (l7sw-0.1.1 and haproxy-1.3.5), you need the CAP_NETADMIN capability | |
to START and to RUN. For human beings, it means that you have to start haproxy | |
as root and keep it running as root, so it must not drop its priviledges. This | |
is somewhat annoying, but we'll try to find a solution later. | |
Also, l7sw-0.1.1 does not yet support TCP window scaling nor SACK. So you have | |
to disable both features on the proxy : | |
$ sudo sysctl -w net.ipv4.tcp_window_scaling=0 | |
$ sudo sysctl -w net.ipv4.tcp_sack=0 | |
$ sudo sysctl -w net.ipv4.tcp_dsack=0 | |
$ sudo sysctl -w net.ipv4.tcp_tw_recycle=1 | |
You can now check that everything works as expected. Run "vmstat 1" or "top" | |
in one terminal, and haproxy in another one : | |
$ sudo ./haproxy -f examples/tcp-splicing-sample.cfg | |
Transfering large file through it should not affect it much. You should observe | |
something like 10% CPU instead of 95% when transferring 1 MB files at full | |
speed. You can play with the tcpsplice option in the configuration to see the | |
effects. | |
Troubleshooting | |
--------------- | |
This software is still beta, and you will probably encounter some caveats. | |
I personnally ran into a few issues that we'll try to address with Alex. First | |
of all, I had occasionnal lockups on my SMP machine which I never had on an UP | |
one. So if you get problems on an SMP machine, please reboot it in UP and do | |
not lose your time on this. | |
I also noticed that sometimes, some sessions remained established even after | |
the end of the program. You might also see some situtations where even after | |
the proxy's exit, the traffic still passes through the system. It may happen | |
when you have a limited source port range and that you reuse a TIME_WAIT | |
session matching exactly the same source and destinations. This will need | |
to be addressed too. | |
You can play with tcp_splice variables and timeouts here in /proc/sys/net/ : | |
$ ls /proc/sys/net/tcp_splice/ | |
debug_level timeout_established timeout_listen timeout_synsent | |
timeout_close timeout_finwait timeout_synack timeout_timewait | |
timeout_closewait timeout_lastack timeout_synrecv | |
$ sysctl net/tcp_splice | |
net.tcp_splice.debug_level = 0 | |
net.tcp_splice.timeout_synack = 120 | |
net.tcp_splice.timeout_listen = 120 | |
net.tcp_splice.timeout_lastack = 30 | |
net.tcp_splice.timeout_closewait = 60 | |
net.tcp_splice.timeout_close = 10 | |
net.tcp_splice.timeout_timewait = 120 | |
net.tcp_splice.timeout_finwait = 120 | |
net.tcp_splice.timeout_synrecv = 60 | |
net.tcp_splice.timeout_synsent = 120 | |
net.tcp_splice.timeout_established = 900 | |
You can also consult the full session list here : | |
$ head /proc/net/tcp_splice_conn | |
FromIP FPrt ToIP TPrt LocalIP LPrt DestIP DPrt State Expires | |
0A000301 4EBB 0A000302 1F40 0A000302 817B 0A000301 0050 CLOSE 7 | |
0A000301 4E9B 0A000302 1F40 0A000302 8165 0A000301 0050 CLOSE 7 | |
Since a session exists at least in CLOSE state for 10 seconds, you just have | |
to consult this entry less than 10 seconds after a test to see a session. | |
Please report your successes, failures, suggestions or fixes to the L7SW | |
mailing list here (do not use the list to report other haproxy bugs) : | |
https://lists.sourceforge.net/lists/listinfo/linux-l7sw-devel | |
Motivations | |
----------- | |
I've always wanted haproxy to be the fastest and most reliable software load | |
balancer available. L7SW is an opportunity to make get a huge performance boost | |
on high traffic sites (eg: photo sharing, streaming, ...). In turn, I find it a | |
shame that Alex wastes his time redevelopping a proxy as a proof of concept for | |
his kernel code. While it is a fun game to enter into, it really becomes harder | |
when you need to get close to customers' needs. So by porting haproxy early to | |
L7SW, I get both the opportunity to get an idea of what it will soon be capable | |
of, and help Alex spend more time on the complex kernel part. | |
Have fun ! | |
Willy |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment