Instantly share code, notes, and snippets.

Embed
What would you like to do?
Reports for netperf's TCP_CRR test

Reports for netperf's TCP_CRR test (i.e TCP accept() performance)

Below is list of results collected with netperf. The interesting value is TCP_CRR, it measures how fast it can TCP connect/request/response/receive. In short, the transaction rate. The test is used to simulate a normal HTTP/1.0 transaction. What's worrying is that this value has very low on Xen virtualized guests. Performance differences between bare metal and virtualization has been as high as 2-3x.

Virt. platformOSCPU# CPU coresTCP_CRRTCP_RRUDP_RRComments
Bare metal Windows 7 Ultimate Intel Core i5 CPU 760 @ 2.80GHz 4 N/A 56319 48682 Couldn't perform TCP_CRR
Xen Linux 2.6.18-164.2.1.el5xen x86_64 Intel Xeon CPU E5345 @ 2.33GHz 4 6494 24134 26852
Bare metal Linux 2.6.38-8-generic x86_64 #42-Ubuntu Intel Core i5-2300 CPU @ 2.80GHz 4 23230 62036 75954
KVM 2.6.38.6 kernel :) AMD Phenom(tm) II X6 1055T Processor 6(vm has 1) 29279.62 104254.60 159134.70 erh... did I do this wrong???
Xen Linux 2.6.36.4 (custom PV kernel) under Xen 3.2.1 Intel Xeon X3323 @ 2.50GHz 1 9636 36644 44443 Production Memset miniserver
Xen dom0 (closer to the bare metal!) Linux 2.6.26 Intel Xeon X3323 @ 2.50GHz 4 9589 38347 47181 dom0 of the above Memset miniserver host
Xen Linux 2.6.38-x86_64-linode17 x86_64 Intel Xeon CPU L5520 @ 2.27GHz 4 4496 18160 20892
Xen FreeBSD dev 8.2-RELEASE i386 Intel(R) Xeon(R) CPU E5430 @ 2.66GHz 1 9317.74 35000.73 37654.41
BareMetal FreeBSD 8.2-RELEASE amd64 VIA Nano processor U2250 (1.6GHz Capable) 1 468.36 17478.24 24971.05
XEN HVM FreeBSD 8.2-RELEASE FreeBSD amd64 - XEN HVM Intel(R) Xeon(R) CPU X5570 @ 2.93GHz 16 5074.47 8969.24 7874.76

Submitting test results

Linux

It's recommended you run these tests on Linux. You can either download prebuilt binaries from your favorite package system or build from source. The latter is quite easy.

Use these steps to build from source:

cd /tmp
curl -O ftp://ftp.netperf.org/netperf/netperf-2.4.5.tar.gz
tar zvxf netperf-2.4.5.tar.gz
cd /netperf-2.4.5
./configure
make
src/netserver -4
src/netperf -H 127.0.0.1 -t TCP_CRR
src/netperf -H 127.0.0.1 -t TCP_RR
src/netperf -H 127.0.0.1 -t UDP_RR

Get OS version with

uname -a

Get processor info with

cat /proc/cpuinfo

Mac OS X

Been having trouble compiling netperf on Mac OS. Might be an easy fix?

FreeBSD

cd /usr/ports/benchmarks/netperf && make && make install

or

pkg_add -r netperf

Then, same as Linux:

sysctl -a | egrep -i 'hw.model|hw.ncpu'
netserver -4
netperf -H 127.0.0.1 -t TCP_CRR
netperf -H 127.0.0.1 -t TCP_RR
netperf -H 127.0.0.1 -t UDP_RR

Windows

Prebuilt binaries for Windows are available. Get them from http://www.chriswolf.com/download/netperf.zip TCP_CRR are problematic to run. Why I'm not sure, but not surprised this is the case. Linux is recommended for doing these tests.

Test results

Please submit your findings in the table above by editing this Gist. Enter your OS version, CPU, number of cores and the results. It's last column in netperf's output you are supposed to report for each test (the transaction rate). Note any special info such as hyperthreading, special sysctl settings etc in the comments field.

@d1b

This comment has been minimized.

Show comment
Hide comment
@d1b

d1b May 24, 2011

I can't edit this gist, but I got

It probably doesn't matter for the loopback test - but I have some additional sysctl settings on the host -->

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0

Virt. platformOSCPU# CPU coresTCP_CRRTCP_RRUDP_RRComments
KVM 2.6.38.6 kernel :) AMD Phenom(tm) II X6 1055T Processor 6(vm has 1) 29279.62 104254.60 159134.70 erh... did I do this wrong???

d1b commented May 24, 2011

I can't edit this gist, but I got

It probably doesn't matter for the loopback test - but I have some additional sysctl settings on the host -->

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0

Virt. platformOSCPU# CPU coresTCP_CRRTCP_RRUDP_RRComments
KVM 2.6.38.6 kernel :) AMD Phenom(tm) II X6 1055T Processor 6(vm has 1) 29279.62 104254.60 159134.70 erh... did I do this wrong???
@cgbystrom

This comment has been minimized.

Show comment
Hide comment
@cgbystrom

cgbystrom May 24, 2011

It's ok, just post it here in the comments. I'll merge it.
You can also fork it and edit it yourself but I think comments are even easier.

Owner

cgbystrom commented May 24, 2011

It's ok, just post it here in the comments. I'll merge it.
You can also fork it and edit it yourself but I think comments are even easier.

@ppanula

This comment has been minimized.

Show comment
Hide comment
@ppanula

ppanula May 25, 2011

Virt. platform: Xen
OS: 2.6.18-238.9.1.el5xen, 64-bit
CPU: Intel X5650 @ 2.67GHz
vCPU cores: 4
TCP_CRR: 9055.86
TCP_RR: 32914.88
UDP_RR: 45758.49
Comments: Running under Citrix XenServer 5.6.0 FP1, GuestOS: CentOS 5.6 64-bit

ppanula commented May 25, 2011

Virt. platform: Xen
OS: 2.6.18-238.9.1.el5xen, 64-bit
CPU: Intel X5650 @ 2.67GHz
vCPU cores: 4
TCP_CRR: 9055.86
TCP_RR: 32914.88
UDP_RR: 45758.49
Comments: Running under Citrix XenServer 5.6.0 FP1, GuestOS: CentOS 5.6 64-bit

@ppanula

This comment has been minimized.

Show comment
Hide comment
@ppanula

ppanula May 25, 2011

Virt. platform: Xen
OS: 2.6.32-71.29.1.el6.x86_64
CPU: Intel Xeon E5420 @ 2.50GHz
vCPU cores: 1
TCP_CRR: 3276.67
TCP_RR: 24117.42
UDP_RR: 30875.78
Comments: Running under Citrix XenServer 6.0 Beta (Boston). GuestOS: Scientific Linux 6 (RHEL6). It was needed to: echo "32768"> /proc/sys/net/nf_conntrack_max

ppanula commented May 25, 2011

Virt. platform: Xen
OS: 2.6.32-71.29.1.el6.x86_64
CPU: Intel Xeon E5420 @ 2.50GHz
vCPU cores: 1
TCP_CRR: 3276.67
TCP_RR: 24117.42
UDP_RR: 30875.78
Comments: Running under Citrix XenServer 6.0 Beta (Boston). GuestOS: Scientific Linux 6 (RHEL6). It was needed to: echo "32768"> /proc/sys/net/nf_conntrack_max

@ppanula

This comment has been minimized.

Show comment
Hide comment
@ppanula

ppanula May 25, 2011

Virt. platform: Xen
OS: 2.6.18-238.9.1.el5xen, 64-bit
CPU: Intel Xeon E5420 @ 2.50GHz
CPU cores: 1
TCP_CRR: 9591.32 (first run did resulted: 3275.99 iptables was on)
TCP_RR: 39908.04
UDP_RR: 45809.07
Comments: Running under Citrix XenServer 6.0 (Boston). GuestOS: CentOS 5.6 64-bit. iptables was stopped because conntrack and other limits, also it seems it slows down connection rates, etc.

ppanula commented May 25, 2011

Virt. platform: Xen
OS: 2.6.18-238.9.1.el5xen, 64-bit
CPU: Intel Xeon E5420 @ 2.50GHz
CPU cores: 1
TCP_CRR: 9591.32 (first run did resulted: 3275.99 iptables was on)
TCP_RR: 39908.04
UDP_RR: 45809.07
Comments: Running under Citrix XenServer 6.0 (Boston). GuestOS: CentOS 5.6 64-bit. iptables was stopped because conntrack and other limits, also it seems it slows down connection rates, etc.

@ppanula

This comment has been minimized.

Show comment
Hide comment
@ppanula

ppanula May 25, 2011

re-running Scientific Linux 6, without iptables on TCP_CRR gives: 7382.91, also TCP_RR somewhat bigger: 26283.32 and UDP_RR: 33780.30. So results differents if iptables is running or not.

ppanula commented May 25, 2011

re-running Scientific Linux 6, without iptables on TCP_CRR gives: 7382.91, also TCP_RR somewhat bigger: 26283.32 and UDP_RR: 33780.30. So results differents if iptables is running or not.

@teknoraver

This comment has been minimized.

Show comment
Hide comment
@teknoraver

teknoraver Aug 21, 2018

What the TCP_CRR column represent? Transactions per second? Time needed per transaction? If so, milliseconds, microseconds?

teknoraver commented Aug 21, 2018

What the TCP_CRR column represent? Transactions per second? Time needed per transaction? If so, milliseconds, microseconds?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment