Benchmark huge pages

gistfile1.txt

Host
====

Physical machine
2 * Intel X5670 @ 2.93 GHz, HT activated, 24 threads, cpufreq deactivated
48 GB

OS: SLES10 SP3
Linux ncegcolnx243 2.6.16.60-0.54.5-smp #1 SMP Fri Sep 4 01:28:03 UTC 2009 x86_64 x86_64 x86_64 GNU/Linux

MemTotal:     49452408 kB
MemFree:       9745596 kB
Buffers:        721068 kB
Cached:        4744648 kB
SwapCached:       1188 kB
Active:        3621452 kB
Inactive:      1953992 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     49452408 kB
LowFree:       9745596 kB
SwapTotal:     8393920 kB
SwapFree:      8391420 kB
Dirty:              12 kB
Writeback:           0 kB
AnonPages:      106088 kB
Mapped:          30396 kB
Slab:           527300 kB
CommitLimit:  16342908 kB
Committed_AS:   580044 kB
PageTables:       3312 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    266784 kB
VmallocChunk: 34359471279 kB
HugePages_Total: 16384
HugePages_Free:  16384
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

processor       : 23
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz
stepping        : 2
cpu MHz         : 2926.082
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 10
cpu cores       : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc nonstop_tsc pni monitor ds_cpl vmx smx est tm2 cx16 xtpr dca popcnt lahf_lm ida arat
bogomips        : 5852.17
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

available: 2 nodes (0-1)
node 0 size: 24240 MB
node 0 free: 2773 MB
node 1 size: 24215 MB
node 1 free: 6730 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10

policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
cpubind: 0 1
nodebind: 0 1
membind: 0 1

/sys/devices/system/node/node0:
total 0
drwxr-xr-x 2 root root    0 Oct  8 13:31 .
drwxr-xr-x 4 root root    0 Aug 25 17:13 ..
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu0 -> ../../../../devices/system/cpu/cpu0
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu10 -> ../../../../devices/system/cpu/cpu10
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu12 -> ../../../../devices/system/cpu/cpu12
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu14 -> ../../../../devices/system/cpu/cpu14
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu16 -> ../../../../devices/system/cpu/cpu16
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu18 -> ../../../../devices/system/cpu/cpu18
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu2 -> ../../../../devices/system/cpu/cpu2
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu20 -> ../../../../devices/system/cpu/cpu20
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu22 -> ../../../../devices/system/cpu/cpu22
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu4 -> ../../../../devices/system/cpu/cpu4
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu6 -> ../../../../devices/system/cpu/cpu6
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu8 -> ../../../../devices/system/cpu/cpu8
-r--r--r-- 1 root root 4096 Oct  8 13:31 cpumap
-r--r--r-- 1 root root 4096 Oct  8 13:30 distance
-r--r--r-- 1 root root 4096 Oct  8 13:30 meminfo
-r--r--r-- 1 root root 4096 Oct  8 13:29 numastat

/sys/devices/system/node/node1:
total 0
drwxr-xr-x 2 root root    0 Oct  8 13:31 .
drwxr-xr-x 4 root root    0 Aug 25 17:13 ..
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu1 -> ../../../../devices/system/cpu/cpu1
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu11 -> ../../../../devices/system/cpu/cpu11
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu13 -> ../../../../devices/system/cpu/cpu13
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu15 -> ../../../../devices/system/cpu/cpu15
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu17 -> ../../../../devices/system/cpu/cpu17
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu19 -> ../../../../devices/system/cpu/cpu19
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu21 -> ../../../../devices/system/cpu/cpu21
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu23 -> ../../../../devices/system/cpu/cpu23
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu3 -> ../../../../devices/system/cpu/cpu3
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu5 -> ../../../../devices/system/cpu/cpu5
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu7 -> ../../../../devices/system/cpu/cpu7
lrwxrwxrwx 1 root root    0 Oct  8 13:41 cpu9 -> ../../../../devices/system/cpu/cpu9
-r--r--r-- 1 root root 4096 Oct  8 13:31 cpumap
-r--r--r-- 1 root root 4096 Oct  8 13:30 distance
-r--r--r-- 1 root root 4096 Oct  8 13:30 meminfo
-r--r--r-- 1 root root 4096 Oct  8 13:29 numastat

Result of redis-benchmark of this machine:
------------------------------------------

Using loopback
PING (inline): 135501.36 requests per second
PING: 136798.91 requests per second
MSET (10 keys): 78864.35 requests per second
SET: 134770.89 requests per second
GET: 135685.22 requests per second
INCR: 133868.81 requests per second
LPUSH: 134952.77 requests per second
LPOP: 134952.77 requests per second
SADD: 134048.27 requests per second
SPOP: 134048.27 requests per second
LPUSH (again, in order to bench LRANGE): 134228.19 requests per second
LRANGE (first 100 elements): 78988.94 requests per second
LRANGE (first 300 elements): 41614.64 requests per second
LRANGE (first 450 elements): 29994.00 requests per second
LRANGE (first 600 elements): 24195.50 requests per second

Using Unix domain socket
PING (inline): 194552.53 requests per second
PING: 194931.77 requests per second
MSET (10 keys): 96805.42 requests per second
SET: 194931.77 requests per second
GET: 193423.59 requests per second
INCR: 194931.77 requests per second
LPUSH: 196463.66 requests per second
LPOP: 194174.77 requests per second
SADD: 194174.77 requests per second
SPOP: 192307.70 requests per second
LPUSH (again, in order to bench LRANGE): 196078.44 requests per second
LRANGE (first 100 elements): 94966.77 requests per second
LRANGE (first 300 elements): 46339.20 requests per second
LRANGE (first 450 elements): 33333.33 requests per second
LRANGE (first 600 elements): 25866.53 requests per second


Redis
=====

Version 2.2.12 + patch to fix LRANGE issue + CPU affinity patch + COW ratio patch
Size of the dump file:    3 GB
Peak memory consumption: 24 GB (working set after filling or loading)

Compiled with huge page support
https://gist.github.com/1240452

Benchmark program
https://gist.github.com/1272522

COW ratio patch
https://gist.github.com/1240427

Note: the COW ratio patch returns meaningless results when huge pages support is activated.
This is due to the fact huge pages are not tracked correctly in the page map of the process.
With huge pages, COW ratio has been evaluated manually with /proc/meminfo by comparing
HugePages_Total and HugePages_Free.

Huge pages area was hard limited to 32 GB (working set was about 24 GB)
So about one third of Redis memory was still available to support COW at bgsave time.

Benchmark
=========

Fill use case generates about 50M of write queries (multiplied by 3 clients)
It is used to build the working set.
 
Read use case generates about 50M of random read queries (multiplied by 3 clients)
It is used to evaluate the read throughput

Update use case generates random write queries, with a tunable throttle to limit the throughput.
It is used to test COW efficiency (by running it concurrencly to a bgsave)

All tests are done using Unix domain sockets.
Clients are bound on the same CPU than Redis server, but on different cores
(enforced using CPU affinity). This is the most efficient configuration.

Fill, 3 clients, with HP
------------------------

ncegcolnx243:genload> x 0 0  68.20s user 2.98s system 15% cpu 7:31.85 total
ncegcolnx243:genload> x 0 1  75.28s user 3.76s system 17% cpu 7:31.90 total
ncegcolnx243:genload> x 0 2  77.30s user 3.48s system 17% cpu 7:31.94 total

=> throughput = 331931 q/s

used_cpu_sys:376.68
used_cpu_user:18.35

used_memory:18810796424
used_memory_human:17.52G
used_memory_rss:25773010944
mem_fragmentation_ratio:1.37

Read queries, 3 clients, with HP
--------------------------------

ncegcolnx243:genload> x 1 0  85.68s user 3.04s system 23% cpu 6:09.86 total
ncegcolnx243:genload> x 1 1  83.82s user 2.98s system 23% cpu 6:09.86 total
ncegcolnx243:genload> x 1 2  90.71s user 1.62s system 24% cpu 6:09.86 total

=> throughput = 405558 q/s

used_cpu_sys:736.58
used_cpu_user:28.31

Bgsave with HP
--------------

[6150] 08 Oct 15:51:51 * Fork: 3893
[6150] 08 Oct 15:51:51 * Background saving started by pid 7487
[7487] 08 Oct 15:52:57 * DB saved on disk

=> fork latency = 4 ms
=> duration = 66 secs

Restart with HP
---------------

[7630] 08 Oct 15:57:22 * Server started, Redis version 2.2.12
[7630] 08 Oct 15:58:22 * DB loaded from disk: 60 seconds

Fill, 3 clients, without HP
---------------------------

ncegcolnx243:genload> x 0 0  68.89s user 2.04s system 15% cpu 7:38.15 total
ncegcolnx243:genload> x 0 2  67.00s user 3.56s system 15% cpu 7:38.22 total
ncegcolnx243:genload> x 0 1  74.39s user 2.95s system 16% cpu 7:38.24 total

=> throughput = 327367 q/s

used_cpu_sys:381.86
used_cpu_user:24.17

used_memory:18810795776
used_memory_human:17.52G
used_memory_rss:25715482624
mem_fragmentation_ratio:1.37

Read queries, 3 clients, without HP
-----------------------------------

ncegcolnx243:genload> x 1 0  81.52s user 2.60s system 22% cpu 6:21.30 total
ncegcolnx243:genload> x 1 1  84.88s user 1.30s system 22% cpu 6:21.30 total
ncegcolnx243:genload> x 1 2  87.23s user 2.46s system 23% cpu 6:21.30 total

=> throughput = 393391 q/s

used_cpu_sys:752.62
used_cpu_user:34.69

Bgsave without HP
-----------------

[17603] 08 Oct 18:34:06 * Fork: 286990
[17603] 08 Oct 18:34:06 * Background saving started by pid 18600
[18600] 08 Oct 18:35:13 * DB saved on disk

=> fork latency = 287 ms
=> duration = 67 secs

Restart without HP
------------------

[19396] 08 Oct 18:48:38 * Server started, Redis version 2.2.12
[19396] 08 Oct 18:49:41 * DB loaded from disk: 63 seconds

COW efficiency evaluation
-------------------------

With huge pages, at only 60 w/s, ratio is about 25%
With higher throughput (for instance 120 w/s), the limit of 32 GB is quickly reached, and Redis is killed,
which means too many pages (more than one third) are copied.

Without huge pages, at 60 w/s, ratio is negligible
Without huge pages, at 5000 w/s, ratio is about 5%

Final results
=============

Here are all the results in a single table:

                             With HP    Without HP   Ratio %

Throughput fill (q/s)         331931        327367    101.39
CPU user                      376.68        381.86     98.64
CPU sys                        18.35         24.17     75.92
CPU total                     395.03        406.03     97.29

Throughput read queries (q/s) 405558        393391    103.09
CPU user                      359.9         370.76     97.07
CPU sys                         9.96         10.52     94.68
CPU total                     369.86        381.28     97.00

Fork latency (ms)                  4           287      1.39
BGSAVE duration (s)               66            67     98.51
Load duration (s)                 60            63     95.24

We can see the gain in throughput due to huge pages is
between 1 and 4 % (i.e. a few percent only). The gain
in system CPU is about 25% at object creation time
(but system CPU accounts for a tiny fraction of the
total CPU consumption). There is also a small gain of
1.5% and 5% at save and load time.

It is clear that activating huge pages to boost general
performance of Redis is not really worth it.

The most noticeable benefit is of course the fork latency
which dramatically drops from 287 to only 4 ms (for a 24 GB
instance).

COW efficiency is abysmal with huge pages. Even with very low
update rate, most of the pages end up duplicated very quickly.
It is almost mandatory to provision twice the memory to support
background save, unless some strong locality of the traffic can
be exploited.