NE1690: Analyse Memory Impact of Pre-Allocated Server Slots for Different Numbers of Routes

NE1690.md

NE-1690 -Analyse Memory Impact of Pre-Allocated Server Slots for Different Numbers of Routes

HAProxy version 2.8.5-aaba8d0 2023/12/07 - https://haproxy.org/

Algorithm=leastconn, Weight=1 maxconn=50000 (T=#Threads B=#Backends)

server-template range: 0..N  T4 B100 RSS (MB)  T4 B1000 RSS (MB)  T4 B10000 RSS (MB)  T64 B100 RSS (MB)  T64 B1000 RSS (MB)  T64 B10000 RSS (MB)
---------------------------  ----------------  -----------------  ------------------  -----------------  ------------------  -------------------
                          0                12                 18                  81                 17                  24                   86
                          1                12                 22                 123                 18                  32                  168
                          2                13                 27                 166                 19                  40                  249
                          5                14                 39                 292                 21                  64                  492
                         10                16                 60                 504                 25                 105                  898
                        100                54                441                4308                 98                 835                 8204
                        200                96                863                8534                180                1647                16322
                        300               139               1286               12761                261                2459                24440

Algorithm=random, Weight=1 maxconn=50000 (T=#Threads B=#Backends)

server-template range: 0..N  T4 B100 RSS (MB)  T4 B1000 RSS (MB)  T4 B10000 RSS (MB)  T64 B100 RSS (MB)  T64 B1000 RSS (MB)  T64 B10000 RSS (MB)
---------------------------  ----------------  -----------------  ------------------  -----------------  ------------------  -------------------
                          0                12                 18                  81                 17                  23                   86
                          1                12                 23                 131                 18                  32                  175
                          2                13                 28                 181                 19                  41                  264
                          5                14                 43                 330                 22                  68                  530
                         10                17                 68                 579                 26                 112                  973
                        100                62                515                5055                106                 910                 8952
                        200               111               1013               10030                194                1797                17817
                        300               161               1510               15004                283                2683                26683

Algorithm=roundrobin, Weight=1 maxconn=50000 (T=#Threads B=#Backends)

server-template range: 0..N  T4 B100 RSS (MB)  T4 B1000 RSS (MB)  T4 B10000 RSS (MB)  T64 B100 RSS (MB)  T64 B1000 RSS (MB)  T64 B10000 RSS (MB)
---------------------------  ----------------  -----------------  ------------------  -----------------  ------------------  -------------------
                          0                12                 18                  80                 17                  24                   86
                          1                12                 22                 123                 18                  32                  168
                          2                13                 26                 166                 19                  40                  249
                          5                14                 39                 293                 21                  64                  492
                         10                16                 60                 504                 25                 105                  898
                        100                54                441                4308                 98                 835                 8204
                        200                96                863                8534                180                1647                16322
                        300               139               1286               12761                261                2459                24440

It would also be interesting to see whether custom certificates, custom HTTP header actions, or other options magnify the cost, but it isn't practical to test everything.

For these numbers, each backend only has balance specified. Literally nothing but balance and some number of server lines.

backend backend-1
    balance {leastconn,random,roundrobin}
    server server_1 127.0.0.1:80 weight 1
    server server_2 127.0.0.1:80 weight 1
    server server_3 127.0.0.1:80 weight 1
    server server_4 127.0.0.1:80 weight 1
    server server_5 127.0.0.1:80 weight 1
    ...

Revision 2 includes updated results, adding 0, 1, and 2 to the #Servers per backend values.

Revision 3 is the same as revision 2 (w.r.t. adding 0,1,2 for '#Servers per backend') but this revision no longer blats out a server line for $N servers and instead uses the server-template:

backend backend-130
    balance roundrobin
    server-template _dynamic-pod- 1-300 172.4.0.4:8765 check disabled

But this is not the end of the story. Simply emitting a server line entry 300 times was ordinarily fine for my configured set of open file limits (1048576) but using the server-template approach I had to significantly raise the limits otherwise HAProxy would fail to start when configured with the upper limits of 10,000 backends, and a server-template _dynamic-pod- 1-300 172.4.0.4:8765 check disabled.

I eventually bumped my limits to:

modified   nixos/configs/user.nix
@@ -62,8 +62,8 @@ in
     # Nix will hit the stack limit when using `nixFlakes`.
     security.pam.loginLimits = [
       { domain = config.users.users.aim.name; item = "stack"; type = "-"; value = "unlimited"; }
-      { domain = config.users.users.aim.name; type = "soft"; item = "nofile"; value = "1048576"; }
-      { domain = config.users.users.aim.name; type = "hard"; item = "nofile"; value = "1048576"; }
+      { domain = config.users.users.aim.name; type = "soft"; item = "nofile"; value = "2097152"; }
+      { domain = config.users.users.aim.name; type = "hard"; item = "nofile"; value = "unlimited"; }
     ];

but it is not clear why the server-template approach appears to need hugely inflated file limits. At the extremes, and without raising file limits, I was seeing:

    [NOTICE]   (60389) : haproxy version is 2.6.13-234aa6d
    [NOTICE]   (60389) : path to executable is /etc/profiles/per-user/aim/bin/ocp-haproxy-2.6.13
    [ALERT]    (60389) : [ocp-haproxy-2.6.13.main()] Cannot raise FD limit to 2100054, limit is 2097152.
    HAProxy failed to start. Exit status: 1

In later revisions I've also noted in the table header that this is when maxconn=50,000.

Revision 4 is using haproxy-2.8.5.

Revision 5 updates the column header to server-template range: 0..N, hopefully clarifying the representation of these numbers. My tooling currently lacks the capability to switch from specifying server-template to emitting a list of individual server lines.