gistfile1.txt

When reloading haproxy too fast on EL7 (RedHat, CentOS) the system is
being filled with orphaned processes.

I encountered this problem on CentOS 7 with
haproxy-1.5.4-4.el7_1.x86_64 but expect it to exist on all systems
using haproxy-systemd-wrapper not just those based on Fedora.

Steps to reproduce:

1) haproxy is running normal.

[root@localhost ~]# ps ax | grep haproxy
 3140 ?        Ss     0:00 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
 3141 ?        S      0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
 3142 ?        Ss     0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

2) Several reloads are executed in quick succession. Problem worsens
when processes happen to execute a reload in parallel.

[root@localhost ~]# while :; do systemctl reload haproxy; done
^C

3) There's multiple haproxy processes running that will never end. As
you can see there's duplicate pids for the -sf arg. Maybe caused by a
race between haproxy-systemd-wrapper reading and the new haproxy
process writing it's pid.

[root@localhost ~]# ps ax | grep haproxy
  423 ?        S      0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 419
  429 ?        S      0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 425
  430 ?        Ss     0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 419
  431 ?        Ss     0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 425
31833 ?        Ss     0:01 /usr/sbin/haproxy-systemd-wrapper -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid
36593 ?        S      0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 36587
36600 ?        Ss     0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 36587
38316 ?        S      0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 38311
38324 ?        Ss     0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 38311
38344 ?        S      0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 38325
38350 ?        Ss     0:00 /usr/sbin/haproxy -f
/etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 38325
...
...


I believe the problem is that there's a race in
haproxy-systemd-wrapper.c line 98 where it's missing a
} else if (nb_pid > 0) { ... block until nb_pid is no longer found in
pidfile. Or something similarly blocking.

Otherwise the parent will accept new SIGUSR2/SIGHUP reloads before the
new haproxy process that was spawned in line 96 has written it's pid
file.

Also note the following from the systemd.service manpage:
"It is strongly recommended to set ExecReload= to a command that not
only triggers a configuration reload of the daemon, but also
synchronously waits for it to complete."
That's currently not the case.