On a test box, which was receiving light amounts of production traffic, we have seen httpd processes starting to die repeatedly from a certain point in time.
At irregular intervals, an unknown event takes place (probably triggered by some client). After this point, some URLs on the box crash consistently, while other URLs remain completely available. The situation persists and is only relieved by restarting Apache.
I don't have the core file of the initial crash. When I kept individual core files, the filesystem was immediately filled. So I can see only the latest one.
Backtrace of two crashes (note that t.voilabot@orange.com is part of the request, it seems part of a common User-Agent.
#0 0x0000000808960040 in acmp_btree_find (node=0x0, letter=114) at acmp.c:267
267 acmp_btree_node_t *bnode = node->btree;
[New Thread 802806400 (LWP 100143/httpd)]
(gdb) bt
#0 0x0000000808960040 in acmp_btree_find (node=0x0, letter=114) at acmp.c:267
#1 0x000000080896001d in acmp_goto (node=0x0, letter=114) at acmp.c:283
#2 0x000000080895ff22 in acmp_process_quick (acmpt=0x7fffffffcf38, match=0x7fffffffcf50,
data=0x80bbdd39a "t.voilabot@orange.com)", len=68) at acmp.c:577
#3 0x00000008089b90cd in msre_op_pm_execute (msr=0x80bbdd598, rule=0x802bfa7b8, var=0x80bb66770,
error_msg=0x7fffffffd060) at re_operators.c:1464
#4 0x00000008089b42a7 in execute_operator (var=0x80bb66770, rule=0x802bfa7b8, msr=0x80bbdd598,
acting_actionset=0x80acd5020, mptmp=0x80bb66028) at re.c:2650
#5 0x00000008089b3dbd in msre_rule_process_normal (rule=0x802bfa7b8, msr=0x80bbdd598) at re.c:3272
#6 0x00000008089b0d08 in msre_rule_process (rule=0x802bfa7b8, msr=0x80bbdd598) at re.c:3353
#7 0x00000008089b06ab in msre_ruleset_process_phase (ruleset=0x80bb53e10, msr=0x80bbdd598) at re.c:1773
#8 0x000000080897cf08 in modsecurity_process_phase_request_body (msr=0x80bbdd598) at modsecurity.c:555
#9 0x000000080897cd0c in modsecurity_process_phase (msr=0x80bbdd598, phase=2) at modsecurity.c:801
#10 0x0000000808979da5 in hook_request_late (r=0x80bbdc0a0) at mod_security2.c:1037
#11 0x00000000004467f9 in ap_process_request_internal ()
#12 0x00000000004629f5 in ap_process_async_request ()
#13 0x0000000000462aa9 in ap_process_request ()
#14 0x000000000045fa41 in ap_expr_yyrealloc ()
#15 0x00000000004589b6 in ap_process_connection ()
#16 0x0000000000469a36 in ap_set_etag ()
#17 0x0000000000469606 in ap_set_etag ()
#18 0x0000000000468f02 in ap_set_etag ()
#19 0x000000000043621d in ap_run_mpm ()
#20 0x000000000042fbbf in main ()
Another crash:
0 0x0000000808960040 in acmp_btree_find (node=0x0, letter=115) at acmp.c:267
267 acmp_btree_node_t *bnode = node->btree;
[New Thread 802806400 (LWP 100138/httpd)]
(gdb) bt
#0 0x0000000808960040 in acmp_btree_find (node=0x0, letter=115) at acmp.c:267
#1 0x000000080896001d in acmp_goto (node=0x0, letter=115) at acmp.c:283
#2 0x000000080895ff22 in acmp_process_quick (acmpt=0x7fffffffcf38, match=0x7fffffffcf50,
data=0x81e45d23d "", len=5) at acmp.c:577
#3 0x00000008089b90cd in msre_op_pm_execute (msr=0x80bbdf840, rule=0x80af89708, var=0x80bb63168,
error_msg=0x7fffffffd060) at re_operators.c:1464
#4 0x00000008089b42a7 in execute_operator (var=0x80bb63168, rule=0x80af89708, msr=0x80bbdf840,
acting_actionset=0x80afbcfb8, mptmp=0x80bb62028) at re.c:2650
#5 0x00000008089b3dbd in msre_rule_process_normal (rule=0x80af89708, msr=0x80bbdf840) at re.c:3272
#6 0x00000008089b0d08 in msre_rule_process (rule=0x80af89708, msr=0x80bbdf840) at re.c:3353
#7 0x00000008089b06ab in msre_ruleset_process_phase (ruleset=0x80bb52148, msr=0x80bbdf840) at re.c:1773
#8 0x000000080897cf08 in modsecurity_process_phase_request_body (msr=0x80bbdf840) at modsecurity.c:555
#9 0x000000080897cd0c in modsecurity_process_phase (msr=0x80bbdf840, phase=2) at modsecurity.c:801
#10 0x0000000808979da5 in hook_request_late (r=0x80bbde0a0) at mod_security2.c:1037
#11 0x00000000004467f9 in ap_process_request_internal ()
#12 0x00000000004629f5 in ap_process_async_request ()
#13 0x0000000000462aa9 in ap_process_request ()
#14 0x000000000045fa41 in ap_expr_yyrealloc ()
#15 0x00000000004589b6 in ap_process_connection ()
#16 0x0000000000469a36 in ap_set_etag ()
#17 0x0000000000469606 in ap_set_etag ()
#18 0x0000000000468f02 in ap_set_etag ()
#19 0x000000000043621d in ap_run_mpm ()
#20 0x000000000042fbbf in main ()
Lots of different children dying:
Nov 24 01:06:47 test kernel: pid 70021 (httpd), uid 80: exited on signal 11
Nov 24 01:06:47 test kernel: pid 70022 (httpd), uid 80: exited on signal 11
Nov 24 01:06:47 test kernel: pid 70024 (httpd), uid 80: exited on signal 11
Nov 24 01:15:23 test kernel: pid 69835 (httpd), uid 80: exited on signal 11
Nov 24 01:15:23 test kernel: pid 69634 (httpd), uid 80: exited on signal 11
Nov 24 01:19:06 test kernel: pid 69839 (httpd), uid 80: exited on signal 11
Nov 24 01:22:08 test kernel: pid 70027 (httpd), uid 80: exited on signal 11
Nov 24 01:22:32 test kernel: pid 69841 (httpd), uid 80: exited on signal 11
Nov 24 01:22:32 test kernel: pid 70238 (httpd), uid 80: exited on signal 11
Nov 24 01:22:32 test kernel: pid 70239 (httpd), uid 80: exited on signal 11
Nov 24 01:22:32 test kernel: pid 70240 (httpd), uid 80: exited on signal 11
...
Running Apache 2.4.10 prefork on FreeBSD 10.0p12.