Explanation of bug with disabling GC after stack overflow exception

explanation.md

There are two primary places for rescuing stack overflow exceptions in Ruby:

1. If ruby stack ends – it will be checked in vm_call0_body in vm_eval.c (https://github.com/ruby/ruby/blob/trunk/vm_eval.c#L173) and thrown in vm_stackoverflow from vm_inshelper.c (https://github.com/ruby/ruby/blob/trunk/vm_insnhelper.c#L38)
2. But if C stack ends – it will be received as a signal BUS on Mac OS X (https://github.com/ruby/ruby/blob/trunk/signal.c#L850) and as SEGV on Linux (https://github.com/ruby/ruby/blob/trunk/signal.c#L882).

Usually the main thread has a 8MB limit (default on *nix) for C stack size and only 1MB for ruby stack (https://github.com/ruby/ruby/blob/trunk/vm_core.h#L540) But for new thread ruby uses 1MB limit for C stack size (https://github.com/ruby/ruby/blob/trunk/vm_core.h#L538), that is how in example1 and example2 we have C's stack overflow before ruby's.

So our problem is in second case. Rescue flow for those signals are the same – checking previously registered signal and if it is same – abort ruby, but if it is first time – just disable ruby GC (https://github.com/ruby/ruby/blob/trunk/signal.c#L934). In ruby 2.1 and before there was a different behavior – was disabled just stress mode for GC (https://github.com/ruby/ruby/commit/0c391a55d3ed4637e17462d9b9b8aa21e64e2340).

After disabling ruby GC we check for stack overflow

• https://github.com/ruby/ruby/blob/trunk/signal.c#L885 ->
• https://github.com/ruby/ruby/blob/trunk/signal.c#L818 ->
• https://github.com/ruby/ruby/blob/trunk/thread.c#L2113

This will raise the same exception in ruby as in case 1. But GC is already disabled! Variable ruby_disable_gc can only be found in a few places in ruby sources:

• https://github.com/ruby/ruby/blob/trunk/gc.c#L781 - initialize variable with 0 (false)
• https://github.com/ruby/ruby/blob/trunk/gc.c#L6061 - check that heap is ready for GC – wihtout it GC will not start
• https://github.com/ruby/ruby/blob/trunk/signal.c#L934 - in signal.c it is turned off after signal

So it can be turned off permanently, never turned on again and there is no way to find it out as it is internal variable and we don't have ruby accessor for it.