Many of our optimizations are speculative in nature. They rely on things that are unlikely to change not changing, and on guard checks being placed along the way to assume that expected types are really the way we expected them to be. As a result, being able to break out of the optimized code and return to the safe but unoptimized version is essential. The cheaper we can do this, the more we can speculate without worrying too much over the costs of doing so. Cheap things: gotos, tweaking a couple of pointers. Costly things: mapping between different frame layouts. The latter is unavoidable if deopt is triggered and we have inlinings to undo, but that can be made mostly "just" memcpy-ing regions.
Overall, this hints at - at least initially - keeping the same frame layout for both JITted and interpreted code. That will keep the frame in the shape the interpreter would have kept it, which is important for deopt. Once that is working, we can start to cheat. Every op that MoarVM has is annotated as to whether it's a deopt point or not. We only need to ensure that things are in sync at those points; we can cheat between them as much as we like - or even spill things back upon realizing we must deopt, for guards.
Most stack traces you'll see at C level in MoarVM are very shallow. A VM level call involves entering MVM_frame_invoke, which updates the location of the interpreter. We then return back into its execution loop. We avoid inferior runloops, as they ruin continuations. Instead, we provide a CPS mechanism, which is why you'll see various bits of the VM internals written in a CPS style. The JIT likely wants to take a similar root.
How often will we move between interpreted code and JITted code? Well, it depends. Clearly, a deopt causes a JIT -> interpreter transition. The other thing that may do so - since we don't wish to do much on the C stack - is returning from a frame that was JITted into its caller which was not. We may also invoke a continuation, which takes us back to an non-JITted place. Going the other way, we may make a call that targets something that was already JITted, invoke a continuation that takes us to JITted code, or do some kind of OSR optimization.
For entering the JIT, we add a secret interpreter op that means "enter the JITted code". When we want to run JITted code, MVM_frame_invoke (or continuation invoke, or OSR) will set up the frame appropriately, and then point the interpreter at a (possibly global) piece of memory that has the single "enter the JITted code" operation. Since we will JIT specializations (or probably that's the way to go), we could have the JITted code hang off frame->spesh_cand, even. Then it's simply invoked, just by calling the function pointer. After that, there is a goto to the top of the interpreter. Put another way, by the time we call out of the JITted code again, we expect the interpreter's PC will have been updated.
Just as MVM_frame_invoke updates the PC, we'll have an equivalent thing that the JIT can call when it wishes to invoke something. If we know the target is already JITted we can, of course, obtain the JITted address and just go goto it. If not, we can call something that works out where we're going. If it's to interpreted code, then it should set up the interpreter and return a NULL, which is the signal to the JITted code that it should just return back to the interpreter. If not, it can do a goto of the JITted code. For a return, it's similar.
This means the JITted code doesn't recurse on the C stack when it calls from one MoarVM frame to another; rather, it just obtains the latest register/lexical base pointers from the MVMThreadContext and does a goto. Or the other order, if that works out better (let the target of the goto get the state in shape) - that may be wiser.