Discovered various interesting things in the process of solving the masking issue, thought it would be good to circulate.
Masking in Unity
So the way masking works in Unity is by doing the following:
- Each object contains a mask object (child) and then a bunch of other child objects
- Mask object is given a masking shader
- At render time, the mask "draws", but doesn't write color to the output buffer, it only writes depth. Another way to visualize this is that it creates an invisible 3D object that still blocks other objects behind it
- When the other objects render, the parts that are behind the invisible objects get masked because the Z-buffer compare fails
- Why can't you have two on the screen at once?
Unity, like most 3D engines, is trying to limit the amount of draw calls, which is the usual bottleneck on performance. To do this, everything in the scene that uses the same shader is essentially batched together and drawn at the same time. Imagine two objects, A and B. In this case, what happens is that Unity draws the mask from A together with the mask from B, and then afterwards draws the other opaque children from A and B. The result of this is that both masks essentially combine additively, which will mean that the mask from B can block the objects from A (or vice versa, depending on Z positioning in the scene).
Render queues and batching
Internally, Unity has a system of "render queues", which basically are buckets that control high-level rendering order - http://docs.unity3d.com/Manual/SL-SubshaderTags.html
Objects are put into render queues, and then everything in the same render queue is eligible to be broken apart and batched. Queues have a integer ID - Background is 1000, Geometry is 2000, AlphaTest is 2450, Transparent is 3000 and Overlay is 4000.
It's important in a 3D engine that partially transparent objects be dynamically Z-sorted front to back (because they don't actually write into the Z-buffer), and apparently in Unity anything that uses a render queue above 2450 is eligible for dynamic Z-sorting. From the Unity docs: "Geometry render queue optimizes the drawing order of the objects for best performance. All other render queues sort objects by distance, starting rendering from the furthest ones and ending with the closest ones."
How to make multiple masks work
To get multiple masks to work right, a few things need to happen:
- Batching has to be disabled, so that the order of the two objects is: mask A draws, objects in A draw, mask B draws, objects in B draw
- A needs to be farther away that B, so that the B objects won't be behind A's mask in terms of Z depth. If A and B are at the same z-level, they won't draw correctly (even with the right draw order) because A and B's mask pixels will be writing to the same Z-buffer zone
To disable batching, we can traverse each object's hierarchy, and assign different render queue ID's to the mask and the non-masked objects. We can also bias this by Z-depth so that the objects that are farther away get lower render queue order (i.e. draw first). So in this example the render queue id's might be:
- A mask - 2500
- A objects - 2501
- B mask - 2502
- B objects - 2503