Skip to content

Instantly share code, notes, and snippets.

Created December 10, 2015 11:39
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save roxlu/b22729eb4d8f9fa50939 to your computer and use it in GitHub Desktop.
Save roxlu/b22729eb4d8f9fa50939 to your computer and use it in GitHub Desktop.
Stencil push / pop.
#include <keynote/Stencil.h>
namespace poly {
Stencil gstencil;
int Stencil::stencil_level = 0;
void Stencil::push() {
stencil_level += 1;
if (1 == stencil_level) {
glStencilFunc(GL_ALWAYS, 0, 0);
void Stencil::pop() {
stencil_level -= 1;
if (0 == stencil_level) {
glStencilFunc(GL_EQUAL, stencil_level, 0xFF);
void Stencil::enable() {
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); /* Enable drawing into the color buffer again. */
glStencilFunc(GL_EQUAL, stencil_level, 0xFF); /* When performing the stencil test, we only allow fragments for which the stencil buffer has `stencil_value` to pass. We don't alter the stencil buffer itself by setting all operations to GL_KEEP. */
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); /* When we start drawing into the color buffer we want to keep the stencil values. */
void Stencil::disable() {
glStencilFunc(GL_ALWAYS, 0, 0);
How the OpenGL stencil buffer works:
The short explanation would be something like: you use the
stencil buffer to mask out certain areas from your color buffer.
For example you can define an area in the stencil buffer, then
set all values in this area to 1, enable stencilling, set the
stencil value to 1 and draw your scene. Only the fragments for
which the corresponding stencil value is 1 will be drawn.
A longer explanation:
The two most important functions that OpenGL provides for
using the stencil buffer are `glStencilFunc` and `glStencilOp`.
There are two modes in which you want to use these functions.
First you use these functions to define how you want to draw
into your stencil buffer and the second mode is used to
tell opengl what masking value you want to use when drawing
the color buffer. To be honest, there is a bit more to it,
but I find this the clearest description of how to use the
stencil buffer; so to recap: you use the stencil functions in
two modes:
1. Define how and what you draw into the stencil buffer
2. Set the stencil-value that you want to use when
drawing the color buffer.
In this class we use a solution that increments and decrements
the values in the stencil buffer. We use `glStencilOp` to tell GL
that it should increment the values in the stencil buffer when we
start drawing our stencil polygon. So lets say we clear the
stencil buffer, then draw a rectangle. The result of this drawing
command is that the values in the stencil buffer are now 1. Then
when we draw the exact same polygon again, the values in the
stencil buffer will be 2. Next we can tell GL that it should
decrement the values in the stencil buffer whenever we draw the
stencil polygon. So lets say we change the mode and draw the same
polygon again; now the values in the stencil buffer are 1
again. To define how GL changes the values in the stencil buffer
we use `glStencilOp`. For example, we can tell GL that it should
increment the stencil values when we draw into it using
`glStencilOp(GL_INCR, GL_INCR, GL_INCR)`. We pass three
parameters; each of these parameters is used in a different
test. e.g. one checks the depth buffer and the stencil buffer and
only writes when that specific test succeeds. We can also tell
that GL should decrement the values in the stencil buffer by
calling `glStencilOp(GL_DECR, GL_DECR, GL_DECR)`. But before
you think this is it, I'm sorry, there is one more thing that
you need to know. The operations that are performed by calling
`glStencilOp()` use the outcome of a *test* which is set by
calling `glStencilFunc()`. So the usage of `glStencilFunc` is
twofold: one we tell what stencil test to use and two: tell
what stencil value to use when performing the actual clipping.
There is also a third aspect to this function, namely the `mask`
that is used when performing the tests. We don't use this in
this class but this `mask` parameter would allow us to implement
a different, solution to perform hierarchical clipping.
When we are drawing the stencil polygon we use
`glStencilFunc(GL_ALWAYS, 0, 0)` and set the stencil operations
to increment. This means that the stencil buffer will (GL_)
ALWAYS be increment when we draw our stencil polygon. The last
two parameters to `glStencilFunc` have no meaning in this case.
Next, when we want to enable the stencil test and draw our color
buffer we need to call `glStencilFunc(GL_EQUAL, stencil_value,
0xFF)`. We need to use GL_EQUAL because we want to make sure that
only the (color) framgents are drawn that have the
`stencil_value` in our stencil buffer. For every pixel that we
draw a test is performed that checks the stencil buffer for the
set stencil_value. But one really *IMPORTANT* thing here is that
this draw call can also influence what is drawn into the stencil
buffer. So we need to make sure that when we draw our
color-buffer (e.g. the visuals that you want to draw), that we
don't change the values in the stencil buffer. We use
`glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP)` to tell that whatever we
draw, we don't want our stencil buffer to change.
After we've drawn our color-buffer we need to draw our stencil
polygon again because we want to make sure that the values in the
stencil buffer are decrement to the values of the "parent". We
do this by calling `glStencilOp(GL_DECR, GL_DECR, GL_DECR)` and
we also need to make sure that when we draw into the stencil
buffer that all tests succeed, using: `glStencilFunc(GL_ALWAYS,
0, 0)`. After calling these functions, we draw our stencil
polygon which undoes the stencil of the current node.
How this stencilling code works:
We're using the OpenGL stencil buffer to enable a hierarchy of
stencil tests. Using the stencil buffer in a hierarchical way is
not as trivial as one might think, though this solution that was
based on the [Kivy][0] project works perfectly in all cases that
I need. This solution does add quite some draw calls, though this
shouldn't really be a problem. Another solution is to use
[masks][2] which makes things a lot more complex, especially when
you want to have a dynamic number of parent > child nodes.
To use this class you should follow the the steps below. Note
that it may seem wierd that you need to draw the stencil polygon
twice, but this is needed because we need to decrement the values
in the stencil buffer when you're ready with drawing your clipped
visuals. Also remember that you don't need to draw your whole
scene when you want to draw the stencil
(draw_stencil()). `draw_stencil()` can use a fragment shader
which is empty, e.g. `main() { }`. The only important thing is
that the vertices are pushed through the pipeline using a draw
1. push()
2. enable();
call_draw_for_all_children() (children start at 1)
3. disable()
draw_stencil() (again, to unset)
4. pop()
In pseude code:
void draw() {
if (use_stencil) {
// 1. Push the stencil-value on the stack and draw the stencil polygon.
// 2. Enable the created stencil and draw your color buffer.
// 3. IMPORTANT: When you have `children` to draw,
// call their `draw()` function. The children,
// will draw their own clipping polygons and
// incrementing / decrementing the values in the
// stencil buffer (done by calling the stencil.*()
// functions).
for each child {
// 4. Clear the stencil mask you created at step 1.
// 5. Pop the stencil-value from the stack.
else {
for each child {
Stencil buffer and opacity:
When you read articles about the using the stencil buffer you'll probably
notice that drawing with opacity is of a concern. Most important thing to
remember is that you want to draw from "back" to "front" whenever you want
to use alpha transparency. Because this `Stencil` class was created to draw
a hierarchy of nodes, this solution will work just fine. But the way we
call the children is important; calling `draw()` on the children will increment
and decrement the values in the stencil buffer;
Call trace when applying two stencil masks:
StencilPush: increment, value is: 1
StencilPush: clear stencil buffer
StencilUse: value is: 1
StencilPush: increment, value is: 2
StencilUse: value is: 2
StencilUnUse: value is: 2
StencilPop: decrement, value is: 1
StencilUnUse: value is: 1
StencilPop: decrement, value is: 0
StencilPop: disable, value is: 0
- See `KeynoteSprite.h`, `KeynoteSpriteTexture`, which make use of this class (gstencil).
- Used in this test:
- @todo the [Kivy project][1] uses a max of 128 layers; I'm not sure why and I'm not using that.
We should be able to use 254 layers.
[0]: "Kivy Stencil Push/Pop/Use/Unuse"
[1]: "Backup of Kivy stencil tests."
[2]: "Nesting and Overlapping Translucent Stenciled Windows"
[3]: "Clipping in OpenGL"
[4]: "Clipping in OpenGL, backup"
[5]: "Extra Buffer (Stencil, Clipping)"
[6]: "Stencil Test"
[7]:!Advanced-OpenGL/Stencil-testing "Stencil Testing"
[8]: "Best approach to draw clipped UI elements in OpenGL"
[9]: "Stencil Example"
[10]: "Clearing stencil buffer and glStencilMask!! "
[11]: "Some info on masking"
#include <glad/glad.h>
#include <poly/Log.h>
namespace poly {
/* --------------------------------------------------------------------------------- */
class Stencil {
void push(); /* Push a new stencil value onto the stencil stack. This makes sure that stencilling is enabled and that the stencil polygon you draw increments the values in the stencil buffer. */
void enable(); /* Enable the stencil mask that you drew after you called push(). */
void disable(); /* Disable the stencil mask, call this before you draw your stencil mask again to clear it in the stencil buffer. */
void pop(); /* Pop decrements the stencil value and resets to the state before calling push(). This is the last function that you call per stencil operation. */
static int stencil_level; /* We keep track of the stencil value in the stencil buffer and use this when you call enable(). enable() basically sets the stencil value that we should use as mask to this value. */
/* --------------------------------------------------------------------------------- */
extern Stencil gstencil; /* We provide a global stencil object which maintains state of a global stencil buffer; this kinda makes sense as you probably will also only have one stencil buffer. */
/* --------------------------------------------------------------------------------- */
} /* namespace stencil. */
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment