Stencil push / pop.

Stencil.cpp

#include <keynote/Stencil.h>

namespace poly {
  Stencil gstencil;

  int Stencil::stencil_level = 0;

  void Stencil::push() {

    stencil_level += 1;
    
    if (1 == stencil_level) {
      glStencilMask(0xFF);
      glClearStencil(0);
      glClear(GL_STENCIL_BUFFER_BIT);
    }

    glEnable(GL_STENCIL_TEST);
    glStencilFunc(GL_ALWAYS, 0, 0);
    glStencilOp(GL_INCR, GL_INCR, GL_INCR);
    glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
  }

  void Stencil::pop() {

    stencil_level -= 1;

    glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
    
    if (0 == stencil_level) {
      glDisable(GL_STENCIL_TEST);
      return;
    }

    glStencilFunc(GL_EQUAL, stencil_level, 0xFF);
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
  }
  
  void Stencil::enable() {
    glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);          /* Enable drawing into the color buffer again. */
    glStencilFunc(GL_EQUAL, stencil_level, 0xFF);             /* When performing the stencil test, we only allow fragments for which the stencil buffer has `stencil_value` to pass. We don't alter the stencil buffer itself by setting all operations to GL_KEEP. */
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);                   /* When we start drawing into the color buffer we want to keep the stencil values. */
  }
  
  void Stencil::disable() {
    glStencilFunc(GL_ALWAYS, 0, 0);
    glStencilOp(GL_DECR, GL_DECR, GL_DECR);
    glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
  }
}

Stencil.h

/*
  How the OpenGL stencil buffer works:

     The short explanation would be something like: you use the 
     stencil buffer to mask out certain areas from your color buffer.
     For example you can define an area in the stencil buffer, then 
     set all values in this area to 1, enable stencilling, set the
     stencil value to 1 and draw your scene. Only the fragments for 
     which the corresponding stencil value is 1 will be drawn.

     A longer explanation:

     The two most important functions that OpenGL provides for 
     using the stencil buffer are `glStencilFunc` and `glStencilOp`.
     There are two modes in which you want to use these functions. 
     First you use these functions to define how you want to draw
     into your stencil buffer and the second mode is used to 
     tell opengl what masking value you want to use when drawing
     the color buffer. To be honest, there is a bit more to it, 
     but I find this the clearest description of how to use the
     stencil buffer; so to recap: you use the stencil functions in 
     two modes:
         
         1. Define how and what you draw into the stencil buffer
         2. Set the stencil-value that you want to use when 
            drawing the color buffer. 

     In this class we use a solution that increments and decrements
     the values in the stencil buffer. We use `glStencilOp` to tell GL
     that it should increment the values in the stencil buffer when we
     start drawing our stencil polygon. So lets say we clear the
     stencil buffer, then draw a rectangle. The result of this drawing
     command is that the values in the stencil buffer are now 1. Then
     when we draw the exact same polygon again, the values in the
     stencil buffer will be 2. Next we can tell GL that it should
     decrement the values in the stencil buffer whenever we draw the
     stencil polygon. So lets say we change the mode and draw the same
     polygon again; now the values in the stencil buffer are 1
     again. To define how GL changes the values in the stencil buffer
     we use `glStencilOp`. For example, we can tell GL that it should
     increment the stencil values when we draw into it using
     `glStencilOp(GL_INCR, GL_INCR, GL_INCR)`.  We pass three
     parameters; each of these parameters is used in a different
     test. e.g. one checks the depth buffer and the stencil buffer and
     only writes when that specific test succeeds. We can also tell
     that GL should decrement the values in the stencil buffer by
     calling `glStencilOp(GL_DECR, GL_DECR, GL_DECR)`.  But before
     you think this is it, I'm sorry, there is one more thing that
     you need to know. The operations that are performed by calling 
     `glStencilOp()` use the outcome of a *test* which is set by 
     calling `glStencilFunc()`.  So the usage of `glStencilFunc` is
     twofold: one we tell what stencil test to use and two: tell
     what stencil value to use when performing the actual clipping. 
     There is also a third aspect to this function, namely the `mask`
     that is used when performing the tests. We don't use this in 
     this class but this `mask` parameter would allow us to implement
     a different, solution to perform hierarchical clipping. 

     When we are drawing the stencil polygon we use
     `glStencilFunc(GL_ALWAYS, 0, 0)` and set the stencil operations
     to increment. This means that the stencil buffer will (GL_)
     ALWAYS be increment when we draw our stencil polygon.  The last
     two parameters to `glStencilFunc` have no meaning in this case.

     Next, when we want to enable the stencil test and draw our color
     buffer we need to call `glStencilFunc(GL_EQUAL, stencil_value,
     0xFF)`. We need to use GL_EQUAL because we want to make sure that
     only the (color) framgents are drawn that have the
     `stencil_value` in our stencil buffer. For every pixel that we
     draw a test is performed that checks the stencil buffer for the
     set stencil_value. But one really *IMPORTANT* thing here is that
     this draw call can also influence what is drawn into the stencil
     buffer. So we need to make sure that when we draw our
     color-buffer (e.g. the visuals that you want to draw), that we
     don't change the values in the stencil buffer. We use
     `glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP)` to tell that whatever we
     draw, we don't want our stencil buffer to change.

     After we've drawn our color-buffer we need to draw our stencil
     polygon again because we want to make sure that the values in the
     stencil buffer are decrement to the values of the "parent".  We
     do this by calling `glStencilOp(GL_DECR, GL_DECR, GL_DECR)` and
     we also need to make sure that when we draw into the stencil
     buffer that all tests succeed, using: `glStencilFunc(GL_ALWAYS,
     0, 0)`.  After calling these functions, we draw our stencil
     polygon which undoes the stencil of the current node.

  How this stencilling code works:
  
     We're using the OpenGL stencil buffer to enable a hierarchy of
     stencil tests. Using the stencil buffer in a hierarchical way is
     not as trivial as one might think, though this solution that was
     based on the [Kivy][0] project works perfectly in all cases that
     I need. This solution does add quite some draw calls, though this
     shouldn't really be a problem. Another solution is to use
     [masks][2] which makes things a lot more complex, especially when
     you want to have a dynamic number of parent > child nodes.

     To use this class you should follow the the steps below. Note
     that it may seem wierd that you need to draw the stencil polygon
     twice, but this is needed because we need to decrement the values
     in the stencil buffer when you're ready with drawing your clipped
     visuals.  Also remember that you don't need to draw your whole
     scene when you want to draw the stencil
     (draw_stencil()). `draw_stencil()` can use a fragment shader
     which is empty, e.g. `main() { }`. The only important thing is
     that the vertices are pushed through the pipeline using a draw
     command.

         1. push()
            draw_stencil()

         2. enable();
            draw_color()
            call_draw_for_all_children()  (children start at 1)

         3. disable()
            draw_stencil() (again, to unset)

         4. pop()

     In pseude code:

     ````c++

         void draw() { 
         
           if (use_stencil) {
               // 1. Push the stencil-value on the stack and draw the stencil polygon.
               
               stencil.push()
               drawStencilPolygon();
               
               // 2. Enable the created stencil and draw your color buffer.
               
               stencil.enable();
               drawColoredScene()
               
                    // 3. IMPORTANT: When you have `children` to draw, 
                    //    call their `draw()` function. The children, 
                    //    will draw their own clipping polygons and 
                    //    incrementing / decrementing the values in the 
                    //    stencil buffer (done by calling the stencil.*()
                    //    functions).
               
                    for each child {
                       draw();
                    }
               
               
               // 4. Clear the stencil mask you created at step 1.
               
               stencil.disable()
               drawStencilPolygon()
               
               // 5. Pop the stencil-value from the stack.
               stencil.pop();
          } 
          else {
            drawColoredScene()
            for each child {
                 draw();
            }
          }

      ````

  Stencil buffer and opacity:

    When you read articles about the using the stencil buffer you'll probably 
    notice that drawing with opacity is of a concern. Most important thing to 
    remember is that you want to draw from "back" to "front" whenever you want
    to use alpha transparency. Because this `Stencil` class was created to draw
    a hierarchy of nodes, this solution will work just fine. But the way we 
    call the children is important; calling `draw()` on the children will increment
    and decrement the values in the stencil buffer; 

  Call trace when applying two stencil masks:

    StencilPush:  increment, value is: 1
    StencilPush:  clear stencil buffer
    StencilUse:   value is: 1
    StencilPush:  increment, value is: 2
    StencilUse:   value is: 2
    StencilUnUse: value is: 2
    StencilPop:   decrement, value is: 1
    StencilUnUse: value is: 1
    StencilPop:   decrement, value is: 0
    StencilPop:   disable, value is: 0


  References:
  
    - See `KeynoteSprite.h`, `KeynoteSpriteTexture`, which make use of this class (gstencil).
    - Used in this test: https://www.flickr.com/photos/diederick/23649658275/in/dateposted-public/

  TODO:
  - @todo the [Kivy project][1] uses a max of 128 layers; I'm not sure why and I'm not using that. 
    We should be able to use 254 layers. 
  
  [0]: https://github.com/kivy/kivy/blob/b72d1735a91b5cabad8343e8d63f84c840df1a96/kivy/graphics/stencil_instructions.pyx "Kivy Stencil Push/Pop/Use/Unuse"
  [1]: https://gist.github.com/roxlu/245d1596e86de9586f9f "Backup of Kivy stencil tests."
  [2]: http://cranialburnout.blogspot.nl/2014/03/nesting-and-overlapping-translucent.html "Nesting and Overlapping Translucent Stenciled Windows"
  [3]: http://cpntools.org/cpn2000/clipping_in_opengl "Clipping in OpenGL" 
  [4]: https://gist.github.com/roxlu/e9056cbfd97d42d5f5ee "Clipping in OpenGL, backup"
  [5]: https://open.gl/depthstencils "Extra Buffer (Stencil, Clipping)"
  [6]: https://www.opengl.org/wiki/Stencil_Test "Stencil Test"
  [7]: http://www.learnopengl.com/#!Advanced-OpenGL/Stencil-testing "Stencil Testing"
  [8]: http://stackoverflow.com/questions/13742556/best-approach-to-draw-clipped-ui-elements-in-opengl "Best approach to draw clipped UI elements in OpenGL"
  [9]: https://gist.github.com/roxlu/6a8785b47e79e3db9e43 "Stencil Example"
  [10]: http://stackoverflow.com/questions/33548848/issue-when-updating-stencil-buffer-in-opengl "Clearing stencil buffer and glStencilMask!! "
  [11]: https://www.opengl.org/discussion_boards/archive/index.php/t-155034.html "Some info on masking"

 */

#ifndef POLY_STENCIL_H
#define POLY_STENCIL_H

#include <glad/glad.h>
#include <poly/Log.h>

namespace poly { 

  /* --------------------------------------------------------------------------------- */
  
  class Stencil {
  public:
    void push();               /* Push a new stencil value onto the stencil stack. This makes sure that stencilling is enabled and that the stencil polygon you draw increments the values in the stencil buffer. */
    void enable();             /* Enable the stencil mask that you drew after you called push(). */
    void disable();            /* Disable the stencil mask, call this before you draw your stencil mask again to clear it in the stencil buffer. */
    void pop();                /* Pop decrements the stencil value and resets to the state before calling push(). This is the last function that you call per stencil operation. */
    
  private:
    static int stencil_level;  /* We keep track of the stencil value in the stencil buffer and use this when you call enable(). enable() basically sets the stencil value that we should use as mask to this value. */
  };

  /* --------------------------------------------------------------------------------- */
  
  extern Stencil gstencil;     /* We provide a global stencil object which maintains state of a global stencil buffer; this kinda makes sense as you probably will also only have one stencil buffer. */

  /* --------------------------------------------------------------------------------- */
  
} /* namespace stencil. */

#endif