So the technique is called Shell Mapping and I got the basics from:
Interactive Smooth and Curved Shell Mapping
https://www.cg.tuwien.ac.at/research/publications/2007/JESCHKE-2007-ISC/
The technique as documented starts with Geometry Shaders which is a really bad idea so I kept things simple and did all the pre-processing on the CPU when meshes were imported. These days I'd look at doing it on-demand in a Compute Shader. Preprocessing steps are:
1. Parameterise a heightmap for your mesh.
2. Duplicate all triangles and extrude them to your max thickness, forming Prisms above each
triangle.
3. Each Prism stores:
* 6 prism vertex positions.
* 3 vertex normals for each vertex at the base of the prism.
* 3 uv co-ordinates for the heightfield at the base.
4. Compress Prism rep as much as possible since there's a lot of BW here.
5. Create a Prism LUT that maps vertices to their parent Prism.
In your shader:
1. Render all Prism triangles
1. Use the current Primitive ID to determine which Prism to load/unpack.
2. Trace rays from the camera into this Prism.
a. Find Prism entry/exit points by direct intersection with all 8 prism triangles.
* Moller-Trumbore is a good start to ensure accuracy but there are faster options for GPU.
* The entry point is actually the object-space pixel position for this primitive.
3. Perform an iterative search to find out how *high* within the Prism each entry/exit point is.
* For surfaces that are close to planar, this is simply the distance from the intersection
point to the plane of the base triangle. This is a *very* useful optimisation to stick on
a material flag.
* All other surfaces will have Prism extrusion normals pointing in different directions and
there will be a non-linear map from Prism height to distance to base plane.
* I used a binary search between min and max shell extrusion heights.
* In practice you can afford a large number of max search iterations as you will find the
result very quickly within reasonable tolerance.
4. Map entry and exit points into texture-space and store in UV. Store Prism height in W.
5. Ray march from entry to exit in UVW space until W goes below the sampled height.
* Sample your material function along the way.
* You actually want this to be some kind of complicated material function and there is
enough BW here to support virtual textures.
6. Write depth to have this nicely intersect with scene geometry.
Optimisation notes:
* This will all happily run on GTX 660 and XBone targets with plenty of frame time for the frame.
* Keep your heightfield textures low-resolution for these targets (e.g. no more than 512) as
the technique will rapidly consume BW and slow things to a crawl.
* Your heightfield rep matters but the best rep actually varies based on your texture source.
* Distance fields/cone stepping only matters on low frequency textures.
* The old trick of larger ray march steps with a binary search around the intersection point
may help but again, it's dependent on source texture frequency.
* Temporal AA with jitter works really well to be reduce your step counts or gain higher accuracy
on thin details like grass or moss.