Address the shadow terminator problem by computing a new shading position

shading_position.hlsl

// Projects the specified position (point) onto the plane with the specified origin and normal.
float3 projectOnPlane(float3 position, float3 origin, float3 normal)
{
    return position - dot(position - origin, normal) * normal;
}

// Computes the shading position of the specified geometric position and vertex positions and
// normals. For a triangle with normals describing a convex surface, this point will be slightly
// above the surface. For a concave surface, the geometry position is used directly.
// NOTE: The difference between the shading position and geometry position is significant when
// casting shadow rays. If the geometric position is used, a triangle may fully shadow itself when
// it should be partly lit based on the shading normals; this is the "shadow terminator" problem.
float3 computeShadingPosition(
    float3 geomPosition, float3 shadingNormal,
    float3 positions[3], float3 normals[3], float3 barycentrics)
{
    // Project the geometric position (inside the triangle) to the planes defined by the vertex
    // positions and normals.
    float3 p0 = projectOnPlane(geomPosition, positions[0], normals[0]);
    float3 p1 = projectOnPlane(geomPosition, positions[1], normals[1]);
    float3 p2 = projectOnPlane(geomPosition, positions[2], normals[2]);

    // Interpolate the projected positions using the barycentric coordinates, which gives the
    // shading position.
    float3 shadingPosition = p0 * barycentrics.x + p1 * barycentrics.y + p2 * barycentrics.z;

    // Return the shading position for a convex triangle, where the shading point is above the
    // triangle based on the shading normal. Otherwise use the geometric position.
    bool convex = dot(shadingPosition - geomPosition, shadingNormal) > 0.0f;
    return convex ? shadingPosition : geomPosition;
}