gistfile1.txt

The scene has a perspective projection.

I want to know if a fragment is inside the cone formed by the camera and a sphere.

sphereCentre is set by mvMatrix*vec3(0,0,0) and radius is 1; I am drawing a unit sphere.

The camera is at 0,0,0 in view space.

The fragment shader just passes the scene position to the fragment shader:

precision mediump float;
attribute vec3 vertex;
uniform mat4 pMatrix, mvMatrix;
varying vec4 pos;
void main() {
	pos = (mvMatrix * vec4(vertex,1.0));
	gl_Position = pMatrix * pos;
}

The fragment shader wants to find the nearest point on the ray from fragment-to-camera to the sphere's centre, and see if that is in the sphere.  I want this point on the fragment-to-camera ray for later which is why I want to do the cone intersection test this way:

precision mediump float;
uniform vec4 fgColour, bgColour;
uniform vec3 sphereCentre;
uniform float sphereRadius2; // always 1 in my game fwiw
varying vec4 pos;
void main() {
	vec3 p = pos.xyz/pos.w;
	float t = dot(sphereCentre,p)/dot(p,p);
	vec3 d = (p*t)-sphereCentre;
	gl_FragColor = (dot(d,d) < sphereRadius2)? fgColour: bgColour;
}

And, predictably, its not working :(  dot(d,d) is always less than sphereRadius2.  It seems to be always more than 0 but a very very small number.