glm bug report glm::axisAngle()

bug report.md

There seems to be a problem with glm::axisAngle(), a regression brought in with commit 26b3e3ed788f5f6adcde11c5ec6a41aa100d29cd (Fixed axisAngle NaN #638).

Take this example code:

#include <cstdio>

#include "glm/vec3.hpp"
#include "glm/mat4x4.hpp"
#include "glm/gtx/matrix_interpolation.hpp"

int main(void) {
	float angle = 0.0f;
	glm::vec3 axis(0.0f);

	glm::mat4 modelTransform(0.0f);

	modelTransform[0][0] =   0.0f; modelTransform[0][1] = 1.0f; modelTransform[0][2] = 0.0f; modelTransform[0][3] = 0.0f;
	modelTransform[1][0] = - 1.0f; modelTransform[1][1] = 0.0f; modelTransform[1][2] = 0.0f; modelTransform[1][3] = 0.0f;
	modelTransform[2][0] =   0.0f; modelTransform[2][1] = 0.0f; modelTransform[2][2] = 1.0f; modelTransform[2][3] = 0.0f;
	modelTransform[3][0] = -14.0f; modelTransform[3][1] = 3.0f; modelTransform[3][2] = 0.0f; modelTransform[3][3] = 1.0f;

	glm::axisAngle(modelTransform, axis, angle);

	std::fprintf(stderr, "- | %5.3f, %5.3f, %5.3f | %5.3f\n", axis.x, axis.y, axis.z, angle);

	modelTransform[0][0] = - 0.707f; modelTransform[0][1] =  0.707f; modelTransform[0][2] = 0.000f; modelTransform[0][3] = 0.000f;
	modelTransform[1][0] = - 0.707f; modelTransform[1][1] = -0.707f; modelTransform[1][2] = 0.000f; modelTransform[1][3] = 0.000f;
	modelTransform[2][0] =   0.000f; modelTransform[2][1] =  0.000f; modelTransform[2][2] = 1.000f; modelTransform[2][3] = 0.000f;
	modelTransform[3][0] = -17.945f; modelTransform[3][1] = -3.345f; modelTransform[3][2] = 0.384f; modelTransform[3][3] = 1.000f;

	glm::axisAngle(modelTransform, axis, angle);

	std::fprintf(stderr, "- | %5.3f, %5.3f, %5.3f | %5.3f\n", axis.x, axis.y, axis.z, angle);

	return 0;
}

With glm 0.9.8.5 (which was some time before 26b3e3ed7), this prints:

- | 0.000, 0.000, 1.000 | 1.571
- | 0.000, 0.000, 1.000 | 2.356

With glm 0.9.9.8 (after said commit), it prints:

- | 0.000, 0.000, 1.000 | 0.785
- | 0.000, 0.000, 1.000 | 0.785

There seems to be two things wrong with it, both with if-clause in gtx/matrix_interpolation.inl:81:

1. acos(0) is pi / 2, not pi / 4. So the true case should return pi * 0.5, not pi * 0.25
2. The if condition itself triggers too eagerly. Possibly connected to negative values? In the second case, angleCos is negative