Skip to content

Instantly share code, notes, and snippets.

@gregcotten
Last active June 13, 2022 16:57
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save gregcotten/2583ba117de2a6c79969b3a964239bc8 to your computer and use it in GitHub Desktop.
Save gregcotten/2583ba117de2a6c79969b3a964239bc8 to your computer and use it in GitHub Desktop.
Performs a test to verify that 8, 10, 12, and 16-bit code values can be converted to normalized 32-bit floats, rounded to only 5 decimal places (!!!), and reconverted losslessly back to the original integer code value.
import numpy as np
def float32PrecisionTest(bitdepth, floatingPointDecimalPlaces):
maximumCodeValue = pow(2, bitdepth) - 1
maximumCodeValueAsFloat = np.float32(maximumCodeValue)
failedCodeValues = []
for codeValue in range(0, pow(2, bitdepth)):
normalizedFloatValue = np.round(np.float32(codeValue) / maximumCodeValueAsFloat, floatingPointDecimalPlaces)
rescaledCodeValue = np.uint16(np.round(normalizedFloatValue * maximumCodeValueAsFloat))
if codeValue != rescaledCodeValue:
failedCodeValues.append(codeValue)
return failedCodeValues
floatingPointDecimalPlaces = 5
for bitdepth in [8, 10, 12, 16]:
result = float32PrecisionTest(bitdepth, floatingPointDecimalPlaces)
if len(result) is 0:
print(str(bitdepth) + "-bit test passed.")
else:
print(str(bitdepth) + "-bit test failed for " + str(len(result)) + " code values")
@gregcotten
Copy link
Author

Output of running this:

8-bit test passed.
10-bit test passed.
12-bit test passed.
16-bit test passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment