Skip to content

Instantly share code, notes, and snippets.

@bdezonia
Last active December 22, 2015 09:59
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bdezonia/6455786 to your computer and use it in GitHub Desktop.
Save bdezonia/6455786 to your computer and use it in GitHub Desktop.
An example of roundoff error in linear scaling
public class Calibration {
private static double m = 0.235;
private static double b = 0.33;
private static double calibrated(double rawValue) {
return m * rawValue + b;
}
private static double scale(double raw1, double raw2) {
return (calibrated(raw2) - calibrated(raw1)) / (raw2 - raw1);
}
public static void run() {
System.out.println(scale(0, 1)); // prints 0.23499999999999993
System.out.println(scale(0.1, 1.1)); // prints 0.235
System.out.println(scale(0, 499)); // prints 0.23499999999999996
}
public static void main(String[] args) {
run();
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment