Created
October 2, 2014 08:37
-
-
Save nothings/06ac1d8ea61635d09e0f to your computer and use it in GitHub Desktop.
ryg on luajit string-tofloat
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Thinking about this some more: the decimal expansion of the halfway point between two numbers is always going to be either | |
...xyz000000 | |
or | |
...xyz500000 | |
We don't want to deal with the 5s if we can avoid it. Say the longest decimal expansion of a binary float is N digits (it's somewhere around 770, haven't calculated it exactly). Then the halfway point is gonna be all zeros after N+1 decimal digits, guaranteed. | |
To round, we just need to know if we're *exactly* on the edge between two numbers (in which case we might need to employ a tie breaker like round to nearest even) or slightly higher - in effect, is the number "1234.50000" or "1234.50000+"? | |
So we read at least N+1 digits exactly (LuaJIT just reads up to 800, which is definitely enough for doubles); beyond that, we just need to know if any of the following digits is nonzero (we have the '+') or not. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment