Here is the code:
local number1 = (100 + 1/3) - 100 local number2 = 1/3 print("values:", number1, number2) if number1 == number2 then print("they are the same") else print("nope") end
Here is the output:
values: 0.33333333333333 0.33333333333333
nope
And no, I don’t know why this is happening. I have a guess that it might be the underlying type.
This is a well known phenomenon. Remember, at the end of the day, these values are stored in a binary encoded format and there will be precision and rounding issues based on the number of bits available for the exponent and mantissa.
https://www.doc.ic.ac.uk/~eedwards/compsys/float/
http://floating-point-gui.de/formats/fp/
http://floating-point-gui.de/errors/comparison/
https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
To solve this, simply apply floor, ceil, or your own rounding :
For your specific example, this fixes the problem:
local function round(val, n) if (n) then return math.floor( (val \* 10^n) + 0.5) / (10^n) else return math.floor(val+0.5) end end local number1 = (100 + 1/3) - 100 local number2 = 1/3 print("1 values:", number1, number2) if number1 == number2 then print("1 they are the same") else print("1 nope") end local number1 = round((100 + 1/3) - 100,14) local number2 = round(1/3,14) print("2 values:", number1, number2) if number1 == number2 then print("2 they are the same") else print("2 nope") endOutput of above:
11:01:02.003 1 values: 0.33333333333333 0.33333333333333 11:01:02.003 1 nope 11:01:02.003 2 values: 0.33333333333333 0.33333333333333 11:01:02.003 2 they are the sameOne more note. If it wasn’t clear, the end number is not the problem. The error is creeping in during the intermediate steps of the two very different calculations.
If I were to guess, this calculation is the one where the errors are creeping in, because it involves both a division (hardware) unit and two runs through an adder, with a sign flip.
local number1 = (100 + 1/3) - 100 -- Divider: 1/3 -\> Store to register -- Adder: 100 + register result from last --\> Store to register -- Adder: Result of last + twos-complement of 100 -\> Store to register -- May be even more involved than this...
Warning: In my ‘example’ above, I rounded to 14 places. That is probably NOT a good idea. I would suggest rounding to 3 or 4 places. Even then, rounding may fail.
Finally, can you tell us what you’re using these numbers for? i.e. What you want to do with these calculations?
Was it just something you noticed while goofing around or were you doing something like trying to compare the position of two objects that had been moved a calculated distance.
If you’re dealing with pixels/position calculations you’re often better off rounding to zero places or using ceil() or floor() to throw away the decimal places.
No. In a Lua forum elsewhere in an old thread somebody was trying to build an algebra “wolfram” mathematical type app. There was a discussion on how hard it would be to build it.
Yesterday I noticed an ad for this problem. I wonder how Lua would handle it since it is very obvious to me the answer. The solution I came up with is more convoluted: looking at the “string” equivalence across the equal sign.
In any case, I figure to put up a diversion since I am tired of reading about ANRs on Android.
This is a well known phenomenon. Remember, at the end of the day, these values are stored in a binary encoded format and there will be precision and rounding issues based on the number of bits available for the exponent and mantissa.
https://www.doc.ic.ac.uk/~eedwards/compsys/float/
http://floating-point-gui.de/formats/fp/
http://floating-point-gui.de/errors/comparison/
https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
To solve this, simply apply floor, ceil, or your own rounding :
For your specific example, this fixes the problem:
local function round(val, n) if (n) then return math.floor( (val \* 10^n) + 0.5) / (10^n) else return math.floor(val+0.5) end end local number1 = (100 + 1/3) - 100 local number2 = 1/3 print("1 values:", number1, number2) if number1 == number2 then print("1 they are the same") else print("1 nope") end local number1 = round((100 + 1/3) - 100,14) local number2 = round(1/3,14) print("2 values:", number1, number2) if number1 == number2 then print("2 they are the same") else print("2 nope") endOutput of above:
11:01:02.003 1 values: 0.33333333333333 0.33333333333333 11:01:02.003 1 nope 11:01:02.003 2 values: 0.33333333333333 0.33333333333333 11:01:02.003 2 they are the sameOne more note. If it wasn’t clear, the end number is not the problem. The error is creeping in during the intermediate steps of the two very different calculations.
If I were to guess, this calculation is the one where the errors are creeping in, because it involves both a division (hardware) unit and two runs through an adder, with a sign flip.
local number1 = (100 + 1/3) - 100 -- Divider: 1/3 -\> Store to register -- Adder: 100 + register result from last --\> Store to register -- Adder: Result of last + twos-complement of 100 -\> Store to register -- May be even more involved than this...
Warning: In my ‘example’ above, I rounded to 14 places. That is probably NOT a good idea. I would suggest rounding to 3 or 4 places. Even then, rounding may fail.
Finally, can you tell us what you’re using these numbers for? i.e. What you want to do with these calculations?
Was it just something you noticed while goofing around or were you doing something like trying to compare the position of two objects that had been moved a calculated distance.
If you’re dealing with pixels/position calculations you’re often better off rounding to zero places or using ceil() or floor() to throw away the decimal places.
No. In a Lua forum elsewhere in an old thread somebody was trying to build an algebra “wolfram” mathematical type app. There was a discussion on how hard it would be to build it.
Yesterday I noticed an ad for this problem. I wonder how Lua would handle it since it is very obvious to me the answer. The solution I came up with is more convoluted: looking at the “string” equivalence across the equal sign.
In any case, I figure to put up a diversion since I am tired of reading about ANRs on Android.