I just submitted a verified bug with json.encode (loses data on long [64 bit] integers)
All I got was a thank-you email with subject saying:
(Case 33039) json.encode loses data on long ints
Is 33039 the actual bug number??
If so, how do I use this number to track the status of the bug
If not, how do I get the bug number so I can track the status of this bug?
This is blocking the release of my app and so I’ll have to be a bit persistent about it…
Thanks
Dewey
There is no method to track the bug AFAIK. You can ask about it here, or send support to ask about it, but the bug DB is basically private at this point. That is the actual bug number if you want to ask about it, though.
We know our visibility into the bug database isn’t good. But Brent and I will do what we can to find out what’s going on with your bug numbers when you post them here.
in this case, looking at your bug, I’m not sure there is a solution to this. Lua doesn’t have discrete numeric types. It only has “double float”. It tries to make numbers look like integers when it can, but there really is no concept of a 8 bit, 16, bit, 32 bit or 64 bit integer.
Rob
Rob…I understand Lua really well…I was working with it for many years before Corona was even formed…and Lua does internally represent its floats like every other language…and this is clearly a bug in the JSON library (hopefully one of the open source ones) in-use by CL.
To see what I’m talking about, simply notice the difference between these two representations:
print(string.format("%d", 3333444455556677))
print(string.format("%.0f", 3333444455556677))
Thanks for your response and for looking into this for me.
Dewey
Hey Dewey.
Have you considered grabbing the open source dkjson.lua file from the web and seeing if the bug exists there? And if so possibly fixing it, if the need is urgent to release your app?
Not saying you should have to do this, but afaik it’s a possible option.
Cheers
Yes, thank you.
That’s exactly what I intend to do just as soon as I:
Rob…can you find out which JSON library you guys are currently using?
Thx
Dewey
oh, perhaps you are telling me: is it dkjson??
We’re currently using DKJSON 2.2 and plan to update to 2.5 soon.
I don’t know if a more recent version of DKJSON fixes this issue (there’s a commit for 2.4 relating to “Fixed encoding and decoding of numbers in different numeric locales” which I guess might be apropos though it sounds unlikely).
Great!! I’ll go check the new version and see if its been fixed…I dont think my C is good enough to fix it if not
Thanks
Dewey
Oh, I forgot to mention …
I would just treat the returned ids as strings. They are opaque values which just happen to look like large numbers. You aren’t going to be doing math with them so just avoid having any part of the software try to convert them between representations.
DKJSON is a pure Lua implementation.
Super…guess I was looking at the wrong repo…here:
https://github.com/LuaDist/dkjson
I’ve got 8 tables that sync with the server…and the foreign key-names are not equivalent across tables…and I have greater-than (old vs new) comparisons (if stmts) that would have to change.
It would be very time consuming and laborious to go add this tostring casting into all my rest calls, queries and conditionals…
I’m hoping I don’t have to go that route…if it’s pure Lua, I’m hopeful…
Oh, if I load my own dkjson source, will it be able to see the CL internal LPEG library??
It will still work if not, albeit a bit slower…
Thanks again for looking at this!
D
This should do the trick:
local lpeg = require("lpeg")Just tested with 2.5…problem is still there.
Will report back later…
D
doohh…I clearly need some sleep 
yeah, he’s just looking for the std lpeg.so – so all good
Think I’ll shut it down tonight and look again with a clear head later in the weekend…
Thanks everyone for the tips and suggestions
Out of curiosity, I looked into the precise limits on integer size in Lua (our Lua, like most, uses 8 byte doubles to represent numbers).
Consider this code fragment:
local maxdouble = 2^53 -- one less than the maximum can be represented precisely print (string.format("%.0f",maxdouble-1)) --\> 9007199254740991 -- the maximum itself can be represented precisely print (string.format("%.0f",maxdouble)) --\> 9007199254740992 -- one more than the maximum gets rounded down print (string.format("%.0f",maxdouble+1)) --\> 9007199254740992 again
Output:
2014-05-27 11:24:20.996 Corona Simulator[60038:303] maxdouble-1 (9007199254740991): 9007199254740991 2014-05-27 11:24:20.996 Corona Simulator[60038:303] maxdouble (9007199254740992): 9007199254740992 2014-05-27 11:24:20.997 Corona Simulator[60038:303] maxdouble+1 (9007199254740992): 9007199254740992
So you can represent integers towards the upper end of 16 digits but the wheels come off when you go greater than 2^53.
Add a leading digit (to the example value supplied):
local mynumber = 23334445556667711 print (string.format("mynumber-1 (23334445556667710): %.0f",mynumber-1)) --\> should be 23334445556667710 print (string.format(" mynumber (23334445556667711): %.0f",mynumber)) --\>should be 23334445556667711 print (string.format("mynumber+1 (23334445556667712): %.0f",mynumber+1)) --\>should be 23334445556667712
And the output is incorrect (actually, and unfortunately, it’s close enough to the original number that you might not notice for a while that you’re in trouble):
2014-05-27 11:24:20.999 Corona Simulator[60038:303] mynumber-1 (23334445556667710): 23334445556667712 2014-05-27 11:24:21.000 Corona Simulator[60038:303] mynumber (23334445556667711): 23334445556667712 2014-05-27 11:24:21.001 Corona Simulator[60038:303] mynumber+1 (23334445556667712): 23334445556667712
Note that the same thing would happen if your 16 digit number happens to have greater than 90 for its first two digits.
So remember that Lua doesn’t actually support 64-bit integers, though it gets fairly close. Things that are opaque ids should be transmitted as strings. If you want to do math with them, then you’ll have make your own arrangements for arbitrary precision numbers.
agreed…and much of this is supposed to get fixed in 5.3 but for now I’ve had to patch dkjson as follows:
change line 183 (approx) in the encoder:
return replace(fsub(strformat("%.16g",num), numfilter, “”), decpoint, “.”) – my fixed version
– return replace(fsub(tostring(num), numfilter, “”), decpoint, “.”) – orig dkjson version of line 183 or thereabouts
– I think I made a comment or two at the top of the file so 183 may not be exact…
That seems to resolve my problems without my having to go cast back and forth (number to string) all over the place…
There is no method to track the bug AFAIK. You can ask about it here, or send support to ask about it, but the bug DB is basically private at this point. That is the actual bug number if you want to ask about it, though.
We know our visibility into the bug database isn’t good. But Brent and I will do what we can to find out what’s going on with your bug numbers when you post them here.
in this case, looking at your bug, I’m not sure there is a solution to this. Lua doesn’t have discrete numeric types. It only has “double float”. It tries to make numbers look like integers when it can, but there really is no concept of a 8 bit, 16, bit, 32 bit or 64 bit integer.
Rob
Rob…I understand Lua really well…I was working with it for many years before Corona was even formed…and Lua does internally represent its floats like every other language…and this is clearly a bug in the JSON library (hopefully one of the open source ones) in-use by CL.
To see what I’m talking about, simply notice the difference between these two representations:
print(string.format("%d", 3333444455556677))
print(string.format("%.0f", 3333444455556677))
Thanks for your response and for looking into this for me.
Dewey