json decode performance

Other
#1

We’ve been using json.decode for a while now, but for large files it can often hog CPU time when decoding large files. We are looking into making an asynchronous json decode function using enterprise, but we would like to know what library are you guys using on the native side?

The main problem is that we need to decode around 20 files dynamically, out of around 200. We do not want to create a loading screen in this scene, since it can easily discourage the user from using it altogether, and cannot load the 200 because they can hog memory very easily.

Decoding time for a string 250000 characters long takes 150 - 160 milliseconds, which if we multiply by 20, we have a 3 second stall, which is significant for animations, transitions and other stuff, and looks sloppy overall. What we are doing to improve the sloppiness is to let the screen refresh after each decode, which improves but definitely does not fix the issue.

It would be ideal to have an asynchronous decode function. Are there any plans for this? What json library does Corona use internally?

#2

The Corona json module is based on dkjson 2.5.1.

#3

Great! now i can use a coroutine to make it “async” Thanks for pointing me on the right direction  :smiley:

EDIT: Any plans on using CJson? 13x - 21x faster seems pretty impressive against dkjsons 1x - 4.1x

#4

+1 @basiliogerman

Wondering the same thing as I rely most on JSON across my apps.

@Michael

Will Corona consider using CJson?

#5

Decoding time for a string 250000 characters long takes 150 - 160 milliseconds

What platform are we talking here?  Even on my slowest device I don’t see it being that slow but my test is probably different to your case.  On a modern device it’s almost 10 times faster.  I mention this because it may be tempting to think a faster JSON decoder will solve all problems but it might make little difference if the bulk of the time is spent in other code.

Just for calibration, an iPhone 6 Plus will decode 7MB of JSON to a table with 5300 nodes in under a second.

There’s a lot to like about dkjson from a compatibility point of view but, obviously, faster is always better so we’ll keep an eye on the available options.

#6

I was able to make the normal json decoder included with dkjson async using coroutines, which is like 5 times slower since it does not use lpeg, which would have been awesome to use the lpeg functions but couldn’t yield in C-call boundaries.

For any of those who are wondering how, i basically create a coroutine, and have an enterframe which resumes the routine. If the routine is taking too long it yields, successfully returning to the enterframe and refreshing the screen. when it is done, it calls a callback listener (onComplete)

Unfortunately, our game architecture would need a lot of changes to support an async json decode, so we had to veer away from that solution. 

@Perry We test on a huge amount of devices, newer devices don’t have much of a problem. We are using a custom super optimized spine engine (Animation). We mostly load our spine objects dynamically which means that if we load 15 different characters that are not yet cached it has to read those 15 jsons, which make the scene transitions seem a bit jerky on some devices, so our next best thing to try would be a faster decoder.

#7

We can look into it, though I don’t see it being an extremely high priority.

That said, someone could make a CJson plugin.

#8

@Michael

Yes, we are looking into starting its development using enterprise next week.

#9

The Corona json module is based on dkjson 2.5.1.

#10

Great! now i can use a coroutine to make it “async” Thanks for pointing me on the right direction  :smiley:

EDIT: Any plans on using CJson? 13x - 21x faster seems pretty impressive against dkjsons 1x - 4.1x

#11

+1 @basiliogerman

Wondering the same thing as I rely most on JSON across my apps.

@Michael

Will Corona consider using CJson?

#12

Decoding time for a string 250000 characters long takes 150 - 160 milliseconds

What platform are we talking here?  Even on my slowest device I don’t see it being that slow but my test is probably different to your case.  On a modern device it’s almost 10 times faster.  I mention this because it may be tempting to think a faster JSON decoder will solve all problems but it might make little difference if the bulk of the time is spent in other code.

Just for calibration, an iPhone 6 Plus will decode 7MB of JSON to a table with 5300 nodes in under a second.

There’s a lot to like about dkjson from a compatibility point of view but, obviously, faster is always better so we’ll keep an eye on the available options.

#13

I was able to make the normal json decoder included with dkjson async using coroutines, which is like 5 times slower since it does not use lpeg, which would have been awesome to use the lpeg functions but couldn’t yield in C-call boundaries.

For any of those who are wondering how, i basically create a coroutine, and have an enterframe which resumes the routine. If the routine is taking too long it yields, successfully returning to the enterframe and refreshing the screen. when it is done, it calls a callback listener (onComplete)

Unfortunately, our game architecture would need a lot of changes to support an async json decode, so we had to veer away from that solution. 

@Perry We test on a huge amount of devices, newer devices don’t have much of a problem. We are using a custom super optimized spine engine (Animation). We mostly load our spine objects dynamically which means that if we load 15 different characters that are not yet cached it has to read those 15 jsons, which make the scene transitions seem a bit jerky on some devices, so our next best thing to try would be a faster decoder.

#14

We can look into it, though I don’t see it being an extremely high priority.

That said, someone could make a CJson plugin.

#15

@Michael

Yes, we are looking into starting its development using enterprise next week.