Hi,
I am needing a fast way to load a lot of data into my app, I was wondering if Json or Lua tables is faster. I am having all my data in external files. I can imagine that Lua tables in an external table is faster? Any pros having any ideas of this?
One more question, I can’t find a good tutorial or samples - on how to load external lua tables? Any one?
Best regards, Joakim
[import]uid: 81188 topic_id: 23933 reply_id: 323933[/import]
Hey Joakim!
If you need to load a lot of data quickly you should use SQLite. there’s a good example supplied with corona and it would be the fastest way possible.
I’m also pretty sure that saving the data in “regular” lua tables would be quicker for simple structures and local data. encoding and decoding the json to a workable table would take some processing time.
If you are planning on using a remote server, I would use JSON because it makes communication easier even if it’s a bit more space consuming.
my 2 cents.
Shahar [import]uid: 13553 topic_id: 23933 reply_id: 96433[/import]
Regardless of what you do, you have to read a file and parse it. SQLLite is fast, but its purpose is to find and retrieve records. If you’re loading the whole database in, there is more overhead with SQL than there would be reading a flat file.
I don’t know how to do external Lua tables and how they are parsed. I’ve created level data as lua tables, but those get compiled directly into the app and you don’t have to open a file and read it in.
Since you have to parse the data, JSON makes this process simple and its reasonably efficient.
Now if you’re reading bits at a time through out the run of your app and don’t want all the data in memory at once, then SQLLite is the best way to go. [import]uid: 19626 topic_id: 23933 reply_id: 96462[/import]
Hi, I am reading maybe 3 to 4 thousands of lines from sqlite and it takes almost 1.5 sec, and thats not acceptable for the purpose of my app. I just tried Json yesterday and that did the loading time in 600ms. But there we have the parsing and conversion to a lua table.
I was hoping to be able to read a flat file with the lua data, I have to try to see what I can come up with!
Thanks, Joakim [import]uid: 81188 topic_id: 23933 reply_id: 96498[/import]
About the same concern ( performance ) I try to get my json code as lua code.
I ll try to explain :
I don t want to load json file and convert it to lua table (i can do that), I just want to write in my code a table I have done in json.
I have about 300 items and hope I don t have to code it by hand 
The idea is :
If your data is one diminsional and simple variables then you could read. In your Json data and using a for loop output the code to make a lua table then cut and paste the console output into a lua file. Seems more work than just reading the Json file and decoding it to a lua. Table
[import]uid: 19626 topic_id: 23933 reply_id: 108491[/import]
Within the simulator you can actually output to a file, so if your data starts as JSON I’d say write a tiny main.lua that exports the data as a LUA table - you won’t get anything quicker than that.
For example, say you have an ‘export.lua’ file next to your main.lua, in main.lua you can do something similar to:
local saveFile = system.pathForFile( “export.lua” )
local fileHandle = io.open(saveFile, “w”)
fileHandle:write(“I am new contents!”)
io.close(fileHandle)
and after running the simulator, export.lua will now contain the data you passed in the write() line.
Obviously it will overwrite each time you run, so don’t forget to copy or rename the exported data file once it contains what you want! [import]uid: 46639 topic_id: 23933 reply_id: 108500[/import]
Thx Rob (by the way one more thank for your loadTable saveTable code that I use to save my tables in json format)
The table I wish to “code” is multidimensional (I guess), it’s something like :
mytable={}
mytable[1]={id=1, otherkey=“name”, otherkey=45, … }
Actually, I first made an sqlite table, having trouble with saves I made it a json table, finding it slow to load, I decided to try to put it in my code…
I tried to use http://jsonviewer.stack.hu/, it is ok but I just have to change : with = … [import]uid: 9328 topic_id: 23933 reply_id: 108498[/import]
* [import]uid: 9328 topic_id: 23933 reply_id: 108497[/import]
If your data is multi dimensional then you will have to test each element to see if it’s a table and then have another loop iterate over each table
[import]uid: 19626 topic_id: 23933 reply_id: 108502[/import]
Thx guys !
When you say :
local saveFile = system.pathForFile( “export.lua” )
local fileHandle = io.open(saveFile, “w”)
fileHandle:write(“I am new contents!”)
io.close(fileHandle)
Where do I put ‘mytable’ ? [import]uid: 9328 topic_id: 23933 reply_id: 108501[/import]
With regards to the original question:
a .lua file would be the absolute fastest, but comes with a number of restrictions.
Corona pre-compiles it’s lua file to byte-code on device, which adds security and speed, at the cost of being able to interpret Lua at runtime. That means you cannot download or write .lua files on device and use them. If it’s read-only data, a quick script to convert your data into a string containing Lua is a great idea, and probably the best performance you will be able to get. If you need to modify your data post-install your only options are SQLite or JSON, and that choice depends on how you plan on using your data, since SQLite is great if you only ever need small parts of a large database, but JSON is probably an easier choice if you need to use your entire database at the same time, or if it’s not a very big file.
Note: (comparing JSON to Lua in Corona) Loading a Trie dictionary it was ~1:1 on simulator but Lua was ~8x faster on device due to Lua being pre-compiled.
Edit: As to how to do that, you will need to write either a quick script or tool that reads in a JSON or other file type and writes the lua string that looks like “return { }”. Write the entire thing to a file with a .lua extension using some of the quick examples in above posts. Then you can just use require(“myFile”) in code and you get your Lua Data Table. [import]uid: 134101 topic_id: 23933 reply_id: 108524[/import]
Well, after almost two months trying to get my animation running, I have tried everything. Json, Sqlite and predefined textfiles parsed into a multidimensional lua table.
For me the fastest way is to load my predefined textfiles. I have one for each animation and there are maybe at most 800 lines of data. Every row contains 10 fields of data. I am loading the animation in runtime and theres no glitch when the animation is changing 
Second best was Json but the parsing made the animation stutter.
As for sqlite, it wasn’t even possible to use a database with all this data. The animation stopped for to long when switching sequence.
Joakim [import]uid: 81188 topic_id: 23933 reply_id: 108529[/import]
To add to what Ntero said,
First benchmark.
Second, ideally, Lua should be faster than JSON assuming you could use dofile() to load the code. But due to Apple restrictions and security concerns about running arbitrary downloaded code, we have crippled dofile() for now so you can’t rely on it to work at runtime. There are legitimate reasons to use dofile that don’t involve arbitrary downloaded code (e.g. you writing out your own save-state tables and reading them back in). I think this latter case is technically in Apple’s rules and not really a security threat, but Apple is not forthcoming about these things so we don’t really know. But what we are afraid of is Corona users abusing this if we re-enable dofile() which prompts Apple to overreact and then ban interpreted languages again (the nightmare scenario).
Third, in the absence of dofile(), my theory is given optimal parsers for JSON and Lua, Lua should be faster because the Lua language was designed to be parsed quickly. But you need a Lua parser. There are various things available from the Lua wiki and general searches. (off the top of my head: ‘lua table serialization’, ‘lua pickle’)
Forth, we are planning to add LPeg after the next release. Our current JSON parser (dkjson) has hooks to automatically use LPeg when available which can have dramatic speed improvements. There is a separate project called “Leg” which is designed to parse Lua using LPeg and can solve the lack of dofile(). My personal expectation is that Leg will beat dkjson for this kind of task. But benchmarking is needed.
Finally, if we do eventually relax the dofile() restriction and you are using Lua tables, it will be easy for you to migrate and you will get a good speed boost.
[import]uid: 7563 topic_id: 23933 reply_id: 108531[/import]
Wow ! Thank you so much for all the top notch answers. Guess this thred will become a reference for everyone wondering about lua/json/sqlite.
Ntero, you’have quite understood me :
“If it’s read-only data, a quick script to convert your data into a string containing Lua is a great idea, and probably the best performance you will be able to get”.
That’s a read-only, I know what to do
“If you need to modify your data post-install … JSON is probably an easier choice if you need to use your entire database at the same time, or if it’s not a very big file.”
I already worked on that, geting a very easy setup thx to robmiracle snippets.
“As to how to do that”.
I made a kind of mix with online tool find and replace ( : to = ) solution.
“Then you can just use require(“myFile”) in code and you get your Lua Data Table”.
So : in my main.lua I do
local test = require("testtable")
print(test[1]) And what is testtable.lua ?
{"boo","baa"}
-- that's not that easy ! Thanks again !
[import]uid: 9328 topic_id: 23933 reply_id: 108727[/import]
testtable.lua would look like this:
return {"boo","baa"} This has to do with how Require works.
Require will load your file, run it, and return any return values. Also it will store those returned values, so if you call require(“file”) twice it only runs file once, but stores the result in a location of packages.loaded[fileName]. This may be getting more technical than you need, but essentially the only part you missed above was that your table needs to be returned from testtable.lua.
[import]uid: 134101 topic_id: 23933 reply_id: 108741[/import]
The *#! return.
Sorry for a such a mistake. [import]uid: 9328 topic_id: 23933 reply_id: 108754[/import]
The following project was announced yesterday on the Lua mailing list as a Lua serializer and pretty printer. I really like everything he did. Below is the announcement.
[blockquote]
Yes, yet another serializer and pretty printer. I carefully studied
existing implementations and described my rationale and examples here:
http://notebook.kulchenko.com/programming/serpent-lua-serializer-pretty-printer.
My requirements for a serializer: (1) pure Lua (need to execute it as
part of a debugger on various platforms including mobile), (2) does
both pretty printing and robust serialization, (3) handles shared and
self-references, (4) serializes keys of various types, including
tables as keys, and (5) is short and doesn’t have too many
dependencies to be included with another module.
To summarize, I want the serialized result that is as readable as
possible and is still a valid fragment that I can load with
loadstring. The implementation is fairly short and sufficiently fast
for my needs, but I’m sure it can be improved on both of those counts.
“Serpent” because it handles self-references and reminds me of a
serpent eating its own tail (http://en.wikipedia.org/wiki/Ouroboros).
Available on github.
Paul.
[/blockquote] [import]uid: 7563 topic_id: 23933 reply_id: 110769[/import]