Problem with big file

Solar2D General Questions/Discussion
#1

Hi there,

I’m having trouble dealing with a big file in my app. Basically I have a dictionary with ~700000 words and I have to load it into a hashtable in the beginning of my words game. The problem is that it’s taking too long to load(~15 sec in my cellphone) and the file is too big (18 mb).

Here’s what I’ve tried:

1-Read everything line by line from a txt file

2-store the dictionary in a json file and then decoding it at the beginning of the program

Both these metods weren´t efficient

3-I then tried to store it in a table inside a lua module and then requiring it.The file was too big and corona didn’t accept that.

4-pre-compile the file to .luac archive and put it in the resourse directory.Corona didn’t accept that either.

Is there a way that reduces drastically the load time and possibly the file size?

thanks

#2

Is there a specific reason you need the entire file to be loaded?  Would you be able to break it up into smaller parts and only load the information when needed? 

#3

No, I have to load it all

#4

I came across similar issue a bit ago. I needed this data to be table, but .lua file was to big and indeed Corona SDK didn’t accept it while building.

If it must be .lua table try to separation of data to different .lua files - try for example first two letters as a criterium.

So if you need a word - by knowing 2 first letters you can use proper table.

Eg. Corona would be in table[“co”]

Other option (this one is not tested by me) is to use SQLite database.

#5

thanks for the advice. I’m going to try thinking some inteligent ideas over this things

#6

Is there a specific reason you need the entire file to be loaded?  Would you be able to break it up into smaller parts and only load the information when needed? 

#7

No, I have to load it all

#8

I came across similar issue a bit ago. I needed this data to be table, but .lua file was to big and indeed Corona SDK didn’t accept it while building.

If it must be .lua table try to separation of data to different .lua files - try for example first two letters as a criterium.

So if you need a word - by knowing 2 first letters you can use proper table.

Eg. Corona would be in table[“co”]

Other option (this one is not tested by me) is to use SQLite database.

#9

thanks for the advice. I’m going to try thinking some inteligent ideas over this things

#10

You must break the dictionary into pieces, and call them “a”, “b” and so on.

It should take less than a second to load each one when needed.

Or if it needs to load faster than that do what @delwing suggests.

It would be simple logic to search for the word in the corresponding file.

#11

This post was created > 2 years ago…

#12

Hmmm, you shouldn’t be able to reply to old posts any more.

We flipped a forum settings blocking posting to old posts a couple of weeks ago. Odd.

Rob

#13

Whoops, I didn’t look at the date!

#14

You must break the dictionary into pieces, and call them “a”, “b” and so on.

It should take less than a second to load each one when needed.

Or if it needs to load faster than that do what @delwing suggests.

It would be simple logic to search for the word in the corresponding file.

#15

This post was created > 2 years ago…

#16

Hmmm, you shouldn’t be able to reply to old posts any more.

We flipped a forum settings blocking posting to old posts a couple of weeks ago. Odd.

Rob

#17

Whoops, I didn’t look at the date!