load file data into table

Solar2D General Questions/Discussion
#1

I’m making a word game where I load a list of words stored in a txt file into a table so I can then search the list for a specific word. This worked fine in flash and the list which is 1.9mb loaded in seconds and I was able to search the list in no time as well. In the simulator it takes about a minute or 30s to load the list and each time I search the list it takes about a minute as well.

I am loading the list into a table and checking the table for the word using the following function:

–Check word against list
local checkWord = function(wordToCheck)
for _,v in pairs(wordList) do
if v == wordToCheck then
return true
end
end
return false
end

here is the load function:

–Load word list into array
local path = system.pathForFile( “wordlist.txt”, system.ResourceDirectory )

for line in io.lines(path) do
table.insert(wordList,line)
end
Is there a better way to do this to cut down on the time it takes? [import]uid: 3018 topic_id: 1215 reply_id: 301215[/import]

#2

According the number of words, I would suggest the use of SQLite instead a huge text file. Also, SQL queries tend to be much more faster than find words in large tables.
[import]uid: 4883 topic_id: 1215 reply_id: 3223[/import]

#3

I’d like to try this, I’ve tried modifying the example code but haven’t been able to figure out how to create and save a sql database and then search it for a string. The example code at http://luasqlite.luaforge.net/lsqlite3.html is quite difficult to follow. Any chance we could get some more examples for using SQLite databases? [import]uid: 3018 topic_id: 1215 reply_id: 3261[/import]

#4

First thing I would do is to update your word search so that it uses hashes instead of walking the list!

I think this will also make the list load a bit faster. I am using something similiar but for another purose (speeding up calcuations using a precalculated table).

It would work like this:

local wlist={}  
  
-- Load the list  
local path = system.pathForFile( "wordlist.txt", system.ResourceDirectory )  
  
for line in io.lines(path) do  
 wlist[line]=true  
end  
  
--Check word against list  
  
if wlist[line] ~= nil then  
 print("word in list")  
end

This will speed up the lookups enormous.

This will be faster than using sqlite3 too.
The second problem is the time it takes to load the data…

But there is a very easy and elegant way to speed this up also!

Create an Lua Module containing all your words like this:

local wlist={  
 ['word1']=true,  
 ['word2']=true,  
 ['word3']=true,  
 ['word4']=true  
}

To create this file you can use any tool … (even creating it with lua while developing)

You would then just require your “wlist.lua” file … like this:

local wlist = require “wlist.lua”

and check for the words with

if wlist[myword] then…

Hope that helps … and inspires some of the other readers :slight_smile:

(the code listed is untested and typed from memory… but it should be obvious what I mean) [import]uid: 6928 topic_id: 1215 reply_id: 3306[/import]

#5

thanks, the hash table works great, I can search it with almost no noticeable delay.

[import]uid: 3018 topic_id: 1215 reply_id: 3367[/import]

#6

Glad to hear it helped!

If you still have some delay, you may try to create more than one hash table!

Like all Starting with ‘A’ in one… all with ‘B’ in the other.

The problem with hash tables is, that the hash algorithm may “clash” for different values. Which will need additional comparisons against the list of data which has the same hash value. “Perfect Hashes” are possible but are usually only generated at compile time for some cases.

I am NOT sure what technique Lua uses… It could also use different techniques to hash its table keys.

Anyway it may (may!) be faster to use multiple hash tables if you have really a lot of data to look up!

But then… it needs additional work and it sounded you have it fast enough :slight_smile: [import]uid: 6928 topic_id: 1215 reply_id: 3370[/import]

#7

A+++

Awesome advice. Thank You. Over 210,000 words and not a bit of lag. [import]uid: 4871 topic_id: 1215 reply_id: 4519[/import]