Help with converting a pure LUA Markov generator?

Getting Started Newbie Questions
#1

Hey all. I’ve been banging my head against this one for a couple days now and am reaching out to the gang for some help.

There’s this cool thing called Markov generation, which can come up with some pretty cool randomly strung together text if you feed it a large enough “sample file”, it will spew out some pretty convincing sounding sentences.

I found a sample actually in LUA from the PIL official docs, but I can’t get it to work properly.

The original’s here: http://www.lua.org/pil/10.2.html

It’s pretty damned small and nice too.

So I converted it over to this ugly beast:

math.randomseed( os.time() )  
  
ourTextTable = {}  
function allwords ()  
 local path = system.pathForFile( "testtext.txt", system.Resources )  
 local fh = io.open( path )  
 for line in fh:lines() do   
 while line do -- repeat while there are lines  
 local s, e = string.find(line, "%w+", pos)  
 if s then -- found a word?  
 pos = e + 1 -- update next position  
 --print("Inserting "..string.sub(line, s, e))  
 table.insert(ourTextTable,string.sub(line, s, e))  
 else  
 line = io.read() -- word not found; try next line  
 pos = 1 -- restart from first position  
 end  
 end  
 end  
 fh:close()  
end  
  
function prefix (w1, w2)  
 return w1 .." ".. w2  
end  
  
allwords()  
local statetab = {}  
  
 function insert (index, value)  
 if not statetab[index] then  
 statetab[index] = {n=0}  
 end  
 table.insert(statetab[index], value)  
 end  
  
  
local N = 2  
-- MAXGEN is the number of words you want to generate in your final sentence  
local MAXGEN = 100  
local NOWORD = "\n"  
  
-- build table  
-- build table  
local w1, w2 = NOWORD, NOWORD  
for i=1,#ourTextTable do  
 insert(prefix(w1, w2), ourTextTable[i])  
 w1 = w2; w2 = ourTextTable[i];  
end  
  
insert(prefix(w1, w2), NOWORD)  
  
local myFinishedText = ""  
  
-- generate text  
w1 = NOWORD; w2 = NOWORD -- reinitialize  
for i=1,MAXGEN do  
 local list = statetab[prefix(w1, w2)]  
 -- choose a random item from list  
 local r = math.random(table.getn(list))  
 local nextword = list[r]  
 myFinishedText = myFinishedText .." "..nextword  
 w1 = w2; w2 = nextword  
end  
  
print(myFinishedText)

You’ll need to plop in a text file named “testtext.txt”, and it’s supposed to go thru it and…well I explained above. I hate to pull a “solve my problem for me”, but what do I have to lose?

I started by yanking out the io.read stuff, converted it to a table, parsed the text, etc…and it’s KINDA working…but instead of doing it in chunks of every 1 word then finding the next logical word, it does it in a 5 word chunk, then another 5 word chunk, really wierd. I tried posting some Dark Tower text in there, and Roland kept doing the same 5 word chunks throughout the finished output.

I’d post some test text but I don’t want the post to get really huge.

I’d really, REALLY appreciate any help anyone can give me. I’m sure someone with the LUA skillz can bust it out, and I’m not averse to paypaling someone a six pack of their favorite beverages or a coffee! :slight_smile:

[import]uid: 11636 topic_id: 9609 reply_id: 309609[/import]

#2

The change you made to the part that went for w in allwords() wasn’t quite right.

Try this:

[code]
local path = system.pathForFile( “testtext.txt”, system.Resources )
io.input(path)

function allwords ()
local line = io.read() – current line
local pos = 1 – current position in the line
return function () – iterator function
while line do – repeat while there are lines
local s, e = string.find(line, “%w+”, pos)
if s then – found a word?
pos = e + 1 – update next position
return string.sub(line, s, e) – return the word
else
line = io.read() – word not found; try next line
pos = 1 – restart from first position
end
end
return nil – no more lines: end of traversal
end
end

function prefix (w1, w2)
return w1 … ’ ’ … w2
end

local statetab

function insert (index, value)
if not statetab[index] then
statetab[index] = {n=0}
end
table.insert(statetab[index], value)
end

local N = 2
local MAXGEN = 1000
local NOWORD = “\n”

– build table
statetab = {}
local w1, w2 = NOWORD, NOWORD

for w in allwords() do
insert(prefix(w1, w2), w)
w1 = w2; w2 = w;
end
insert(prefix(w1, w2), NOWORD)

– generate text
local function makegibberish()
local result = “”
w1 = NOWORD; w2 = NOWORD – reinitialize
for i = 1, MAXGEN do
local list = statetab[prefix(w1, w2)]
– choose a random item from list
local r = math.random(table.getn(list))
local nextword = list[r]
if nextword == NOWORD then
return result
end
result = result … " " … nextword
w1 = w2; w2 = nextword
end
return result
end

print ("Generated: " … makegibberish())
[/code] [import]uid: 6787 topic_id: 9609 reply_id: 35120[/import]

#3

GOT IT!!!

We had to add the “return result” to the end of the “makegibberish” function.

-- generate text  
local function makegibberish()  
 local result = ""  
 w1 = NOWORD; w2 = NOWORD -- reinitialize  
 for i = 1, MAXGEN do  
 local list = statetab[prefix(w1, w2)]  
 -- choose a random item from list  
 local r = math.random(table.getn(list))  
 local nextword = list[r]   
 if nextword == NOWORD then   
 return result  
 end  
 result = result .. " " .. nextword  
 w1 = w2; w2 = nextword  
 end   
 return result  
end

Edit your post when you have a chance and it’ll be saved for posterity!!

Thanks again!!! It looks to be working like a champ!!!
W00000T!!!

So I’ll buy you a beverage on my pilgrimage to your offices!! [import]uid: 11636 topic_id: 9609 reply_id: 35142[/import]

#4

looking forwards! [import]uid: 6787 topic_id: 9609 reply_id: 35981[/import]