Looking for method to force screen update before end of my portion of a frame

[sieler2]

Hi,

I’m saving a large file to “disk”, and it takes 4 to 10 seconds …

I’d like to display a message to that effect, but my attempts have failed (i.e., no message appears).  (#1)

(My terminology is going to be wrong, sorry.)

My understanding is that the physical screen is updated when I come to the “end” of my code (“fall off”),

and logically return to the Corona jacket (and wait for some event to occur).  (Or, perhaps its updated at the start of the frame…same effect for me.) .

For uploads/downloads to the network, that’s fine … because those (somehow) get handled 

asynchronously.

But, that’s not case when I’m using Lua code to write thousands of lines to a file.

(Yes, I could implement an incredibly ugly kludge: save some lines, set a timer interrupt for a 

few milliseconds from now, and effectively loop by “exiting” my Lua code every now and then,

and I know that would cause the screen to be updated.)

I’d like to find something like: display.renderNow ()

Back in 2015, Rob said:

   It usually creates problem when you try to do a progress bar,

   start a bunch of things loading in the same block of code and wonder

   why the progress bar didn’t update. The frame buffer won’t update again to

   show the new progress bar until that block finishes.

…and that’s what I’m running into

thanks,

SS

---

#1. In my current attempt, I mark all the current on-screen objects as “invisible”, and mark a text block

I created earlier (for holding my “wait, I’m saving a file” text) as “visible”.  

I know that works in and of itself, via separate testing.

(At the end of the file saving, I mark the wait text block “invisible”, and all the old on-screen objects as “visible” … my version of scenes.)

[nick_sherman]

Corona is single-threaded. If the thread is busy, you can’t update the UI.

The only option is to break up your task into chunks. If those thousands of lines written to a file are from a table, it’s easy to setup an index for where you currently are in the table, and choose a number to process per frame that keeps the UI responsive. Just use an enterFrame listener and use a loop to go between currentIndex and currentIndex + processPerFrame each frame.

In what format are you saving these lines? I’d suggest switching to something like SQLlite, which will write thousands of table rows much quicker than writing to a txt or JSON file.

[roaminggamer]

If you have work that takes a long time (seconds), you need to break that work up into slices and do a little at a time, then release the thread.

This is super easy to do if you’re simply iterating over a table to write to a file.

Using SSK2 it would be something like this:

// This saves table to file, but does not block thread. local function saveBigTableToFile( tbl, dst, base, onComplete ) local index = 1 -- temp function to write content every frame local function writer() local stopIndex = index + 1000 local lastCall = false if (stopIndex \> #tbl ) then stopIndex = #tbl end for i = index, stopIndex do if( i == 1 ) then io.writeFile( tbl[i] .. "\n", dst, base or system.DocumentsDirectory ) else io.appendFile( tbl[i] .. "\n", dst, base or system.DocumentsDirectory ) end end index = stopIndex + 1 if( lastCall ) then if ( onComplete ) then onComplete() end return end timer.performWithDelay( 1, writer ) end writer end

Then, assuming you have a big table called listOfStuff, you could do this:

saveBigTableToFile( listOfStuff, "listOfStuff.txt" )

In this example, onComplete is a optional function that will be called when the writing is done.  You can use it to do anything that needs to be done when writing is complete.

Also, choose a number greater than 1000 for stepping.  That was just a random number I chose.  Work out a good number on your own.

[davebollinger]

this is one of those rare cases where coroutines make sense

[roaminggamer]

@sieler2,

One more note.  You used the word ‘kludge’ above when describing the idea of using a timer based approach.  This is the wrong way to think about it.

You must embrace the concept of event-driven game logic.  That is the core of how Corona works.  Timers are nothing more than events.

In fact, if you don’t embrace this you will, time and time again, run up against the fact that Corona is single threaded and blocking actions will plague you.

[roaminggamer]

@davebollinger,

I feel like the approach I suggested (self-contained function call with a timer and temporary local helper) would be simpler than @siesler learning co-routines.  Also, you still need a way to wake the co-routine on a regular basis and that seems like it would require a timer or enterFrame listener.

That said, I don’t use co-routines a lot.  If you have time and are willing, can you elaborate on the use of co-routines for this kind of task?

[davebollinger]

coroutines are 100% worth learning IFF you have a valid use for them.  there’s always a way to do the same thing WITHOUT them, but sometimes the “bookkeeping” needed to maintain state WITHOUT a coroutine outweighs that learning effort.  THAT is when you might have a good use case.  maybe something like this:

local function saveBigTableToFile(path, lines) local co co = coroutine.create(function() local file = io.open(path,"wt") for i = 1, #lines do file:write(lines[i].."\n") if (i%1000==0) then timer.performWithDelay(1, function() coroutine.resume(co) end) coroutine.yield() end end file:close() end) coroutine.resume(co) end

(aside:  am unsure what the worry was about needing a timer/enterFrame to resume it - your method does also)

[sieler2]

Thanks for your posts!

I probably should have made it clearer in my original post that I was 

really hoping to avoid the kludge (and, rg, yeah, it is one :).

@roaminggamer … thanks for your example.  

I have no problem with event driven logic … but Corona doesn’t really support a robust form of that.

(Robustness would require asynchronous tasking, which we lack).

From a programming esthetics viewpoint, having to break a block of multiple writes (or other actions)

into a series of interrupted work just to allow display updates is … well, kludgy is the kindest work I can think of.

Additionally, since there doesn’t seem to be a way to say "schedule a fake event for immediately after

I exit the ‘frame’", I’d have to wait until a timer pops, which delays/slows-down the already slow writing process.

Worse yet, it’s not clear if a “frame” would be started when my timer actually pops, or if I have to wait

until the next frame interval to get invoked (i.e., if I schedule a timer for 1 millisecond from now, will it be

triggered in 1 millisecond, or in (about) 1/30th of a second … if I get time, I’ll test and report).   

The Corona internals need a lot of explanation

I chuckled at “learning coroutines” …

IIRC, my first use of coroutines was in 1971, on a B6700 (using Burroughs ALGOL),

I might have been the first to implement coroutines in FORTH (on a PC, in our Next Generation Systems FORTH, circa 1987).

I designed the process handling in the 1980s for the HPE operating system (some of my code still survives in MPE/iX).

Unless I’m missing something, there’s no way coroutines would help in this case.

Remember that they’re synchronous with your main “thread/process/task”, and

the Corona wrapper won’t update the display until you’ve “returned” from all code.

Oh, Corona Labs could implement a limited form of true asynchronous processing

in Lua (which would solve my problem), but they don’t seem interested in improving

the Corona environment (html/linux support aside). 

I’d be happy to share some ideas with them, since if I don’t get to be CTO of Niantic,

Corona Labs is my next choice

The annoying and disappointing thing is that a form of “please update the display now” could

be implemented by CoronaLabs easily, precisely because it would be a request from

a call/function, and therefore the Lua environment would be in a known/stable state (#1)

during the display update.  (Note: I’m not the only person requesting this, as google shows.)

thanks,

Stan

#1. If we had asynchronous processing, then it would be more difficult, and some form

of locking with the Lua interpreter(s) would be needed.  But (sadly), we don’t, so

we don’t.

[sieler2]

@rick_sherman I’m writing in text, creating a CSV (comma separated values) file.

It doesn’t need to be csv, and if I had the ability to simply save a Lua table as a blob (re-readable later, of course), I’d much much rather do that.  (That’s an item on my Lua enhancement list, if I get to be CTO

[davebollinger]

@sieler2

 (i.e., if I schedule a timer for 1 millisecond from now, will it be

triggered in 1 millisecond, or in (about) 1/30th of a second … if I get time, I’ll test and report).   

pretend that timer.performWithDelay() is instead named timer.performOnNextFrameAfterAtLeastThisMuchDelay()

(because that is literally how it functions – you will NOT get an intra-frame callback into the Lua interpreter)

[davebollinger]

@sieler2

Unless I’m missing something, there’s no way coroutines would help in this case.

Remember that they’re synchronous with your main “thread/process/task”, and

the Corona wrapper won’t update the display until you’ve “returned” from all code.

yes, everything about Lua is single-threaded, including coroutines.  however, you appear to misunderstand that a yield/resume cycle IS essentially a “return” from Lua code (or how about:  it’s a “return that remembers state and can thus be ‘un-return-ed’ from”) at which time the frame loop is able to “come up for air” and refresh.  On next frame, you resume and yield again, display refreshes again, repeat until coroutine complete.

[roaminggamer]

@davebollinger

pretend that timer.performWithDelay() is instead named timer.performOnNextFrameAfterAtLeastThisMuchDelay()

(because that is literally how it functions – you will NOT get an intra-frame callback into the Lua interpreter)

Dave is right.  My use of 1 as the time really means, execute at the next earliest possible time.  This will always be the next frame.

As for the rest, you have to do something like this if you don’t want to block.  You can easily make this optimal.  It will just take a little more work to keep track of how much time has been spent in the loop and then stop the loop at that time, resuming later.  I’d use a while loop for this instead of a for.  

[davebollinger]

@sieler2

a form of “please update the display now” could be implemented by CoronaLabs easily

hardly!  :D  take a few seconds to consider the impact on just the physics library alone.  (and let’s not even open up the display capture can of worms)

I get what you want, it’s something like old VB6’s DoEvents method – so that you can write huge monolithic loops and YOU get to be in control of when the host takes a breath.  Problem is, that’s entirely backwards to the way things actually work in Corona, and is unlikely to change.

[roaminggamer]

UPDATE: Video is a little wrong. Blocking page should say ‘blocking’ not ‘non-blocking’.

Code is updated, but I’m not redoing the video.

https://www.youtube.com/watch?v=iteRNqjHWOA&feature=youtu.be

https://github.com/roaminggamer/RG_FreeStuff/raw/master/AskEd/2019/02/nonBlockingSave.zip

key file: scripts/saver.lua

[roaminggamer]

Once you download the updated example you will see two options:

‘blocking’ - Demonstrates saving all in one go.  It blocks. Spinner does not work.

‘non blocking’ - Demonstrates asynchronous save.  It does not block and spinner animation plays correctly.

[StarCrunch]

I covered some of the coroutine-related ideas in my tutorial a few years back. The “wait” functions you see about a third of the way down could be applied with a file:write() wrapper as the update, maybe using some of the yield helpers further down to get in several lines per frame. This would be a variant of what Dave posted earlier.

I’ve been dabbling in the Corona source. Corona’s display objects get boiled down into commands and run in another thread outside the control of Lua. A stop-and-update would be non-trivial. 

I do have a plugin for one of the Lua multithreading libraries: luaproc, with some extensions. Threads are definitely no silver bullet, though–much more often they make things harder!–and I think the techniques others have mentioned are better fits.

[sieler2]

thanks all for some very interesting posts, giving me a lot to try out!

Stan

[nick_sherman]

Corona is single-threaded. If the thread is busy, you can’t update the UI.

The only option is to break up your task into chunks. If those thousands of lines written to a file are from a table, it’s easy to setup an index for where you currently are in the table, and choose a number to process per frame that keeps the UI responsive. Just use an enterFrame listener and use a loop to go between currentIndex and currentIndex + processPerFrame each frame.

In what format are you saving these lines? I’d suggest switching to something like SQLlite, which will write thousands of table rows much quicker than writing to a txt or JSON file.

[roaminggamer]

If you have work that takes a long time (seconds), you need to break that work up into slices and do a little at a time, then release the thread.

This is super easy to do if you’re simply iterating over a table to write to a file.

Using SSK2 it would be something like this:

// This saves table to file, but does not block thread. local function saveBigTableToFile( tbl, dst, base, onComplete ) local index = 1 -- temp function to write content every frame local function writer() local stopIndex = index + 1000 local lastCall = false if (stopIndex \> #tbl ) then stopIndex = #tbl end for i = index, stopIndex do if( i == 1 ) then io.writeFile( tbl[i] .. "\n", dst, base or system.DocumentsDirectory ) else io.appendFile( tbl[i] .. "\n", dst, base or system.DocumentsDirectory ) end end index = stopIndex + 1 if( lastCall ) then if ( onComplete ) then onComplete() end return end timer.performWithDelay( 1, writer ) end writer end

Then, assuming you have a big table called listOfStuff, you could do this:

saveBigTableToFile( listOfStuff, "listOfStuff.txt" )

In this example, onComplete is a optional function that will be called when the writing is done.  You can use it to do anything that needs to be done when writing is complete.

Also, choose a number greater than 1000 for stepping.  That was just a random number I chose.  Work out a good number on your own.

[davebollinger]

this is one of those rare cases where coroutines make sense