What's wrong with my save function? Randomly crashes on devices - HELP!

Solar2D General Questions/Discussion
#1

Hey friends!  I’m having a terrible time diagnosing why my game save function is either randomly crashing devices or causing 0-length save files.  FYI, my save game files are json-based and can get pretty big (6-10MB+), but I’m not sure that should cause an issue.  Here’s what I’ve got going on and I would most definitely appreciate any ideas!

SaveGame function - checks for valid GameID and then makes a backup of primary save file in case something goes wrong, then saves the file. Also creates an info file that shows key info about the save without having to load the whole big thing (used to show list of saves in main menu):

function SaveGame() local tmpT, y local showSave="N" local lfs = require "lfs" GameData.SaveGroup.alpha=1 timer.performWithDelay(10000, function() GameData.SaveGroup.alpha=0 end,1) if \_Game.GameID~=nil and \_Game.GameID\>0 then print("SAVING GAME!") tmpT={Team=\_Game.UserTeamID, Year=\_Game.SeasonYear, W=\_Game.Teams[\_Game.UserTeamID].Wins, L=\_Game.Teams[\_Game.UserTeamID].Losses, CW=\_Game.Teams[\_Game.UserTeamID].ConfWins, CL=\_Game.Teams[\_Game.UserTeamID].ConfLosses, C=\_Game.CoachName,R=\_Game.Teams[\_Game.UserTeamID].Rank, CY=\_Game.CareerYears} ---Copy existing table as backup local destDir = system.DocumentsDirectory -- Location where the file is stored local size = lfs.attributes (system.pathForFile( "game"..\_Game.GameID..".json", destDir ), "size") if size~=nil and size\>0 then os.remove( system.pathForFile( "game"..\_Game.GameID..".bak", destDir ) ) local result, reason = os.rename( system.pathForFile( "game"..\_Game.GameID..".json", destDir ), system.pathForFile( "game"..\_Game.GameID..".bak", destDir ) ) else os.remove( system.pathForFile( "game"..\_Game.GameID..".json", destDir ) ) end saveTable(tmpT, "game"..\_Game.GameID.."info.json") if saveTable(\_Game, "game"..\_Game.GameID..".json")==true then print("SAVED SUCCESSFULLY") else print("SAVE FAILED 1") --TRY AGAIN if saveTable(\_Game, "game"..\_Game.GameID..".json")~=true then print("SAVE FAILED 2") os.remove( system.pathForFile( "game"..\_Game.GameID..".json", destDir ) ) os.rename( system.pathForFile( "game"..\_Game.GameID..".bak", destDir ), system.pathForFile( "game"..\_Game.GameID..".json", destDir ) ) end end end end

Here’s the saveTable function that I use in the function above, as well. Yes, It’s listed in my code before that SaveGame function!

function saveTable(t, filename) local path = system.pathForFile( filename, system.DocumentsDirectory) local file = io.open(path, "w") if file then local contents = json.encode(t) file:write( contents ) io.close( file ) --GameData.SaveGroup.alpha=0 return true else --GameData.SaveGroup.alpha=0 return false end end
#2

FYI, _Game is the table that holds all of my game data.

#3

According to Lua docs, functions like os.remove and os.rename will tell you if they failed - you’re just ignoring the errors (if any).

http://lua-users.org/wiki/OsLibraryTutorial

You may want to add some vars that capture those return values, and hopefully see where things are failing.

#4

So, I went ahead and changed my save function to iterate through saves to automatically create backups to avoid any os library calls that might be creating issues on certain devices.  Only a few people are telling me that their game is crashing during the save process, but I want to eliminate any crashing.  Here is my updated SaveGame function:

function SaveGame() if \_Game.SaveID==nil then \_Game.SaveID=0 end if \_Game.GameID~=nil and \_Game.GameID\>0 then if \_Game.SaveID==0 or \_Game.SaveID==3 then print("SAVING GAME! SaveID=1") \_Game.SaveID=1 elseif \_Game.SaveID==1 then print("SAVING GAME! SaveID=2") \_Game.SaveID=2 else print("SAVING GAME! SaveID=3") \_Game.SaveID=3 end tmpT={Team=\_Game.UserTeamID, Year=\_Game.SeasonYear, W=\_Game.Teams[\_Game.UserTeamID].Wins, L=\_Game.Teams[\_Game.UserTeamID].Losses, CW=\_Game.Teams[\_Game.UserTeamID].ConfWins, CL=\_Game.Teams[\_Game.UserTeamID].ConfLosses, C=\_Game.CoachName,R=\_Game.Teams[\_Game.UserTeamID].Rank, CY=\_Game.CareerYears, SaveID=\_Game.SaveID} saveTable(tmpT, "game"..\_Game.GameID.."info.json") if saveTable(\_Game, "game"..\_Game.GameID..\_Game.SaveID..".json")==true then print("Game"..\_Game.GameID..\_Game.SaveID.." SAVED SUCCESSFULLY") GameData.Changed="N" timer.performWithDelay(1000, function() GameData.SaveGroup.alpha=0 end,1) else timer.performWithDelay(2000, function() GameData.SaveGroup.alpha=0 end,1) end end end

Is there an easy way to “validate” my json data perhaps before saving?  I’ve really tried to make this as simple as possible, but for some reason, some people are still having crashing issues while saving… Thoughts?

#5

Just for clarity on how that new save function works, if a user opens save Game #1, it would will create up to 3 save file iterations as the user saves each time - first game11.json, then game12.json and then game13.json and will then re-save game11.json the fourth time, etc…

#6

So you’re checking the return value on all os calls now and you still don’t see where it’s failing?  Or, are you even able to replicate this issue on your own test devices?

Also, when are you trying to save these games?  Is it at application exit?  If so, it may be taking too long and your process gets killed.

#7

As you can see by my updated Save function, I have removed all os calls and streamlined into just iterating the save process.  The save is triggered either automatically after each “week” in game or on-demand via a save button (depending on which option a user chooses in settings).  The game does not autosave upon exit, so that is not an issue. It does prompt a user to save if they have not saved upon exit, but that’s not the issue. I do fear there is something getting corrupted in the json conversion, so curious if there are ways to check and validate json data/keys as it is being written out to the file…

#8

OK I see it’s just io.open and write now.  You didn’t mention if you’re able to replicate the problem on your own devices?  Or are you just relying on reports from testers?

I’ve read and written JSON files to and from Lua tables many times, but have never had an issue like what you’re describing.  And your saveTable code is almost exactly what is in the docs at https://docs.coronalabs.com/tutorial/data/jsonSaveLoad/index.html other than discarding any possible error string when opening the file.

According to the intro message here http://dkolf.de/src/dkjson-lua.fsl/home it’s possible to have Lua tables that can’t be encoded into JSON, but your tmpT table looks OK to me.  Maybe you can use rPrint (or tPrint, whatever you have like https://gist.github.com/ripter/4270799 ) to print your table out to console before saving it, then you’ll see what’s about to be saved.  Maybe the CoachName contains something that is breaking the JSOn encoding?

#9

You can use many different online JSON validators to make sure your JSON file is valid. This is of course not practical in a distributed app, but while developing it, or even if you can catch the crash, have it email you the JSON file and then you can check it for errors. I’m pretty sure that our JSON library doesn’t like mixed index types, such as a numerical index and a key index at the same level.

But what is likely happening is something is nil when you create the table to save:

tmpT={ Team=\_Game.UserTeamID, Year=\_Game.SeasonYear, W=\_Game.Teams[\_Game.UserTeamID].Wins, L=\_Game.Teams[\_Game.UserTeamID].Losses, CW=\_Game.Teams[\_Game.UserTeamID].ConfWins, CL=\_Game.Teams[\_Game.UserTeamID].ConfLosses, C=\_Game.CoachName, R=\_Game.Teams[\_Game.UserTeamID].Rank, CY=\_Game.CareerYears }

If _Game.UserTeamID is nil or a value that doesn’t exist in Game.Teams[], you’re going to get an error that will crash Corona if not caught. Maybe check to make sure it’s valid before trying to save things.

Something like:

if \_Game.Teams and \_Game.Teams[\_Game.UserTeamID] and \_Game.Teams[\_Game.UserTeamID].Wins then     -- create the temp table and save end

might help harden things.

Rob

#10

Thanks, Rob!  I always like your perspective!   I’ll look at your suggestion, but I believe the problem lies in saving the larger _Game table and not the data that I create and save in tmpT.  Is there a function that you’ve seen that will let me iterate through my _Game table (and sub-tables) and validate the data for json?  I fear something is not being set right in that table (which is pretty large - 3,500+ entries, with numerous values within each entry/row) and when things go wrong, the data is not being saved and is generating a 0-length save file.  So weird…

#11

For that amount of data is it worth moving to SqlLite? I don’t like using JSON when that many rows are involved, plus you’ll more easily be able to see what is being written out.

#12

If you think that the json encoding goes wrong, a starting point is to check if the encoding was successful (eg. not nil) and log an error if not successful.

function saveTable(t, filename) ...     local contents = json.encode(t, {exception = function(...) for i,v in ipairs(arg) do print("Exception: " .. tostring(v)) end }) if contents then     file:write( contents ) ... end

Edit: Didn’t check if this works, small adjustments might be necessary… :slight_smile:

#13

FYI, I used sqlLite in my first app and bulk updates were SOOOO slow.  Really made the app hang and created other issues.  I thought moving to in-memory and json might make it easier…

#14

Yoger, I like your suggestion but your code throws back errors…  “Unexpected symbol near }”

#15

Shouldn’t be slow at all if you’re wrapping your transactions in ‘begin transaction’ and ‘commit’.

#16

nick, maybe it was because I was using a file-based database throughout.  Would it be better to load the entire database into memory and only write it back out to a file-based database from memory all at once during the “save” process if I decided to use sqllite again.   

Yoger, still want to get your exception check working… Any tweaks to fix that error?

#17

I haven’t used in-memory db in Corona, just file-based. Multiple tables with 100,000+ records.

The slowest part of writing to SQLlite using lua is the string concatenation to build the query, so I had to optimise that quite a lot, building queries from strings in a table, using table.concat to build the final string.

The actual disk operations are pretty fast, especially on mobile. Windows desktop is 2-3 times slower.

Looks like you’re working on a sports sim too so the use case is similar, I don’t store much in memory - just grabbing and saving data when required, so I don’t have a save game function.

#18

Yes, the function end statement was missing. 

Refactored slightly, now the code looks like:

local function exceptionHandling(...) print("json encode failed") for i,v in ipairs(arg) do print("Exception: " .. tostring(v)) end end local contents = json.encode(t, {exception = exceptionHandling}) if contents then file:write( contents ) end

To test the function you can call 

exceptionHandling("test", "test2", "test3")

I tried but couldn’t produce a lua table that triggered an exception in json encode.

#19

If your save files are 6-10 MB+, then it might be that between the “contents” variable and all the intermediate data (the JSON library’s table, unique strings along the way, etc.) plus game state, you’re just hitting a memory limit.

By the looks of it, encode will just table.concat() the final result, like what Nick described, at the very end. But apparently you can also pass in your own table through the options, via the buffer field, with bufferlen set to the size afterward. (This would distinguish the case where fewer than # buffer  were written, if you recycled the table.) If memory is failing but only because it needs one big chunk, you could try this approach, manually concatenate sub-ranges, then send them to file:write() one by one.

Also, from a cursory examination it doesn’t seem like the library is doing much with the table contents along the way, so you might even be able to proxy it right to your file (opened beforehand, of course), maybe with some tuned buffering. In theory this would lower the memory load a bit, at the cost of some complexity.

That said, all of the above is just an educated guess.  :slight_smile:

#20

Wow, StarCrunch!  That’s some good stuff… Some of this is new to me. I’ve never played around with the buffer.  Any suggestions on a function that would better “chunk” my save data or handle that buffering better?