A few clarifications regarding tables and removeSelf

[pirx]

Hi, I’ve been programming in Corona for a couple of years but I still encounter bug-causing little things that I don’t fully understand. So could someone please explain these to me?

1. If I do  

   local object1 = display.newCircle(…) table[1] = object1

then what is stored in the table field? Is it a copy of the object, or is it a reference to the original object? 

E.g. when I call table[1]:removeSelf() does it affect the original object1?

2. Why does an object, after calling removeSelf() on it, still have a removeSelf method (provided I didn’t nil it)?

3. Is it safe to nil a table field? (won’t it remove the field i.e. shorten the table?)

4.  If I have like 5000 objects that I removed, but not nilled, how does it affect performance/memory? (they CAN be nilled, but I’m trying to get a better understanding of this)

That’s it for now. Thank you!

[davebollinger]

  First, I’d suggest using display.remove(object1) instead of object1:removeSelf().  Second, once removed there is no reason not to nil the reference (while there are good reasons TO nil all references to it)

  Tables are assigned by reference.  Watch out for “stragglers” – is that “object1” in your code an unintentional global?  If so, it will retain a reference and prevent release of that memory EVEN IF you later do table[1]=nil.  Either use “local” and let it fall out of scope, or, in your code as given, there’s no reason not to just do table[1] = display.newCircle(…) directly, now you only have the one reference to worry about.

  when Corona creates a DisplayObject (like newCircle, newRect, etc) it first creates an internal “black box” representation of it, one that you never directly “experience”, but the internal engine uses.  It then wraps a pointer (a “userdata” in Lua terms) to that black box inside a table, with its assorted fields/methods, that it returns to you.  When you remove a DisplayObject from the display the internal black box is destroyed.  That is, the “userdata” becomes meaningless, so the table “wrapper” returned to you also becomes meaningless.  You should not attempt to keep accessing the wrapper’s fields or calling the wrapper’s methods when its userdata no longer exists.  Nil’ing those references will prevent you from even attempting that, also free up the Lua memory required to store that now-useless “wrapper”.

[roaminggamer]

1a. The variable object1 contains a reference to the circle object.
 
1b. table[1] contains a reference to the circle object.
Tip: I know this was just a question, but ‘table’ is a invalid name for any variable since it is a reserved word.
 
 
2a. I agree this is problematic, but the object isn’t cleaned up till the next frame.

--- Dang local obj = display.newCircle( 10, 10, 10 ) print("here ==\> ", obj.removeSelf) obj:removeSelf() print("not gone? (not nil) ==\> ", obj.removeSelf)

-- Ahh local obj = display.newCircle( 10, 10, 10 ) print("here ==\> ", obj.removeSelf) obj:removeSelf() timer.performWithDelay(1, function() print("gone now (nil) ==\> ", obj.removeSelf) end )

2b.  Having said that, I hate ‘removeSelf’ for this very reason.  It is unsafe and constantly confuses new users.  Thus I agree w/ @davebollinger and suggest that you use display.remove(obj).
 
Note: If you run into a crash however (caused by double call to obj:removeSelf(), it means you may have a problem in your game design.
 
 
3a.  No… This is not safe because it causes holes:

local objs = {} objs[1] = display.newCircle( 10, 10, 10 ) objs[2] = display.newCircle( 10, 10, 10 ) objs[3] = display.newCircle( 10, 10, 10 ) display.remove( objs[2] ) objs[2] = nil -- Now objs contains { reference, nil, reference } print( #objs ) -- Prints 1, not 2 as you might expect.

3b.  Yes… I suggest instead of using numeric indexes, use reference indexes.

-- Just an example to demonstrate concept... local objs = {} local tmp = display.newCircle( 10, 10, 10 ) objs[tmp] = tmp local tmp2 = display.newCircle( 10, 10, 10 ) objs[tmp2] = tmp2 local tmp = display.newCircle( 10, 10, 10 ) objs[tmp] = tmp objs[tmp2] = nil display.remove( tmp2 ) local count = 0 for k,v in pairs( objs ) do    count = count + 1 end print( count ) -- prints 2

 
 
4. I don’t understand this question.  Sorry.

[davebollinger]

  @roaminggamer hit a good point that I missed (you have a lot of questions!) regarding “sparse numeric array” tables.  If part of your question is “should I avoid sparse tables”, then yes, probably.  (because you had to ask!  ie, unless you have a known reason for using them, then you probably don’t want them; fe indexing primes or fibonnaci’s or etc for first-pass quick “isPrime()” -type testing, a sparse table works well for that, et al).  So, while it’s entirely “safe” to do (OP) it may not be “wise” to do.

  Mainly what you’d like to avoid is “tricking” lua into first creating a giant array table, then later causing it to convert the sparse remnants to a hash table.  table.remove() could remedy that and keep your numeric indices, or use non-numeric keys from the start; though having said all that, you can still iterate a sparse numeric table with pairs().  Though as @roaminggamer points out # (and things that depend on it, like table.sort) won’t work with sparse tables.

[pirx]

One always learns. This is super-interesting. Thanks for the suggestions.

These questions come from the general field of instantiation of objects so I’m no longer sure of my usual approach to it. Could you have look at the following?

Let’s assume I have a screen full of circles; also new ones are created periodically. Every circle is based on a prototype “class” in a module so that I can do e.g. 

local someNewCircle = circleModule.new() 

My goal is for each circle to be draggable and  when dropped on another one, both are destroyed. The collision is not to be handled by physics but by a function that measures distance and radius (that doesn’t really matter here).

Now, here’s how I would usually approach this:

1. create a table, say, local myCircles = {}

2. Each time a new circle is to be created, the program does something like 

   myCircles[#myCircles+1] = circleModule.new()

3. Each time a circle is dropped after being dragged, I go through the table of all my objects ( for i=1, #myCircles do ) and see if another object is occupying the same space. If this is the case, destroy both. 

Issues:

The problems begin when I want to destroy the two overlapping circles. If I do myCircles[5].removeSelf() (or display.remove) on them, then the objects disappear but their data remains in the table so after some time the table grows insanely long. On the other hand, I don’t want to nil these fields (myCircles[i] = nil) not to create “holes” in the table. Using table.remove() for this purpose, in turn, immediately changes the table length so it can get messy during iteration (unless I’m iterating backwards) or when a lot of references to length (#myCircles) are used around. So I’m no longer sure what is the best way to delete instances of objects stored in a numbered table.

What I usually did was just myCircles[i]:removeSelf() and then myCircles[i].active = false, so that I can skip the destroyed objects later. But, as I understand, this is kind of inefficient and leaves behind unused “stuff” in memory.

Is there a better approach to creating/storing and removing objects (sprites, enemies, bonuses, database objects etc.)? For example, is it better, as in roaminggamer’s suggestion, to have a refence indexes based table and iterate in pairs?

[davebollinger]

regardless of the language, mutating an “array” structure while iterating is usually problematic.  (lua’s tables and java’s arraylist are similar in that regard, for example)

for deletion, the rule-of-thumb is backwards traversal.  for insertion, the rule-of-thumb is a secondary array to merge into the original after traversal.  for deletion, you can also nil as you go, then a secondary traversal (using original #), to “shift down” one at a time as needed – one of the rare times where native lua can be a win over a built-in like table.remove(), but only if you’re doing LOTS of removes in a single iteration, O(n) vs O(n^2).

hash tables are easier, you CAN set existing entries to nil during traversal (see docs for next() and pairs()), you’d still need to do after-the-fact insertion tho.  or roll your own doubly-linked list (trivial) and then both insert/delete become non-issues during traversal (“contains()” -type functionality becomes the loser here),

[pirx]

Thank you, I have a clearer picture of how this is supposed to be done now. 

For my purposes it actually solves all the problems - I can table.remove() while going backwards (so that iteration doesn’t go sideways), and as for insertion, in the scenario I described earlier new fields are always added at the end of the table and never during looping through, so this is a non issue.

[Lava_Level]

@pirx, One thing I have found useful, and I am not sure if this would work for your case, is to create a proxy object that is actually being dragged, and if the conditions are correct when it lands, then I usually change all of mytables and redraw everyone in their correct positions. 

re:3a:

Also here is a neat trick I do, but I haven’t tried it on display objects, mainly game data. I don’t do much spawning of new actorObjects. 

local function \_deepcopy( t ) local cpy = {} for k, v in pairs( t ) do if type( v ) == "table" then cpy[k] = \_deepcopy( v ) else cpy[k] = v end end setmetatable( cpy, getmetatable( t ) ) return cpy end local objs = {} objs[1] = {ObjectName="Thing1",objectX=50, objectY=5, objectColor=(1,0,0)} objs[2] = {ObjectName="Thing2",objectX=50, objectY=5, objectColor=(0,1,0)} objs[3] = {ObjectName="Thing3",objectX=50, objectY=5, objectColor=(0,0,1)} -- Next the number you want to delete should be home to the end object's data objs[2] = \_deepcopy( objs[#objs] ) -- then i always delete the guy at the end display.remove( objs[#objs] ) objs[#objs]=nil print( #objs ) for i=1, #objs do print ("Object Name:".. objs[i].ObjectName) end

I wonder if there is a way to create a deepCopy for a display object? Because the method above moves the table at the end to the deleted point. And then just zaps the final entry. So basically it prevents tables from having nested *nils*.  It’s a sure fire way for me to know that the number of items matches the length of a table. Lets me do for loops on tables faster than in pairs.

@daveboillinger, Speaking of unintentional Globals… 

There was a corona build that at one time would highlight every global variable that it came across. 

I guess it was removed because it was so intensive on the console. It will highlight all _knownGlobals. So after I get my game/code set up to a major point, I like to re-run my app through CoronaBuild-2866 to check if something has accidentally become global through  my own faults. I find that handy, sort of wish I knew of a codeline to turn that functionality on in later builds.

[pirx]

@Lava Level, this is a clever little function, I will try it out today. 

As to the globals, the first example wasn’t an unintentional global, it was just taken out of context for simplicity. 

As a sidenote: I’ve recently watched an interview on the Corona blog where a developer said he ‘believed in responsible globals’. I tend to agree with that approach. I mean, because the use of globals is generally discouraged here, there has been a bit of a paranoya in the spirit of “globals are evil”. I think there are situations when making something global is not harmful and simpler than e.g. creating an additional module or passing it around in arguments. For example, I have a amountOfCoins variable that gets used all over my game. It just sits there as a global and functions fine, while being unlikely to confilct with an internal system reserved word or something. I really don’t think there is any reason to do anything further with it.

That said, I think maybe it would be safer if lua required variables to be declared with either ‘local’ or ‘global’, and threw errors in the case of assiging a value to a previously undeclared one.

[Lava_Level]

Totally. I agree, I am still a bit of stickler. One thing for me though, if I have a gameState global, I like to keep them tidy by stuffing them into a global table like: 

_gameState = {}

  _gameState.player = “not loaded”

  _gameState.background = “not initialized”

  _gameState.firstloaded = true

  _gameState.Talk=false

  _gameState.gameState = “main menu”

  _gameState.npcActionTakingPlace = false

  _gameState.console={}

  _gameState.console[1]="…"

  _gameState.howManyMoves=0

  

Super easy to save out as a data json file in event of systemEvent makes returning to them super smooth in some cases.

re:declared vars, I know right?  At least it’s keeping my CoronaBuild-2866 still active during memory testing. :D 

[davebollinger]

@Lava\_Level

  At least it’s keeping my CoronaBuild-2866 still active during memory testing. :D 

ditch 2866, just grab Niklas Frykholm’s GLOBAL_lock from here (or strict.lua for 5.1, if you prefer)

[pirx]

@Lava Level  – I’m often actually doing the exact same thing: using one global gameData table with fields like “isUserActive”, “gameLocation” etc. for easy saving purposes. ;)  

@davebollinger – This is brilliant!

[davebollinger]

  First, I’d suggest using display.remove(object1) instead of object1:removeSelf().  Second, once removed there is no reason not to nil the reference (while there are good reasons TO nil all references to it)

  Tables are assigned by reference.  Watch out for “stragglers” – is that “object1” in your code an unintentional global?  If so, it will retain a reference and prevent release of that memory EVEN IF you later do table[1]=nil.  Either use “local” and let it fall out of scope, or, in your code as given, there’s no reason not to just do table[1] = display.newCircle(…) directly, now you only have the one reference to worry about.

  when Corona creates a DisplayObject (like newCircle, newRect, etc) it first creates an internal “black box” representation of it, one that you never directly “experience”, but the internal engine uses.  It then wraps a pointer (a “userdata” in Lua terms) to that black box inside a table, with its assorted fields/methods, that it returns to you.  When you remove a DisplayObject from the display the internal black box is destroyed.  That is, the “userdata” becomes meaningless, so the table “wrapper” returned to you also becomes meaningless.  You should not attempt to keep accessing the wrapper’s fields or calling the wrapper’s methods when its userdata no longer exists.  Nil’ing those references will prevent you from even attempting that, also free up the Lua memory required to store that now-useless “wrapper”.

[roaminggamer]

1a. The variable object1 contains a reference to the circle object.
 
1b. table[1] contains a reference to the circle object.
Tip: I know this was just a question, but ‘table’ is a invalid name for any variable since it is a reserved word.
 
 
2a. I agree this is problematic, but the object isn’t cleaned up till the next frame.

--- Dang local obj = display.newCircle( 10, 10, 10 ) print("here ==\> ", obj.removeSelf) obj:removeSelf() print("not gone? (not nil) ==\> ", obj.removeSelf)

-- Ahh local obj = display.newCircle( 10, 10, 10 ) print("here ==\> ", obj.removeSelf) obj:removeSelf() timer.performWithDelay(1, function() print("gone now (nil) ==\> ", obj.removeSelf) end )

2b.  Having said that, I hate ‘removeSelf’ for this very reason.  It is unsafe and constantly confuses new users.  Thus I agree w/ @davebollinger and suggest that you use display.remove(obj).
 
Note: If you run into a crash however (caused by double call to obj:removeSelf(), it means you may have a problem in your game design.
 
 
3a.  No… This is not safe because it causes holes:

local objs = {} objs[1] = display.newCircle( 10, 10, 10 ) objs[2] = display.newCircle( 10, 10, 10 ) objs[3] = display.newCircle( 10, 10, 10 ) display.remove( objs[2] ) objs[2] = nil -- Now objs contains { reference, nil, reference } print( #objs ) -- Prints 1, not 2 as you might expect.

3b.  Yes… I suggest instead of using numeric indexes, use reference indexes.

-- Just an example to demonstrate concept... local objs = {} local tmp = display.newCircle( 10, 10, 10 ) objs[tmp] = tmp local tmp2 = display.newCircle( 10, 10, 10 ) objs[tmp2] = tmp2 local tmp = display.newCircle( 10, 10, 10 ) objs[tmp] = tmp objs[tmp2] = nil display.remove( tmp2 ) local count = 0 for k,v in pairs( objs ) do    count = count + 1 end print( count ) -- prints 2

 
 
4. I don’t understand this question.  Sorry.

[davebollinger]

  @roaminggamer hit a good point that I missed (you have a lot of questions!) regarding “sparse numeric array” tables.  If part of your question is “should I avoid sparse tables”, then yes, probably.  (because you had to ask!  ie, unless you have a known reason for using them, then you probably don’t want them; fe indexing primes or fibonnaci’s or etc for first-pass quick “isPrime()” -type testing, a sparse table works well for that, et al).  So, while it’s entirely “safe” to do (OP) it may not be “wise” to do.

  Mainly what you’d like to avoid is “tricking” lua into first creating a giant array table, then later causing it to convert the sparse remnants to a hash table.  table.remove() could remedy that and keep your numeric indices, or use non-numeric keys from the start; though having said all that, you can still iterate a sparse numeric table with pairs().  Though as @roaminggamer points out # (and things that depend on it, like table.sort) won’t work with sparse tables.

[pirx]

One always learns. This is super-interesting. Thanks for the suggestions.

These questions come from the general field of instantiation of objects so I’m no longer sure of my usual approach to it. Could you have look at the following?

Let’s assume I have a screen full of circles; also new ones are created periodically. Every circle is based on a prototype “class” in a module so that I can do e.g. 

local someNewCircle = circleModule.new() 

My goal is for each circle to be draggable and  when dropped on another one, both are destroyed. The collision is not to be handled by physics but by a function that measures distance and radius (that doesn’t really matter here).

Now, here’s how I would usually approach this:

1. create a table, say, local myCircles = {}

2. Each time a new circle is to be created, the program does something like 

   myCircles[#myCircles+1] = circleModule.new()

3. Each time a circle is dropped after being dragged, I go through the table of all my objects ( for i=1, #myCircles do ) and see if another object is occupying the same space. If this is the case, destroy both. 

Issues:

The problems begin when I want to destroy the two overlapping circles. If I do myCircles[5].removeSelf() (or display.remove) on them, then the objects disappear but their data remains in the table so after some time the table grows insanely long. On the other hand, I don’t want to nil these fields (myCircles[i] = nil) not to create “holes” in the table. Using table.remove() for this purpose, in turn, immediately changes the table length so it can get messy during iteration (unless I’m iterating backwards) or when a lot of references to length (#myCircles) are used around. So I’m no longer sure what is the best way to delete instances of objects stored in a numbered table.

What I usually did was just myCircles[i]:removeSelf() and then myCircles[i].active = false, so that I can skip the destroyed objects later. But, as I understand, this is kind of inefficient and leaves behind unused “stuff” in memory.

Is there a better approach to creating/storing and removing objects (sprites, enemies, bonuses, database objects etc.)? For example, is it better, as in roaminggamer’s suggestion, to have a refence indexes based table and iterate in pairs?

[davebollinger]

regardless of the language, mutating an “array” structure while iterating is usually problematic.  (lua’s tables and java’s arraylist are similar in that regard, for example)

for deletion, the rule-of-thumb is backwards traversal.  for insertion, the rule-of-thumb is a secondary array to merge into the original after traversal.  for deletion, you can also nil as you go, then a secondary traversal (using original #), to “shift down” one at a time as needed – one of the rare times where native lua can be a win over a built-in like table.remove(), but only if you’re doing LOTS of removes in a single iteration, O(n) vs O(n^2).

hash tables are easier, you CAN set existing entries to nil during traversal (see docs for next() and pairs()), you’d still need to do after-the-fact insertion tho.  or roll your own doubly-linked list (trivial) and then both insert/delete become non-issues during traversal (“contains()” -type functionality becomes the loser here),

[pirx]

Thank you, I have a clearer picture of how this is supposed to be done now. 

For my purposes it actually solves all the problems - I can table.remove() while going backwards (so that iteration doesn’t go sideways), and as for insertion, in the scenario I described earlier new fields are always added at the end of the table and never during looping through, so this is a non issue.

[Lava_Level]

@pirx, One thing I have found useful, and I am not sure if this would work for your case, is to create a proxy object that is actually being dragged, and if the conditions are correct when it lands, then I usually change all of mytables and redraw everyone in their correct positions. 

re:3a:

Also here is a neat trick I do, but I haven’t tried it on display objects, mainly game data. I don’t do much spawning of new actorObjects. 

local function \_deepcopy( t ) local cpy = {} for k, v in pairs( t ) do if type( v ) == "table" then cpy[k] = \_deepcopy( v ) else cpy[k] = v end end setmetatable( cpy, getmetatable( t ) ) return cpy end local objs = {} objs[1] = {ObjectName="Thing1",objectX=50, objectY=5, objectColor=(1,0,0)} objs[2] = {ObjectName="Thing2",objectX=50, objectY=5, objectColor=(0,1,0)} objs[3] = {ObjectName="Thing3",objectX=50, objectY=5, objectColor=(0,0,1)} -- Next the number you want to delete should be home to the end object's data objs[2] = \_deepcopy( objs[#objs] ) -- then i always delete the guy at the end display.remove( objs[#objs] ) objs[#objs]=nil print( #objs ) for i=1, #objs do print ("Object Name:".. objs[i].ObjectName) end

I wonder if there is a way to create a deepCopy for a display object? Because the method above moves the table at the end to the deleted point. And then just zaps the final entry. So basically it prevents tables from having nested *nils*.  It’s a sure fire way for me to know that the number of items matches the length of a table. Lets me do for loops on tables faster than in pairs.

@daveboillinger, Speaking of unintentional Globals… 

There was a corona build that at one time would highlight every global variable that it came across. 

I guess it was removed because it was so intensive on the console. It will highlight all _knownGlobals. So after I get my game/code set up to a major point, I like to re-run my app through CoronaBuild-2866 to check if something has accidentally become global through  my own faults. I find that handy, sort of wish I knew of a codeline to turn that functionality on in later builds.

[pirx]

@Lava Level, this is a clever little function, I will try it out today. 

As to the globals, the first example wasn’t an unintentional global, it was just taken out of context for simplicity. 

As a sidenote: I’ve recently watched an interview on the Corona blog where a developer said he ‘believed in responsible globals’. I tend to agree with that approach. I mean, because the use of globals is generally discouraged here, there has been a bit of a paranoya in the spirit of “globals are evil”. I think there are situations when making something global is not harmful and simpler than e.g. creating an additional module or passing it around in arguments. For example, I have a amountOfCoins variable that gets used all over my game. It just sits there as a global and functions fine, while being unlikely to confilct with an internal system reserved word or something. I really don’t think there is any reason to do anything further with it.

That said, I think maybe it would be safer if lua required variables to be declared with either ‘local’ or ‘global’, and threw errors in the case of assiging a value to a previously undeclared one.