My iOS game, after 20 minutes of play, log: "was killed by Jetsam."

Solar2D iOS
#1

My iOS game, after 20 minutes of play, log: “was killed by Jetsam.” And suddenly my App/game is gone and I find myself on the iOS home screen.

 

Jetsam is a system that monitors memory use in OSX and iOS. It keeps a list of processes on the device, to monitor for situations where a device is running out of free RAM and looks for things to kill to free up RAM.

JetSamEvents are low RAM notifications. And report processes that are likely to be killed to free RAM.

 

I have attached 3 logs showing what was occuring just prior to “was killed by Jetsam”,

Log before kill #1:
Apr  3 07:46:43 iPhone containermanagerd[3651] <Error>: 0x1a1267000 -[MCMGlobalConfiguration init]: Current mobile user ID is 501
Apr  3 07:46:43 iPhone pkd[3642] <Warning>: assigning plug-in com.apple.reminders.todaywidget(1.0) to plugin sandbox
Apr  3 07:46:43 iPhone pkd[3642] <Warning>: enabling pid=58 for plug-in com.apple.reminders.todaywidget(1.0) 5DE646A4-A991-4491-AD10-92A3DF3AEA38 /Applications/Reminders.app/PlugIns/RemindersWidget.appex
Apr  3 07:46:43 iPhone apsd[100] <Warning>: Low Priority Push: com.apple.icloud-container.com.apple.mobilenotes - App killed
Apr  3 07:46:44 iPhone kernel[0] <Notice>: xpcproxy[3652] Container: /private/var/mobile/Containers/Data/PluginKitPlugin/73FB9634-243F-484F-8D21-D7A704AE4478 (sandbox)
Apr  3 07:46:44 iPhone backboardd[60] <Warning>: Unable to bootstrap_look_up port with name com.apple.reminders.todaywidget.gsEvents: unknown error code (1102)
Apr  3 07:46:44 iPhone SpringBoard[58] <Warning>: HW kbd: Failed to set com.apple.reminders.todaywidget as keyboard focus
Apr  3 07:46:44 iPhone tccd[1557] <Notice>: Refusing client without bundle identifier (/usr/libexec/coreduetd)
Apr  3 07:46:44 iPhone tccd[1557] <Notice>: Refusing client without bundle identifier (/usr/libexec/duetexpertd)
Apr  3 07:46:44 iPhone tccd[1557] <Notice>: Refusing client without bundle identifier (/usr/libexec/duetexpertd)
Apr  3 07:46:44 iPhone tccd[1557] <Notice>: Refusing client without bundle identifier (/usr/libexec/routined)
Apr  3 07:46:46 iPhone SpringBoard[58] <Warning>: plugin com.apple.reminders.todaywidget invalidated
Apr  3 07:46:54 iPhone SpringBoard[58] <Warning>: HW kbd: Failed to set (null) as keyboard focus
Apr  3 07:46:54 iPhone SpringBoard[58] <Warning>: UNNotificationRegistrarConnectionListener connection invalidated
Apr  3 07:46:54 iPhone mediaserverd[2458] <Notice>: ‘’ com.playthebible.rdgames(pid = 3615) setting DiscoveryMode = DiscoveryMode_None, currentDiscoveryMode = DiscoveryMode_None
Apr  3 07:46:54 iPhone mediaserverd[2458] <Notice>: ‘’ (pid = 3615) setting DiscoveryMode = DiscoveryMode_None, currentDiscoveryMode = DiscoveryMode_None
Apr  3 07:46:54 iPhone UserEventAgent[26] <Notice>: jetsam: kernel termination snapshot being created
Apr  3 07:46:54 iPhone SpringBoard[58] <Warning>: Application ‘UIKitApplication:com.playthebible.rdgames[0x12c2]’ was killed by jetsam.
Apr  3 07:46:54 iPhone UserEventAgent[26] <Warning>: 11358580884165: id=com.playthebible.rdgames pid=3615, state=0
Apr  3 07:46:54 iPhone ReportCrash[3655] <Warning>: Saved type ‘298(298)’ report (1 of max 25) at /var/mobile/Library/Logs/CrashReporter/JetsamEvent-2016-04-03-074654.ips

Log before kill #2:

Apr  3 20:42:26 iPhone SpringBoard[58] <Warning>: HW kbd: Failed to set (null) as keyboard focus
Apr  3 20:42:27 iPhone SpringBoard[58] <Warning>: UNNotificationRegistrarConnectionListener connection invalidated
Apr  3 20:42:27 iPhone UserEventAgent[26] <Notice>: jetsam: kernel termination snapshot being created
Apr  3 20:42:27 iPhone mediaserverd[2458] <Notice>: ‘’ com.playthebible.rdgames(pid = 4389) setting DiscoveryMode = DiscoveryMode_None, currentDiscoveryMode = DiscoveryMode_None
Apr  3 20:42:27 iPhone mediaserverd[2458] <Notice>: ‘’ (pid = 4389) setting DiscoveryMode = DiscoveryMode_None, currentDiscoveryMode = DiscoveryMode_None
Apr  3 20:42:27 iPhone SpringBoard[58] <Warning>: Application ‘UIKitApplication:com.playthebible.rdgames[0xfddf]’ was killed by jetsam.
Apr  3 20:42:27 iPhone kernel[0] <Notice>: [Speaker] AppleCS35L19Amp: audio muted for system load _aliveLateCounter=1
Apr  3 20:42:27 iPhone UserEventAgent[26] <Warning>: 12350243397818: id=com.playthebible.rdgames pid=4389, state=0
Apr  3 20:42:27 iPhone kernel[0] <Notice>: LwVM::_generateMappedExtentsList - failed to map unmap range to partition
Apr  3 20:42:27 iPhone ReportCrash[4487] <Warning>: Saved type ‘298(298)’ report (5 of max 25) at /var/mobile/Library/Logs/CrashReporter/JetsamEvent-2016-04-03-204227.ips
 

Log before Kill #3:

Apr  3 20:28:07 iPhone SpringBoard[58] <Warning>: HW kbd: Failed to set (null) as keyboard focus
Apr  3 20:28:07 iPhone SpringBoard[58] <Warning>: UNNotificationRegistrarConnectionListener connection invalidated
Apr  3 20:28:07 iPhone kernel[0] <Notice>: [Speaker] AppleCS35L19Amp: audio muted for system load _aliveLateCounter=1
Apr  3 20:28:07 iPhone mediaserverd[2458] <Notice>: ‘’ com.playthebible.rdgames(pid = 4317) setting DiscoveryMode = DiscoveryMode_None, currentDiscoveryMode = DiscoveryMode_None
Apr  3 20:28:07 iPhone UserEventAgent[26] <Notice>: jetsam: kernel termination snapshot being created
Apr  3 20:28:07 iPhone SpringBoard[58] <Warning>: Application ‘UIKitApplication:com.playthebible.rdgames[0x5b10]’ was killed by jetsam.
Apr  3 20:28:07 iPhone UserEventAgent[26] <Warning>: 12329603350966: id=com.playthebible.rdgames pid=4317, state=0
Apr  3 20:28:07 iPhone ReportCrash[4376] <Warning>: Saved type ‘298(298)’ report (4 of max 25) at /var/mobile/Library/Logs/CrashReporter/JetsamEvent-2016-04-03-202807.ips

#2

I still have no idea why this is happening and if is serious. My memory values do not change - although the App is tasking the iOS system at around 235MB during full play. But there is no explanation as to why at these levels the game plays fine for 20 minutes and then kicks out. Of course, on my iPad 2 with less memory, this occurs more frequently.

#3

So I’ve now done a lot of checking out how my game uses memory - typically it has 5-8MB in variables, as I use a lot of globals. And textures, accounts for the rest. At the time of my first post above, the game would typically hit between 215MB and 250MB, depending upon what was going on. I’ve since reduced this number by around 60-70MB, so the average total memory usage is about 150MB, much, much less than before.

But, Jetsam still kicks out my app - looks like a crash bug, but it isn’t. This is a very serious problem that I first thought was related to the amount of memory used. But even still, the iOS memory warning messages seem to occur at the same place, despite the fact that memory usage is 65-100MB less. This tells me that there is a behavior in my app that causes Jetsam to kill it, and the behavior isn’t as simple as using memory. But perhaps, how I’m using memory.

My app does use many global variables, but since when is that a situation that could cause the OS to kill it (Jetsam)?

Anyone, please help.

#4

Without Jetsam running does your app ever get killed?  

#5

Jetsam is part of Apple’s iOS. Here is some information that is VERY troubling. My App uses around 15% of the total memory of an iPhone 6 (150MB = 15% of 1GB RAM). So this really shouldn’t be happening.

I have code well-written to remove display groups and objects all the time and you can consistently see how the memory will nearly reset to around 115MB after every level, only going higher as the level is built - to typically around 150MB. So why this is happening must mean one important thing.

“Jetsam” is watching HOW your App is using memory, not just how much. In fact, before I reduced the App usage by 65-70MB, I would get my first low-memory warning event from the iOS at about when 200MB of memory was reached. But now, that threshold seems to be lower - around 150MB. Seriously!

So, there is a mystery here worth more pursuit not only because it is interesting, but because I can’t release my game until Jetsam leaves my App alone. Arg!

#6

How many other apps do you have suspended in the background?  Maybe that is causing the pool of memory for apps to be gobbled up?

#7

Yes, FeatTec. I didn’t check that - I should count the other apps in memory before testing further.

Do you or does anyone online here have any idea what restrictions exist in “using memory” such that iOS’ Jetsam feels the need to kill my app? Looking online, I’m not the only soul with this issue. It seems to be a mystery. But I can tell you my app will swing down 30-50 MB at a time during in-between levels during cleanup and removal of display objects and groups.

The fact that the App runs without error on Android makes the mystery a little deeper. Help! Anyone?

#8

i have profiled memory with this before http://www.mydevelopersgames.com/profiler.html

#9

Thanks for this, FearTec.

#10

I have confirmed that the problem with my app, and the reason jetsam kills it, is because I do not respond to the memory warning. To solve this, I need to know what the maximum texture memory I can use is, and then I can use scaling for my images to keep within the proper maximum while still maximizing the resolution for higher end devices.

Does anyone know how to get the total maximum texture memory value (not the individual texture limit)? For example, it’s 273MB on my iPhone 6 Plus and less on my iPad.

If I can’t get this value, then I’ll have to run up the objects at initial launch until I get a memory warning, and then record the memory usage at that time. This seems a bit ridiculous - but is it the only way to see what a specific device can handle?

#11

Further, when I try and run up the value to find the maximum, on my HTC One X, granted an older phone, but still running Android 4.1.1, even if I run up the texture memory 1MB at a time, I RECEIVE NO MEMORY WARNING. Here’s what happens:

I/Corona  (20318): Memory Currently Used:139.978515625
I/Corona  (20318): Memory Currently Used:140.916015625
I/Corona  (20318): Memory Currently Used:141.853515625
I/Corona  (20318): Memory Currently Used:142.7900390625
D/Corona  (20318): Error in drawing our frame! Surface is bad!

No warning. Just crashed - and worse, it leaves the App in memory running so the crash is evident to the user without any message popping up. They will have to remove it from memory. And in today’s world, that’s the death nail of every hoping to bring a customer back.

So please, can ANYONE help me determine the GPU memory maximum, or the ‘texture memory’ maximum? Both are different values as explained in the next post.

#12

On iOS, the method of testing memory (but increasing GPU memory use in 5MB increments), will generate a memory warning error before jetsam kills it. I’ve verified that if, when you receive the memory warning in your code, then respond immediately by eliminating display objects on screen (reducing texture memory), jetsam will never kill it. However, if the increment of additional GPU memory is too high, it will cause jetsam to kill it before a memory warning can occur.

To do this kind of profile testing, as I’m doing with a test project dedicated to this problem, requires at least 7 seconds at the start of the App on an iOS device to determine the max GPU use before receiving a memory warning. Is there no other setting or way to get the GPU memory maximum? Or must I check what the limits are on every device?

Another point worth mentioning is that the GPU memory appears to be unrelated to texture memory, but actually in terms of calculations, its a ‘subset’ of the ‘texture’ memory because texture memory holds all objects sent to the GPU. For example, our texture memory use can go over 1GB - seriously, that’s right - as long as you keep the objects created off-screen using (alpha=0, isVisible=false or its coordinates out of range). It’s only when you attempt to bring those objects using texture memory onto the screen (via the GPU) that the GPU’s memory is tasked - causing a memory warning.

With these facts in mind, am I missing something about Corona? Have they already determined this and fully understand how and why this happens? If so, why can’t I find it documented anywhere? For example, imagine you have a large image that you create using parts that make up a scrolling background game that is 100 screens wide. Corona and the OS will allow this. But it’s only when those objects are brought onto the screen that they end up using the ‘GPU memory’ - so with nothing more than moving the y coordinate of a group of objects (group.y=0), a crash can occur if GPU memory is exceeded - and if the change in GPU memory exceeds 5MB, it could even be killed by jetsam without a memory warning.

So again, does anyone have any idea where I can find what the GPU memory available is, or must I create a table based upon the iOS models available? With every new iOS release, I would have to update the table on my game server (and my code would have to download it) so that my app knows the GPU memory available on every new iOS device. Crazy to go to such lengths, but it would work.

It just seems to me that Corona would have found another way to handle this, I just can’t find it anywhere.

For clarification, GPU Memory Issue #1 is based upon when jetsam kills an app.

In addition to this need to know the GPU memory max, the secondary problem encountered as a result of GPU memory max amounts on Android platform are more severe. As mentioned above, once the GPU memory limit is reached on an Android 4.1.1, you don’t get any warning - Corona outputs to the console “D/Corona  (20318): Error in drawing our frame! Surface is bad!” and your app is toast. (the first part of this issue was located at: https://forums.coronalabs.com/topic/62032-crash-adb-logcat-dcorona-13417-error-in-drawing-our-frame-surface-is-bad/ and it has now been brought here as a related and relevant GPU Memory Issue.

So, for clarification, GPU Memory Issue #2 is based upon what occurs on Android 4.1.1, when the GPU memory limit is reached, there is no warning message and Corona crashes.

To solve this issue entirely, I’m attempting to perfect the idea of consolidating objects spread across many screens to try and flatten them all into one consolidated object. I’m still working on that, but recommendations by other developers haven’t resolved the issue of taking a snapshot of something that is both A) in GPU memory; with B) something not yet in GPU memory.

#13

I don’t know if there is such a thing as a table of what GPU memory is available, nor whether it would really help, because Jetsam is adaptive, and picks on apps that are not good at freeing memory when asked.  Apparently this is essentially more important than the absolute amount of memory used - it needs to be able to give some memory back when asked in order to avoid being killed by Jetsam. 

Also, while it may be true that GPU memory is unrelated to texture memory, Jetsam does react to apps that use large memory resources which come from an external source and require substantial CPU cycles to recreate. It doesn’t like ‘memory splashes’. e.g. you’re using 40 Mb of RAM, and then allocating 80 Mb’s more for some short computation.

I don’t know anything about your game, of course, but are you really saying you can use over 1gb of texture memory? That sounds…incredible. Presumably you’re lazy loading the various parts of your possibly hypothetical 100 screen wide scrolling background as needed and releasing as soon as you don’t? and reusing as much of the component parts as possible? And using appropriate versions of your textures based on device resolutions?

Not trying to teach you to suck eggs, by any means, but I’ve written some pretty hefty games with thousands of animated assets from a library of vast texture atlases on screen and never encountered anywhere near the memory usage you’re quoting.

Only once did I fall foul of Jetsam, on lower end devices, and it just required a bit of rejigging of asset usage - since it’s better to try and avoid Jetsams scrutiny at all!

(In fact, to the point that I’m confident enough to have never actually needed to include any “memoryWarning” listeners nor handlers in production versions)

#14

Hi JWiow, I’m not actually using 1GB of texture memory. My point was that if your objects are off-screen, the iOS (and jetsam) have no problem at all giving  you 1GB of texture memory - proving that the issue isn’t texture memory, it is “display/GPU” memory.

Yes, I agree that when a memory warning is given, something must be done about it on iOS and you are fine. But of course, this is NOT the case on Android - using too much texture memory causes a crash without any memory warning (Android 4.1.1).

I’m trying to consolidate many objects, but another issue with Corona is that there is no apparent way to flatten a group that is larger than the resolution of the device screen - so even if I have a group of objects 10 screens wide, it will flatten it but at a reduced resolution of 1/10th. So you see, this problem will occur for any game that ‘draws’ objects more than loads them - as does my game.

#15

So, I found the bug in Corona that causes jetsam!!!

There is a bug in Corona that fails to properly clean-up particles that have finished. So, if you don’t remove particle objects yourself, iOS will eventually kill your app via jetsam. It took a long time to find this.

There is no documentation in Corona regarding this problem and it needs to be added to save others from relying upon Corona’s memory management to handle finished particles.

In my case, if my effect is 3 seconds long, I simply use timer.performWithDelay(3500,removeAndNilObject)  – notice I just add 1/2 second to the particle length to make sure on very slow devices that the effect has finished.

#16

Can you file a bug report on that Troy?
 

#17

Hi Rob, to file a bug report, don’t I need to create a sample that it can be duplicated with? Unfortunately, I’m on deadlines and don’t have the time now. But as I described it, the issue really will not be seen by most people unless they have a complex app with hundreds particles - and simultaneously are tasking the device’s memory. So, I’m not sure how to easily create a template without spending some time on it - perhaps I can have my asst. developer help with that if it is necessary. You think?
 

#18

If it’s a case that we are not cleaning up particles, I suppose a demo app would maybe be 10 emitters using particles like you are and let it run for a while monitoring memory. Meet your deadline since you have a work around for it, but afterwards, consider filing a report on it.

Rob

#19

Hi Rob, just wanted you to know that I had my asst. developer create an isolated case to present the bug and Reported Bug online here. In the interim, may I suggest that you update the documentation for particles to inform all developers that they hold the burden to remove a particle, just as they would an object, or eventually jetsam on Mac will kick their app out or on Android, it may even crash their device? Perhaps you don’t need the consequences text, but certainly letting them know that if a firework runs for 5 seconds, perhaps they should remove it automatically after say, 10 seconds (to account for slower systems) to ‘avoid memory leaking issues’. The bug demonstrates that Corona isn’t able to handle this properly, even when my code has garbage collection in memory warning messages. The only fix is to remove the emitter. Hope this helps someone.

#20

Do you have a CaseID for the bug report?