Lots of ANRs on Android (PackageStateChangedService, android.intent.action.SCREEN_OFF)

Solar2D Android
#41

@Rob Miracle - thank you so much for the info. We updated to 3315 just to be safe.

Were wake lock issues a problem all along, or was this an issue with one of the more recent daily builds that ended up being patched? The reason I ask is because I had one app with a 3% rate and google tagged it as “Bad Behavior” but other apps don’t seem to have as high of an occurrence.

I prefer not to go through and update those apps again unless the rate is over the threshold, but lmk if you have any info on when this error started happening. Thank you!

#42

@JBean, I’d need to get an engineer to try and explain the problem. This is just me speculating, but I don’t think this is something new we created in a daily build. I think it’s more likely that Google is “*finding*” more problems by refining that they now consider “Bad Behavior”. It’s creating a moving target.

Rob

#43

@Rob Miracle

I think you are right on google finding more problems. It seems that they started tracking that back in late May of this year, as I see some apps had a spike in bad behavior around that time. Thank you for clarifying!

#44

It is most definitely a Corona problem @Rob.  I only get this on 2 of 3 apps and the one that hasn’t got any bad behaviour hasn’t been updated in the past 6 weeks so it is using a pre GPDR build.

FYR, the “problem” was introduced with build 3306.

#45

@SGS:

It’s not just a Corona problem. We’ve seen this issue pop up around the same timeline as Corona built apps compared to other apps built with other engines. In fact, one of the apps where the issue popped up (not built with Corona) hadn’t been updated in over 12 months, and this issue became a new one. We took a peek into the android vitals, and it has been an issue since before March of this year, but it was never targeted until recently as a bad behavior issue.

My guess is that many apps have had this issue all along,  but google has recently decided to add that as part of the bad behavior stats, and will down-rank apps as a result.

It is most definitely a newly targeted issue by Google play and not specific to this engine. After having compared the data, this is almost a 100% certainty.

#46

The “wakelock bug” was introduced with daily 3306 (maybe changes made surfaced the issue, I don’t know), specifically “changing how audio library memory is handled, should reduce amount of AL related crashes”.  Prior to his build I was running at 0.01% for the previous 3 months.

It was then (mostly) fixed - after we all alerted Corona - with 3311, specifically “Suspending audio thread in background”.  3314 promises to fix it completely but it is too early for conclusive data as yet.

If it was purely a Google change then all my apps would have been affected (same plugins, same backend, same code libraries, etc.)

​Maybe the truth is somewhere in between?

#47

@SGS

Yeah it could be, but given my specific scenario, I’m not 100% sure.

One of our apps that we hadn’t updated in awhile though was also impacted (in greater %), and was a Gamesalad built app. This one shows in the stats that the partial wake lock has been an issue for at least 3 months (It won’t let me look back further on the timeline).

It leads me to believe that it has been an issue all along, but perhaps google has recently tagged this issue as “bad behavior” and is now asking developers to fix it. 

The interesting thing is, this app has the same string of partial wake lock issues since March, but didn’t impact the apps ranking until late May. So it leads me to conclude that Google (during late May) decided to add this to the list of issues that devs need to fix in order to keep their app rank in good standing, even though this has been an issue for quite some time.

I would advise digging into your stats under Android Vitals - Overview - then click on “View details” under the Stuck Partial Wake Locks stat and see how long some of your apps have had that isuse over time. There’s a grid there, and you can look back as far as 3 months. That should answer your question as to whether or not it’s a new or old issue.

#48

The important thing is we think we have it fixed. 

#49

I think you can see when I published with 3306…

And you can probably also tell when I pushed a panic fix with 3311 too (which seems to be really helping).

When I said it was build related I really meant it lol

#50

@SGS:

Yes, yours clearly indicates that an update could have impacted the release with that new issue…

It is very bizarre for sure, because I’m seeing apps that have this issue dating back to March of 2018. (Both Corona and Gamesalad).

Oh well, at least in the end it is fixed!

#51

@Rob Miracle: I’m still geting about a 1.71% rate of partial wake locks after updating with the latest daily build (3315).

We are using Applovin and Admob paid ad plugins as well as the google play iAP plugin.

Can you please check this for us, and make sure these may not be causing these issues?

#52

@JBean how long have your apps been published with 3315? It seems like it can take a couple of days for Google’s Vitals to settle out.

Rob

#53

the “overall rate” can take a long time to decrease, particularly if you have lots of samples from older releases that exhibit the problem relative to the activity on your newest release, especially if the older-releases are still actively sending in samples (fe user doesn’t update)

you’ll gradually see the daily graph start to “wash out”, but the overall rate may not change significantly - you may have to wait for those older-release samples to “age out”.  look for a decreasing trend in the daily graph after say a week, then expect a corresponding change to the overall rate after perhaps a month? (all depending on volume of downloads/updates of new release vs existing installs of problem releases)

#54

Yeah, it could be that the other releases are impacting the statistics.

I don’t know if I was reading the dashboard correctly, but it appeared that the “latest” apk I had published about 4 days ago still had over 1% rate of issues with the wake lock stat. I think it said specifically that version was still having issues vs prior versions. Again, I could be reading the dashboard incorrectly.

I updated it again today for safe measure, and will report back next week if the stats are lowering. Thanks for the help!

#55

I’ve recently dived into the ANR issue on my apps and although my latest update is using build 3362, I still see quite a few ANRs especially the ones with: Broadcast of Intent { act=android.intent.action.SCREEN_OFF flg=0x50200010

Is there something we can do on our end to reduce these?

#56

@rune7 can you use a service like pasetbin, copy and paste the entire backtrace to one of these ANR’s to it and share a link to the pastebin here?

Rob

#57

Sure, here is an example - https://pastebin.com/6AeG0JP6

It may not be as clear when pasting all threads. If its enough to get specific threads please let me know.

It was taken from: Sony Xperia Z5 Compact (E5823), Android 7.1

#58

Hello! Thank you for your report. I have several questions:

  1. What is the name of ANR?

  2. How many ANRs of this type you get (percentage, or if absolute numbers, please provide total number of launches)"

  3. Do you have a reproduction of this ANR?

  4. What plugins are you using?

#59

@vlad, wake locks and ANRs from one of my games…

ANR over bad threshold level and wake locks are still not fixed yet.  This is with 3335 - last 30 days.

Mostly Broadcast of Intent { act=android.intent.action.SCREEN_OFF flg=0x50000010 } or Input dispatching timed out (Waiting to send non-key event because the touched window has not finished processing certain input events that were delivered to it over 500.0ms ago. Wait queue length: 4. Wait queue head age: 378834.9ms.)

Raised from  com.ansca.corona.CoronaActivity 

#60

Hi @vlads,

  1. The one I pasted the error stack for was titled on Google Developer console: 

Broadcast of Intent { act=android.intent.action.SCREEN_OFF flg=0x50200010.

Looking at what STS sent I see these two as the dominant ones as well.

  1. My ANR ratio is still high ~1.35% which is way over their bad behavior threshold of ~0.5%. Out of the total I’d say these two take over 90% of the total. So, overall I’d say they affect around 1.3%+ of our total sessions.

  2. On our test devices I could not reproduce these. But we only have 2 different devices to test on.

  3. Plugin used on Google: advertisingId, flurry, iap.v3

List of ANRs from last 3 days (note that we don’t have a lot of daily users):

Amount: 15

Input dispatching timed out (Waiting to send non-key event because the touched window has not finished processing certain input events that were delivered to it over 500.0ms ago. Wait queue length: 19. Wait queue head age: 99153.1ms.)

Amount: 5

Broadcast of Intent { act=android.intent.action.SCREEN_OFF flg=0x50200010 (has extras) }

They consist 20 out of 21 total reports for the period.

On a side note, it seems that our last update introduced the issue of wake lock which was not there on our previous build from last year. I don’t have enough data yet to give good confidence percentage but I can say that it exist while it didn’t previously.

I currently use build 3362. 

Just to note that latest builds have vastly improved ANR situation compared to last year, so well done!

Adi