Excessive increase "crashes rate" from Build 3692

orangegstudios · September 8, 2023, 1:34pm

For a while now I’ve been trying to identify a significant increase in the crash rate in my apps when I compile with build 3692 or later, even with 3696.

With build 3691 or lower the crash rate of my apps is below the threshold, but when compiling with later versions without making any source code changes, the crash rate is upper 3%, even up to 8% in some apps!

Discarding possible causes, I have eliminated the use of the “media” library that in some parts of the code had remained (now use only the “audio” library throughout the code), but still compiling with Build 3695 and 3696 the failure rate keep up.

As an example with one of my apps, this is what happens to me:

Although in this app I went from Build 3690 to 3695. In other of my apps from compiling with Build 3692 it is where the increase in failures occurs.

Even without making any change in the source code. If compile with Build 3691 there are no problems. When compiling with Build 3692 or later it is where the increase occurs.

This has happened to someone? Is any SDK problem?

These are the most relevant failures of aab 38 (Build 3696)

**Stack tracking: java.lang.NullPointerException**

Exception java.lang.NullPointerException: Attempt to invoke virtual method 'android.content.res.Configuration android.content.res.Resources.getConfiguration()' on a null object reference
  at android.app.ActivityThread.updateLocaleListFromAppContext (ActivityThread.java:6472)
  at android.app.ActivityThread.handleBindApplication (ActivityThread.java:6710)
  at android.app.ActivityThread.access$1500 (ActivityThread.java:258)
  at android.app.ActivityThread$H.handleMessage (ActivityThread.java:2042)
  at android.os.Handler.dispatchMessage (Handler.java:107)
  at android.os.Looper.loop (Looper.java:213)
  at android.app.ActivityThread.main (ActivityThread.java:7967)
  at java.lang.reflect.Method.invoke
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:503)
  at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:938)

**Stack tracking: java.lang.IllegalArgumentException**

Exception java.lang.IllegalArgumentException: parameter must be a descendant of this view
  at android.view.ViewGroup.offsetRectBetweenParentAndChild (ViewGroup.java:6305)
  at android.view.ViewGroup.offsetDescendantRectToMyCoords (ViewGroup.java:6224)
  at android.view.ViewRootImpl.scrollToRectOrFocus (ViewRootImpl.java:4480)
  at android.view.ViewRootImpl.draw (ViewRootImpl.java:4046)
  at android.view.ViewRootImpl.performDraw (ViewRootImpl.java:3890)
  at android.view.ViewRootImpl.performTraversals (ViewRootImpl.java:3152)
  at android.view.ViewRootImpl.doTraversal (ViewRootImpl.java:2005)
  at android.view.ViewRootImpl$TraversalRunnable.run (ViewRootImpl.java:8234)
  at android.view.Choreographer$CallbackRecord.run (Choreographer.java:972)
  at android.view.Choreographer.doCallbacks (Choreographer.java:796)
  at android.view.Choreographer.doFrame (Choreographer.java:731)
  at android.view.Choreographer$FrameDisplayEventReceiver.run (Choreographer.java:957)
  at android.os.Handler.handleCallback (Handler.java:938)
  at android.os.Handler.dispatchMessage (Handler.java:99)
  at android.os.Looper.loop (Looper.java:223)
  at android.app.ActivityThread.main (ActivityThread.java:7945)
  at java.lang.reflect.Method.invoke
  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:603)
  at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:947)

**Stack tracking:java.lang.IllegalStateException (This also happens with Build 3691 or previous)**

Exception java.lang.IllegalStateException: The file system on the device is in a bad state. WorkManager cannot access the app's internal data store.
  at androidx.work.impl.utils.ForceStopRunnable.run (ForceStopRunnable.java:128)
  at androidx.work.impl.utils.SerialExecutor$Task.run (SerialExecutor.java:91)
  at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641)
  at java.lang.Thread.run (Thread.java:764)
Caused by android.database.sqlite.SQLiteCantOpenDatabaseException: unknown error (code 14 SQLITE_CANTOPEN): Could not open database
  at android.database.sqlite.SQLiteConnection.nativeOpen
  at android.database.sqlite.SQLiteConnection.open (SQLiteConnection.java:211)
  at android.database.sqlite.SQLiteConnection.open (SQLiteConnection.java:195)
  at android.database.sqlite.SQLiteConnectionPool.openConnectionLocked (SQLiteConnectionPool.java:503)
  at android.database.sqlite.SQLiteConnectionPool.open (SQLiteConnectionPool.java:204)
  at android.database.sqlite.SQLiteConnectionPool.open (SQLiteConnectionPool.java:196)
  at android.database.sqlite.SQLiteDatabase.openInner (SQLiteDatabase.java:880)
  at android.database.sqlite.SQLiteDatabase.open (SQLiteDatabase.java:865)
  at android.database.sqlite.SQLiteDatabase.openDatabase (SQLiteDatabase.java:739)
  at android.database.sqlite.SQLiteDatabase.openDatabase (SQLiteDatabase.java:729)
  at android.database.sqlite.SQLiteOpenHelper.getDatabaseLocked (SQLiteOpenHelper.java:355)
  at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase (SQLiteOpenHelper.java:298)
  at androidx.sqlite.db.framework.FrameworkSQLiteOpenHelper$OpenHelper.getWritableSupportDatabase (FrameworkSQLiteOpenHelper.java:145)
  at androidx.sqlite.db.framework.FrameworkSQLiteOpenHelper.getWritableDatabase (FrameworkSQLiteOpenHelper.java:106)
  at androidx.room.RoomDatabase.inTransaction (RoomDatabase.java:476)
  at androidx.room.RoomDatabase.assertNotSuspendingTransaction (RoomDatabase.java:281)
  at androidx.work.impl.model.SystemIdInfoDao_Impl.getWorkSpecIds (SystemIdInfoDao_Impl.java:120)
  at androidx.work.impl.background.systemjob.SystemJobScheduler.reconcileJobs (SystemJobScheduler.java:298)
  at androidx.work.impl.utils.ForceStopRunnable.cleanUp (ForceStopRunnable.java:249)
  at androidx.work.impl.utils.ForceStopRunnable.forceStopRunnable (ForceStopRunnable.java:215)
  at androidx.work.impl.utils.ForceStopRunnable.run (ForceStopRunnable.java:110)

chris_raz · September 8, 2023, 10:38pm

Regarding the second crash you have there. We have had a HUGE jump in that crash

Exception java.lang.IllegalArgumentException: parameter must be a descendant of this view
  at android.view.ViewGroup.offsetRectBetweenParentAndChild (ViewGroup.java:6305)
  at android.view.ViewGroup.offsetDescendantRectToMyCoords (ViewGroup.java:6224)

After doing some research some people claim it could be admob banner which causes this. Seems to be a bug when the app suspends and reopens with the banner view displayed from what I can see. I posted in another thread regarding it… Do you use admob and banner with them? If so it might be a clue that that is in fact the cause…

In unity I have noticed the Admob plugin has depreciated the adaptive banner and now use smart banner. I’m not sure if this was a fix for the problem or not

orangegstudios · September 8, 2023, 10:59pm

Hi, Yes I am using admob and banners in my apps, in addition to interstitials.

lmgualandi · September 9, 2023, 1:59am

I didn’t update to target 33 yet but I do have the second crash (offsetRectBetweenParentAndChild) for some time now representing the vast majority of my crashes in vitals. My user-perceived crash rate is at .14% so not that worrying in my case. I’m more close to danger in ANRs at .17%.
I do have Admob and banners.
The other crashes you cited I’m not seeing (yet)
Are you using any crash analytics plugin like Crashlytics or Bugsnag? That could give more information.

orangegstudios · September 9, 2023, 11:28am

I’m not using any crash analytics plugin.

orangegstudios · September 11, 2023, 2:39pm

Any clue that may be happening?

markus-bc · September 11, 2023, 6:24pm

Hi! A similar issue occurred recently with one of our apps, built with Solar2D Native:

The app had some bugs that we had not noticed ourselves, and for a reason or another the
Solar2D runtime errors got automatically enabled when we built an update with a new version of Solar2D.

First the crash rate went up about 2%, then we added

application =
{
showRuntimeErrors = false,
}

to config.lua as instructed here:
(Solar2D Documentation — Developer Guides | Basics)
… and the error rate on Google Play dashboard went back down.

Not sure if this helps, but your issue sounds very much the same so I wanted to share this.

orangegstudios · September 11, 2023, 6:51pm

Hi, I will try it and tell you what result I get.
Thank you so much!

anon63346430 · September 11, 2023, 10:32pm

This is automatically false for production builds. Even if this was now broken the popup only reports the crash, the crash has already happened.

@orangegstudios Remove admob and see if the crashes go away?

clang · September 12, 2023, 1:36am

orangegstudios:

Exception java.lang.NullPointerException: Attempt to invoke virtual method 'android.content.res.Configuration android.content.res.Resources.getConfiguration()' on a null object reference

I am not sure if this issue related to non-SDK restrictions on Android 13, can you share the Android version distribution of the issue?
I recommend using veridex-tool to check your apk. It will give you a table that tells you the java interface used by the apk and the blocking level, as well as the final summary. If it is unsupported, it can be use. If it is anything else, be on the lookout.

orangegstudios · September 12, 2023, 2:27pm

I share my build.settings file, in one of the last compilations that I uploaded to Google Play (Build 3696), I setting the version of the admob complement to version 30 to see if the last version of the complement could be bringing problems. But still the failure rate remains high.

Below I share the failures reported by Android Vitals of the last upload version.

settings =
{
	android =
   {
      supportsScreens =
      {
         resizeable    = true,
         smallScreens  = true,
         normalScreens = true,
         largeScreens  = true,
         xlargeScreens = true,
      },

      minSdkVersion = "16",
      largeHeap = true,      

      -- Para habilitar Compras Integradas --
      usesPermissions = {"com.android.vending.BILLING",
                         "android.permission.WRITE_EXTERNAL_STORAGE",
      }, 

      applicationChildElements =
      {
          [[
              <meta-data android:name="com.google.android.gms.ads.APPLICATION_ID"
                  android:value="xxxxxxxxxxxxxxxxxxxxxxxxxx"/> 
          ]],

          [[
               <provider
                  android:name="androidx.startup.InitializationProvider"
                  android:authorities="${applicationId}.androidx-startup"
                  tools:node="remove">
               </provider>            
          ]],
      },
   },

	orientation = {
				default = "landscapeRight",
				supported = { "landscapeLeft", "landscapeRight" },
			   },

  plugins =
  {
    -- enable the admob plugin
    ["plugin.admob"] = {publisherId = "com.coronalabs",
                        version = "v30",},
   
    -- enable the native.popup.social plugin PARA BOTON COMPARTIR
    ["CoronaProvider.native.popup.social"] = {publisherId = "com.coronalabs"},   
   
    -- Para habilitar Compras Integradas --
    ["plugin.google.iap.billing.v2"] = {publisherId = "com.solar2d"},        
  },    	
}

Android Vitals:

orangegstudios · September 12, 2023, 5:19pm

@clang Do you mean sharing the apk file?

chris_raz · September 12, 2023, 10:19pm

you are using the exact same plugins i use. The Viewgroup.offsetRectBetweenParentAndChild Crash has always been a problem, only since i switched to api 33 has it become more pronounced to the point it is pushing it over the bad behaviour threshold. I dont think its going to help going back with the admob plugin. something in api 33 is causing this crash to be more pronounced, again from what i have read you can do i quick google search, this error commonly happens with admob banners. Which admob seem to have addressed. not sure if its the banner type, as they now use smart banners over adaptive banners, or its something entirely different.

Quick question, what does this do in your build settings?

 <provider
                  android:name="androidx.startup.InitializationProvider"
                  android:authorities="${applicationId}.androidx-startup"
                  tools:node="remove">
               </provider>

anon63346430 · September 12, 2023, 10:57pm

You’ve barely any crashes there? I have much more than that per day yet I am around 0.05% crash rate with 3695. If you are really concerned about this then use another ad network… +1 for unity ads.

Also, banner ads only make any sense if you have many thousands of DAU as the eCPMs are awful, not to mention the player experience. When was the last time you played “a big game” that had banner ads?

My advice is remove the banners and look into a reward ad-style of gameplay. You’ll thank me later

chris_raz · September 12, 2023, 11:27pm

We do have thousands of DAU (one of our apps is about to hit 100m downloads) and ECPMs are awful for us given that our app is in the family section of GP. Also admob does give us some open doors into google itself which allows for other potential possibilities. With admob being one of the largest ad networks you would think it should be viable and less prone to crashes but currently its not the case here on solar2d. but something that really needs to be rectified here instead of just removing admob for the reason stated above.

anon63346430 · September 13, 2023, 9:45am

I completely agree. But then again, the ANR issues haven’t been fixed in the past 2 years.

Sometimes it is easier to remove the banners, take a small hit to revenue, wait for a future solution and avoid being deranked by Google for bad behaviour.

High ANR rate doesn’t seem to affect ranking but a high crash rate can kill your visibility. I know loads of devs (not solar) that have been punished for a high crash rate.

orangegstudios · September 13, 2023, 12:37pm

Hi, this information that I was showing corresponds to one of my apps with fewer downloads, as long as I make updates as in this case I try my less critical apps. My two most important apps have about 270k average daily downloads, and 22.5 million active users between the two.

All my apps, including the most importants, have the same configuration and use the same plugins, so I have not updated them yet until this problem is not solved.

Although in this app that I have used to compile with build 3696, vitals reports a minimum number of users and sessions affected, the crash rate is 2.73% which is worrisome.

This is also something that I still can’t understand the metric, I suppose it informs it is about a representative sample of users, because with such a low affected sessions I do not understand the 2.73% crash rate.

orangegstudios · September 13, 2023, 12:42pm

@chris_raz, my situation is similar. I agree that the solution is not to eliminate admob, but to solve the problem of blockages.

orangegstudios · September 13, 2023, 1:05pm

@chris_raz , this entry in build.settings added it by a suggestion of Vlad to see if one of the faults could be resolved (java.lang.NullPointerException).

jdsmedeirosbr · September 13, 2023, 4:45pm

Is anyone using Appodeal having these Admob problems?