Excessive increase "crashes rate" from Build 3692

Due to “errors in callbacks and listeners are not caught”,
Lua coding error will cause Lua pcall crashed.
see unhandledError.

Mostly Redmi. And RAM starts from 2GB, so I don’t think problem with low RAM.

Same update was released on Appstore and there are no such crashes. So it’s platform dependent I think. Also errors in callbacks and listeners never crashed whole app. It was just error. But here app crashed completely without any messages.

Here are plugins we’re using:

['plugin.att'] =
{
    publisherId = 'com.solar2d',
    supportedPlatforms = {iphone = true}
},
["plugin.firebaseAnalytics"] =
{
    publisherId = "tech.scotth",
    marketplaceId = "xxxxx",
    supportedPlatforms = {android=true}
},
["plugin.firebaseCrashlytics"] =
{
    publisherId="tech.scotth",
    marketplaceId = "xxxxx",
    supportedPlatforms = {android=true}
},
["plugin.gpgs.v2"] =
{
    publisherId = "com.coronalabs",
    supportedPlatforms = {android=true}
},
["CoronaProvider.gameNetwork.apple"] =
{
    publisherId = "com.coronalabs",
    supportedPlatforms = { iphone=true, ["iphone-sim"]=true },
},
["plugin.google.iap.billing.v2"] =
{
    publisherId = "com.solar2d",
    supportedPlatforms = { android=true }
},
["plugin.openssl"] = {
    publisherId = "com.coronalabs",
    supportedPlatforms = { android=true}
},
["plugin.utf8"] =
{
    publisherId = "com.coronalabs"
},
["plugin.notifications.v2.firebase"] =
{
    publisherId = "com.coronalabs"
},
["plugin.advertisingId"] =
{
    publisherId = "com.coronalabs",
},
["plugin.pasteboard"] =
{
    publisherId = "com.coronalabs",
},
['plugin.appodeal.beta.base'] = { publisherId = 'com.coronalabs' },
['plugin.appodeal.beta.AppLovin'] = { publisherId = 'com.coronalabs' },
['plugin.appodeal.beta.Bidmachine'] = { publisherId = 'com.coronalabs',
   supportedPlatforms = {
      android = true,
      iphone = true,
   }
},
['plugin.appodeal.beta.GoogleAdMob'] = { publisherId = 'com.coronalabs',
   supportedPlatforms = {
      android = true,
      iphone = true,
   },
},
['plugin.appodeal.beta.Unity'] = { publisherId = 'com.coronalabs' },
['plugin.appodeal.beta.Yandex'] = { publisherId = 'com.coronalabs' },		

We already tested without Appodeal, GPGS, Crashlytics, Firebase Analytics and Notifications V2. Nothing changed.

My user perceived crash rate jumped to 11.6% in an app that only has the following plugins:
[“plugin.google.iap.billing.v2”]
[“CoronaProvider.native.popup.social”]
[“CoronaProvider.native.popup.activity”]
[“plugin.iap_badger”]

Hello, I have results of new tests I did.

I upload an app that has no integrated purchases, that is, I do not use [“plugin.google.iap.billing.v2”]. I compiled it with 3696. And although the crash rate went up a little, it was not as abrupt as in the other apps where I use this plugin.

And the ANR rate down:

What catches my attention is that @anon63346430 uses this plugin but had no increase in crash rate.

But this app also uses native social popup plugin… Therefore, it could infer that the main problem is the integrated purchasing plugin …

On the other hand I am waiting for the results of the test I did with another app where i eliminated the native social popup plugin. This result may help determine what is the plugin that is having problems

I have the results of the app where I eliminated the “Native Social Popup Plugin” (this app now only uses the in-app purchases plugin and Admob).

As can be seen in the graph, the crash rate did not lower!

The only failure reporting so far in version 45 is:

[split_config.armeabi_v7a.apk!libcorona.so]
SIGSEGV

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***

backtrace:
  #00  pc 0x000000000003651c  /apex/com.android.runtime/lib/bionic/libc.so (__memcpy_a53+96)
  #01  pc 0x0000000000061cbb  /vendor/lib/egl/libGLESv2_mtk.so
  #02  pc 0x0000000000061f99  /vendor/lib/egl/libGLESv2_mtk.so
  #03  pc 0x00000000000ebdd3  /vendor/lib/egl/libGLESv2_mtk.so
  #04  pc 0x00000000000eaca1  /vendor/lib/egl/libGLESv2_mtk.so (glTexImage2D+164)
  #05  pc 0x00000000000c5c35  /data/app/~~NF3NKwOONq9nFE-uNRKWhQ==/com.orange.kids.game.coloring-6a_os8I-dQh7GubUH_DQBg==/split_config.armeabi_v7a.apk!libcorona.so (BuildId: 0e49b121c8829ecf7bfe306c247185d7d9399613)
  #06  pc 0x00000000000c88f9  /data/app/~~NF3NKwOONq9nFE-uNRKWhQ==/com.orange.kids.game.coloring-6a_os8I-dQh7GubUH_DQBg==/split_config.armeabi_v7a.apk!libcorona.so (BuildId: 0e49b121c8829ecf7bfe306c247185d7d9399613)
  #07  pc 0x0000000000071afd  /data/app/~~NF3NKwOONq9nFE-uNRKWhQ==/com.orange.kids.game.coloring-6a_os8I-dQh7GubUH_DQBg==/split_config.armeabi_v7a.apk!libcorona.so (BuildId: 0e49b121c8829ecf7bfe306c247185d7d9399613)
  #08  pc 0x0000000000061ad1  /data/app/~~NF3NKwOONq9nFE-uNRKWhQ==/com.orange.kids.game.coloring-6a_os8I-dQh7GubUH_DQBg==/split_config.armeabi_v7a.apk!libcorona.so (BuildId: 0e49b121c8829ecf7bfe306c247185d7d9399613)
  #09  pc 0x00000000000bfa4d  /data/app/~~NF3NKwOONq9nFE-uNRKWhQ==/com.orange.kids.game.coloring-6a_os8I-dQh7GubUH_DQBg==/split_config.armeabi_v7a.apk!libcorona.so (BuildId: 0e49b121c8829ecf7bfe306c247185d7d9399613)
  #10  pc 0x000000000004206f  /data/app/~~NF3NKwOONq9nFE-uNRKWhQ==/com.orange.kids.game.coloring-6a_os8I-dQh7GubUH_DQBg==/oat/arm/base.odex (art_jni_trampoline+102)
  #11  pc 0x00000000000476d1  /data/app/~~NF3NKwOONq9nFE-uNRKWhQ==/com.orange.kids.game.coloring-6a_os8I-dQh7GubUH_DQBg==/oat/arm/base.odex (com.ansca.corona.Controller.updateRuntimeState+1544)
  #12  pc 0x00000000000547fb  /data/app/~~NF3NKwOONq9nFE-uNRKWhQ==/com.orange.kids.game.coloring-6a_os8I-dQh7GubUH_DQBg==/oat/arm/base.odex (com.ansca.corona.graphics.opengl.GLSurfaceView$GLThread.guardedRun+2482)
  #13  pc 0x00000000000d39d5  /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68)
  #14  pc 0x00000000004eac6b  /apex/com.android.art/lib/libart.so (art_quick_invoke_stub+282)
  #15  pc 0x000000000012bd2d  /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+148)
  #16  pc 0x000000000023f1ef  /apex/com.android.art/lib/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+254)
  #17  pc 0x0000000000236eaf  /apex/com.android.art/lib/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+738)
  #18  pc 0x00000000004de3eb  /apex/com.android.art/lib/libart.so (MterpInvokeDirect+514)
  #19  pc 0x00000000000ce514  /apex/com.android.art/lib/libart.so (mterp_op_invoke_direct+20)
  #20  pc 0x0000000000219560  /data/app/~~NF3NKwOONq9nFE-uNRKWhQ==/com.orange.kids.game.coloring-6a_os8I-dQh7GubUH_DQBg==/oat/arm/base.vdex (com.ansca.corona.graphics.opengl.GLSurfaceView$GLThread.run+42)
  #21  pc 0x000000000022fea9  /apex/com.android.art/lib/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.llvm.18134065572441359314)+248)
  #22  pc 0x0000000000236619  /apex/com.android.art/lib/libart.so (art::interpreter::EnterInterpreterFromEntryPoint(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*)+120)
  #23  pc 0x00000000004cdc63  /apex/com.android.art/lib/libart.so (artQuickToInterpreterBridge+698)
  #24  pc 0x00000000000d8561  /apex/com.android.art/lib/libart.so (art_quick_to_interpreter_bridge+32)
  #25  pc 0x00000000000d39d5  /apex/com.android.art/lib/libart.so (art_quick_invoke_stub_internal+68)
  #26  pc 0x00000000004eac6b  /apex/com.android.art/lib/libart.so (art_quick_invoke_stub+282)
  #27  pc 0x000000000012bd2d  /apex/com.android.art/lib/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+148)
  #28  pc 0x00000000003f84e7  /apex/com.android.art/lib/libart.so (art::JValue art::InvokeVirtualOrInterfaceWithJValues<art::ArtMethod*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, art::ArtMethod*, jvalue const*)+374)
  #29  pc 0x00000000003f85f7  /apex/com.android.art/lib/libart.so (art::JValue art::InvokeVirtualOrInterfaceWithJValues<_jmethodID*>(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue const*)+42)
  #30  pc 0x00000000004396f5  /apex/com.android.art/lib/libart.so (art::Thread::CreateCallback(void*)+1056)
  #31  pc 0x0000000000080b13  /apex/com.android.runtime/lib/bionic/libc.so (__pthread_start(void*)+40)
  #32  pc 0x0000000000039d23  /apex/com.android.runtime/lib/bionic/libc.so (__start_thread+30)

With these results, and the results of previous tests, where I eliminated Admob and the crash rate did not go down, and others of my apps that do not use the in-app Purchasse plugin and compiled with 3696 the crash rate was barely increased , everything seems to indicate that the problem is with the billing plugin [“plugin.google.iap.billing.v2”].

If this plugin is the cause, we are in trouble solving it…

@chris_raz you have mentioned that you have apps that use the same plugins, but that the best performance apps are the ones that had the greatest impact on the shock rate. These are the ones that generate the greatest income in integrated purchases?

1 Like

Thanks for continuing to update this thread with more info. It’s annoying when things are out of our control.

Hopefully it gets tracked down/patched ASAP.

Have you been on their discord? Wonder if this is being considered at all for fixing. Now with the iOS/Mac builds also not working, I wish we could get an update from vlad even if it’s just a brief on what what they are currently prioritising.

Yes that would be correct, our top performers would definately generate more iap’s. But I don’t know if that’s the problem. I’m scratching my head constantly trying to figure it out. I’m sure @anon63346430 is getting a good amount of iap’s and he is not being affected by the crashes. So there either must be some implementation type of problem of ours which API 33 exposes or it is some other core API we are using in the app not related to iap’s.

We are in play pass with our affected apps so that could be a thought, are you in play pass with the affected apps?

I noticed a message pasted below on the play console which I doubt is connected to this

Play Billing Library requirements for Android 14 compatibility

Oct 12, 2023 04:40
Android 14 has introduced new [security changes] which affect apps targeting Android 14. In order to ensure compatibility, Google has released [Play Billing Library version 6.0.1 and 5.2.1]
When updating your app to target Android 14, following these [migration steps] you will need to update your app to either of these versions of the Play Billing Library to be compatible with the recent security changes.
These changes only affect your app when it starts targeting Android 14. Users running Android 14 while your app is not targeting Android 14 are unaffected

Doesn’t look like IAP is the issue. Less than 3% of sessions have purchases, unless is something in initialization. Probably some other API is guilty. If you could tie the crashes to a specific action in your app would be the ideal outcome. As I said before Crashlytics and Bugsnag can help with that.
Also at 3-12% crash rate its feasable to simulate the crash on your own device?

Hello, I have results from another app that a few days ago i compiled with 3696, This is another app that not make use of the in-app purchasses plugin.

Only use Admob and Native Social Popup plugins.

plugins =
  {
    -- enable the admob plugin
    ["plugin.admob"] = {publisherId = "com.coronalabs"},

    -- enable the native.popup.social plugin PARA BOTON COMPARTIR
    ["CoronaProvider.native.popup.social"] = {publisherId = "com.coronalabs"},      
  },        
}

In this case, the crash rate not only did not go up, but went down. (Step from 0.52% to 0.37%)

This new result continues to give more strength to my hypothesis that the problem is the billing library.

I have compiled with build 3696 in total 7 apps, I have performed the tests of eliminating admob, eliminate social native popup, change the library “media” for “audio”, add in build.settings several inputs suggested by Vlad, and all these the only ones who have not had increase the crash rate are the two that do not use the billing library.

1 Like

Dropping by to appreciate your regular updates on this. We have the same issues but I’m too tied down of late with other things to be able to run tests for comparisons. Thanks for pursuing this.

Just FYI, we also use IAPs and subscriptions in our game and our crash rate is 0.14% - well below the threshold. If IAPs are the cause of your crashes, my best guess would be that it’s an implementation problem rather than the billing plugin being inherently broken.

Edit: We used the iap_badger module rather than writing the IAP handling stuff ourselves, just in case that’s useful to know.

Thanks for the information.
I will continue testing this.

For statistics: in my app (where crashes have increased) there is no any IAPs. So I think it’s unlikely that this is the problem.
I’m also currently working on app updates and collecting crash statistics. If anything becomes clearer, I’ll write here.

Thank you, among all we will discover what the problem is.

Added BugSnag to our build. Almost any crash looks like this one.

Trim memory: running critical and on scene change it crashes. “Will” phase of scene:hide manages to complete and “Did” phase never happens.

I’m thinking about adding memory dumping.

Considering billing.v2 plug-in, we’re using it but 99.9% of our users just watching ads and make no purchases. And all crashes, as I mention before, happen on scene change.

when I was using bugsnag it was the same… all ANR’s/crashes happen mainly after suspend/resume and memory trims. Seems like important memory (like game objects and classes) somehow get corrupted?

Maybe. Strange it’s all started so suddenly. Like, if API 33 is enabled then Android garbage collector is working more aggressively. In our case these crashes are memory independent. Some devices have 8-10 GBs and still experiencing these crashes.