@Rob, you mentioned in here and in your blog entry just now, that the issue only occurs weekdays and from a specific time. Sounds to me then like this is a Cron job, or a backup Daemon of some sort. Something that runs on a schedule. Perhaps locking database tables while it operates. Could even be that a database table has grown larger than the available memory and the engine is no longer properly handling the temp table creations that certain queries would invoke, or maybe an index has corrupted. Could also be that a recent server update has replaced MySQL with MariaDB since modern Linux distributions tend to favour that as a drop-in replacement - there are a few subtle differences in how MariaDB handles certain queries. ORDER clauses on inner queries are ignored if no LIMIT clause is set, for example.
We’ve already scrubbed the cron jobs and looked at various backup jobs and such. I don’t believe we’ve made any changes, if so that would have been one of the first things we backed out. We don’t have any auto-updates, so I don’t believe a MariaDB would have been dropped in without us knowing it.
But then today, I don’t believe we had an outage and it’s in the middle fo the week.
It may not even be the server at all. Potential DNS or CDN routing issues, or some form of attack that a dynamic firewall is kicking in on too trigger happily. Is Fail2Ban running and blocking the other servers during peak usage? Or the server could be sitting behind a load balancer that’s turned iffy.
Again just throwing ideas in the air. Even a seemingly obvious thing to check might have been missed during the panic. I’ve reconfigured and recompiled all sorts of server software in the past before thinking to check the file I’d just edited for missing semicolons!
I’m seeing mixed messages here and its creating confusion for me.
Rob is talking about random server outages, but others (including myself) are experiencing build failures. Is one a symptom of the other? Are these separate issues?
I’ve been trying to make a successful build every night for the past 2 weeks, with no success. The build servers appear to be available and responsive, but my problem is in the build/compilation process. I’m talking about errors like “There was a problem linking the app” and “Builder failed: Undefined symbols for architecture armv7”
People seem to think it’s related to their specific plugin, but it appears to occur with ANY plugin. Add any plugin to one of the bundled sample code projects and try an iOS build. It will fail. Take the plugin out and the build will succeed. Based on a note on the last daily build, it looks like there were attempts to fix some build related issues.
Rob and team, please clarify what sort of errors are expected as a result of these “outages”. This will be helpful to know if I should continue waiting for a fix or look to other avenues.
@viktor Builds require server authentication. Having plugins also need server authentication. I think in general your build settings get sent to Corona servers, so if the server isnt working at the time you try to build, then the build fails. ie it is the same thing.
The server problems are defined by very long build times before the process times out. Generally the Forums have problems at the same time. If you can build and the build times are reasonable, you’re not hitting server problems.
If it’s not an apparent time out issue, i.e. you build in reasonable times, a 256 error is generally plugin related. The best test you can do is to build the HelloWorld sample app that comes with Corona. If you’re getting a 256 error there, then it’s not plugin related.
So lets assume it’s not a server problem. If you get a 256 error without plugins, we need to know if you’re on Windows or a Mac and perhaps specifically the macOS version if you’re on a Mac.
If you do not need Appodeal 2.6.1, I would suggest dropping back to the build 2020.3567 or earlier. Appodeal 2.6.0 should work with that build.
The team is aware of these issues with 3569 and 3570 and hopefully we will get a resolved build soon.
Seems to be all okay and nippy right now on forums and docs.
Im not sure if it helps but I turn off internet as I build, and then back on again to deploy using Xcode+Native Corona. Just in case it fails on Corona Builder while checking Corona server. Might be barking up the wrong tree but Im not taming any chances.
I’m on macOS Catalina 10.15.3 and I get the same error building for Android. I don’t know if those having success are on Windows or macOS, but I do know that Apple in the past few days changed their notarization processes. I don’t know if this is the impact.
I’ve notified Engineering and I’m waiting to hear back.
Not sure whether this is related, but I notice the openssl plugin keeps on download when I re-launch my simulator. If not mistaken, the openssl file size is ~800kb.
00:38:53.319 @ver \> NUL 00:38:53.319 Exception in thread "DisconnectableInputStream source reader" org.gradle.api.UncheckedIOException: java.io.IOException: Access is denied 00:38:53.319 at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:61) 00:38:53.319 at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:41) 00:38:53.319 at org.gradle.util.DisconnectableInputStream$1.run(DisconnectableInputStream.java:125) 00:38:53.319 at java.lang.Thread.run(Thread.java:748) 00:38:53.319 Caused by: java.io.IOException: Access is denied 00:38:53.319 at java.io.FileInputStream.readBytes(Native Method) 00:38:53.319 at java.io.FileInputStream.read(FileInputStream.java:255) 00:38:53.319 at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) 00:38:53.319 at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) 00:38:53.319 at java.io.BufferedInputStream.read(BufferedInputStream.java:345) 00:38:53.319 at org.gradle.util.DisconnectableInputStream$1.run(DisconnectableInputStream.java:96) 00:38:53.319 ... 1 more 00:38:53.319 ERROR: There was a problem downloading plugin 'shared.firebase.messaging\_com.coronalabs'. Please, try again. 00:38:53.319 00:38:53.319 FAILURE: Build failed with an exception. ... 00:38:53.319 \* What went wrong: 00:38:53.319 Connect to s3.amazonaws.com:443 [s3.amazonaws.com/52.217.43.190] failed: Connection timed out: connect 00:38:53.319 \> Connection timed out: connect