USE THIS THREAD: Is the Build Server Down?

Solar2D Android
#161

@Rob, you mentioned in here and in your blog entry just now, that the issue only occurs weekdays and from a specific time. Sounds to me then like this is a Cron job, or a backup Daemon of some sort. Something that runs on a schedule. Perhaps locking database tables while it operates. Could even be that a database table has grown larger than the available memory and the engine is no longer properly handling the temp table creations that certain queries would invoke, or maybe an index has corrupted. Could also be that a recent server update has replaced MySQL with MariaDB since modern Linux distributions tend to favour that as a drop-in replacement - there are a few subtle differences in how MariaDB handles certain queries. ORDER clauses on inner queries are ignored if no LIMIT clause is set, for example.

Just throwing ideas out there…

#162

We’ve already scrubbed the cron jobs and looked at various backup jobs and such. I don’t believe we’ve made any changes, if so that would have been one of the first things we backed out. We don’t have any auto-updates, so I don’t believe a MariaDB would have been dropped in without us knowing it. 

But then today, I don’t believe we had an outage and it’s in the middle fo the week. 

Baffling.

#163

It may not even be the server at all. Potential DNS or CDN routing issues, or some form of attack that a dynamic firewall is kicking in on too trigger happily. Is Fail2Ban running and blocking the other servers during peak usage? Or the server could be sitting behind a load balancer that’s turned iffy.

Again just throwing ideas in the air. Even a seemingly obvious thing to check might have been missed during the panic. I’ve reconfigured and recompiled all sorts of server software in the past before thinking to check the file I’d just edited for missing semicolons!

#164

Not sure DNS/CDN issues would be time-based.  More likely some date-based code is broken?

As with all these issues, you simply check the change log and see what changed Jan 10/11th and focus on that?

If no code has changed then you check the pipeline and investigate what changed there.

#165

I’m seeing mixed messages here and its creating confusion for me.

Rob is talking about random server outages, but others (including myself) are experiencing build failures. Is one a symptom of the other? Are these separate issues?

I’ve been trying to make a successful build every night for the past 2 weeks, with no success. The build servers appear to be available and responsive, but my problem is in the build/compilation process. I’m talking about errors like “There was a problem linking the app” and “Builder failed: Undefined symbols for architecture armv7”

People seem to think it’s related to their specific plugin, but it appears to occur with ANY plugin. Add any plugin to one of the bundled sample code projects and try an iOS build. It will fail. Take the plugin out and the build will succeed. Based on a note on the last daily build, it looks like there were attempts to fix some build related issues.

Rob and team, please clarify what sort of errors are expected as a result of these “outages”. This will be helpful to know if I should continue waiting for a fix or look to other avenues.

#166

@viktor Builds require server authentication. Having plugins also need server authentication. I think in general your build settings get sent to Corona servers, so if the server isnt working at the time you try to build, then the build fails. ie it is the same thing.

#167

They are experiencing multiple problems right now.

Even the latest daily builds 2020.3569 and builds 2020.3570 have issues for many people.

2020.3569 is giving error code 256 (even with empty projects without plugins).

2020.3570 has authorization problem and asks you to deauthorize and login again (it doesn’t help though).

2020.3567 works fine but it has no support for Appodeal SDK 2.6.0.

It’s a complete mess.

Everything is falling apart and I hope they do something soon.

#168

Couldn’t build for the past few hours. It was working around 10 hours ago. It gets stuck at “Packaging app…”.

Now, it returns an error ‘error while fetching plugins’

#169

Vlad mentioned that they experience build issues.

https://forums.coronalabs.com/topic/76819-appodeal-sdk-260/

Info will be minimum as always and we just have to wait.

#170

The server problems are defined by very long build times before the process times out. Generally the Forums have problems at the same time. If you can build and the build times are reasonable, you’re not hitting server problems.

If it’s not an apparent time out issue, i.e. you build in reasonable times, a 256 error is generally plugin related. The best test you can do is to build the HelloWorld sample app that comes with Corona. If you’re getting a 256 error there, then it’s not plugin related. 

So lets assume it’s not a server problem. If you get a 256 error without plugins, we need to know if you’re on Windows or a Mac and perhaps specifically the macOS version if you’re on a Mac.

If you do not need Appodeal 2.6.1, I would suggest dropping back to the build 2020.3567 or earlier. Appodeal 2.6.0 should work with that build.

The team is aware of these issues with 3569 and 3570 and hopefully we will get a resolved build soon.

Rob

#171

Rob thanks for the reply!

HelloWorld is giving me the same 256 error.

I need Appodeal 2.6.0 for android (don’t care for 2.6.1 for iOS).

I’m on Mac, macOS Catalina version 10.15.2

#172

I just realized that you are saying that I can use 357. I need it for Android and appodeal SDK version is 2.6.0 there!

#173

Seems to be all okay and nippy right now on forums and docs.

Im not sure if it helps but I turn off internet as I build, and then back on again to deploy using Xcode+Native Corona. Just in case it fails on Corona Builder while checking Corona server. Might be barking up the wrong tree but Im not taming any chances. 

#174

I’m on macOS Catalina 10.15.3 and I get the same error building for Android. I don’t know if those having success are on Windows or macOS, but I do know that Apple in the past few days changed their notarization processes. I don’t know if this is the impact.

I’ve notified Engineering and I’m waiting to hear back.

Rob

#175

Maybe unrelated, but while searching the forum and clicked on a linked with domain anscamobile.com, it lead me to a porn site.

Ben

#176

Corona Labs used to be Ansca Mobile…

They should have kept their old domain name; stumbling upon a porn site doesn’t contribute to spreading positive notoriety :-/

#177

Sorry about that. DO NOT visit the old anscamobile website. It’s NSFW.

Should we have let it go? Probably not, but in the various ownership transfers, culling domain names you don’t use any more is a common practice.  

We can’t fix it now, so just don’t visit it.

Rob

#178

I brought the issue up because:

  1. why would porn site owners buy a domain called anscamobile.com, unless “ansca” relates to porn in some way.

  2. i was concerned Coronalabs was hacked through something they own but have forgotten about it.

#179

Build seems down now. I have been trying to build on windows simulator 2019.3559 and keep on getting errors saying s3 bucket does not exist ( http://s3.amazonaws.com/52.216.147.21, http://s3.amazonaws.com/52.217.43.190)..)

Not sure whether this is related, but I notice the openssl plugin keeps on download when I re-launch my simulator. If not mistaken, the openssl file size is ~800kb.

00:38:53.319 @ver \> NUL 00:38:53.319 Exception in thread "DisconnectableInputStream source reader" org.gradle.api.UncheckedIOException: java.io.IOException: Access is denied 00:38:53.319 at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:61) 00:38:53.319 at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:41) 00:38:53.319 at org.gradle.util.DisconnectableInputStream$1.run(DisconnectableInputStream.java:125) 00:38:53.319 at java.lang.Thread.run(Thread.java:748) 00:38:53.319 Caused by: java.io.IOException: Access is denied 00:38:53.319 at java.io.FileInputStream.readBytes(Native Method) 00:38:53.319 at java.io.FileInputStream.read(FileInputStream.java:255) 00:38:53.319 at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) 00:38:53.319 at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) 00:38:53.319 at java.io.BufferedInputStream.read(BufferedInputStream.java:345) 00:38:53.319 at org.gradle.util.DisconnectableInputStream$1.run(DisconnectableInputStream.java:96) 00:38:53.319 ... 1 more 00:38:53.319 ERROR: There was a problem downloading plugin 'shared.firebase.messaging\_com.coronalabs'. Please, try again. 00:38:53.319 00:38:53.319 FAILURE: Build failed with an exception. ... 00:38:53.319 \* What went wrong: 00:38:53.319 Connect to s3.amazonaws.com:443 [s3.amazonaws.com/52.217.43.190] failed: Connection timed out: connect 00:38:53.319 \> Connection timed out: connect
#180

A new iOS server build error has appeared:

iOS build failed (5) after 12 seconds
Feb 10 12:15:03.880 ERROR: Build Failed: A device build error occurred on the server.
                    
                    Error: Template not found.
                    
                    Build Number: 2020.3573
                    Platform: iphone-sim
                    Platform Version: 13.2
                    Template Type: basic
                    BuildID: 5e413b34b065c