USE THIS THREAD: Is the Build Server Down?

I keep getting Gateway Timeout when compiling for Android.
It’s day 2 right now.

Is it just my Corona or is it the server?

We had some errors earlier today (starting around 3am ET). I’m not sure what time it resolved itself or if it’s fully resolved yet, but I just did builds for iOS and Android and they seemed normal. 

Rob

I have been getting Build Failed; HTTP error 504 frequently for the past several days.

ERROR: XMLRPC: The operation couldn’t be completed. (HTTP error 504.)
Jan 15 04:25:26.712 ERROR: build failed
                    
                    The operation couldn’t be completed. (HTTP error 504.)

Our server issues are still ongoing. They have been stable for the last 6 hours, but what ever is triggering it, could come back. Please be patient and try again laster. Our team is working to find the root cause of it.

Rob

build for IOS successful. Android failed

FAILURE: Build failed with an exception.

* Where:

Build file ‘/private/var/folders/1_/k18c3y356_l9dypldfd0xfg40000gp/T/CLtmpwTEYQR/template/app/build.gradle.kts’ line: 467

* What went wrong:

Not Found

* Try:

Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 14s

This problem is likely going to continue. We have not found the root cause yet and are making our best effort to solve it. The only advice is to try back periodically.

Rob

Hi, Rob! Any news about Build Server availability?

My advice is the opposite. All stop trying to make builds, and let Corona try to discover/correct the problem. Come back in forum to check when it’s ready. I bet all programmers right now are flooding resquests to make builds (I was one of them) not helping the problem.

http://status.coronalabs.com/

fwiw, I’ve had no problems building for Android last night (1/16) and early this morning (1/17).

Let me try and explain what’s happening.

But first, I’ve not gotten an update today. All services seem to be functional at this point. This is basically what I posted in Slack yesterday. Nothing has changed.

We have a sever known as developer.coronalabs.com. It is our main user database. It’s where authorizations are (for those who remember when we allowed only two authorized devices). It’s where we used to turn on Enterprise licenses. Self-hosted plugins are there and a bunch of other stuff. But more importantly this is where all logins happen. This database also has other bits, like when you sign up for a new account you’re registered and there are tie-ins to the website and it also drives forum logins, the name that shows in the forums and a bunch of other stuff. I suspect (I don’t have access to our full architecture) that the forum database may be on the same server. If this server doesn’t work, pretty much nothing is going to work.

Servers like Marketplace are running on a totally different server. Logins should be tied to developer, but I suspect that because cookies keep you logged in, it doesn’t have to hit developer that often. Plugin licensing is handled through the marketplace (except for self-hosted plugins).

Now, we have not found what’s causing developer to stop responding. We know that some of the server worker processes just stop running. The database is healthy. The web servers are healthy. We have gone down a lot of rabbit holes trying to find the root cause, but nothing so far is showing up. We thought we had it solved on Friday. One of the things we tried to fix created a reverse proxy problem which turned into the 413 error that iOS developers saw over the weekend. We got that solved, but now the developer problems are showing up sometime in the morning (east coast USA time) and magically clearing up sometime in the afternoon. But that is a vague general pattern and I don’t think we can really draw anything from it (because I’ve seen www/forums drop offline during evening hours.)

We are trying to bring in a specialist to give us a fresh set of eyes and see if we can find the root cause.I wish I had more of an updated. We know how frustrating this is. We are putting our best effort forward to solve it.

Now today’s outage seemed to follow the same time frame. I don’t have exact times that it goes down and there is some variability on when it comes back and then there are other sporadic outages. I’ve asked the server admins to look at background jobs that may be starting at a particular time and see if something there is amiss. Sadly I don’t have visibility to go look at those things myself. I’m waiting on my next update from them.

Rob

IOS build failing now
error: There was a problem linking the app. 
https://pastebin.com/BJswwZ6B

Just successfully build my project.

Rob,

Thank you for the rundown.

My experience today (Saturday, Jan 18) is that Pingdom is reporting 100% uptime for Build 2.0, and I was able to build for Android, no problem. Unfortunately, I’m still getting a “Problem linking the app” error when I try to build for iOS.

Fingers crossed the problem gets sorted out soon!

@anna.krepchenko, it looks like you’re trying to build to a folder in the same folder as your project. This is generally not recommended. Please output your build to a folder outside your project and remove the .build folder from your project folder.

It would also be helpful if people having build issues for iOS share your build.settings.

Rob

@Rob i found build is ok, when remove firebase analytics from the build.settings
seems its smth with analytics again. Worked 2 days ago=(

I can confirm that deleting firebase plugin from my projects fixes problem with iOS build (it was working couple days ago).

In a couple of other threads, Scott said he patched the various firebase plugins. You might want to try again.

Rob

Android and iOS build is failing now (it gets hang during authorizing build).