Build Issues

Solar2D General Questions/Discussion
#1

Is anyone else still having build issues?

Not only are builds constantly failing throughout the day (they sometimes work when the servers are available, but I mostly get gateway timeouts), it appears sections of the website are extremely slow to load too.

I even tried downloading Corona to a new PC today, and got a gateway timeout…

(https://developer.coronalabs.com/sites/default/files/Corona-2018.3326.msi)

Even now as I write this, elements of this web page are still trying to load from developer.coronalabs.com

I’ve tried connecting from multiple machines on multiple networks, so I’m guessing I’m not the problem… Unless somehow all UK based ISPs are struggling to connect to Corona servers!

If nobody can even download the software, how can we expect more users to work with it?

Surely this is something that needs to be addressed ASAP?

Does anyone have any update on when we might start to see a more reliable service? I love Corona and have used it for years. But as things currently stand I’m starting to worry that it’s only a matter of time before you all vanish over night never to be seen again. When I’m working with Apps that are likely to need updating regularly I simply can’t take that risk on a platform, especially when I’m reliant on builds being generated on a third party server.

Hopefully can comment and hope to alleviate my fears.

#2

Just in case it helps anyone (although it is a simple timeout):

13:15:54.513  Picked up JAVA_TOOL_OPTIONS: -Duser.language=en
13:22:35.087  WebServices: Call system.connect failed in 1580736155 seconds: XmlRpcImplementation::execute: Gateway Time-out
13:22:35.087  Request headers:
13:22:35.087  
13:22:35.087  HTTP/1.1 504 Gateway Time-out
13:22:35.087  Server: nginx/1.17.7
13:22:35.087  Date: Mon, 03 Feb 2020 12:01:47 GMT
13:22:35.087  Content-Type: text/html
13:22:35.087  Content-Length: 167
13:22:35.087  Connection: keep-alive

#3

This has been going on for around 4 weeks-- there was temporary respite for a few days and the problem is now back. Though it’s just speculation and no one from their team has given any concrete information on what the hell is going on, it’s quite possible that Corona are winding down. You can see the discussion here- https://forums.coronalabs.com/topic/76957-use-this-thread-is-the-build-server-down/page-6#entry406764

Most serious developers are porting to Unity or other engines at this point.

#4

I am not porting anywhere. I don’t think Corona is going anywhere, but that is just my opinion.

#5

I wholeheartedly agree with agramonte, but I do fully understand the frustrations going around.

#6

What is the worse that can happen to Corona? The worse that I can think of is that the licensing server gets shutdown. The source code is on github. How hard could it be to remove the license check for native builds? A week at most. A couple of days at best.

But the worse isn’t happening. They are just having build server issues.

I am from a third world country so my perspective might be slightly different. My mom always tells me: “We don’t drown in a cup of water. We adapt!” If you can’t build during the day plan to build at night. Switch to native builds. Once you build once natively the license check doesn’t happen that often. Plan all the plugins you need to use ahead of time. Take the opportunity when the servers are up to run the download plugin script. These are some of the steps I have taken, I don’t have a build problem anymore.

#7

We’re not porting yet, but we are indeed researching other engines and getting bids for porting our games.  It’s very sad.  We love Corona and hope to keep our games on the engine, but as a business we have to start thinking about our survival.

#8

…

Does anyone have any update on when we might start to see a more reliable service? I love Corona and have used it for years. But as things currently stand I’m starting to worry that it’s only a matter of time before you all vanish over night never to be seen again. When I’m working with Apps that are likely to need updating regularly I simply can’t take that risk on a platform, especially when I’m reliant on builds being generated on a third party server.

Hopefully can comment and hope to alleviate my fears.

Any news on this? What are the plans and Roadmap for Corona?

 

#9

I can see your point but the proposals that you offer for working around the issue are not what business who rely on Corona might be willing to do. Building during the night when the server is down during the day is fine if I’m just messing around with some solo projects but it’s too much of a risk for anyone who’s relying on Corona to support their business. 

#10

If somebody is running a “real business”:

  1. Repeatedly posting on these forums about problems that are well documented and received confirmation that they are working on it is not productive for said business.

  2. Depending on CoronaLabs and CoronaLabs alone for the success of said business especially when Coronalabs is mostly free, open source and probably correctly staffed for its revenue is not productive.

  3. Panicking and starting to spend money and effort in migrating to an unknown engine when the current apps don’t need critical changes and are producing ROI is not productive.

  4. Starting an continuous integration solution that builds when available (somebody provided the scripts in these forums) is not a hobby solution but a real mitigation for a vendor problem.

  5. It is a business. Don’t start a business to be the victim of a random company. Adapt or perish. Get to work to mitigate the perceived issue. Don’t drown in a cup of water.

Vlad and Rob already posted all they can or are willing to say about the issue:

  1. Some sort of server problem nobody can figure out. I personally don’t know and don’t care about the details nor have the time to solve it for them.

  2. They’ll publish the Corona 2020 roadmap as soon as they are ready and they are working on it. Roadmap are just best guess estimates in Corona case so I wouldn’t depend the success of your company on anything on that roadmap.

#11

To not depend on CoronaLabs alone, one needs to have alternatives and adopting new technologies comes at a cost in time and money-- it’s not just about critical updates to existing apps so suggesting that migrating to another platform might seem like the only option left for some is not ridiculous. It is what some people on here have suggested they are working on-- it wasn’t a dig at Corona but just a fact. 

It’s also not farfetched that there are small business that may have relied exclusively on Corona (more or less) for a substantial period of time. 

Of course, no one wants to be a victim of a random company but it is clear that most users here still value Corona quite highly for what it offers and in the absence of regular updates during a crisis period, it is quite natural for people to get anxious and frustrated. 

Well documented? What is well documented? A sporadic message every now and then from Rob that there is a server problem justifies what’s going on? Though I didn’t personally post anything to amplify any speculations, enough number of experienced users have suggested that they fear the worst-- if everything’s moving smoothly, surely someone from CLabs would’ve dismissed all the nonsense by now…

#12

I’m not sure how you can say in one breath we shouldn’t depend on Corona, but in the next say it’s not productive to put time and money in moving to an unknown engine. Which one is it?

We are a game company, and we have built all our apps on Corona for the last 8 years. We’re heavily invested into the platform. If the platform is no longer receiving frequent updates, and it’s build servers are spotty, that is a threat to our business.

#13

@kbradford:

My opinion but it is not a radical opinion and pretty much industry standard. You don’t dismantle anything until you have to. So if you have an app that brings revenue you do not migrate it to the new framework until it is absolutely necessary. You hobble along until the cost to make necessary changes is more than migrating to the new framework. Sometimes the app is not making much money. In that case you leave it be until and when you reach the change/cost threshold. You can give your users some time so it won’t be a surprise, push them to migrate to something else or any other options you can come up with.

Depending on how much risk I deem from the facts as presented to me on where Corona is then I would start any new project/s on Corona or some other framework. (I only do casual games for myself; I can reproduce them easily in other frameworks; and I can hobble along for months if not years after something is dead) but for my clients based on their apps and circumstances I would present the facts and let them decide.

@shashwat

I can find multiple instances in these forums and definitely in Slackware from Rob and Vlad themselves. To summarize:

  1. No root cause identify. Just making random changes to try and fix it.

  2. Do a fix and it appears to work only to come down again.

  3. The build is issue is not related to Corona going out of business or Corona shutting down the servers or Corona being swallowed by a sink hole.

From Rob:

coronarob  5:42 PM

I know my responses have not been satisfactory to you, but I feel I’ve shared all that really needs stated. Discussions about our infrastructure, in particular in a public place is somewhat of a security risk. It also adds nothing to the conversation other than satisfying your curiosity or contributing to conspiracy theories. But a longer post-mortem is in order and I will share what I can share.First, I am very limited to what I can see. I have zero access to servers to look at logs, configurations, etc. I don’t even have a good grasp on where things like the forum database or the database for our main website/blog resides. I can only put my years of experience to draw conclusions about the symptoms and explanations. Since they went out with the service usually, I can speculate they share the same server: developer.coronalabs.com.developer.coronalabs.com is our user database, web interface and APIs to that database. The forums talk to this database. The simulator talks to this database. During the build process, authentication/authorization talks to it. Many parts of the process talk to APIs that connect back to developer.coronalabs.com.That server and code has been very stable. It’s not gotten code changes in months. For the most part, it’s the same system that was in place in 2009. When this server starts acting up, it’s usually due to an external force: network connectivity, running out of disc space, DDOS attacks etc. Of course those are the first things we look at when there is a service disruption. If we can’t find the problem there, then it’s time to start digging through log files to find errors. There are many places to have to hunt for problem and the more obscure the problem, the harder it is to find.Sometime on Friday, January 10th, something broke. We still don’t know what it was. We immediatly started looking at the common causes, but in the course of looking for a cause and solution, we changed something in our reverse proxy that broke iOS builds with the 413 error. But the main problem cleared up on it’s own sometime later in the day. This reverse proxy fix unfortunately wasn’t the fix but it coincided with the service coming back, so we thought we fixed it. Problem solved, go home for the weekend. The 413’s started coming in so our sysadmin team had to get back online during the weekend and solve this. The undid the reverse-proxy problem. The service was stable and as such back to finishing out the weekend.Monday, Jan 13th, the problem came back. At this point we had no idea that this outage had a definite pattern. We went offline again. We went back to try and find it. Then it cleared up. Maybe we solved the problem? Maybe it went away on it’s own. Tuesday comes. The service goes out in the morning, comes back in the afternoon. It probably wasn’t until Wednesday or Thursday before the pattern of out in the morning and back in the afternoon became apparent. We obviously started looking at things that would trigger events based on time, but nothing was apparent. Research continued. Friday afternoon comes, the server comes up, and it’s running like a charm for the weekend. At this point, we still don’t have a “business day” pattern established and maybe it’s solved.Once the pattern was established, that helped us focus more on time sensitive actions, but after scouring all the things that run periodically, nothing appeared to be the culprit. On Wednesday we started looking at non-obvious things and something the sysadmins changed broke the 7-12 hour cycle, and the service started responding in the middle of what was previously an outage period. We have now been stable for 2.9 days when we have outages.Clearly something that used to work, stopped. Cleary it was a time sensitive issue. And perhaps less clearly, what we changed seems to have helped.At this point, all I can do is express our sincere apologies for this and in our history, an outage like this has never happened before. I can only hope it doesn’t happen again.

#14

Please use this thread:

https://forums.coronalabs.com/topic/76957-use-this-thread-is-the-build-server-down/

Thank you, 

Rob