Question for Walter: Health Status of Daily Builds ?

Solar2D General Questions/Discussion
#1

Walter,

I have been using public builds for 1 yr. Now I have access to Daily builds. When I get ready to submit my apps I want to ensure that it is built with a Stable version of the SDK.

I assume – public builds are QA tested more then Daily Builds ? correct ?
I assume you do not have the resources to run full regression tests on every daily build
I assume that the stability of daily builds vary based on the amount of new features …

Questions:
How do I determine the health of a particular Daily build ?
Is their a known bug list for each build?
How do you suggest I go about selecting a target build to use for app submission once I am code code/feature complete?

Since each customer has to go through this every time they submit an app have you considered …

  1. Corona Labs adopting a promotion strategy (Daily build -> Stable Build -> Public Build ) each with increasing level of stability/quality defined by Corona Labs.
  2. Commit to a delivering a stable build every ~6-8 weeks ( one or two between public builds)
  3. Publish regression test results for your builds – helps customers to make more informed decisions on what SDK version to use.

I am just looking for some guidance as I now have the option to play with a number of daily builds.

Thanks for you consideration
Ken Cardita
Curved Light Solutions LLC
[import]uid: 112538 topic_id: 34572 reply_id: 334572[/import]

#2

Great Topic, Ken

I like the idea of having a way to gauge the stability of a build.

I try to read the ‘release notes’ for each build, but sometimes the description of the fix do not give you the whole story. For example:

Build 2013:998 -
“Fixes bugs #19950, #14050 - Removes dot files and dot directories from app builds on iOS”

So, where can i look for the original description of Bug# 19950 & 14050? I would like to figure out what a dot file or dot directory is, etc.

(I tried bugs.corona.com and created an account, but could not find logs there either)… looks like that site is only up to date to build 781? [import]uid: 135765 topic_id: 34572 reply_id: 137470[/import]

#3

Ken, R.delia,

Thanks for the questions. These are great questions and very valid. Walter and/or Perry will also chime in in the next couple of days. But here is a quick answer to some of these things:

>
>I assume – public builds are QA tested more then Daily Builds ? correct ?
>

Yes, that is correct. We do indeed do more testing for a Public Build. Not a huge amount more, but there is more.

>
>I assume you do not have the resources to run full regression tests on every daily build
>

All daily builds go through a battery of automated tests, not only for new features, but also for existing features/functionality. This is probably not yet as extensive as it could be, but it is pretty significant.

>
>I assume that the stability of daily builds vary based on the amount of new features …
>

It is exceedingly rare for a daily build that actually gets posted to be “unstable” (e.g., does not build, or apps crash when run). We test for that before posting. And I would say that it is rare for something that has been in the builds more than a couple of weeks to break. For sure, new features may have some bugs when they get put in, but we try to stamp them out as people start using them and things surface in the first week or two.

>
>How do I determine the health of a particular Daily build ?
>Is their a known bug list for each build?
>

Each build’s notes should say what got added and what bugs got fixed.
But today there is no “objective measure” of that build’s health. It’s an interesting question and we’ll discuss internally.

>
>How do you suggest I go about selecting a target build to use for app submission once I am code
>code/feature complete?
>

I would be fine with shipping on a daily build - and I know many of our developers do that all the time. However, if the last public build has all the features you need, then it would make sense to go with that one. If you need to use a more recent build, then I think it just requires some basic testing on your side - which anyone would recommend anyway.

>
>1. Corona Labs adopting a promotion strategy (Daily build -> Stable Build -> Public Build ) each with
>increasing level of stability/quality defined by Corona Labs.
>2. Commit to a delivering a stable build every ~6-8 weeks ( one or two between public builds)
>3. Publish regression test results for your builds – helps customers to make more informed decisions
>on what SDK version to use.
>

We have actually been recently discussing the overall cycle and how it can be improved. As we make any changes, we plan on letting you know. I’ll also bring up the possibility of publishing test results.
However, as mentioned above, if any build does not pass a set of automated tests, it simply does not get published (this is one of the reasons you don’t always see a build each and every day).
In any case, that’s an initial answer. It’s a great set of questions. We’ll chime back in with more info soon.

Thanks,

David
[import]uid: 10668 topic_id: 34572 reply_id: 137561[/import]

#4

David ,

– Thanks for you quick and candid reply.

I really appreciate your transparency regarding testing and letting us peek behind the curtain a bit :slight_smile:
Thanks for addressing my questions… I look forward to hearing from engineering on this as well.

Glad to hear that you have some automated testing in place …
BTW - Does anyone outside of government funded software have 100% regression testing :slight_smile: ?

While I am at it…
— Could you start adding your test results to the daily build notes ?
If you broke them down into high level categories with # passed , # failed for each category ,that would give us a way to "objectively measure " the health of a build .
----A list of failed tests would also help developers know if the build would be good for them.

Having this data would allow customers to see the trends in quality of the SDK over time as well.

I assume you are collecting this data to some degree already … hopefully its a matter of cleaning the data up for external consumption and adding a bit of automated reporting to the release notes.

Once again - the fact that you addressed my questions quickly and candidly instills confidence in my decision to pull the trigger and subscribe recently.

Ken [import]uid: 112538 topic_id: 34572 reply_id: 137579[/import]

#5

Here is my two cents on the Daily Builds.

The builds are automatically generated every night if there has been any check-ins that day. We have a set of automated tests that run and will keep the build from being posted if any of these tests fail. The automated tests verify and test most of the APIs but can’t verify visual or touch APIs. There is no manual testing of the Daily Build before it’s posted. We do use the latest builds when verifying bugs and creating new test cases so we are “eating our own dog food.”

Since we are located in California and we have users all around the world, we heard pretty quickly if there are any major problems with the posted build and will remove it if that’s the case. I have a couple of large projects that I try to run on the Daily Builds to make sure everything is working correctly but this is after the build gets published. In the past we’ve treated “regression bugs” in the Daily Builds with a high priority and try to fix the issues ASAP.

I’ve used Daily Builds for my own projects releases and just watch the forums for any issues users find. I could see having a Daily Build status page that lists any issues found in the builds as a useful feature. This page would be a summary page of issues we found and issues brought to our attention by users.

-Tom [import]uid: 7559 topic_id: 34572 reply_id: 137726[/import]

#6

Tom,

Thanks for describing the daily build , automated “smoke test” , handling of regression issues as well as what gets done when uncaught errors get released. This gives us all a better understanding of what goes into the daily build process at Corona Labs.

Again I appreciate the transparency here… I and hopefully other customers would appreciate test results added to the release notes.

Thanks again

Ken [import]uid: 112538 topic_id: 34572 reply_id: 137825[/import]

#7

Great Topic, Ken

I like the idea of having a way to gauge the stability of a build.

I try to read the ‘release notes’ for each build, but sometimes the description of the fix do not give you the whole story. For example:

Build 2013:998 -
“Fixes bugs #19950, #14050 - Removes dot files and dot directories from app builds on iOS”

So, where can i look for the original description of Bug# 19950 & 14050? I would like to figure out what a dot file or dot directory is, etc.

(I tried bugs.corona.com and created an account, but could not find logs there either)… looks like that site is only up to date to build 781? [import]uid: 135765 topic_id: 34572 reply_id: 137470[/import]

#8

Ken, R.delia,

Thanks for the questions. These are great questions and very valid. Walter and/or Perry will also chime in in the next couple of days. But here is a quick answer to some of these things:

>
>I assume – public builds are QA tested more then Daily Builds ? correct ?
>

Yes, that is correct. We do indeed do more testing for a Public Build. Not a huge amount more, but there is more.

>
>I assume you do not have the resources to run full regression tests on every daily build
>

All daily builds go through a battery of automated tests, not only for new features, but also for existing features/functionality. This is probably not yet as extensive as it could be, but it is pretty significant.

>
>I assume that the stability of daily builds vary based on the amount of new features …
>

It is exceedingly rare for a daily build that actually gets posted to be “unstable” (e.g., does not build, or apps crash when run). We test for that before posting. And I would say that it is rare for something that has been in the builds more than a couple of weeks to break. For sure, new features may have some bugs when they get put in, but we try to stamp them out as people start using them and things surface in the first week or two.

>
>How do I determine the health of a particular Daily build ?
>Is their a known bug list for each build?
>

Each build’s notes should say what got added and what bugs got fixed.
But today there is no “objective measure” of that build’s health. It’s an interesting question and we’ll discuss internally.

>
>How do you suggest I go about selecting a target build to use for app submission once I am code
>code/feature complete?
>

I would be fine with shipping on a daily build - and I know many of our developers do that all the time. However, if the last public build has all the features you need, then it would make sense to go with that one. If you need to use a more recent build, then I think it just requires some basic testing on your side - which anyone would recommend anyway.

>
>1. Corona Labs adopting a promotion strategy (Daily build -> Stable Build -> Public Build ) each with
>increasing level of stability/quality defined by Corona Labs.
>2. Commit to a delivering a stable build every ~6-8 weeks ( one or two between public builds)
>3. Publish regression test results for your builds – helps customers to make more informed decisions
>on what SDK version to use.
>

We have actually been recently discussing the overall cycle and how it can be improved. As we make any changes, we plan on letting you know. I’ll also bring up the possibility of publishing test results.
However, as mentioned above, if any build does not pass a set of automated tests, it simply does not get published (this is one of the reasons you don’t always see a build each and every day).
In any case, that’s an initial answer. It’s a great set of questions. We’ll chime back in with more info soon.

Thanks,

David
[import]uid: 10668 topic_id: 34572 reply_id: 137561[/import]

#9

David ,

– Thanks for you quick and candid reply.

I really appreciate your transparency regarding testing and letting us peek behind the curtain a bit :slight_smile:
Thanks for addressing my questions… I look forward to hearing from engineering on this as well.

Glad to hear that you have some automated testing in place …
BTW - Does anyone outside of government funded software have 100% regression testing :slight_smile: ?

While I am at it…
— Could you start adding your test results to the daily build notes ?
If you broke them down into high level categories with # passed , # failed for each category ,that would give us a way to "objectively measure " the health of a build .
----A list of failed tests would also help developers know if the build would be good for them.

Having this data would allow customers to see the trends in quality of the SDK over time as well.

I assume you are collecting this data to some degree already … hopefully its a matter of cleaning the data up for external consumption and adding a bit of automated reporting to the release notes.

Once again - the fact that you addressed my questions quickly and candidly instills confidence in my decision to pull the trigger and subscribe recently.

Ken [import]uid: 112538 topic_id: 34572 reply_id: 137579[/import]

#10

Here is my two cents on the Daily Builds.

The builds are automatically generated every night if there has been any check-ins that day. We have a set of automated tests that run and will keep the build from being posted if any of these tests fail. The automated tests verify and test most of the APIs but can’t verify visual or touch APIs. There is no manual testing of the Daily Build before it’s posted. We do use the latest builds when verifying bugs and creating new test cases so we are “eating our own dog food.”

Since we are located in California and we have users all around the world, we heard pretty quickly if there are any major problems with the posted build and will remove it if that’s the case. I have a couple of large projects that I try to run on the Daily Builds to make sure everything is working correctly but this is after the build gets published. In the past we’ve treated “regression bugs” in the Daily Builds with a high priority and try to fix the issues ASAP.

I’ve used Daily Builds for my own projects releases and just watch the forums for any issues users find. I could see having a Daily Build status page that lists any issues found in the builds as a useful feature. This page would be a summary page of issues we found and issues brought to our attention by users.

-Tom [import]uid: 7559 topic_id: 34572 reply_id: 137726[/import]

#11

Tom,

Thanks for describing the daily build , automated “smoke test” , handling of regression issues as well as what gets done when uncaught errors get released. This gives us all a better understanding of what goes into the daily build process at Corona Labs.

Again I appreciate the transparency here… I and hopefully other customers would appreciate test results added to the release notes.

Thanks again

Ken [import]uid: 112538 topic_id: 34572 reply_id: 137825[/import]