Re: 8.5 release timetable, again

On Wed, Aug 26, 2009 at 7:39 PM, Bruce Momjian<bruce@momjian.us> wrote:
> Peter Eisentraut wrote:
>> Much of the delay and uncertainty during beta in my mind comes from the
>> situation that we wait for negative results and don't trust the release
>> until we have seen and fixed enough of them.  Instead of waiting for
>> concrete, positive results and producing the release with confidence.
>
> Yep, that is our dilemma.

To get positive results in which you can have confidence, you have to
know that the testing which was done actually did a reasonably good
job exercising the code in a way that would have flushed out bugs, had
any been present.  That sounds a lot like the definition of a
regression test suite.  Of course, we have that already, but it's
nowhere near comprehensive.  Maybe we should be looking at an expanded
test suite that runs on a time scale of hours rather than seconds.
Actually, didn't Peter talk about something like this at PGCon?

I don't think that any test suite is going to eliminate the need for
beta-testing.  But if we could say that we had a regression test suite
which covered X% of our code, and it passed on all Y platforms tested,
that would certainly be a confidence booster, especially for large
values of X.  What we're basically doing now is hoping that
beta-testers come up with some novel tests, and that's a bit
hit-or-miss.

Part of the question, of course, is how to build up such a regression
test suite.  One way to do it is try to add a test every time you fix
a bug, such that the new test fails on the unpatched code and passes
with the fix...  or we could have a expanded-regression-test only
commitfest during beta, or something.  I think people would be willing
to put in some work on this.  I certainly would.  I run the regression
tests frequently during development but unless I'm making system
catalog changes they rarely catch anything.  I'm not prepared to write
the whole thing myself, but I would definitely be willing to develop
and contribute some tests.

...Robert