Re: New regression test time

-----Original Message-----
From: pgsql-hackers-owner@postgresql.org [mailto:pgsql-hackers-owner@postgresql.org] On Behalf Of Josh Berkus
Sent: Saturday, June 29, 2013 3:00 PM
To: Andrew Dunstan
Cc: Alvaro Herrera; pgsql-hackers@postgresql.org; Robins Tharakan
Subject: Re: [HACKERS] New regression test time

On 06/29/2013 02:14 PM, Andrew Dunstan wrote:
> AIUI: They do test feature use and errors that have cropped up in the 
> past that we need to beware of. They don't test every bug we've ever 
> had, nor do they exercise every piece of code.

If we don't have a test for it, then we can break it in the future and not know we've broken it until .0 is released.
Isthat really a direction we're happy going in?
 

> 
> Maybe there is a good case for these last two in a different set of tests.

If we had a different set of tests, that would be a valid argument.  But we don't, so it's not.  And nobody has offered
towrite a feature to split our tests either.
 

I have to say, I'm really surprised at the level of resistance people on this list are showing to the idea of
increasingtest coverage. I thought
 
that Postgres was all about reliability?   For a project as mature as we
are, our test coverage is abysmal, and I think I'm starting to see why.
>>
An ounce of prevention is worth a pound of cure.
The cost of a bug rises exponentially, starting at requirements gathering and ending at the customer.
Where I work, we have two computer  rooms full of machines that run tests around the clock, 24x365.
Even so, a full regression takes well over a week because we perform hundreds of thousands of tests.
All choices of this kind are trade-offs.  But in such situations, my motto is:
"Do whatever will make the customer prosper the most."
IMO-YMMV
<<