Re: Test code is worth the space

On Fri, Aug 14, 2015 at 12:47:49PM +0100, Simon Riggs wrote:
> On 13 August 2015 at 00:31, Robert Haas <robertmhaas@gmail.com> wrote:
> > On Wed, Aug 12, 2015 at 7:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> > > We've talked about having some sort of second rank of tests that
> > > people wouldn't necessarily run before committing, and that would
> > > be allowed to eat more time than the core regression tests would.
> > > I think that might be a valuable direction to pursue if people start
> > > submitting very bulky tests.
> >
> > Maybe.  Adding a whole new test suite is significantly more
> > administratively complex, because the BF client has to get updated to
> > run it.  And if expected outputs in that test suite change very often
> > at all, then committers will have to run it before committing anyway.
> >
> > The value of a core regression suite that takes less time to run has
> > to be weighed against the possibility that a better core regression
> > suite might cause us to find more bugs before committing.  That could
> > easily be worth the price in runtime.
> 
> Seems like a simple fix. We maintain all regression tests in full, but keep
> slow tests in separate files accessed only by a different schedule.
> 
> make check == fast-parallel_schedule
> make check-full == parallel_schedule

+1 for a split, though I would do "make quickcheck" and "make check".  Using
fewer tests should be a conscious decision, and "check" is the widely-known
Makefile target.  In particular, non-hackers building production binaries need
the thorough test battery.  (As a bonus, the buildfarm wouldn't miss a beat.)