On Wed, Aug 12, 2015 at 7:20 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > FWIW, I've objected in the past to tests that would significantly > increase the runtime of "make check", unless I thought they were > especially valuable (which enumerating every minor behavior of a > feature patch generally isn't IMO). I still think that that's an > important consideration: every second you add to "make check" is > multiplied many times over when you consider how many developers > run that how many times a day. > > We've talked about having some sort of second rank of tests that > people wouldn't necessarily run before committing, and that would > be allowed to eat more time than the core regression tests would. > I think that might be a valuable direction to pursue if people start > submitting very bulky tests.
Maybe. Adding a whole new test suite is significantly more administratively complex, because the BF client has to get updated to run it. And if expected outputs in that test suite change very often at all, then committers will have to run it before committing anyway.
We could always do that the other way - meaning put everything in "make check", and invent a "make quickcheck" for developers with the smaller subset.
The value of a core regression suite that takes less time to run has to be weighed against the possibility that a better core regression suite might cause us to find more bugs before committing. That could easily be worth the price in runtime.
Or have a quickcheck you run "all the time" and then run the bigger one once before committing perhaps?