> FWIW, I've objected in the past to tests that would significantly
> increase the runtime of "make check", unless I thought they were
> especially valuable (which enumerating every minor behavior of a
> feature patch generally isn't IMO). I still think that that's an
> important consideration: every second you add to "make check" is
> multiplied many times over when you consider how many developers
> run that how many times a day.
Sure. These are developer's tests run 50 times per day just to check that
nothing was just completly broken. It's just a kind of test.
I agree that having ISN conversions tested everytime does not make much
sense, especially as the source file is touched every two years. On the
other hand, when I tried to fix ISN bugs and figure out that there was no
single tests for the module, then it is hard to spot regressions, so the
tests should be there even if they are not run often: Testing the obvious
is useful when you start meddling in the code.
> We've talked about having some sort of second rank of tests that
> people wouldn't necessarily run before committing, and that would
> be allowed to eat more time than the core regression tests would.
> I think that might be a valuable direction to pursue if people start
> submitting very bulky tests.
Yep.
For regression tests, the list of tests run is maintained in the
*_schedule files. There could be a "large_parallel_schedule" which
included more tests. This is already more or less the case, as there are
"big*" targets which run numeric_big in addition to the others.
This could be expanded and taken into account by the build farm.
I recall test submissions which were rejected on the ground of 'it takes 1
second of my time every day' and is 'not very useful as the feature is
known to work'.
I think that a key change needed is that committers are more open to such
additions and the discussion rather focus on (1) are the tests portable
(2) do the test cover features not already tested (3) should they be in
the default list of tests run under "make check". But some should be
accepted in *core* nevertheless, and not push out in the bin.
--
Fabien.