Re: TAP tests are badly named

On Thu, Jul 30, 2015 at 07:54:27PM -0400, Andrew Dunstan wrote:
> On 07/30/2015 12:40 PM, Andrew Dunstan wrote:
> >We should describe test sets by what they test, not by how they test. TAP

I agree with that philosophy.  I also respect the practicality of grouping by
test harness as a shorthand.  Tests sharing a harness tend to share
dependencies, harness bugs, logging mechanisms, etc.

> >is a testing tool/protocol. The current set of tests we have test the
> >programs in src/bin, and we should really name the test set by a name that
> >reflects that, rather than the fact that we are using TAP tools to run the
> >tests. What if we decide to test something else using TAP? Would we call
> >that set of tests TAP tests too?

Yes.  "TAP test" refers to any test in a suite written for the "prove"
harness, including the existing suite outside src/bin:

$ find . -name t
./src/bin/pg_controldata/t
./src/bin/scripts/t
./src/bin/pg_rewind/t
./src/bin/pg_basebackup/t
./src/bin/pg_ctl/t
./src/bin/pg_config/t
./src/bin/initdb/t
./src/test/ssl/t

> >--enable-tap-tests is a reasonable configuration setting, because it's
> >about whether or not we have a TAP testing framework available, but I
> >think we should stop calling the bin tests "TAP tests" and we should
> >change the test name in vcregress.pl to a more appropriate name. In the
> >buildfarm I'm calling the step "bin-check":
<http://www.pgbuildfarm.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2015-07-30%2012%3A25%3A58&stg=bin-check>
> >
> >Thoughts?

While lack of granularity in vcregress.pl is hostile, with so much about MSVC
development being hostile, this one is below my noise floor.

> In fact, looking more closely at the changes that have been made to
> vcregress.pl, I don't really like the way this has been done. I'm putting
> together some changes to bring things more into line with how the Makefiles
> work.

The commit history of vcregress.pl shows that doing good here is difficult,
and the rewards are low.  If you wish to write tightly-focused patches to
align vcregress.pl with the experience of testing with GNU make, I am
cautiously optimistic.