Re: TAP test breakage on MacOS X

On Mon, Oct 27, 2014 at 12:31:56PM -0400, Tom Lane wrote:
> Heikki Linnakangas <hlinnakangas@vmware.com> writes:
> > On 10/27/2014 05:41 PM, Robert Haas wrote:
> >> Beyond all that, I have serious doubts about whether, even if we
> >> eventually get these tests mostly working in most places, whether they
> >> will actually catch any bugs.
> 
> > The existing tests are not very useful, but it certainly would be nice 
> > to have some infrastructure like TAP to write useful tests.
> 
> Yeah.  I think this is an "if you build it they will come" situation.
> The existing tests are really only proof-of-concept; if we can get the
> infrastructure to the point of being widely usable, we can certainly
> use it to write many more-useful tests.

Agreed.  Having this framework when the pg_upgrade test suite originated would
have prevented acquiring parallel implementations in Perl and shell.

> But let's stop talking in generalities and try to quantify where we think
> the bar needs to be set.  Is it okay to depend on non-core CPAN modules
> at all?  How old versions of Perl, Test::More, etc do we need it to work
> with?
> 
> For my own purposes, I'd be satisfied if it ran on RHEL6 and reasonably
> recent OS X.  For RHEL6, I'd not want to have to install any CPAN modules
> other than the versions supplied by Red Hat (which means no Test::More
> subplans).  This platform is only four years old, if you can't cope with
> that then you're definitely too bleeding-edge to be a portable test
> infrastructure.

I want developers to have an inviting framework for writing tests, but I also
want everyone to run the tests.  Setting this bar entails picking a compromise
between those goals.  Consequences of losing subplans include (1) poorer
visual structure in the output; (2) more kinds of test code changes will
require updating the top-level test counts; (3) can't skip tests as a group.
Those losses seem minor enough.  My gut feeling is to target running on perl
5.10 with bundled module versions only and on perl 5.8 with additional
modules.  That is to say, assume 5.10 with respect to bundled modules and
assume 5.8 with respect to core language features.  Concretely, that means
discontinuing use of subplans and replacing IPC::Run with IPC::Cmd.  (Windows
systems probably need to install IPC::Run so IPC::Cmd can use it internally;
that is okay.)  If those restrictions make the test case developer experience
much worse, though, I'll backpedal toward less portability.