Re: Buildfarm TAP testing is useless as currently implemented

On 07/27/2015 05:06 PM, Tom Lane wrote:
> I challenge anybody to figure out what happened here:
> http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet&dt=2015-07-27%2010%3A25%3A17
> or here:
> http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hamster&dt=2015-07-04%2016%3A00%3A23
> or here:
> http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2015-07-07%2016%3A35%3A06
>
> With no visibility of pg_ctl's output, and no copy of the postmaster log,
> there is no chance of debugging intermittent failures like this one.
> This isn't entirely the buildfarm's fault --- AFAICS, prove-based testing
> has inadequate error reporting by design.  If "not ok" isn't enough
> information for you, tough beans.  (It might help if the farm script
> captured the postmaster log after a failure, but that would do nothing
> for prove's unwillingness to pass through client-side messages.)

Yep.

> I think we should disable TAP testing in the buildfarm until there is
> some credible form of error reporting for it.  I've grown tired of
> looking into buildfarm failure reports only to meet a dead end.
> Aside from the wasted investigation time, which admittedly isn't huge,
> there's an opportunity cost in that subsequent test steps didn't get run.

Commit 1ea06203b - Improve logging of TAP tests - made it a lot better. 
The pg_ctl log should be in the log file now. The buildfarm doesn't seem 
to capture those logs at the moment, but that should be easy to fix.

- Heikki