Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is

Fabien COELHO <coelho@cri.ensmp.fr> writes:
> So this means somehow giving up on coverage, because of one host which can 
> fail under very unusual testing conditions once in a while.

You are attacking a straw man.  There is no reason whatever to believe
that the problem is confined to one host.  We have quite a number of slow
buildfarm hosts: some are using CLOBBER_CACHE_ALWAYS, some are overloaded,
some are just plain slow.  Also, regression test failures outside the
context of our buildfarm are just as bad if not worse.  I still remember
the pain I had with mysql's regression tests, which routinely fell over
in Red Hat's package buildfarm even though they always succeeded on a
personal workstation with nothing else going on.  I do not want our own
packagers to have that kind of experience with Postgres.

> I would prefer to keep the test and have a warning instead, something
> like "ok, although a test which is allowed to rarely fail failed", but at 
> least the feature are tested most of the time.

If the failures are ignored, how much of a test have you really got?
Nobody inquires into nominally-passed buildfarm runs to see what
actually happened.

In the TAP-test context, you do have quite a bit of freedom to decide
what is a pass and what is a fail.  So maybe the path forward is to be
more flexible about the pass conditions for this test.  But I can't
help thinking that these results have exposed some outright bugs too.
        regards, tom lane


-- 
Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-committers