Re: performance-test farm

Tom Lane wrote:
> There's no such thing as a useful performance test that runs quickly
> enough to be sane to incorporate in our standard regression tests.
>   

To throw a hard number out:  I can get a moderately useful performance 
test on a SELECT-only workload from pgbench in about one minute.  That's 
the bare minimum, stepping up to 5 minutes is really necessary before 
I'd want to draw any real conclusions.

More importantly, a large portion of the time I'd expect regression test 
runs to be happening with debug/assert on.  We've well established this 
trashes pgbench performance.  One of the uglier bits of code added to 
add the "performance farm" feature to the buildfarm code was hacking in 
a whole different set of build options for it.

Anyway, what I was envisioning here was that performance farm systems 
would also execute the standard buildfarm tests, but not the other way 
around.  We don't want performance numbers from some platform that is 
failing the basic tests.  I would just expect that systems running the 
performance tests would cycle through regression testing much less 
often, as they might miss a commit because they were running a longer 
test then.

-- 
Greg Smith   2ndQuadrant US    greg@2ndQuadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us