Re: Recording test runtimes with the buildfarm

Greetings,

* Andrew Dunstan (andrew.dunstan@2ndquadrant.com) wrote:
> On 6/10/20 10:13 AM, Tom Lane wrote:
> > Andrew Dunstan <andrew.dunstan@2ndquadrant.com> writes:
> >> Alternatively, people with access to the database could extract the logs
> >> and post-process them using perl or python. That would involve no work
> >> on my part :-) But it would not be automated.
> > Yeah, we could easily extract per-test-script runtimes, since pg_regress
> > started to print those.  But ...
> >
> >> What we do record (in build_status_log) is the time each step took. So
> >> any regression test that suddenly blew out should likewise cause a
> >> blowout in the time the whole "make check" took.
> > I have in the past scraped the latter results and tried to make sense of
> > them.  They are *mighty* noisy, even when considering just one animal
> > that I know to be running on a machine with little else to do.  Maybe
> > averaging across the whole buildfarm could reduce the noise level, but
> > I'm not very hopeful.  Per-test-script times would likely be even
> > noisier (ISTM anyway, maybe I'm wrong).
> >
> > The entire reason we've been discussing a separate performance farm
> > is the expectation that buildfarm timings will be too noisy to be
> > useful to detect any but the most obvious performance effects.
>
> Yes, but will the performance farm be testing regression timings?

We are not currently envisioning that, no.

> Maybe we're going to need several test suites, one of which could be
> regression tests. But the regression tests themselves are not really
> intended for performance testing.

Agree with this- better would be tests which are specifically written to
test performance instead.

Thanks,

Stephen