Re: Recording test runtimes with the buildfarm

On 6/10/20 6:00 PM, Andres Freund wrote:
> On June 10, 2020 2:13:51 PM PDT, David Rowley <dgrowleyml@gmail.com> wrote:
>>On Thu, 11 Jun 2020 at 02:13, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> I have in the past scraped the latter results and tried to make sense
>>of
>>> them.  They are *mighty* noisy, even when considering just one animal
>>> that I know to be running on a machine with little else to do.
>>
>>Do you recall if you looked at the parallel results or the serially
>>executed ones?
>>
>>I imagine that the parallel ones will have much more noise since we
>>run the tests up to 20 at a time. I imagine probably none, or at most
>>not many of the animals have enough CPU cores not to be context
>>switching a lot during those the parallel runs.  I thought the serial
>>ones would be better but didn't have an idea of they'd be good enough
>>to be useful.
> 
> I'd assume that a rolling average (maybe 10 runs or so) would hide noise enough to see at least some trends even for
parallelruns.
 
> 
> We should be able to prototype this with a few queries over the bf database, right?


This seems to me like a perfect use case for control charts:

  https://en.wikipedia.org/wiki/Control_chart

They are designed specifically to detect systematic changes in an environment
with random noise.

Joe

-- 
Crunchy Data - http://crunchydata.com
PostgreSQL Support for Secure Enterprises
Consulting, Training, & Open Source Development