Re: Review: Revise parallel pg_restore's scheduling heuristic

On Mon, Aug 03, 2009 at 10:03:47AM -0500, Kevin Grittner wrote:
> That's about 0.52% slower with the patch.  Because there was over 10%
> variation in the numbers with the patch, I tried leaving out the four
> highest outliers on both, in case it was the result of some other
> activity on the system (even though this machine should have been
> pretty quiet over the weekend) and the difference fell to 0.09%.

What do people do when testing this?  I think I'd look to something like
Student's t-test to check for statistical significance.  My working
would go something like:
 I assume the variance is the same because it's being tested on the same machine.
 samples = 20 stddev  = 144.26 avg1    = 4783.13 avg2    = 4758.46 t       = 0.54  ((avg1 - avg2) / (stddev *
sqrt(2/samples)))

We then have to choose how certain we want to be that they're actually
different, 90% is a reasonably easy level to hit (i.e. one part in ten,
with 95% being more commonly quoted).  For 20 samples we have 19 degrees
of freedom--giving us a cut-off[1] of 1.328.  0.54 is obviously well
below this allowing us to say that there's no "statistical significance"
between the two samples at a 90% level.

--  Sam  http://samason.me.uk/
[1] http://en.wikipedia.org/wiki/Student's_t-distribution#Table_of_selected_values