Re: Performance testing framework..

On Wed, 7 Oct 2009, Michael Renner wrote:

> I haven't thought about result aggregation & rendering/UI part of the whole 
> thing so far, so if anyone has some ideas in that direction they'd be very 
> much appreciated when the time has come.

What I did in pgbench-tools (now available at http://git.postgresql.org/ ) 
is just dump all the individual run results, tagged by the name of test 
and similar key information, into a database.  That lets people aggregate 
however they want using SQL, and cleanly separates out the data from the 
UI.  My current rendering approach just pulls data from there into 
columnar text files I feed into gnuplot, but you could easily render with 
pretty much anything else instead just by tweaking how you pull things 
from the database.

All of the dbt-X tools (also in that git repo) instead use simple columnar 
text files (and their memory resident equivalent) to hold everything, 
feeding the data through the appropriate Perl libraries to compute stats 
and render things.  Since it drives me crazy to manipulate tables of data 
that could go into a database in a scripting language, I never bought into 
that approach.  But it's completely viable too for a standalone tool.

I'm not aware of anyone working on the job management side of the 
"performance farm" yet, so I don't think you duplicated anybody else's 
work.  Mark Wong (who manages the dbt stuff) and I have both been doing 
work on the benchmark results collection and organization side of things, 
but basically one test at a time.  The specific piece I've been working on 
lately is spawning off system monitoring daemons to collect information 
during the test, I think I'm on my 3rd generation of trying to get a 
solution I'm happy with to that problem.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD