Re: Google SoC--Idea Request

On 4/18/06, Jim C. Nasby <jnasby@pervasive.com> wrote:
> On Tue, Apr 18, 2006 at 11:27:40AM -0700, Mark Wong wrote:
> > Jim C. Nasby wrote:
> > >On Sat, Apr 15, 2006 at 03:05:20PM -0400, Jonah H. Harris wrote:
> > >>All ideas welcome!
> > >
> > >I know it's not directly PostgreSQL related, but I'd love to see the
> > >dbt* code improved. Items on my wish-list:
> > >
> > >- make it easy to run the test framework and clients on a seperate
> > >  machine from the database server
> > >- keep results in a database
> > >- provide a front-end to allow users to schedule tests in a queue
> > >- add support for windows, at least for the database (theoretically
> > >  possible to run that way now, but you have to do everything by hand)
> > >
> > >Another idea: afaik, spikesource is still offering a bounty for
> > >improvements to OSS test suites, something that'd fit well with SoC.
> >
> > I second this. :)  There are also the TPC-App (Java) fair-use
> > implementation that I've started and the TPC-E (next gen OLTP) that I
> > would like to start.
>
> Maybe before starting on TPC-E it makes sense to try and get a common
> framework for all the different tests built? AFAIK most of the
> benchmarks all use a fairly standard client-server infrastructure, so we
> should hopefully be able to share that between the different tests...

I agree with Jim.  A framework would really help out here.  All of the
tests are basically the same and would benefit from a framework.

However, Mark, do you think Java is a reliable benchmarking platform?
At EnterpriseDB, we've tried several Java benchmarks and could never
get as repeatable or reliable of a benchmark as DBT2 gives you.

--
Jonah H. Harris, Database Internals Architect
EnterpriseDB Corporation
732.331.1324