Обсуждение: Postgres vr.s Oracle

Thought I'd point this blog post out to the list:
http://www.miketec.org/serendipity/index.php?/archives/7-Oracle-and-Postgres-Redux.html

Brian

On Wed, Dec 10, 2008 at 11:08 AM, Brian Hurt <bhurt@janestcapital.com> wrote:

> Thought I'd point this blog post out to the list:
> http://www.miketec.org/serendipity/index.php?/archives/7-Oracle-and-Postgres-Redux.html

I've always heard the performance comparisons violated Oracle
agreements.  Does that not apply in this case?


--
Regards,
Richard Broersma Jr.

Visit the Los Angeles PostgreSQL Users Group (LAPUG)
http://pugs.postgresql.org/lapug

Richard Broersma wrote:
> On Wed, Dec 10, 2008 at 11:08 AM, Brian Hurt <bhurt@janestcapital.com> wrote:
>
>
>> Thought I'd point this blog post out to the list:
>> http://www.miketec.org/serendipity/index.php?/archives/7-Oracle-and-Postgres-Redux.html
>>
>
> I've always heard the performance comparisons violated Oracle
> agreements.  Does that not apply in this case?
>
>
>
I'm not the person to ask.

Brian

> I'm not the person to ask.

I suspect that if the legal department in your company saw that they'd ask
you to take it down, especially after Oracle contacts them.  And I
wouldn't be surprised if Oracle contacted them, especially with the
results you posted.

Thankfully, those of us whose browsers have those materials cached on disk
aren't subject to any Oracle agreements.  :)

--

-----
http://www.globalherald.net/jb01
GlobalHerald.NET, the Smarter Social Network! (tm)

Interesting, but, 8i?

It would have been more interesting if it was a somewhat less
prehistoric version of Oracle...

bhurt@janestcapital.com (Brian Hurt) writes:
> Thought I'd point this blog post out to the list:
> http://www.miketec.org/serendipity/index.php?/archives/7-Oracle-and-Postgres-Redux.html

There's actually something unfair about the comparison, in an "unfair
to Oracle" way, namely that he's using  version 8i, which dates
back to the PostgreSQL 6.5 days ;-).

And vis-a-vis the comments about it possibly being not-permissible to
release numbers, it's quite possible that 8i is old enough that when
the guy bought his licenses, there wasn't such a clause in the
purchase contract.

And that's purest speculation, so it could indeed be not-permissible,
and the guy may look forward to having lawyers-with-samurai-swords
knock on his door ;-).
--
let name="cbbrowne" and tld="linuxfinances.info" in name ^ "@" ^ tld;;
http://linuxfinances.info/info/internet.html
The software isn't finished until the last user is dead.

On Wed, 10 Dec 2008, Chris Browne wrote:

> And vis-a-vis the comments about it possibly being not-permissible to
> release numbers, it's quite possible that 8i is old enough that when
> the guy bought his licenses, there wasn't such a clause in the
> purchase contract.

Doubt it.  8i shipped in March of 1999.  July 30, 1999 was when the
infamous "See The Test That Wasn't" article in PC Magazine talked about
how Oracle wouldn't allow them to publish the benchmarks they'd done.
http://www.infoworld.com/articles/op/xml/00/01/24/000124opfoster.html
talks about it (that's where I remember it from, who reads PC Mag?), the
commentary there says "Because Oracle's license is not a shrinkwrap per se
but an actual signed contract, PC Magazine didn't have a way to challenge
the non-disclosure term".

While I was poking around looking that up again, I actually found a couple
of places what look like legit Oracle contracts are posted at; glancing at
these two:

http://contracts.onecle.com/bearingpoint/oracle.collab.1995.11.22.shtml
http://www.techagreements.com/agreement-preview.aspx?num=23093

Suggests that it's very unlikely the publication of results was within the
realm of any contract with them.

Mike's earlier blog posting has some choice comments in it too:
http://www.miketec.org/serendipity/index.php?/archives/5-Oracle-prices-themselves-out.html
"I have one app running on Oracle and three that run on a Postgres
database. They run on similar hardware, yet the Oracle server is the one
that gives me the performance problems. Postgres gets hammered all day
long by the user's poorly written reports and handles it just fine."

I get the feeling he's not real concerned about if Oracle gets mad at him.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

On Wed, Dec 10, 2008 at 2:08 PM, Brian Hurt <bhurt@janestcapital.com> wrote:
> Thought I'd point this blog post out to the list:
> http://www.miketec.org/serendipity/index.php?/archives/7-Oracle-and-Postgres-Redux.html

Hmm... I wonder how scientific his benchmarking is and how well his
Oracle system was tuned.  Because I've done quite a few performance
comparisons between Postgres 8.3 and 8.4-dev against a well-tuned
Oracle8i instance on Linux, and 8i (from 1999) beats the latest
versions of Postgres quite handily on the same hardware.  And, while I
have received permission from Oracle to publish the result, I haven't
had the time to write up the blog entry yet.

Outside of simple curiosity, my reason for running the benchmark was
simply to show that in terms of performance, Oracle had it right over
10 years ago and that our continual discussions about leaving things
to the OS and file system developers (because they know how to manage
memory/data better than we do) is pointless.  It illustrates that if
Postgres ever wants to step into this century and take advantage of
newer hardware configurations, we need to accept the fact that PG's
inherent design has serious performance-related flaws which need to be
addressed sooner rather than later.  Similarly, I ran the same tests
against Oracle 10g and 11g, and a properly tuned Oracle system is
10-100x faster than Postgres on lots of operations in both OLTP and
DSS workloads, but because I didn't expect Postgres to be close to
Oracle these days, I went back to comparing against 8i (Standard
Edition) just to make my point.

Regardless of what some people on this list tell you, one of the main
reasons Oracle and other vendors don't like people performing external
benchmarks is because the majority of people screw them up.  Proper
benchmarking is something that takes time to do and you have to have a
good amount of experience ensuring the SUT environment is exactly the
same for both databases.  Similarly, the majority of people don't know
how to tune an Oracle system properly, which is why they get bad
results.  Whether that's the case in this test or not, is unknown.
There certainly isn't enough information in that blog entry to
determine if the system was tuned properly or whether the tests were
even performed correctly, yet it was forwarded on as gospel.
Similarly, the benchmark was performed by someone a bit jaded against
Oracle, which doesn't bode well for presuming he was unbiased.

I can't believe the number of people on these lists who continually
forward piss-poor benchmarks just because they think it somehow
solidifies their belief that PG is somehow equal to or better than
commercial databases in terms of overall performance.  Sure, there are
cases where PG is faster than Oracle... but they few and far between.
If Postgres works well for you, use it.  If not, use something else...
just please don't push half-baked benchmarks performed by jaded people
who are so uninformed that they think Don Burleson is the
be-all-end-all of Oracle tuning consultants.

-Jonah

On Sat, 2008-12-13 at 13:06 -0500, Jonah H. Harris wrote:
> On Wed, Dec 10, 2008 at 2:08 PM, Brian Hurt <bhurt@janestcapital.com> wrote:
> > Thought I'd point this blog post out to the list:
> > http://www.miketec.org/serendipity/index.php?/archives/7-Oracle-and-Postgres-Redux.html
>
> Hmm... I wonder how scientific his benchmarking is and how well his
> Oracle system was tuned.  Because I've done quite a few performance
> comparisons between Postgres 8.3 and 8.4-dev against a well-tuned
> Oracle8i instance on Linux, and 8i (from 1999) beats the latest
> versions of Postgres quite handily on the same hardware.  And, while I
> have received permission from Oracle to publish the result, I haven't
> had the time to write up the blog entry yet.

<major snippage>

> who are so uninformed that they think Don Burleson is the
> be-all-end-all of Oracle tuning consultants.

Where is your patch?

Joshua D. Drake


>
> -Jonah
>
--
PostgreSQL
   Consulting, Development, Support, Training
   503-667-4564 - http://www.commandprompt.com/
   The PostgreSQL Company, serving since 1997

On Sat, Dec 13, 2008 at 1:09 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
> Where is your patch?

What's the point?  I've had the same arguments and discussion for
years and (per you and almost everyone else) anything significant has
to be discussed on-list prior to acceptance.  It's almost pointless to
bring this stuff up anymore.

-Jonah

On Sat, 2008-12-13 at 14:09 -0500, Jonah H. Harris wrote:
> On Sat, Dec 13, 2008 at 1:09 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
> > Where is your patch?
>
> What's the point?  I've had the same arguments and discussion for
> years and (per you and almost everyone else) anything significant has
> to be discussed on-list prior to acceptance.  It's almost pointless to
> bring this stuff up anymore.

My point is your signal to noise ratio is off. You very well could be
correct (in fact I think you probably are) but it is irrelevant because
all you do is hand wave. I am sure there are plenty of people in this
community that would chew up and digest a valid benchmark from a
Oracle/PostgreSQL expert but since we never see them, it is really hard
to justify the work it would take to make significant architectural
changes.

Sincerely,

Joshua D. Drake

--
PostgreSQL
   Consulting, Development, Support, Training
   503-667-4564 - http://www.commandprompt.com/
   The PostgreSQL Company, serving since 1997

On Saturday 13 December 2008 13:06:33 Jonah H. Harris wrote:
> On Wed, Dec 10, 2008 at 2:08 PM, Brian Hurt <bhurt@janestcapital.com> wrote:
> > Thought I'd point this blog post out to the list:
> > http://www.miketec.org/serendipity/index.php?/archives/7-Oracle-and-Postg
> >res-Redux.html
>
> Hmm... I wonder how scientific his benchmarking is and how well his
> Oracle system was tuned.  Because I've done quite a few performance
> comparisons between Postgres 8.3 and 8.4-dev against a well-tuned
> Oracle8i instance on Linux, and 8i (from 1999) beats the latest
> versions of Postgres quite handily on the same hardware.  And, while I
> have received permission from Oracle to publish the result, I haven't
> had the time to write up the blog entry yet.
>

Curious, but does your "well tuned" system include query hints? I've certainly
seen cases where Oracles hinting system allowed them to get to places that
Postgres couldn't get to, but without hinting things tend to be much closer
(not necessarily even; it still has other advantages like advanced sql
syntax)  In my mind this is an argument for adding query hinting to postgres,
but I hate to even mention that idea :-)

> Outside of simple curiosity, my reason for running the benchmark was
> simply to show that in terms of performance, Oracle had it right over
> 10 years ago and that our continual discussions about leaving things
> to the OS and file system developers (because they know how to manage
> memory/data better than we do) is pointless.

I think you have misjudged the argument, which would be (IMHO) more accurately
stated that given this projects resources, we will get better performance by
leveraging os/filesystem improvements rather than circumventing them. One
such example is a patch for doing auto-alignment of columns at the disk layer
for optimal performance. This patch is relatively simple (for this
discussion), yet the idea has been around for years; if we can't knock that
out, do you really think we have the resources to move towards direct
hardware interaction?

<snip>
> Regardless of what some people on this list tell you, one of the main
> reasons Oracle and other vendors don't like people performing external
> benchmarks is because the majority of people screw them up.  Proper
> benchmarking is something that takes time to do and you have to have a
> good amount of experience ensuring the SUT environment is exactly the
> same for both databases.  Similarly, the majority of people don't know
> how to tune an Oracle system properly, which is why they get bad
> results.  Whether that's the case in this test or not, is unknown.

Well, there is something to be said for having a database system so complex
that properly tuning it can't be done even by experienced users. Postgres is
an order of magnitude easier to configure (imho) and we still get beat up for
it.

--
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

On Sat, Dec 13, 2008 at 2:28 PM, Joshua D. Drake <jd@commandprompt.com> wrote:
> My point is your signal to noise ratio is off. You very well could be
> correct (in fact I think you probably are) but it is irrelevant because
> all you do is hand wave.

I agree with that, which is why I did it and asked Oracle for
permission to publish it.  My goal was to point out that the benchmark
was far from being authoritative, repeatable, or well-understood and
that such things are almost baseless as a valid comparison.

> I am sure there are plenty of people in this
> community that would chew up and digest a valid benchmark from a
> Oracle/PostgreSQL expert but since we never see them, it is really hard
> to justify the work it would take to make significant architectural
> changes.

I likely did get carried away.  I hope to find time to write up my
findings along with everything needed to repeat the tests soon.

-Jonah

On Sat, Dec 13, 2008 at 3:38 PM, Robert Treat
<xzilla@users.sourceforge.net> wrote:
> Curious, but does your "well tuned" system include query hints?

No.

> I've certainly seen cases where Oracles hinting system allowed them to get to places that
> Postgres couldn't get to, but without hinting things tend to be much closer
> (not necessarily even; it still has other advantages like advanced sql
> syntax)  In my mind this is an argument for adding query hinting to postgres,
> but I hate to even mention that idea :-)

I've always agreed with adding hints... but as we all know, that's a
different topic altogether.

> I think you have misjudged the argument, which would be (IMHO) more accurately
> stated that given this projects resources, we will get better performance by
> leveraging os/filesystem improvements rather than circumventing them. One
> such example is a patch for doing auto-alignment of columns at the disk layer
> for optimal performance. This patch is relatively simple (for this
> discussion), yet the idea has been around for years; if we can't knock that
> out, do you really think we have the resources to move towards direct
> hardware interaction?

I understand the resource limitation.  That's not the problem.  The
problem, from my point of view, is that there's a near-constant
rejection against the possibility of using fairly well-known and
greatly accepted implementations.  Examples are the "Features We Do
Not Want" in the TODO, and the "Why don't you use threads, raw
devices, async-I/O, <insert your favorite wizz-bang feature here>?"
section of the FAQ.  For examples, we use threads and async I/O.
Threading has been supported by every major OS for how long now?  How
buggy is it really?  A heck of a lot of stuff is written using threads
and it runs continuously and under heavy load without a single
problem.  Asynchronous I/O... Oracle8i supported it in 1999 (and in
earlier versions if you knew how to enable it).  My point is simply
that this isn't new or untested technology, and that we should at
least be open to it.  But when our own FAQ calls threading buggy,
crash-prone, overly complex, and not worth it from a performance
standpoint, it makes a lot of people question the amount of Postgres
community experience related to performance engineering.  Similarly,
it makes it difficult for those of us with experience using these
technologies to even have discussions about them on-list.

> Well, there is something to be said for having a database system so complex
> that properly tuning it can't be done even by experienced users. Postgres is
> an order of magnitude easier to configure (imho) and we still get beat up for
> it.

Well, people will always find something to complain about with
Postgres, Oracle, and every other thing :)

Oracle has tons of options because no workload can be characterized by
a small number of knobs.  If you know how something needs to work,
nine times out of ten Oracle has a way for you to alter the system to
make it work that way.  Yes, that does make it more complex, and I'm
not denying that at all.  My point was simply that individuals without
extensive experience tuning Oracle and performing scientific
benchmarks aren't generally the best people to make valid overall
comparisons.  My main concern is based on talking to people at PG
conferences who put far too much faith and belief into results derived
from bad benchmarks.

Again, I do not want this thread to be an argument of Oracle vs.
Postgres.  I just wanted to say that we should look at benchmarks
which have some resemblance of scientific validity before simply
passing them out or believing in them ourselves.  This is open source
and believing in our own benchmarketing is just going to hurt us
long-term.

-Jonah

"Jonah H. Harris" <jonah.harris@gmail.com> writes:

> Threading has been supported by every major OS for how long now?  How
> buggy is it really?  A heck of a lot of stuff is written using threads
> and it runs continuously and under heavy load without a single
> problem.  Asynchronous I/O... Oracle8i supported it in 1999 (and in
> earlier versions if you knew how to enable it).

And it was buggy as hell. In any case 1999 is pretty recent, we support plenty
of platforms which predate 1999.

Threading is a programming model. If it's convenient to program using it then
you use it. If it isn't there are several other ways to multi-thread your
program without using OS threads.

On modern operating systems (ie, excluding Windows) there's no functional
difference between multiple processes with shared memory and threads. All the
major operating systems use a single-level scheduler where processes and
threads are treated identically. The only difference between them is the
programmer convenience of having all memory shared by default instead of
having all memory private by default.

You pick your programming tools based on what's convenient for the program
you're writing. You don't structure your whole program around wanting to use
your favourite API whether that's threads or async i/o or whatever. There are
plenty of other ways to get the same functionality.

> My point is simply that this isn't new or untested technology, and that we
> should at least be open to it. But when our own FAQ calls threading buggy,
> crash-prone, overly complex, and not worth it from a performance standpoint,
> it makes a lot of people question the amount of Postgres community
> experience related to performance engineering.

There are always people out there with all kinds of wacky ideas.

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's PostGIS support!

On Saturday 13 December 2008 23:34:33 Jonah H. Harris wrote:
> Again, I do not want this thread to be an argument of Oracle vs.
> Postgres.  I just wanted to say that we should look at benchmarks
> which have some resemblance of scientific validity before simply
> passing them out or believing in them ourselves.  This is open source
> and believing in our own benchmarketing is just going to hurt us
> long-term.
>

Fair points, but one also needs to consider the audience of this list. The
information wasn't posted on -performance or -hackers as some scientific
validation, it was posted on -advocacy, where even a bad benchmark carries
wieght because it is interesting for those of use who do advocacy related
work to know what is being discussed even if the exact discussion is not
exactly accurate. That said, it is very good that we have people willing to
point out those inaccuracies; hopefully that keeps everyone on thier toes
when talking with others.

--
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

Jonah,

> Hmm... I wonder how scientific his benchmarking is and how well his
> Oracle system was tuned.  Because I've done quite a few performance
> comparisons between Postgres 8.3 and 8.4-dev against a well-tuned
> Oracle8i instance on Linux, and 8i (from 1999) beats the latest
> versions of Postgres quite handily on the same hardware.

What tests are you running?  There are certainly things,
performance-wise, which Oracle does better than us.  There are also
things they do worse.

I'll point out that, the last time we had public comparables
head-to-head, with *Oracle* doing Oracle's tuning on SpecJAppserver,
PostgreSQL was around 90% of Oracle *10* on analogous hardware.  In
internal tests at Sun I can't quote directly, that comparison held on
succeeding generations of hardware I wasn't allowed to publish :-(

And I'd match the Sun performance labs' knowledge of Oracle tuning
against yours any day.

Not, on TPC-E on the other hand, Oracle is way ahead of us ... we were
like 50% last I checked, due mostly to the amount of time it takes us to
resolve lock conflicts vs. Oracle.  Of course, Microsoft is beating
Oracle by 50%, so Oracle is not the one to match on that benchmark.

--Josh

Josh Berkus wrote:
> Not, on TPC-E on the other hand, Oracle is way ahead of us ... we were
> like 50% last I checked, due mostly to the amount of time it takes us
> to resolve lock conflicts vs. Oracle.  Of course, Microsoft is beating
> Oracle by 50%, so Oracle is not the one to match on that benchmark.
>
> --Josh

Here is the current  TPC-E top 10
http://www.tpc.org/tpce/results/tpce_perf_results.asp
where is oracle???

TCP-H
http://www.tpc.org/tpch/results/tpch_perf_results.asp


TCP-C
 http://www.tpc.org/tpcc/results/tpcc_perf_results.asp

On Sun, Dec 14, 2008 at 8:38 PM, justin <justin@emproshunts.com> wrote:
> Here is the current  TPC-E [H, C] top 10
> where is oracle???

Where you should be looking is at the price/performance benchmarks,
because that's where Postgres plays.  Last time I checked Postgres on
a TPC-C, albeit being 100% free, was anywhere from $4.00 to $6.00 per
transaction depending on the hardware.  Compare that to Oracle's $0.68
or SQL Server's $0.84.  Yeah, I expect the normal it's just an
industry benchmark, it's not fair, it's not representative of real
workloads or real performance response.

Or, just for the fun of it, run Postgres on the 100GB TPC-H and let me
know what you get for price/performance... then compare that to SQL
Server's result from 2006.

I do want to caution everyone though.  The OSDL-DBT kits are *not*
spec-compliant and have several flaws which make the results fairly
untrustworthy for comparison purposes.  The best TPC-C kit for
Postgres I've seen is EnterpriseDB's version of the DBT-2.  While it's
still not spec-compliant, it fixes several major bugs and includes a
more optimized schema.  If you want a copy, you could petition them to
release their modifications to it.

--
Jonah H. Harris, Senior DBA
myYearbook.com

On Sun, Dec 14, 2008 at 2:33 PM, Josh Berkus <josh@agliodbs.com> wrote:
> What tests are you running?  There are certainly things, performance-wise,
> which Oracle does better than us.  There are also things they do worse.

Various benchmarks.  So far I've fully performed DBT-2 and DBT-3.
I've also been toying with a couple of the OSS benchmarks, but I'm not
sure which I'd like to use.

> I'll point out that, the last time we had public comparables head-to-head,
> with *Oracle* doing Oracle's tuning on SpecJAppserver, PostgreSQL was around
> 90% of Oracle *10* on analogous hardware.  In internal tests at Sun I can't
> quote directly, that comparison held on succeeding generations of hardware I
> wasn't allowed to publish :-(

Sucks that you weren't able to publish them before leaving :(

> And I'd match the Sun performance labs' knowledge of Oracle tuning against
> yours any day.

Perhaps.  I don't know anyone on Sun's performance team, so I can't
speak to that directly.  Though, I do seem to recall going over the
Postgres configuration used on those benchmarks, and that it wasn't as
optimized as it could've been.

> Not, on TPC-E on the other hand, Oracle is way ahead of us ... we were like
> 50% last I checked, due mostly to the amount of time it takes us to resolve
> lock conflicts vs. Oracle.  Of course, Microsoft is beating Oracle by 50%,
> so Oracle is not the one to match on that benchmark.

Yeah, Microsoft is doing quite well at TPC-E.

--
Jonah H. Harris, Senior DBA
myYearbook.com

Josh Berkus <josh@agliodbs.com> writes:

> Not, on TPC-E on the other hand, Oracle is way ahead of us ... we were like 50%
> last I checked, due mostly to the amount of time it takes us to resolve lock
> conflicts vs. Oracle.  Of course, Microsoft is beating Oracle by 50%, so Oracle
> is not the one to match on that benchmark.

We've done essentially no benchmark optimization so on any sufficiently
complex benchmark there will undoubtedly be some piece that we basically are
not handling correctly at all. The degree to which the problem, which is
essentially a bug, causes a slowdown isn't really relevant. Whether it's 50%
or 100% or more isn't really interesting.

On the flip side Microsoft and Oracle spend a lot of effort into optimizing
for the cases the benchmarks cover. Oracle beat Microsoft over the head with
updateable views for a long time because it basically broke some of the older
benchmarks and made them thousands of times faster according to those
benchmarks. It doesn't surprise me that Microsoft is doing it to Oracle now.

I'm amazed at how many people in this thread claim to have diagnosed
performance problems that we've never heard about on-list though. What do you
mean "due mostly to the amount of time it takes us to resolve lock
conflicts"?!?

Are you sure it isn't due to the cycles we spend buffering RI triggers and
checking them one-by-one? 8.4 has some microoptimizations in this area but I'm
thinking there are some algorithmic improvements we could make for batch
updates.

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's RemoteDBA services!

Jonah H. Harris wrote:
> On Sun, Dec 14, 2008 at 8:38 PM, justin <justin@emproshunts.com> wrote:
>
>> Here is the current  TPC-E [H, C] top 10
>> where is oracle???
>>
>
> Where you should be looking is at the price/performance benchmarks,
> because that's where Postgres plays.  Last time I checked Postgres on
> a TPC-C, albeit being 100% free, was anywhere from $4.00 to $6.00 per
> transaction depending on the hardware.  Compare that to Oracle's $0.68
> or SQL Server's $0.84.
Where are you getting the $4 and $6 per transaction for PostgreSql.  i
just search through this list
http://www.tpc.org/tpcc/results/tpcc_results.asp?orderby=dbms
it does not contain PostgreSql entry.
all of the less than $1.00 per transaction  are last the few years.

Oracle only just in the last year dropped to below $1.00 it mixed bag
from $3 to $52 (back on 2001)
> Yeah, I expect the normal it's just an
> industry benchmark, it's not fair, it's not representative of real
> workloads or real performance response.
>
First Step in testing and comparing is agree on a Standard that everyone
can agree to.  Second step test the system without cheating which
numerous software including Oracle, MS, and IBM have.
> Or, just for the fun of it, run Postgres on the 100GB TPC-H and let me
> know what you get for price/performance... then compare that to SQL
> Server's result from 2006.
>
>

On Sun, Dec 14, 2008 at 10:17 PM, justin <justin@emproshunts.com> wrote:
> Where are you getting the $4 and $6 per transaction for PostgreSql.  i just
> search through this list

You won't see it there because none of the PG vendors see a reason to
spend the time and money officially benchmarking Postgres only for it
to place last.  Similar to Josh Berkus, I know where it ranks because
I (and several other EnterpriseDB developers) ran the tests to try and
get Postgres a TPC-C.

> Oracle only just in the last year dropped to below $1.00 it mixed bag from
> $3 to $52 (back on 2001)

Oracle hadn't run price/performance in awhile prior to that due to the
cost of the software.  Still, Microsoft has had it below $1 since
2005.

> First Step in testing and comparing is agree on a Standard that everyone can
> agree to.  Second step test the system without cheating which numerous
> software including Oracle, MS, and IBM have.

Cheating?  It's an industry standard benchmark.  And, for the record,
when I compared PG to Oracle on TPC-H, I didn't use Oracle's
additional features, I did a one-to-one comparison using the exact
same schema and indexes.

-Jonah

Jonah H. Harris wrote:
> Outside of simple curiosity, my reason for running the benchmark was
> simply to show that in terms of performance, Oracle had it right over
> 10 years ago and that our continual discussions about leaving things
> to the OS and file system developers (because they know how to manage
> memory/data better than we do) is pointless.  It illustrates that if
> Postgres ever wants to step into this century and take advantage of
> newer hardware configurations, we need to accept the fact that PG's
> inherent design has serious performance-related flaws which need to be
> addressed sooner rather than later.  Similarly, I ran the same tests
> against Oracle 10g and 11g, and a properly tuned Oracle system is
> 10-100x faster than Postgres on lots of operations in both OLTP and
> DSS workloads, but because I didn't expect Postgres to be close to
> Oracle these days, I went back to comparing against 8i (Standard
> Edition) just to make my point.

10-100x?

I am confused because sometimes I hear that Postgres has bad performance
from ex-Oracle users, but in general I hear that Oracle and Postgres
have similar performance behavior from people porting applications.

--
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

On Sun, Dec 14, 2008 at 10:17 PM, justin <justin@emproshunts.com> wrote: 

Where are you getting the $4 and $6 per transaction for PostgreSql.  i just
search through this list   

You won't see it there because none of the PG vendors see a reason to
spend the time and money officially benchmarking Postgres only for it
to place last.  Similar to Josh Berkus, I know where it ranks because
I (and several other EnterpriseDB developers) ran the tests to try and
get Postgres a TPC-C. 

Oracle only just in the last year dropped to below $1.00 it mixed bag from
$3 to $52 (back on 2001)   

Oracle hadn't run price/performance in awhile prior to that due to the
cost of the software.  Still, Microsoft has had it below $1 since
2005.
 

First Step in testing and comparing is agree on a Standard that everyone can
agree to.  Second step test the system without cheating which numerous
software including Oracle, MS, and IBM have.   

Cheating?  It's an industry standard benchmark.  And, for the record,
when I compared PG to Oracle on TPC-H, I didn't use Oracle's
additional features, I did a one-to-one comparison using the exact
same schema and indexes.

-Jonah 

On Sun, Dec 14, 2008 at 11:05 PM, justin <justin@emproshunts.com> wrote:
> I would like to see them even if PostgreSql comes dead last.

Well, running an official TPC benchmark is an interesting process.
First, you have to have a spec-compliant benchmark kit, which we
don't.  Second, you have to work with a hardware vendor, and none of
the vendors are going to work with you if you can't make their
hardware look good; they don't want to come in last even if it's due
to your software.  I know, I talked to three of the major hardware
vendors.  Lastly, it costs a fair amount of money and time to perform
an audited benchmark, which I don't think the community wants to pay
for.

> Does that not prove the point Oracle is over price software compared to the
> competition.

That depends on whether you need the performance or not.

> The only way to verify the results is duplicate the test and given these
> servers cost from $100K to $10million  its unlikely these test are verified
> by a 3rd party running independent hardware.  From where i come from for
> result to be proven  someone else must duplicate the results independently
> with only instructions  given by the original tester.

Perhaps you should download a full disclosure report of a TPC
benchmark.  First of all, they're all audited.  Second of all, every
configuration detail is provided for you to be able to run the test
yourself.  In fact, if you have a good enough reason to do so (i.e.
looking to buy a good amount of software), both Oracle and Microsoft
will give you copies of their benchmark kits for evaluation purposes.

--
Jonah H. Harris, Senior DBA
myYearbook.com

On Sun, Dec 14, 2008 at 10:47 PM, Bruce Momjian <bruce@momjian.us> wrote:
> 10-100x?
>
> I am confused because sometimes I hear that Postgres has bad performance
> from ex-Oracle users, but in general I hear that Oracle and Postgres
> have similar performance behavior from people porting applications.

It depends on the application, whether they had tuned Oracle or not,
and whether the performance of their application was really noticeable
to begin with.  Many times at EDB, we ran into people who had stock,
untuned Oracle installs.  Personally, I don't think that comparing a
tuned Postgres system to an untuned Oracle system is a very fair
comparison.  Though, there were a couple customer applications that
did run faster on Postgres (due to ip4r or Postgres' native
integer/floating point data types not present in the customers version
of Oracle).

The OLTP cases where Oracle starts to outperform Postgres is usually
around 25 heavy concurrent sessions.  When you start scaling into
hundreds of sessions, Oracle really starts to shine.  If, however,
you're running 10 or so concurrent users, Postgres will generally be
faster simply because Oracle isn't optimized for small systems.  This
statement obviously varies based on what those 10 people are actually
doing, because you could likely optimize lots of things using specific
Oracle features such as materialized views, more access paths,
parallelism, etc.

Of course, if an application works equally well on both Oracle and
Postgres, I'd say go with Postgres.  There's no need to pay for Oracle
features you're not going to use.

Arrg!  I feel like I'm bashing on Postgres when I'm simply trying to
defend Oracle from a poorly written benchmark blog.  The fact is, both
systems are good and one should choose the tool that best fits their
need, be that Oracle, Postgres, SQL Server, or (yes, even) MySQL.  I
know this is the advocacy list, and my only hope was that we have a
modicum of science behind a benchmark before putting too much into it.

-Jonah

Jonah H. Harris wrote: <blockquote cite="mid:36e682920812142016x21d31a57q41894aa6b20a0108@mail.gmail.com"
type="cite"><prewrap="">On Sun, Dec 14, 2008 at 11:05 PM, justin <a class="moz-txt-link-rfc2396E"
href="mailto:justin@emproshunts.com"><justin@emproshunts.com></a>wrote: </pre><blockquote type="cite"><pre
wrap="">Iwould like to see them even if PostgreSql comes dead last.   </pre></blockquote><pre wrap="">
 
Well, running an official TPC benchmark is an interesting process.
First, you have to have a spec-compliant benchmark kit, which we
don't.  Second, you have to work with a hardware vendor, and none of
the vendors are going to work with you if you can't make their
hardware look good; they don't want to come in last even if it's due
to your software.  I know, I talked to three of the major hardware
vendors.  Lastly, it costs a fair amount of money and time to perform
an audited benchmark, which I don't think the community wants to pay
for.
 </pre></blockquote><br /> Thats a problem with testing.  Its not cheap or easy.  I do this all the time with
electricalequipment.  The company I work for has 20K in a environmental testing chamber, 55K in power supplies  60K in
DMMs which must be certified to NIST costing another 10K every year.  This is only for the lab and QC.<br /><br /> So
yesi know testing is very expensive but it can be done and should be done independently of the Hardware/Software
manufactures. If not how are we to know that this hardware/Software is really a production item.  Example of
independentlab is Consumer Reports, created to test products without the manufactures influence over the net result as
themanufactures were hoisting bogus testing reports on the public.  <br /><br /> The more i learn how the tests are run
themore suspicious i become its totally black bagged.  The test method must be easy to understand be realistic to a
realmodel that anyone can run with X time and X money on hand.  Since you have stated that is near impossible to do, i
wouldsay the testing methods need to be rethought. <br /><br /> Any one can test our product if they went out spend 40K
onelectrical lab equipment and has a basic understand of electrical principles.  There is no need to have an
understandingof ISO 17025 <a class="moz-txt-link-freetext"
href="http://en.wikipedia.org/wiki/ISO_17025">http://en.wikipedia.org/wiki/ISO_17025</a><br/><br /><br /><blockquote
cite="mid:36e682920812142016x21d31a57q41894aa6b20a0108@mail.gmail.com"type="cite"><pre wrap=""></pre><blockquote
type="cite"><prewrap="">Does that not prove the point Oracle is over price software compared to the
 
competition.   </pre></blockquote><pre wrap="">
That depends on whether you need the performance or not. </pre></blockquote> That don't make sense???? Its all about
costif the other software is cheaper per transaction means you can spend the same amount of money as one would on
Oracleon lets say MSSQL then blow by Oracle's performance<br /><blockquote
cite="mid:36e682920812142016x21d31a57q41894aa6b20a0108@mail.gmail.com"type="cite"><pre wrap=""> </pre><blockquote
type="cite"><prewrap="">The only way to verify the results is duplicate the test and given these
 
servers cost from $100K to $10million  its unlikely these test are verified
by a 3rd party running independent hardware.  From where i come from for
result to be proven  someone else must duplicate the results independently
with only instructions  given by the original tester.   </pre></blockquote><pre wrap="">
Perhaps you should download a full disclosure report of a TPC
benchmark.  First of all, they're all audited.  Second of all, every
configuration detail is provided for you to be able to run the test
yourself.  In fact, if you have a good enough reason to do so (i.e.
looking to buy a good amount of software), both Oracle and Microsoft
will give you copies of their benchmark kits for evaluation purposes</pre></blockquote><br /> Used to be work under ISO
9000writing standards and dealing with Auditors.  I know how auditing works and ways to make yourself look good and in
compliancewhen its completely penciled wiped into shape. Given no 3rd party does completely independent testing i view
itas an all around joke. <br /><br /> This is the reason we chose to drop ISO certification as it had nothing to do
aboutdoing the job right <br /> 

On Mon, Dec 15, 2008 at 12:11 AM, justin <justin@emproshunts.com> wrote:
> Thats a problem with testing.  Its not cheap or easy.
...
> So yes i know testing is very expensive but it can be done and should be
> done independently of the Hardware/Software manufactures.

Well, this is an open source community, and it doesn't have the money
or the resources to perform such a benchmark.  Similarly, regardless
of your opinion regarding TPC database benchmarks, I (and several
others) have run these tests ourselves and know for a fact that both
Oracle and SQL Server's results aren't fake.  Likewise, I (and other
PG companies) know where PG stands in comparison.

I'm not quite sure where you're coming from, because you obviously
haven't performed any of these benchmarks yourself and are therefore a
bit uninformed in regard to questioning their validity.

--
Jonah H. Harris, Senior DBA
myYearbook.com

Jonah H. Harris wrote:
> Well, running an official TPC benchmark is an interesting process.
> First, you have to have a spec-compliant benchmark kit, which we
> don't.

Well, then the first order of business would be to write a benchmark
kit.  I have been thinking for a while that we should make our own
maintained version of the DBT+* suite, or whatever other suite is
appropriate.  And then start running it.

The rest of this discussion, while interesting, cannot really lead to
any improvements, because none of the results and analyses can be
published.  Like, you know, "I have these great insights into important
problems, but I cannot tell you about them."

Is it possible to write a possibly eventually compliant benchmark kit
under open-source terms?

On Mon, Dec 15, 2008 at 3:25 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
> Well, then the first order of business would be to write a benchmark kit.  I
> have been thinking for a while that we should make our own maintained
> version of the DBT+* suite, or whatever other suite is appropriate.  And
> then start running it.

Agreed.  For a start on TPC-C, EnterpriseDB has a closer version of the kit.

> The rest of this discussion, while interesting, cannot really lead to any
> improvements, because none of the results and analyses can be published.
>  Like, you know, "I have these great insights into important problems, but I
> cannot tell you about them."

I will soon tell you, because Oracle gave me permission to publish.

> Is it possible to write a possibly eventually compliant benchmark kit under
> open-source terms?

I believe so.

--
Jonah H. Harris, Senior DBA
myYearbook.com

"Jonah H. Harris" <jonah.harris@gmail.com> writes:

> On Sun, Dec 14, 2008 at 10:47 PM, Bruce Momjian <bruce@momjian.us> wrote:
>> 10-100x?
>>
>> I am confused because sometimes I hear that Postgres has bad performance
>> from ex-Oracle users, but in general I hear that Oracle and Postgres
>> have similar performance behavior from people porting applications.

I think there are problem cases where Postgres really doesn't perform well.
That's the only time you're going to see a 10-100x factor anyways. Even if you
compared SQL-Lite to Oracle you would expect to see vaguely comparable results
unless you're hitting on a case that SQL-Lite just isn't intended to handle.

For example there are some queries in the DBT-3 power test which Postgres
doesn't optimize well. In some cases Postgres is just entirely lacking the
ideal plan type. It's kind of pointless to talk about *how* much slower some
random inappropriate plan is than some entirely different plan Oracle uses.
Who cares if it's 10x or 100x or even 1000x slower. No amount of optimization
is going to fix it unless we address the algorithmic problem anyways.

> The OLTP cases where Oracle starts to outperform Postgres is usually
> around 25 heavy concurrent sessions.  When you start scaling into
> hundreds of sessions, Oracle really starts to shine.

Postgres is definitely not optimized for systems with many concurrent
sessions. In the past we've basically dismissed such systems as misconfigured
on the basis that there's not much point in having many more sessions than the
number of processors in your system. If you have a big RAID array then it's
possible the number of spindles might be more relevant than number of
processors. But the point is that there's no performance advantage to
configuring more sessions than the system can actually make progress on
concurrently rather than queueing up the requests and processing them
sequentially.

There is a counter-argument to this however. If you have a system with complex
multi-request transactions then it's a question of programming convenience.
The easiest model to work with is to start a database session and use the
database to maintain transaction state -- which is what it's for after all.
That's more robust and convenient than using autocommit or otherwise trying to
manage short stateless requests. But it does risk leaving you with relatively
long-lived sessions which are often idle and forces you to have a large number
of backends running.

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's Slony Replication support!

Jonah,

> Where you should be looking is at the price/performance benchmarks,
> because that's where Postgres plays.  Last time I checked Postgres on
> a TPC-C, albeit being 100% free, was anywhere from $4.00 to $6.00 per
> transaction depending on the hardware.  Compare that to Oracle's $0.68
> or SQL Server's $0.84.

You can't compare DBT2 with TPCC.  They're not the same benchmark.

--Josh

Josh Berkus <josh@agliodbs.com> writes:

> Jonah,
>
>> Where you should be looking is at the price/performance benchmarks,
>> because that's where Postgres plays.  Last time I checked Postgres on
>> a TPC-C, albeit being 100% free, was anywhere from $4.00 to $6.00 per
>> transaction depending on the hardware.  Compare that to Oracle's $0.68
>> or SQL Server's $0.84.
>
> You can't compare DBT2 with TPCC.  They're not the same benchmark.

Eh? DBT2 is a (partial) implementation of TPC-C. The nature of benchmarks is
that small infelicities could invalidate the whole result though and there are
definitely infelicities in DBT2's implementation of TPC-C.

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training!

On Mon, Dec 15, 2008 at 3:26 PM, Josh Berkus <josh@agliodbs.com> wrote:
> You can't compare DBT2 with TPCC.  They're not the same benchmark.

Agreed.  Somewhere earlier in the thread I mentioned that the OSDL
kits are non-spec-compliant, a little buggy, and sometimes give better
results than the real tests.

--
Jonah H. Harris, Senior DBA
myYearbook.com

Gregory Stark escreveu:

> Eh? DBT2 is a (partial) implementation of TPC-C. The nature of benchmarks is
> that small infelicities could invalidate the whole result though and there are
> definitely infelicities in DBT2's implementation of TPC-C.
>
[I did't read the spec carefully but ...] Could you elaborate what
infelicities are? I would be really nice to have some benchmark kit as close
as possible from the specs.


--
  Euler Taveira de Oliveira
  http://www.timbira.com/

On Mon, Dec 15, 2008 at 6:17 AM, Jonah H. Harris <jonah.harris@gmail.com> wrote:
> On Mon, Dec 15, 2008 at 3:25 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
>> Well, then the first order of business would be to write a benchmark kit.  I
>> have been thinking for a while that we should make our own maintained
>> version of the DBT+* suite, or whatever other suite is appropriate.  And
>> then start running it.
>
> Agreed.  For a start on TPC-C, EnterpriseDB has a closer version of the kit.

I don't recall turning away patches to make the dbt kits less TPC spec
compliant...

Regards,
Mark

On Mon, Dec 15, 2008 at 12:25 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
> Jonah H. Harris wrote:
>>
>> Well, running an official TPC benchmark is an interesting process.
>> First, you have to have a spec-compliant benchmark kit, which we
>> don't.
>
> Well, then the first order of business would be to write a benchmark kit.  I
> have been thinking for a while that we should make our own maintained
> version of the DBT+* suite, or whatever other suite is appropriate.  And
> then start running it.

We're slowing tuning a system for the dbt2 kit in Portland with the
hardware from HP.

> The rest of this discussion, while interesting, cannot really lead to any
> improvements, because none of the results and analyses can be published.
>  Like, you know, "I have these great insights into important problems, but I
> cannot tell you about them."
>
> Is it possible to write a possibly eventually compliant benchmark kit under
> open-source terms?

Yes.  I believe the only restriction is to not use the kit in a way
that competes with the TPC.  Using the kits for our developing efforts
is ok.

Regards,
Mark

"Mark Wong" <markwkm@gmail.com> writes:

> On Mon, Dec 15, 2008 at 6:17 AM, Jonah H. Harris <jonah.harris@gmail.com> wrote:
>> On Mon, Dec 15, 2008 at 3:25 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
>>> Well, then the first order of business would be to write a benchmark kit.  I
>>> have been thinking for a while that we should make our own maintained
>>> version of the DBT+* suite, or whatever other suite is appropriate.  And
>>> then start running it.
>>
>> Agreed.  For a start on TPC-C, EnterpriseDB has a closer version of the kit.
>
> I don't recall turning away patches to make the dbt kits less TPC spec
> compliant...

Uhm, why would we send them to you? Are you the maintainer for DBT2 now? I
looked recently and couldn't find any upstream source that had been touched
any time within the last few years. Where do i find it?

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's RemoteDBA services!

On Tue, Dec 16, 2008 at 12:14 PM, Gregory Stark <stark@enterprisedb.com> wrote:
> "Mark Wong" <markwkm@gmail.com> writes:
>
>> On Mon, Dec 15, 2008 at 6:17 AM, Jonah H. Harris <jonah.harris@gmail.com> wrote:
>>> On Mon, Dec 15, 2008 at 3:25 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
>>>> Well, then the first order of business would be to write a benchmark kit.  I
>>>> have been thinking for a while that we should make our own maintained
>>>> version of the DBT+* suite, or whatever other suite is appropriate.  And
>>>> then start running it.
>>>
>>> Agreed.  For a start on TPC-C, EnterpriseDB has a closer version of the kit.
>>
>> I don't recall turning away patches to make the dbt kits less TPC spec
>> compliant...
>
> Uhm, why would we send them to you? Are you the maintainer for DBT2 now? I
> looked recently and couldn't find any upstream source that had been touched
> any time within the last few years. Where do i find it?

http://sourceforge.net/projects/osdldbt/

Mark has been maintaining the DBT2 kit for a while now - since he was at OSDL.

As far as recent updates, he's been focused on updating the results
parsing and graphing libraries, which were written in Perl. I've been
helping with that a little

We've been presenting results -- first some IO tests at Linux Plumbers
Conf, and the very beginnings of postgresql tuning results at Pg West.
 That's Mark, Gabrielle Roth and me.

I'll be presenting some of our more recent results, and the donated HP
hardware, at FOSDEM. (assuming my talk got accepted! :D)

-selena

--
Selena Deckelmann
Open Source Bridge - http://www.opensourcebridge.org
PDXPUG - http://pugs.postgresql.org/pdx
Me - http://www.chesnok.com/daily

On Tue, Dec 16, 2008 at 12:29 PM, Selena Deckelmann
<selenamarie@gmail.com> wrote:
> On Tue, Dec 16, 2008 at 12:14 PM, Gregory Stark <stark@enterprisedb.com> wrote:

>> Uhm, why would we send them to you? Are you the maintainer for DBT2 now? I
>> looked recently and couldn't find any upstream source that had been touched
>> any time within the last few years. Where do i find it?
>
> http://sourceforge.net/projects/osdldbt/
>
> Mark has been maintaining the DBT2 kit for a while now - since he was at OSDL.

Correction -- Mark created this kit in 2003.

-selena



--
Selena Deckelmann
Open Source Bridge - http://www.opensourcebridge.org
PDXPUG - http://pugs.postgresql.org/pdx
Me - http://www.chesnok.com/daily

On Monday 15 December 2008 23:56:08 Jonah H. Harris wrote:
> On Mon, Dec 15, 2008 at 3:26 PM, Josh Berkus <josh@agliodbs.com> wrote:
> > You can't compare DBT2 with TPCC.  They're not the same benchmark.
>
> Agreed.  Somewhere earlier in the thread I mentioned that the OSDL
> kits are non-spec-compliant, a little buggy, and sometimes give better
> results than the real tests.

Could we consolidate everyone's concerns into a project outline, if one is
necessary?

- Everyone would like to have a TPC-like benchmark kit.
- OSDL script are not compliant.
- OSDL scripts are outdated/appear unmaintained.
- OSDL scripts need POstgreSQL-specific improvements.
- EnterpriseDB is apparently maintaining their own fork.

What can we do?

I found the TPC-C spec on the web, so it's freely available for
implementation, it appears.

Peter,

> I found the TPC-C spec on the web, so it's freely available for
> implementation, it appears.

For some definition of "free".  It's not open source, and you can't use
it however you want.  For example, you can't use it to publish a
benchmark, and as far as the TPC is concerned, putting it on a wiki is
"publishing".


This is why I was focussing on SpecJAppserver instead.  It has a
derivitative wersion (EAStress), which does not have such restrictions.

--Josh

On Tuesday 16 December 2008 15:58:15 Peter Eisentraut wrote:
> On Monday 15 December 2008 23:56:08 Jonah H. Harris wrote:
> > On Mon, Dec 15, 2008 at 3:26 PM, Josh Berkus <josh@agliodbs.com> wrote:
> > > You can't compare DBT2 with TPCC.  They're not the same benchmark.
> >
> > Agreed.  Somewhere earlier in the thread I mentioned that the OSDL
> > kits are non-spec-compliant, a little buggy, and sometimes give better
> > results than the real tests.
>
> Could we consolidate everyone's concerns into a project outline, if one is
> necessary?
>
> - Everyone would like to have a TPC-like benchmark kit.
> - OSDL script are not compliant.
> - OSDL scripts are outdated/appear unmaintained.
> - OSDL scripts need POstgreSQL-specific improvements.
> - EnterpriseDB is apparently maintaining their own fork.
>

Don't forget Jan's TPC-W benchmark, available at
http://pgfoundry.org/projects/tpc-w-php/. Not sure if it still works, but Jan
claims it did at one point.

--
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

"Selena Deckelmann" <selenamarie@gmail.com> writes:

> http://sourceforge.net/projects/osdldbt/
>
> Mark has been maintaining the DBT2 kit for a while now - since he was at OSDL.
>
> We've been presenting results -- first some IO tests at Linux Plumbers
> Conf, and the very beginnings of postgresql tuning results at Pg West.
>  That's Mark, Gabrielle Roth and me.

That SVN repository hasn't been touched in almost two years. It doesn't even
work with the current TPC-C datagen without editing the data files.

What drives me nuts trying to get these kits running is how they have so many
scripts running other scripts and they all depend on having an environment set
up with specific users, paths, and environment variables set.

Ideally it would just be a simple binary like pgbench which takes command-line
options to tell it what to do and can be run without setting up any special
environment.

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's Slony Replication support!

On Tue, Dec 16, 2008 at 3:38 PM, Gregory Stark <stark@enterprisedb.com> wrote:
>
> What drives me nuts trying to get these kits running is how they have so many
> scripts running other scripts and they all depend on having an environment set
> up with specific users, paths, and environment variables set.
>
> Ideally it would just be a simple binary like pgbench which takes command-line
> options to tell it what to do and can be run without setting up any special
> environment.

Fair enough.

We'd love some help.  Or maybe patches from the fork of Mark's code. :)

Currently, we're focusing on updating the reporting end.

-selena

--
Selena Deckelmann
Open Source Bridge - http://www.opensourcebridge.org
PDXPUG - http://pugs.postgresql.org/pdx
Me - http://www.chesnok.com/daily

On Tue, Dec 16, 2008 at 12:14 PM, Gregory Stark <stark@enterprisedb.com> wrote:
> "Mark Wong" <markwkm@gmail.com> writes:
>
>> On Mon, Dec 15, 2008 at 6:17 AM, Jonah H. Harris <jonah.harris@gmail.com> wrote:
>>> On Mon, Dec 15, 2008 at 3:25 AM, Peter Eisentraut <peter_e@gmx.net> wrote:
>>>> Well, then the first order of business would be to write a benchmark kit.  I
>>>> have been thinking for a while that we should make our own maintained
>>>> version of the DBT+* suite, or whatever other suite is appropriate.  And
>>>> then start running it.
>>>
>>> Agreed.  For a start on TPC-C, EnterpriseDB has a closer version of the kit.
>>
>> I don't recall turning away patches to make the dbt kits less TPC spec
>> compliant...
>
> Uhm, why would we send them to you? Are you the maintainer for DBT2 now? I
> looked recently and couldn't find any upstream source that had been touched
> any time within the last few years. Where do i find it?

Honestly I would have expected to see them sent to the osdldbt-general
mailing list, not to me personally.  But I guess that was never
advertised in the kit.  I don't think I stopped maintaining it, but
some people have been sending patches to the list that apply to
releases older than what is in the svn repository.  That makes it a
little hard for me to get it right, especially when they are for
databases I don't use.  And I admit fault at never announcing that
I've move the source code from svn to mercurial to git, which is not
on git.postgresql.org.

Regards,
Mark

On Tue, Dec 16, 2008 at 3:38 PM, Gregory Stark <stark@enterprisedb.com> wrote:
> "Selena Deckelmann" <selenamarie@gmail.com> writes:
>
>> http://sourceforge.net/projects/osdldbt/
>>
>> Mark has been maintaining the DBT2 kit for a while now - since he was at OSDL.
>>
>> We've been presenting results -- first some IO tests at Linux Plumbers
>> Conf, and the very beginnings of postgresql tuning results at Pg West.
>>  That's Mark, Gabrielle Roth and me.
>
> That SVN repository hasn't been touched in almost two years. It doesn't even
> work with the current TPC-C datagen without editing the data files.

Please explain further.  TPC doesn't have a tpc-c datagen program like
they do for the H, W, and E.  Otherwise the code in the git repository
appears to be working with 8.3.5.

> What drives me nuts trying to get these kits running is how they have so many
> scripts running other scripts and they all depend on having an environment set
> up with specific users, paths, and environment variables set.

I agree.  I've refactored them a little bit.  But it's not that much better.

> Ideally it would just be a simple binary like pgbench which takes command-line
> options to tell it what to do and can be run without setting up any special
> environment.

Anything after a the The C, E, H, and newer are not likely to be that
simple.  The results will be of limited used without physically
configuring the disks for example, IMHO.

Regards,
Mark

On Wed, Dec 17, 2008 at 1:26 AM, Mark Wong <markwkm@gmail.com> wrote:
>
>> That SVN repository hasn't been touched in almost two years. It doesn't even
>> work with the current TPC-C datagen without editing the data files.
>
> Please explain further.  TPC doesn't have a tpc-c datagen program like
> they do for the H, W, and E.

Sorry, the discrepancy with datagen that I remembered was with DBT3.

So uh, where is the current source repository?

--
greg

On Tue, Dec 16, 2008 at 5:38 PM, Greg Stark <stark@enterprisedb.com> wrote:
> On Wed, Dec 17, 2008 at 1:26 AM, Mark Wong <markwkm@gmail.com> wrote:
>>
>>> That SVN repository hasn't been touched in almost two years. It doesn't even
>>> work with the current TPC-C datagen without editing the data files.
>>
>> Please explain further.  TPC doesn't have a tpc-c datagen program like
>> they do for the H, W, and E.
>
> Sorry, the discrepancy with datagen that I remembered was with DBT3.

dbgen always needed editing.  The changes are supplied with the kit.

> So uh, where is the current source repository?

On git.postgresql.org.

http://git.postgresql.org/?p=~markwkm/dbt2.git;a=summary

And since you bring up dbt3:

http://git.postgresql.org/?p=~markwkm/dbt3.git;a=summary

Regards,
Mark

On Tuesday 16 December 2008 20:19:28 Mark Wong wrote:
> And I admit fault at never announcing that
> I've move the source code from svn to mercurial to git, which is not
> on git.postgresql.org.
>

that should read "now on", not "not on", right ?

--
Robert Treat
Conjecture: http://www.xzilla.net
Consulting: http://www.omniti.com

On Dec 16, 2008, at 9:10 PM, Robert Treat wrote:

> On Tuesday 16 December 2008 20:19:28 Mark Wong wrote:
>> And I admit fault at never announcing that
>> I've move the source code from svn to mercurial to git, which is not
>> on git.postgresql.org.
>>
>
> that should read "now on", not "not on", right ?

Yeah, my bad on spelling.

Regards,
Mark