more filesystem benchmarks

In our last installment, we saw that JFS provides higher pgbench
performance than either XFS or ext3.  Using a direct-I/O patch stolen
from 8.1, JFS achieved 105 tps with 100 clients.

To refresh, the machine in question has 5 7200RPM SATA disks, an Areca
RAID controller with 128MB cache, and 1GB of main memory.  pgbench is
being run with a scale factor of 1000 and 100000 total transactions.

At the suggestion of Andreas Dilger of clusterfs, I tried modulating the
size of the ext3 journal, and the mount options (data=journal,
writeback, and ordered).  I turns out that you can achieve a substantial
improvement (almost 50%) by simply mounting the ext3 volume with
data=writeback instead of data=ordered (the default).  Changing the
journal size did not seem to make a difference, except that 256MB is for
some reason pathological (9% slower than the best time).  128MB, the
default for a large volume, gave the same performance as 400MB (the max)
or 32MB.

In the end, the ext3 volume mounted with -o noatime,data=writeback
yielded 88 tps with 100 clients.  This is about 16% off the performance
of JFS with default options.

Andreas pointed me to experimental patches to ext3's block allocation
code and writeback strategy.  I will test these, but I expect the
database community, which seems so attached to its data, will be very
interested in code that has not yet entered mainstream use.

Another frequent suggestion is to put the xlog on a separate device.  I
tried this, and, for a given number of disks, it appears to be
counter-productive.  A RAID5 of 5 disks holding both logs and data is
about 15% faster than a RAID5 of 3 disks with the data, and a mirror of
two disks holding the xlog.

Here are the pgbench results for each permutation of ext3:

Journal Size | Journal Mode | 1 Client | 10 Clients | 100 Clients
------------------------------------------------------------------
32             ordered        28         51           57
32             writeback      34         70           88
64             ordered        29         52           61
64             writeback      32         69           87
128            ordered        32         54           62
128            writeback      34         70           88
256            ordered        28         51           60
256            writeback      29         64           79
400            ordered        26         49           59
400            writeback      32         70           87

-jwb