Re: WIP: bufmgr rewrite per recent discussions

"Mark Cave-Ayland" <m.cave-ayland@webbased.co.uk> writes:
> I compiled and tested your patch on a dual Opteron server with 12GB RAM
> running FC3. Here are the results I get with pgbench with a scale-factor of
> 10 over an average of 6 runs.

Thanks for posting these results.  What -c and -t settings were you using
with pgbench?  (I like to use -c equal to the scale factor and -t of at
least 1000 ... much less than that gives fairly unstable results in my
experience.)

> The interesting thing to note was that going up to 100000 buffers seemed to
> cause the performance to go down again which is something I wouldn't have
> expected on a server with such a large amount of RAM given that it was
> suggested ARC would perform better with more shared buffers.

I think what is probably happening here is the background writer is
eating too many cycles.  As of the patch I posted, the bgwriter is still
using its 8.0 control parameters, in which the minimum scan percentage
is 1% of all the buffers (so 1000 buffers scanned in each round in your
last test) and it's willing to write up to 100 dirty buffers per round
by default.  I was looking at this yesterday and thinking it seemed
clearly excessive.  With a default bgwriter_delay of 200 msec, this
allows the entire buffer array to be scanned every 20 sec, so we're in
effect keeping the thing under constant syncer load.

If you have time to redo your experiment, would you try knocking
bgwriter_maxpages down to 10 to see if it helps at the larger
shared_buffer settings?

Since yesterday I've improved my patch by converting the bgwriter
percentage variable into a float, so that values smaller than 1% can be
selected, and I've split the two variables into four so that people can
independently control the effort spent on the whole buffer array versus
the buffers just in front of nextVictimBuffer (see BgBufferSync in the
patch).  I'm not sure how important that is, but I do think that the 1%
/ 100 default settings are way too high for larger buffer pools.  Once
that's in, it will be hard to compare the patch directly against CVS tip,
so trying it now with a smaller maxpages setting for both would be a
fairer comparison.

I have another couple of small ideas for improving the patch --- I'll
try to get those done and post a revised version this evening.

            regards, tom lane