brk() function and performance

Hi,

We're running PostgreSQL 7.1.3 (I know, I know) on Solaris 7 on two
Sun E4500s with 8 CPUs and 16 Gig of RAM.

We have noticed that one of the machines is considerably slower than
the other.  We have traced the problem to the brk() funciton call.

We were having some trouble with certain queries, because we were
spending a lot of time moving blocks between the OS filesystem
buffers and the Postgres shared buffer.  That was on server A.  So,
we played with some settings, and settled on shared_buffers = 262144.

On server B, we did not see the same problem.  There,
shared_buffers=8192.  That is the only difference between the
machines, except that server A also sees a lot more interactive
traffic (server B is a replicated copy, and handles a number of
read-only queries).

We increased the shared_buffers setting on server A a few weeks ago,
and saw an immediate (albeit slight, but enough for us) improvement
in the system.  But in the past few days, we have experienced
sluggish behaviour.

By selecting a single line from a frequently-accessed, relatively
small table (< 300 rows), and selecting on an indexed field, we get
the following difference in the truss output:

Server A: the query takes 700-800 ms.
syscall      seconds   calls
brk              .27      62

Server B: the query takes 200-300 ms.
syscall      seconds   calls
brk              .02      64

Everything else is the same.

The backend has been running since the shared memory change.  I was
wondering if perhaps the problem is what brk() is doing.  Maybe it
needs a contiguous segment, and when it goes to allocate more of its
reserved memory, it has to shift the whole thing around?  (If so,
this is a clear reason why not to use huge shared buffers.)  The
Solaris man page doesn't make clear how this works (and in fact seem
to suggest that brk() shouldn't be used).

Any remarks, pointers, or suggestions would be welcome.  I'm stumped.
This is very puzzling.

A

--
----
Andrew Sullivan                               87 Mowat Avenue
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M6K 3E3
                                         +1 416 646 3304 x110