Re: 100% CPU pg processes that don't die.

On Sun, 10 Aug 2008, Scott Marlowe wrote:

> The good news is that both Centos 5.2 and Ubuntu 7.10 seem immune to
> this particular bug, and have been running 13 hours now without a
> hitch.

Not sure if it's relevant here, but you do know that I've been kicking
back to lkml that pgbench has issues on the 2.6.24 kernel, right?  I
haven't tried simulating 1000 clients like you're case, but it fails
miserably to scale to even 10 with the select-only workload.

The CFS scheduler used in 2.6.23+ is only about a year old right now, and
it sure seems like it's still a baby going through its share of teething
pains.  Lest you think it's just me complaining, read
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/188226 Maybe your
problem is somewhere else instead, but particularly given the known
pgbench issues it may be worth spending a minute investigating that area.

One bit of black magic I was told is that you can turn off some of the new
scheduling "features" (one of which was the main cause of the pgbench
incompatibility) by updating a kernel tunable.  You can turn off all the
fancy features with:

echo 0 > /proc/sys/kernel/sched_features

See http://lkml.org/lkml/2008/5/26/288 for more details.

Whether the problem still shows up with that change should help narrow
whether your issue is likely related to the scheduler changes or likely
something else that's different with the newer kernel.  Your pgbench
results should be higher after that change, too, but you shouldn't run the
system like that normally--it's optimizing for something bad pgbench does
but will hurt other, more realistic workloads.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD