Re: High CPU Utilization

On Fri, Mar 20, 2009 at 2:26 PM, Joe Uhl <joeuhl@gmail.com> wrote:
> On Mar 17, 2009, at 12:19 AM, Greg Smith wrote:
>
>> On Tue, 17 Mar 2009, Gregory Stark wrote:
>>
>>> Hm, well the tests I ran for posix_fadvise were actually on a Perc5 --
>>> though
>>> who knows if it was the same under the hood -- and I saw better
>>> performance
>>> than this. I saw about 4MB/s for a single drive and up to about 35MB/s
>>> for 15
>>> drives. However this was using linux md raid-0, not hardware raid.
>>
>> Right, it's the hardware RAID on the Perc5 I think people mainly complain
>> about.  If you use it in JBOD mode and let the higher performance CPU in
>> your main system drive the RAID functions it's not so bad.
>>
>> --
>> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
>
> I have not yet had a chance to try software raid on the standby server
> (still planning to) but wanted to follow up to see if there was any good way
> to figure out what the postgresql processes are spending their CPU time on.
>
> We are under peak load right now, and I have Zabbix plotting CPU utilization
> and CPU wait (from vmstat output) along with all sorts of other vitals on
> charts.  CPU utilization is a sustained 90% - 95% and CPU Wait is hanging
> below 10%.  Since being pointed at vmstat by this list I have been watching
> CPU Wait and it does get high at times (hence still wanting to try Perc5 in
> JBOD) but then there are sustained periods, right now included, where our
> CPUs are just getting crushed while wait and IO (only doing about 1.5 MB/sec
> right now) are very low.
>
> This high CPU utilization only occurs when under peak load and when our JDBC
> pools are fully loaded.  We are moving more things into our cache and
> constantly tuning indexes/tables but just want to see if there is some
> underlying cause that is killing us.
>
> Any recommendations for figuring out what our database is spending its CPU
> time on?

What does the cs entry on vmstat say at this time?  If you're cs is
skyrocketing then you're getting a context switch storm, which is
usually a sign that there are just too many things going on at once /
you've got an old kernel things like that.