Re: CPU spikes and transactions

On Mon, Oct 14, 2013 at 6:45 PM, Tomas Vondra <tv@fuzzy.cz> wrote:
> On 15.10.2013 01:26, Tony Kay wrote:
>> Hi Calvin,
>>
>> Yes, I have sar data on all systems going back for years.
>>
>> Since others are going to probably want to be assured I am really
>> "reading the data" right:
>>
>> - This is 92% user CPU time, 5% sys, and 1% soft
>> - On some of the problems, I _do_ see a short spike of pgswpout's
>> (memory pressure), but again, not enough to end up using much system time
>> - The database disks are idle (all data being used is in RAM)..and are
>> SSDs....average service times are barely measurable in ms.
>
> OK. Can you share the data? Maybe we'll notice something suspicious.
>
>> If I had to guess, I'd say it was spinlock misbehavior....I cannot
>> understand why ekse a transaction blocking other things would drive
>> the CPUs so hard into the ground with user time.
>
> Have you tried running perf, to verify the time is actually spent on
> spinlocks?

+1 this.  It is almost certainly spinlocks, but we need to know which
one and why.  plz install debug symbols and run a perf during normal
and high load conditions.

merlin