Re: Sudden crazy high CPU usage

On Mon, Mar 31, 2014 at 5:25 AM, Niels Kristian Schjødt
<nielskristian@autouncle.com> wrote:
> I'm running postgresql 9.3 on a production server. An hour ago, out of the "blue", I ran into an issue I have never
encounteredbefore: my server started to use CPU as crazy. The server is a standard ubuntu 12.04 LTE installation
runningonly Postgres and Redis. 
>
> The incident can be seen on the in numbers below:
>
> https://s3-eu-west-1.amazonaws.com/autouncle-public/other/cpu.png
>
> I imidiatly took a look at pg_stat_activity but nothing in there seemed suspicious. I also had a look at the postgres
log,but nothing was in there too. I have pg_stat_statements running, so I reseted that one, and nothing really
suspiciousoccurred in there, expect for the fact, that all queries were taking 100x times longer than usual. 
>
> I have tried the following with no luck:
>
>         * Restart clients connecting to the db
>         * Restart postgres
>         * Restart the whole server
>
> I have run memory tests on the server as well, and nothing seems to be wrong.
>
> No changes in any software running on the servers has been made within the last 24 hours.
>
> The question is: I have a streaming replication server running, which I have now done a failover to, and it runs
fine.However I still have no clue why my master suddenly has become so CPU consuming, and how I can debug / trace it
furtherdown? 

Using linux 6? One possible culprit is "Transparent Huge Page
Compaction".  It tends to hit severs with a lot of memory, especially
if they've configured a lot of shared buffers.  Google it a for a lot
of info.

There may be other issues masquerading as this one but it's the first
thing to rule out.  Symptoms are very high cpu utilization and poor
performance that strikes without warning and then resolves also
without warning (typically seconds or minutes after the event).

For starters, take a look at the value of:

/sys/kernel/mm/redhat_transparent_hugepage/enabled

And do some due diligence research.

merlin