Re: upgrade from 9.2.x to 9.3 causes significant performance degradation

On Wed, Sep 18, 2013 at 2:02 AM, Kevin Grittner <kgrittn@ymail.com> wrote:
> Lonni J Friedman <netllama@gmail.com> wrote:
>
>> top shows over 90% of the load is in sys space.  vmstat output
>> seems to suggest that its CPU bound (or bouncing back & forth):
>
> Can you run `perf top` during an episode and see what kernel
> functions are using all that CPU?

Oddly, the problem went away on its own yesterday just after 4PM, and
performance has remained 'normal' since that time.  I changed
absolutely nothing.  If/when it returns, I'll certainly capture that
output.

>
> This looks similar to cases I've seen of THP defrag going wild.
> Did the OS version or configuration change?  Did the PostgreSQL
> memory settings (like shared_buffers) change?

Nothing changed other than the version of postgres.  I re-used the
same postgresql.conf that was in place when running 9.2.x.

Anyway, here are the current THP related settings on the server:
[root@cuda-db7 ~]# grep AnonHugePages /proc/meminfo
AnonHugePages:    548864 kB
[root@cuda-db7 ~]# egrep 'trans|thp' /proc/vmstat
nr_anon_transparent_hugepages 272
thp_fault_alloc 129173889
thp_fault_fallback 17462551
thp_collapse_alloc 148437
thp_collapse_alloc_failed 15143
thp_split 242