Re: hung backends stuck in spinlock heavy endless loop

On Wed, Jan 14, 2015 at 9:30 AM, Andres Freund <andres@2ndquadrant.com> wrote:
> If you gdb in, and type 'fin' a couple times, to wait till the function
> finishes, is there actually any progress? I'm wondering whether it's
> just many catalog accesses + contention, or some other
> problem. Alternatively set a breakpoint on ScanPgRelation() or so and
> see how often it's hit.

well, i restarted the database, forgetting my looper was running which
immediately spun up and it got stuck again with a similar profile
(lots of cpu in spinlock):

Samples: 3K of event 'cycles', Event count (approx.): 269572322831.16%  postgres            [.] s_lock22.32%  postgres
         [.] tas12.13%  postgres            [.] tas 5.93%  postgres            [.] spin_delay 5.69%  postgres
[.] LWLockRelease 3.75%  postgres            [.] LWLockAcquireCommon 3.61%  perf                [.] 0x00000000000526c4
2.51% postgres            [.] FunctionCall2Coll 1.48%  libc-2.17.so        [.] 0x000000000016a132
 

> If you gdb in, and type 'fin' a couple times,
(gdb) fin
Run till exit from #0  0x00007ff4c63f7a97 in semop () from
/lib/x86_64-linux-gnu/libc.so.6
0x00000000006de073 in PGSemaphoreLock ()
(gdb) fin
Run till exit from #0  0x00000000006de073 in PGSemaphoreLock ()

It returned once.  Second time, it didn't at least so far (minute or so).

>> I can send the code off-list if you guys think it'd help.
>
> Might be interesting.

sent (off-list).

merlin