Re: "stuck spinlock"

On 2013-12-12 21:15:29 -0500, Tom Lane wrote:
> Christophe Pettus <xof@thebuild.com> writes:
> > On Dec 12, 2013, at 5:45 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> >> Presumably, we are seeing the victim rather than the perpetrator of
> >> whatever is going wrong.
> 
> > This is probing about a bit blindly, but the only thing I can see about this system that is in some way unique (and
thisis happening on multiple machines, so it's unlikely to be hardware) is that there are a relatively large number of
relations(like, 440,000+) distributed over many schemas.  Is there anything that pins a buffer that is O(N) to the
numberof relations?
 
> 
> It's not a buffer *pin* that's at issue, it's a buffer header spinlock.
> And there are no loops, of any sort, that are executed while holding
> such a spinlock.  At least not in the core PG code.  Are you possibly
> using any nonstandard extensions?

It could maybe be explained by a buffer aborting while performing
IO. Until it has call AbortBufferIO(), other backends will happily loop
in WaitIO(), constantly taking the the buffer header spinlock and
locking io_in_progress_lock in shared mode, thereby preventing
AbortBufferIO() from succeeding.

Christophe: are there any "unusual" ERROR messages preceding the crash,
possibly some minutes before?

Greetings,

Andres Freund

-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services