Re: Pg stuck at 100% cpu, for multiple days

On Tue, Aug 31, 2021 at 11:41 AM hubert depesz lubaczewski
<depesz@depesz.com> wrote:
>
> On Mon, Aug 30, 2021 at 09:09:20PM +0200, Laurenz Albe wrote:
> > On Mon, 2021-08-30 at 17:18 +0200, hubert depesz lubaczewski wrote:
> > > The thing is - I can't close it with pg_terminate_backend(), and I'd
> > > rather not kill -9, as it will, I think, close all other connections,
> > > and this is prod server.
> >
> > Of course the cause should be fixed, but to serve your immediate need:
> >
> > After calling pg_terminate_backend(), you can attach gdb to the backend and then run
> >
> >   print ProcessInterrupts()
> >
> > That will cause the backend to exit normally without crashing the server.
>
> I got this mail too late, and the decision was made to restart Pg. After
> restart all cleaned up nicely.
>
> So, while I can't help more with diagnosing the problem, I think it
> might be good to try to figure out what could have happened.
>

One possibility could be there are quite a few DDLs happening in this
application at some particular point in time which can lead to high
CPU usage. Prior to commit d7eb52d718 in PG-14, we use to execute all
invalidations at each command end during logical decoding which might
lead to such behavior temporarily. I think a bit of debugging when it
shows this symptom could help us to identify if it is the problem I am
speculating here.

-- 
With Regards,
Amit Kapila.