Re: PG in container w/ pid namespace is init, process exits cause restart

On 5/3/21 3:07 PM, Andres Freund wrote:
> Hi,
>
> A colleague debugged an issue where their postgres was occasionally
> crash-restarting under load.
>
> The cause turned out to be that a relatively complex archive_command was
> used, which could in some rare circumstances have a bash subshell
> pipeline not succeed.  It wasn't at all obvious why that'd cause a crash
> though - the archive command handles the error.
>
> The issue turns out to be that postgres was in a container, with pid
> namespaces enabled. Because postgres was run directly in the container,
> without a parent process inside, it thus becomes pid 1. Which mostly
> works without a problem. Until, as the case here with the archive
> command, a sub-sub process exits while it still has a child. Then that
> child gets re-parented to postmaster (as init).
>
> Such a child is likely to have exited not just with 0 or 1, but
> something else. As the pid won't match anything in reaper(), we'll go to
> CleanupBackend(). Where any exit status but 0/1 will unconditionally
> trigger a restart:
>
>     if (!EXIT_STATUS_0(exitstatus) && !EXIT_STATUS_1(exitstatus))
>     {
>         HandleChildCrash(pid, exitstatus, _("server process"));
>         return;
>     }
>
>
> This kind of thing is pretty hard to debug, because it's not easy to
> even figure out what the "crashing" pid belonged to.
>
> I wonder if we should work a bit harder to try to identify whether an
> exiting process was a "server process" before identifying it as such?
>
> And perhaps we ought to warn about postgres running as "init" unless we
> make that robust?
>

Hmm, my initial reaction was if we detect very early on we're PID 1 then
fork and do all our work in the child, and in the parent just wait until
there are no more children. Not sure if that's feasible but I thought
I'd throw it out there.


cheers


andrew


--
Andrew Dunstan
EDB: https://www.enterprisedb.com