Re: postmaster dead but backends still running?

Charles Hornberger <charlie@hss.caltech.edu> writes:
> Other things I perhaps ought to mention: Trying to stop the postmaster
> using pg_ctl fails (unsurprisingly, since pg_ctl relies on
> /var/pgsql/data/postmaster.pid, which contains a nonexistent PID); I
> haven't tried to start a new postmaster yet, because the old backends
> are hanging around.

In theory a new postmaster would detect the old backends and refuse to
start anyway.  I don't trust that interlock unreservedly though.  (But
please test it while you have the opportunity...)

> Nor have I attempted to restart the web server, which might allow the
> hanging-round backends to die by closing the old connections it's
> holding to them. I'm tempted to go ahead and do this, though I'm not
> sure whether I ought to until I've diagnosed what's going on right now.

You will need to close all the existing connections before the new
postmaster can be started.  I'd recommend doing so sooner instead of
later, because with no postmaster you aren't getting any checkpoints
done, and your WAL space is going to start ballooning.

As far as diagnosing the problem goes: if you have a postmaster log
file, look to see if the postmaster wrote an ERROR or FATAL message
before it exited.  (Finding it among all the backend-level messages
might be painful though.)  Also look in the directory the postmaster
was started in to see if there's a core file.  Save away any evidence
you can find before trying to start a new postmaster.

Because the postmaster doesn't actually do much, crashes are pretty
unusual.  I'm interested in whatever you can find.

            regards, tom lane