Re: BUG #13459: Replaying WAL logs can hang on startup

chris+postgresql@qwirx.com writes:
> I restored a standby from a pg_basebackup made from the master the previous
> morning. As a result, it had a lot of WAL logs to catch up on.
> At one point it hangs while restoring logs. It normally takes a few seconds
> to process a 16 MB WAL segment, but on this one, it was "recovering
> 0000000100000CEC00000025" for 7 minutes now with no log output at all.

Hmm ...

> My guess is that ForwardFsyncRequest() is continually returning false, and
> this code is stuck forever. I noticed that it says that "I'm inclined to
> assume that the checkpointer
> will always empty the queue soon", but there is no checkpointer running
> during recovery, is there?

There is supposed to be one once we have reached a consistent state; see
SetForwardFsyncRequests().  AFAICS it should be impossible to reach the
wait you're seeing unless the startup process's local pendingOpsTable has
been removed by SetForwardFsyncRequests(), and the caller of that should
have pinged the postmaster to start up a checkpointer.

If you can repro this easily, please look to see whether there's a
checkpointer, and if not, what state the postmaster is in (pmState,
CheckpointerPID, Shutdown, FatalError, RecoveryError might be
interesting).  If there is a checkpointer, then that's what to be
looking at.

            regards, tom lane