Hot standby, prepared xacts, locks

The Hot Standby patch changes lock_twophase_recover() so that when we're
starting up from Hot Standby mode to normal operation, as opposed to
crash recovery, we assume that all AccessExcusiveLocks are already held
by the startup process and instead of acquiring them anew they are
transferred from the startup process to the dummy PGPROC entry.

As the patch stands, that works. However, if we don't see a
running-xacts record, but only some XLOG_RELATION_LOCK records followed
by a PREPARE record, the startup process will hold some of the
transactions locks but not all of them. That's OK because we're not
letting read-only backends in yet. If we then failover and bring the
standby server up as a new master, lock_twophase_recover() will grant
the locks directly for the dummy process. But the startup process is
already holding some of them, so we briefly have a situation where two
processes - the dummy process and the startup process - are both holding
the same AccessExclusiveLock. AFAICS, the lock manager can cope with
that situation just fine, even though it can't normally happen, but it's
worth a mention.

A while back in your branch you changed ProcArrayApplyRecoverInfo() so
that the standby is put into "pending" mode when it sees a running-xacts
record marked as 'locks overflowed', instead of just ignoring the
record. If we see such a record, go into pending mode, and then try to
bring up the standby as the new master, we will run into this error in
lock_twophase_recover():

>         /*
>          * We should already hold the desired lock.
>          */
>         if (!(proclock->holdMask & LOCKBIT_ON(lockmode)))
>             elog(ERROR, "lock %s on object %u/%u/%u by xid %u was not held by startup process",
>                  lockMethodTable->lockModeNames[lockmode],
>                  locktag->locktag_field1, locktag->locktag_field2,
>                  locktag->locktag_field3,
>                  xid);
> 

lock_twophase_recover() assumes that all locks held by the prepared
transactions are already held by the startup process, but that might not
be true in the pending state.

Given that even without that change in ProcArrayApplyRecoveryInfo(), we
can end up in a situation where both the dummy process and the startup
process hold the same AccessExclusiveLock, I think we could simply
revert all the changes in lock_twophase_recover() and rely on that
behavior whenever we end recovery.

BTW, this self-proclaimed ugly hack in lock_twophase_recover() looks
utterly broken:

>         /*
>          * Transfer the lock->procLocks entry so that the lock is now owned
>          * by the new gxact proc.  Nothing else need change.
>          * This can happen safely because the startup process has the lock
>          * and wishes to give the lock to a gxact dummy process that has
>          * no associated backend. So neither proc can change concurrently.
>          * This technique might normally be considered an ugly hack, yet
>          * managing dummy procs all the way through recovery resulted in
>          * more complex code, which was worse. Chiedere il permesso.
>          */
>         SHMQueueInit(&lock->procLocks);
>         SHMQueueInsertBefore(&lock->procLocks, &proclock->lockLink);
>         SHMQueueInsertBefore(&(proc->myProcLocks[partition]),
>                              &proclock->procLink);

It will blow away any PROCLOCKs belonging to other backends waiting for
the lock, and you can't just transfer a PROCLOCK entry from one backend
to another anyway, because the PGPROC pointer is part of its the hash
key. But then again, I don't think it's actually transferring the
startup process' PROCLOCK entry anyway, because the PROCLOCK entry
searched/created just above that belongs to the dummy proc, not the
startup process. So this is really just blowing away all other PROCLOCKs
from the lock's procLock queue, and possibly inserting a duplicate entry
into proc->myProcLocks[partition]. Ugh.

So, I'm quite eager to just revert all those lock_twophase_recover()
changes, and always rely on the "grant lock to dummy proc, then release
it in startup process" method. If we don't want to rely on that,
PostPrepare_Locks is an example of how to transfer lock ownership from
one process to another correctly.

--  Heikki Linnakangas EnterpriseDB   http://www.enterprisedb.com