Re: Re: Synch Rep: direct transfer of WAL file from the primary to the standby

Hi,

Thanks for the brilliant comments!

On Wed, Jul 8, 2009 at 4:00 AM, Heikki
Linnakangas<heikki.linnakangas@enterprisedb.com> wrote:
>> There are still some interesting questions in this about exactly how you
>> switch over from "catchup mode" to following the live WAL broadcast.
>> With the above design it would be the master's responsibility to manage
>> that, since presumably the requested start position will almost always
>> be somewhat behind the live end of WAL.  It might be nicer to push that
>> complexity to the slave side, but then you do need two data paths
>> somehow (ie, retrieving the slightly-stale WAL is separated from
>> tracking live events).  Which is what you're saying we should avoid,
>> and I do see the point there.
>
> Yeah, that logic belongs to the master.
>
> We'll want to send message from the master to the slave when the catchup
> is done, so that the slave knows it's up-to-date. For logging, if for no
> other reason.

This seems to be a main difference between us. You and Tom think
that the catchup (transferring the old XLOG file) and WAL streaming
(shipping the latest XLOG records continuously) are performed in
serial by using the same connection. I think that in parallel by using
more than one connection. I'd like to build consensus which design
should be chosen. If my design is worse, I'll change the patch
according to the other design.

In my design, WAL streaming is performed between walsender and
walreceiver. In parallel with that, the startup process requests the
old XLOG file to a normal backend if it's not found during recovery.
If the startup process has reached the WAL streaming start position,
it's guaranteed that all the XLOG files required for recovery exist in
the standby, which means that it's up-to-date. After that, the startup
process replays only the records shipped by WAL streaming.

The advantage of my design is:

- It's guaranteed that the standby can catch up with the primary within a reasonable period.
- We can keep walsender simple. It has only to take care of the latest XLOG records (ie. doesn't need to control the
oldrecords and some history files). And, it doesn't need to calculate whether the standby is already up-to-date or not
bycomparing some LSNs. 
- In the future, in order to make the standby catch up more quickly, we can easily extend the mechanism so that two or
moreold XLOG files might be transferred concurrently by using multiple connections. 

What is your opinion?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center