Re: WAL "low watermark" during base backup

On Sep9, 2011, at 13:48 , Magnus Hagander wrote:
> On Fri, Sep 9, 2011 at 13:40, Dimitri Fontaine <dimitri@2ndquadrant.fr> wrote:
>> Magnus Hagander <magnus@hagander.net> writes:
>>>> If you must have this then make pg_basebackup copy xlog files
>>>> regularly during the backup. That way your backup can take forever and
>>>> your primary disk won't fill up. In many cases it actually will take
>>>> forever, but at least we don't take down the primary.
>>>
>>> There is a patch to do something like that as well sitting on the CF
>>> page. I don't believe one necessarily excludes the other.
>>
>> I'm not getting why we need the later one when we have this older one?
>
> One of them is for the simple case. It requires a single connection to
> the server, and it supports things like writing to tarfiles and
> compression.
>
> The other one is more compelx. It uses multiple connections (one for
> the base, one for the xlog), and as such doesn't support writing to
> files, only directories.

I guess the real question is, why can't we stream the WALs as they are
generated instead of at the end even over a single connection and when
writing tarfiles?

Couldn't we send all available WAL after each single data-file instead
of waiting for all data files to be transferred before sending WAL?

best regards,
Florian Pflug