Re: Streaming base backups

On 01/05/2011 02:54 PM, Magnus Hagander wrote:
[..]
> Some remaining thoughts and must-dos:
>
> * Compression: Do we want to be able to compress the backups server-side? Or
>    defer that to whenever we get compression in libpq? (you can still tunnel it
>    through for example SSH to get compression if you want to) My thinking is
>    defer it.
> * Compression: We could still implement compression of the tar files in
>    pg_streamrecv (probably easier, possibly more useful?)

hmm compression would be nice but I don't think it is required for this 
initial implementation.


> * Windows support (need to implement readlink)
> * Tar code is copied from pg_dump and modified. Should we try to factor it out
>    into port/? There are changes in the middle of it so it can't be done with
>    the current calling points, it would need a refactor. I think it's not worth
>    it, given how simple it is.
>
> Improvements I want to add, but that aren't required for basic operation:
>
> * Stefan mentiond it might be useful to put some
> posix_fadvise(POSIX_FADV_DONTNEED)
>    in the process that streams all the files out. Seems useful, as long as that
>    doesn't kick them out of the cache *completely*, for other backends as well.
>    Do we know if that is the case?

well my main concern is that a basebackup done that way might blew up 
the buffercache of the OS causing temporary performance issues.
This might be more serious with an in-core solution than with what 
people use now because a number of backup software and tools (like some 
of the commercial backup solutions) employ various tricks to avoid that.
One interesting tidbit i found was:

http://insights.oetiker.ch/linux/fadvise/

which is very Linux specific but interesting nevertheless...




Stefan