Re: Streaming a base backup from master

On Sat, Sep 04, 2010 at 02:42:40PM +0100, Greg Stark wrote:
> On Fri, Sep 3, 2010 at 8:30 PM, Martijn van Oosterhout
> <kleptog@svana.org> wrote:
> >
> > rsync is not rocket science. All you need is for the receiving end to
> > send a checksum for each block it has. The server side does the same
> > checksum and for each block sends back "same" or "new data".
>
> Well rsync is closer to rocket science than that. It does rolling
> checksums and can handle data being moved around, which vacuum does do
> so it's probably worthwhile.

Not sure. When vacuum moves rows around the chance that it will move
rows as a block and that the line pointers will be the same is
practically nil. I don't think rsync will pick up on blocks the size of
a typical row. Vacuum changes the headers so you never have a copied
block.

> *However* I tihnk you're all headed in the wrong direction here. I
> don't think rsync is what anyone should be doing with their backups at
> all. It still requires scanning through *all* your data even if you've
> only changed a small percentage (which it seems is the use case you're
> concerned about) and it results in corrupting your backup while the
> rsync is in progress and having a window with no usable backup. You
> could address that with rsync --compare-dest but then you're back to
> needing space and i/o for whole backups every time even if you're only
> changing small parts of the database.

If you're working from a known good version of the database at some
point, yes you are right you have more interesting options. If you
don't you want something that will fix it.

Have a nice day,
--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patriotism is when love of your own people comes first; nationalism,
> when hate for people other than your own comes first.
>                                       - Charles de Gaulle