Re: Implementing incremental backup

> On Wed, Jun 19, 2013 at 6:20 PM, Stephen Frost <sfrost@snowman.net> wrote:
>> * Claudio Freire (klaussfreire@gmail.com) wrote:
>>> I don't see how this is better than snapshotting at the filesystem
>>> level. I have no experience with TB scale databases (I've been limited
>>> to only hundreds of GB), but from my limited mid-size db experience,
>>> filesystem snapshotting is pretty much the same thing you propose
>>> there (xfs_freeze), and it works pretty well. There's even automated
>>> tools to do that, like bacula, and they can handle incremental
>>> snapshots.
>>
>> Large databases tend to have multiple filesystems and getting a single,
>> consistent, snapshot across all of them while under load is..
>> 'challenging'.  It's fine if you use pg_start/stop_backup() and you're
>> saving the XLOGs off, but if you can't do that..
> 
> Good point there.
> 
> I still don't like the idea of having to mark each modified page. The
> WAL compressor idea sounds a lot more workable. As in scalable.

Why do you think WAL compressor idea is more scalable? I really want
to know why. Besides the unlogged tables issue, I can accept the idea
if WAL based solution is much more efficient. If there's no perfect,
ideal solution, we need to prioritize things. My #1 priority is
allowing to create incremental backup against TB database, and the
backup file should be small enough and the time to create it is
acceptable. I just don't know why scanning WAL stream is much cheaper
than recording modified page information.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp