Re: Implementing incremental backup

On 6/19/13 11:02 AM, Claudio Freire wrote:
> On Wed, Jun 19, 2013 at 7:13 AM, Tatsuo Ishii <ishii@postgresql.org> wrote:
>>
>> For now, my idea is pretty vague.
>>
>> - Record info about modified blocks. We don't need to remember the
>>    whole history of a block if the block was modified multiple times.
>>    We just remember that the block was modified since the last
>>    incremental backup was taken.
>>
>> - The info could be obtained by trapping calls to mdwrite() etc. We need
>>    to be careful to avoid such blocks used in xlogs and temporary
>>    tables to not waste resource.
>>
>> - If many blocks were modified in a file, we may be able to condense
>>    the info as "the whole file was modified" to reduce the amount of
>>    info.
>>
>> - How to take a consistent incremental backup is an issue. I can't
>>    think of a clean way other than "locking whole cluster", which is
>>    obviously unacceptable. Maybe we should give up "hot backup"?
>
>
> I don't see how this is better than snapshotting at the filesystem
> level. I have no experience with TB scale databases (I've been limited
> to only hundreds of GB), but from my limited mid-size db experience,
> filesystem snapshotting is pretty much the same thing you propose
> there (xfs_freeze), and it works pretty well. There's even automated
> tools to do that, like bacula, and they can handle incremental
> snapshots.

A snapshot is not the same as an incremental backup; it presents itself as a full copy of the filesystem. Actually,
sinceit's on the same underlying storage a snapshot isn't really a good backup at all.
 

The proposal (at least as I read it) is to provide a means to easily deal with *only* the data that has actually
*changed*since the last backup was taken.
 
-- 
Jim C. Nasby, Data Architect                       jim@nasby.net
512.569.9461 (cell)                         http://jim.nasby.net