Re: Extending BASE_BACKUP in replication protocol: incremental backup and backup format

On 01/15/2014 08:46 AM, Jim Nasby wrote:
> On 1/14/14, 7:41 AM, Magnus Hagander wrote:
>>     Yes, it would be necessary to scan the whole database as the LSN
>> to be
>>     checked is kept in PageHeaderData :). Perhaps it is not that
>>     performant, but my initial thought was that perhaps the amount of
>> data
>>     necessary to maintain incremental backups could balance with the
>>     amount of WAL necessary to keep and limit the whole amount on disk.
>>
>> It wouldn't be worse performance wise than a full backup. That one
>> also has to read all the blocks after all... You're decreasing network
>> traffic and client storage, with the same I/O on the server side.
>> Seems worthwhile.
>
> If there's enough demand, it probably wouldn't be that hard to keep a
> copy of the page LSNs in a fork; you only need to ensure that the LSN in
> the fork must be older than the LSN on disk could possibly be, and you
> wouldn't have to update the fork every time.

That's backwards. You need to ensure that the LSN in the fork >= that on 
disk. Otherwise the backup will incorrectly conclude that a page doesn't 
need to be backed up because it hasn't been modified.

> BTW, an incremental backup could possibly be useful as a way to catch a
> streaming replica up that's fallen way behind. The write IO would be
> sequential instead of trying to random-write while processing each WAL
> record.

Yeah. And it would work even if you no longer have all the WAL files 
available.

- Heikki