Re: block-level incremental backup

Hi Vignesh,

Please find my comments inline below:

1) If relation file has changed due to truncate or vacuum.
    During incremental backup the new files will be copied.
    There are chances that both the old  file and new file
    will be present. I'm not sure if cleaning up of the
    old file is handled.

When an incremental backup is taken it either copies the file in its entirety if

a file is changed more than 90%, or writes .partial with changed blocks bitmap

and actual data. For the files that are unchanged, it writes 0 bytes and still

creates a .partial file for unchanged files too. This means there is a .partitial

file for all the files that are to be looked up in full backup.

While composing a synthetic backup from incremental backup the pg_combinebackup

tool will only look for those relation files in full(parent) backup which are

having .partial files in the incremental backup. So, if vacuum/truncate happened

between full and incremental backup, then the incremental backup image will not

have a 0-length .partial file for that relation, and so the synthetic backup

that is restored using pg_combinebackup will not have that file as well.

 

2) Just a small thought on building the bitmap,
    can the bitmap be built and maintained as
    and when the changes are happening in the system.
    If we are building the bitmap while doing the incremental backup,
    Scanning through each file might take more time.
    This can be a configurable parameter, the system can run
    without capturing this information by default, but if there are some
    of them who will be taking incremental backup frequently this
    configuration can be enabled which should track the modified blocks.

IIUC, this will need changes in the backend. Honestly, I think backup is a

maintenance task and hampering the backend for this does not look like a good

idea. But, having said that even if we have to provide this as a switch for some

of the users, it will need a different infrastructure than what we are building

here for constructing bitmap, where we scan all the files one by one. Maybe for

the initial version, we can go with the current proposal that Robert has suggested,

and add this switch at a later point as an enhancement.