Re: finding changed blocks using WAL scanning

On Sat, Apr 20, 2019 at 12:09:42AM -0400, Robert Haas wrote:
> On Thu, Apr 18, 2019 at 5:47 PM Bruce Momjian <bruce@momjian.us> wrote:
> > How would the modblock file record all the modified blocks across
> > restarts and crashes?  I assume that 1G of WAL would not be available
> > for scanning.  I suppose that writing a modblock file to some PGDATA
> > location when WAL is removed would work since during a crash the
> > modblock file could be updated with the contents of the existing pg_wal
> > files.
> 
> I think you've got to prevent the WAL from being removed until a
> .modblock file has been written.  In more detail, you should (a) scan
> all the WAL segments that will be summarized in the .modblock file,
> (b) write the file under a temporary name, (c) fsync it, (d) rename it
> into place, (e) fsync it again, and (f) then allow those WAL segments
> to be removed, if they are otherwise eligible to be removed.

Makes sense.  So when you are about to remove WAL, you create the
.modblock files for all complete WAL files and only create a new one
when you are about to remove a WAL that was not in a previous .modblock
file.

-- 
  Bruce Momjian  <bruce@momjian.us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

+ As you are, so once was I.  As I am, so you will be. +
+                      Ancient Roman grave inscription +