RE: xlog loose ends, continued

> > Should we try to read log up to the *physical end* - ie end of last
> > log file - regardless invalid CRC-s/zero pages with attempt to
> > re-apply interim valid records? (Or do we already do this?)
> 
> That doesn't seem like a good idea --- once we fail to read an XLOG
> record, it's probably best to stop there rather than continue on.
> I think we want to try for a consistent recovery to a past point in
> time (ie, wherever the xlog gap is) not a partial recovery to a later
> time.

No way for consistent recovery if there is gap in log due to
disk write re-ordering anyway (and we can't know what was
the reason of the gap). I thought that you wanted apply as much of log
as we can. If you don't then I missed your point in first message:

> xlog.c's ReadRecord formerly contained code that would zero 
> out the rest of the log segment (and delete the next log segment,
> if any) upon detecting a missing or corrupted xlog record.
> I removed that code because I considered it horribly dangerous
> where it was. If there is anything wrong with either the xlog or
> pg_control's pointers to it, that code was quite capable of wiping
> out all hope of recovery *and* all evidence of what went wrong. ^^^^^^^^^^^^^^^^^^^^^^^^

So, if we are not going to re-apply as much valid records as
we can read from log then zeroing is no more dangerous than
SUI in headers. But I totaly agreed that SUI is much better.

> > Anyway I like idea of StartUpID in page headers - this will help
> > if some log files disappeared. Should we add CRC to page header?
> 
> That seems like overkill. I was hoping to keep the page 
> header overhead at eight bytes. We could do that either by storing just
> the two LSBs of StartUpID (and doing the sequence checking mod 64k) or
> by reducing the magic number to two bytes so there's room for four bytes
of
> StartUpID.  I think I like the first alternative better --- comments?

I don't think a few additional bytes in header is a problem.
BTW, why not use CRC32 in header instead of magic?
Or just StartUpID instead of magic if you don't want to calculate
CRC for header - xlp_magic doesn't seem to be more useful than SUI.

> > Hm, maybe XLogFileInit should initialize files with StartUpID & CRC
> > in pages? We would avoid reading empty files.
> 
> We already stop when we hit a zeroed page (because it's not got the
> right magic number).  That seems sufficient.

What if the next page after zeroed one is correct (due to write
re-ordering)?
(But I take back SUI+CRC in XLogFileInit - useless -:))

Vadim