Обсуждение: CRCs (was Re: [GENERAL] Re: Loading optimization)
"Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes:
>> Actually, I'd expect the CRC check to catch an all-zeroes page (if
>> it fails to complain, then you misimplemented the CRC), so that would
>> be the place to deal with it now.
> I've used standard CRC32 implementation you pointed me to -:)
> But CRC is used in WAL records only.
Oh. I thought we'd agreed that a CRC on each stored disk block would
be a good idea as well. I take it you didn't do that.
Do we want to consider doing this (and forcing another initdb)?
Or shall we say "too late for 7.1"?
regards, tom lane
> "Mikheev, Vadim" <vmikheev@SECTORBASE.COM> writes: > >> Actually, I'd expect the CRC check to catch an all-zeroes page (if > >> it fails to complain, then you misimplemented the CRC), so that would > >> be the place to deal with it now. > > > I've used standard CRC32 implementation you pointed me to -:) > > But CRC is used in WAL records only. > > Oh. I thought we'd agreed that a CRC on each stored disk block would > be a good idea as well. I take it you didn't do that. No, I thought we agreed disk block CRC was way overkill. If the CRC on the WAL log checks for errors that are not checked anywhere else, then fine, but I thought disk CRC would just duplicate the I/O subsystem/disk checks. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> Oh. I thought we'd agreed that a CRC on each stored disk block would
>> be a good idea as well. I take it you didn't do that.
> No, I thought we agreed disk block CRC was way overkill. If the CRC on
> the WAL log checks for errors that are not checked anywhere else, then
> fine, but I thought disk CRC would just duplicate the I/O subsystem/disk
> checks.
A disk-block CRC would detect partially written blocks (ie, power drops
after disk has written M of the N sectors in a block). The disk's own
checks will NOT consider this condition a failure. I'm not convinced
that WAL will reliably detect it either (Vadim?). Certainly WAL will
not help for corruption caused by external agents, away from any updates
that are actually being performed/logged.
regards, tom lane
At 21:55 11/01/01 -0500, Tom Lane wrote: > >Oh. I thought we'd agreed that a CRC on each stored disk block would >be a good idea as well. I take it you didn't do that. > >Do we want to consider doing this (and forcing another initdb)? >Or shall we say "too late for 7.1"? > I thought it was coming too. I'd like to see it - if it's not too hard in this release. ---------------------------------------------------------------- Philip Warner | __---_____ Albatross Consulting Pty. Ltd. |----/ - \ (A.B.N. 75 008 659 498) | /(@) ______---_ Tel: (+61) 0500 83 82 81 | _________ \ Fax: (+61) 0500 83 82 82 | ___________ | Http://www.rhyme.com.au | / \| | --________-- PGP key available upon request, | / and from pgp5.ai.mit.edu:11371 |/
> > But CRC is used in WAL records only. > > Oh. I thought we'd agreed that a CRC on each stored disk block would > be a good idea as well. I take it you didn't do that. > > Do we want to consider doing this (and forcing another initdb)? > Or shall we say "too late for 7.1"? I personally was never agreed to this. Reasons? Vadim
> > No, I thought we agreed disk block CRC was way overkill. If the CRC on > > the WAL log checks for errors that are not checked anywhere else, then > > fine, but I thought disk CRC would just duplicate the I/O subsystem/disk > > checks. > > A disk-block CRC would detect partially written blocks (ie, power drops > after disk has written M of the N sectors in a block). The disk's own > checks will NOT consider this condition a failure. I'm not convinced > that WAL will reliably detect it either (Vadim?). Certainly WAL will Idea proposed by Andreas about "physical log" is implemented! Now WAL saves whole data blocks on first after checkpoint modification. This way on recovery modified data blocks will be first restored *as a whole*. Isn't it much better than just detection of partially writes? Only one type of modification isn't covered at the moment - updated t_infomask of heap tuples. > not help for corruption caused by external agents, away from any updates > that are actually being performed/logged. What do you mean by "external agents"? Vadim