Re: Unable to startup postgres: Could not read from file"pg_clog/00EC"

Thank you for the feedback, Alvaro.

Unfortunately, the database is no longer "dumpable". We were able to do 
a pg_dump yesterday morning (12 hours after the crash + purging the 
pg_clog) but if we try one now, we get the following error:

    unexpected chunk number 1 (expected 0) for toast value 8282331 in 
pg_toast_38651

Looking at our data, there seem to be 6 tables that have corrupt 
records. Doing a SELECT * for one of those records, will return a 
similar error:

    missing chunk number 0 for toast value 8288522 in pg_toast_5572299


What is the best way to go from here? Is tracking down these corrupt 
records and deleting them the best / only solution?
Is there a way to determine of there are issues with new data (after the 
crash)?

Any help and advice is very much appreciated.

Thanks,


Nick Renders


On 5 Feb 2020, at 12:51, Alvaro Herrera wrote:

> On 2020-Feb-05, Nick Renders wrote:
>
>> Is there anything specific I should check in our postgres 
>> installation /
>> database to make sure it is running ok now? Anyway to see what the
>> consequences were of purging that one pg_clog file?
>
> Losing pg_clog files is pretty bad, and should not happen; then again,
> this might have been something else (ie. the file was maybe not lost).
> That said, wrongly overwriting files is even worse.
>
> By zeroing an existing pg_clog file, you marked a bunch of 
> transactions
> as aborted.  Your data is now probably inconsistent, if not downright
> corrupt.  I would be looking for my most recent backup ...
>
> If you're very lucky, your database might be pg_dumpable.  I would try
> that, followed by restoring it in a separate clean instance and seeing
> what happens.
>
> -- 
> Álvaro Herrera                https://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services