Re: recover corrupt DB?

Craig Ringer wrote:
> Peter Eisentraut wrote:
>> On Thursday 23 April 2009 18:30:27 Dan Armbrust wrote:
>>> I had a test system (read as not backed up, sigh) which had the disk
>>> go full while PostgreSQL was loaded, consequently, PostgreSQL will no
>>> longer start.
>>>
>>> It is logging an error about detecting an invalid shutdown, trying to
>>> replay something, and then an error about not being able to open a
>>> file it is looking for.
>> Knowing what file would help analyze this.  In general, pg_resetxlog would be
>> the tool to try here.  Don't panic yet. ;-)
>
> I've been wondering about this for a while. Why does Pg end up with the
> database in an unusable, unrecoverable state after a disk-full error? Is
> there no way it can efficiently defend against issues writing to the
> WAL? Is it, in fact, issues with appending to the current WAL segment
> that're the problem anyway?
>
> This may come up even on fairly well managed databases if users have
> direct access. To me, with a largely user-and-admin perspective, it
> seems like something that really should be handled a bit more cleanly.
>
> --
> Craig Ringer
>
Hmm...

On all our servers we have a cron job that runs daily and reports disk usage stats.
Maybe you need something similar.