Re: UTC4115FATAL: the database system is in recovery mode

Craig Ringer <craig@postnewspapers.com.au> writes:
> On 05/30/2011 10:29 PM, Mathew Samuel wrote:
>> 2011-03-28 10:44:42 UTC3932ERROR: canceling statement due to statement
>> timeout
>> 2011-03-28 10:44:42 UTC3932STATEMENT: vacuum full analyze _zamboni.sl_log_1
>> 2011-03-28 10:44:42 UTC3932PANIC: cannot abort transaction 1827110275,
>> it was already committed
>> 2011-03-28 10:44:42 UTC3566LOG: server process (PID 3932) was terminated
>> by signal 6

> Interesting. It almost looks like a VACUUM FULL ANALYZE was cancelled by
> statement_timeout, couldn't be aborted (assuming it was in fact
> 1827110275) and then the backend crashed with a signal 6 (SIGABRT).

Yeah, that seems highly likely.  There's a long-known problem in the
pre-9.0 implementation of VACUUM FULL, that it marks itself as committed
well before the vacuuming is actually done.  Any error that occurs after
that point results in exactly the above symptom.  There's a hack
solution for that in releases made after Nov 2009, and getting rid of
the problem in a cleaner fashion was one of the motivations for
replacing the VACUUM FULL implementation in 9.0.  But I suppose the OP
is running something not too up-to-date :-(.

> Wow. Four hours recovery with default checkpoint settings.

Ouch ... the reason for that needs investigation.

            regards, tom lane