The database system is in recovery mode

Our database just experienced the problem in the subject line. After the
error, the database was still up, but would issue the error to any new
connections. The stats collector process, a vacuum and one other
connection were all in an uninterruptable state and the machine had to be
rebooted.

 Could this be the linux kernel randomly killing processes under heavy
load issue?  I've seen that happen on other machines before, but in those
cases the kernel logged when it was killing processes in syslog... There
were no messages in syslog in this case.

System is postgresql 7.2.1 on redhat 7.2. Here's the logs:

2003-05-01 16:54:08 DEBUG:  server process (pid 2599) was terminated by signal 11
2003-05-01 16:54:08 DEBUG:  terminating any other active server processes
2003-05-01 16:54:08 NOTICE:  Message from PostgreSQL backend:
        The Postmaster has informed me that some other backend
        died abnormally and possibly corrupted shared memory.
        I have rolled back the current transaction and am
        going to terminate your database system connection and exit.
        Please reconnect to the database system and repeat your query.

After a bunch of these, the database goes in recovery mode:

2003-05-01 16:54:08 FATAL 1:  The database system is in recovery mode


Then after the machine is rebooted and while it is starting up, there is
these messages:

2003-05-01 17:35:49 DEBUG:  ReadRecord: unexpected pageaddr 21/37D94000 in log file 33, segment 63, offset 14237696
2003-05-01 17:35:49 DEBUG:  redo done at 21/3FD92564

I presume this is rerunning the WAL? Is the message serious...could there
be database corruption or just lost transactions?


Thanks for any help.


Regards,

Trevor Astrope
astrope@e-corp.net