Re: production server down

Joe Conway <mail@joeconway.com> writes:
> Any theories on how we screwed up?

I hesitate to suggest this, but maybe a cron job blindly copying data
from point A to point B?

I'm not sure that that could entirely explain the facts.  My
recollection of the xlog.c logic is that the pg_control file is read
into shared memory during postmaster boot, and after that it's
write-only: at checkpoint times we update the file image in shared
memory and then write it out to pg_control.

Offhand my bets would revolve around (a) multiple postmasters trying to
run the same PGDATA directory (we have interlocks to protect against
this, but I have no faith that they work against an NFS-mounted data
directory), or (b) you somehow wiped a PGDATA directory and restored it
from backup tapes underneath a running postmaster.
        regards, tom lane