Re: production server down

Alvaro Herrera wrote:
> I can't help remembering the fact that the init script executes an
> initdb automatically if it finds an empty data directory (the ones I
> know of at least -- does the one you are running?).  Maybe what happened
> was that it found the empty mount point, executed an initdb, and then
> the NFS drive came online.  Later, the pg_control file was sync'ed to
> the "empty database" settings.  It'd be interesting to know if the
> mount point does have some files on it.

Good point! I'll take a look at the first opportunity.

> These values (from the corrupt pg_control file) are strange:
> 
>>pg_control last modified:             Tue Dec 14 15:39:26 2004
>>Time of latest checkpoint:            Tue Nov  2 17:05:32 2004
> 
> Maybe the latest checkpoint date has some interesting bit pattern that
> could explain it somehow.
> 

The last modified corresponds to just prior to the PANIC. See the logs:

2004-12-14 15:39:26 LOG:  received smart shutdown request
2004-12-14 15:39:26 LOG:  shutting down
2004-12-14 15:39:28 PANIC:  could not open file 
"/replica/pgdata/pg_xlog/0000000000000000" (log file 0, segment 0): No 
such file or directory

The Tue Nov  2 17:05:32 2004 seems to be related to the *previous* 
restart; from /var/log/messages:

8<----------------------------------
...
Nov  2 17:04:20 csdfds1 syslogd 1.4.1: restart.
...
Nov  2 17:05:22 csdfds1 su: pam_unix2: session started for user 
postgres, service su

...
Nov  2 17:05:33 csdfds1 su: (to postgres) root on /dev/pts/5
Nov  2 17:05:33 csdfds1 su: pam_unix2: session started for user 
postgres, service su
Nov  2 17:05:33 csdfds1 su: pam_unix2: session finished for user 
postgres, service su
...
8<----------------------------------

Can you make any sense out of that?

Joe