Dear all,
I am novice to postgres admin world.
I am using postgresl 9.1.11 on Ubutu 12.04 and this is used by application server running on tomcat.
The application slowed down considerably and the postgresql-9.1-main.log reported pgstat wait timeout as below.
2015-07-30 09:38:20 IST WARNING: pgstat wait timeout
2015-07-30 09:50:20 IST WARNING: pgstat wait timeout
2015-07-30 09:50:26 IST WARNING: pgstat wait timeout
2015-07-30 10:00:55 IST WARNING: pgstat wait timeout
2015-07-30 10:04:51 IST WARNING: pgstat wait timeout
2015-07-30 10:06:51 IST WARNING: pgstat wait timeout
2015-07-30 10:10:21 IST WARNING: pgstat wait timeout
2015-07-30 10:12:21 IST WARNING: pgstat wait timeout
I attempted application stop and database shutdown using init scripts followed by start.
The database failed to restart with the following error.
2015-07-30 14:14:17 IST LOG: received immediate shutdown request
2015-07-30 14:14:17 IST WARNING: terminating connection because of crash of another server process
2015-07-30 14:14:17 IST DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-07-30 14:14:17 IST HINT: In a moment you should be able to reconnect to the database and repeat your command.
2015-07-30 14:14:17 IST LOG: could not send data to client: Broken pipe
2015-07-30 14:14:17 IST WARNING: terminating connection because of crash of another server process
2015-07-30 14:14:17 IST DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2015-07-30 14:14:17 IST HINT: In a moment you should be able to reconnect to the database and repeat your command.
2015-07-30 14:14:17 IST LOG: could not send data to client: Broken pipe
The whole recovery took more than 2.5 hours to complete to come back to a state to accept connections back. This caused considerable disruption in normal service.
Hardware support says after analysis that there is no hardware faults related to network or IO (disks). Also could not figure out any network errors like IP conflicts/IP changes etc as well.
I would like to troubleshoot this further to figure out what caused this problem.
Any help is appreciated in this regard and precautions I can take to understand the problems if occurs in future.
Regards,
James