I'm investigating a problem that happened last night and I would
appreciate any recommendations. The logs indicate that the disks were
full, but I truly doubt that since we only use about 14GB out of the
available 65GB.
I found entries like this in the logs:
ERROR: could not write block 2354 of temporary file: No space left on device
HINT: Perhaps out of disk space?
....
ERROR: could not extend relation "parent_table": No space left on device
HINT: Check free disk space.
....
LOG: could not close temporary statistics file "/var/lib/postgres/data/global/pgstat.tmp.1464": No space left on
device
According to the logs, the problem went away after a reboot. I wonder
if the kernel or the RAID device got confused and postgres was simply
echoing what it was told. We run a couple hundred postgres servers and
we have not seen this before (except when the disks truly were full).
Everything is in the root filesystem, which has plenty of room.
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 67756724 14344392 49970408 23% /
tmpfs 1034768 0 1034768 0% /dev/shm
PostgreSQL 7.4.7 on i386-pc-linux-gnu, compiled by GCC i386-linux-gcc (GCC) 3.3.5 (Debian 1:3.3.5-12)
Debian Sarge with Linux kernel 2.4.27-2-686-smp
Dell PowerEdge 1800
Dell MegaRAID PERC 4/DC RAID Controller, 128MB cache w/BBU
2x SEAGATE Cheetah 10K.7 ST373207LC in RAID 1 (mirroring)
Folks are a little jittery because our customers do very heavy
business this month and we don't want frantic support calls when we
should be drinking eggnog.
-Mike