Обсуждение: cannot read block errors !
Hi All; we're getting these errors on our system today: Sep 9 00:53:45 ux-rqs04-atl2 ri_load[5846]: Caught Exception! Shutting down: ERROR: could not read block 703769 of relation 37698/17627/38565: Input/output error CONTEXT: SQL statement "SELECT 1 FROM ONLY "our_schema"."big_table" x WHERE "id" OPERATOR(pg_catalog.=) $1 FOR SHARE OF x" Sep 9 15:34:36 ux-rqs04-atl2 ri_load[23717]: Caught Exception! Shutting down: PANIC: right sibling's left-link doesn't match: block 1377627 links to 3120580 instead of expected 1736900 in index "no_dups" CONTEXT: COPY big_table, line 1563: "157945230 x/1739441607/false/p_1378267744=0" \copy: unexpected response (7) Anyone have any ideas what's gone wrong and how to fix it? Thanks in advance
Kevin Kempter <kevink@consistentstate.com> writes:
> we're getting these errors on our system today:
> Sep 9 00:53:45 ux-rqs04-atl2 ri_load[5846]: Caught Exception!
> Shutting down: ERROR: could not read block 703769 of relation
> 37698/17627/38565: Input/output error CONTEXT: SQL statement "SELECT
> 1 FROM ONLY "our_schema"."big_table" x WHERE "id" OPERATOR(pg_catalog.=) $1
> FOR SHARE OF x"
> Sep 9 15:34:36 ux-rqs04-atl2 ri_load[23717]: Caught Exception! Shutting
> down: PANIC: right sibling's left-link doesn't match: block 1377627
> links to 3120580 instead of expected 1736900 in index "no_dups" CONTEXT: COPY
> big_table, line 1563: "157945230 x/1739441607/false/p_1378267744=0"
> \copy: unexpected response (7)
> Anyone have any ideas what's gone wrong and how to fix it?
"Input/output error" generally means you've got hardware problems.
Have you looked into the kernel logs to see if there are device driver
complaints?
The other one indicates index corruption, which I might think suggested
a Postgres bug if I hadn't seen the first one. You can fix that index
with REINDEX, but you'd better deal with the underlying issue first
before it corrupts data that's not so easily fixed.
regards, tom lane
On Wednesday 09 September 2009 12:22:29 Tom Lane wrote: > Kevin Kempter <kevink@consistentstate.com> writes: > > we're getting these errors on our system today: > > > > Sep 9 00:53:45 ux-rqs04-atl2 ri_load[5846]: Caught Exception! > > Shutting down: ERROR: could not read block 703769 of relation > > 37698/17627/38565: Input/output error CONTEXT: SQL statement "SELECT > > 1 FROM ONLY "our_schema"."big_table" x WHERE "id" OPERATOR(pg_catalog.=) > > $1 FOR SHARE OF x" > > > > Sep 9 15:34:36 ux-rqs04-atl2 ri_load[23717]: Caught Exception! Shutting > > down: PANIC: right sibling's left-link doesn't match: block 1377627 > > links to 3120580 instead of expected 1736900 in index "no_dups" CONTEXT: > > COPY big_table, line 1563: "157945230 > > x/1739441607/false/p_1378267744=0" \copy: unexpected response (7) > > > > Anyone have any ideas what's gone wrong and how to fix it? > > "Input/output error" generally means you've got hardware problems. > Have you looked into the kernel logs to see if there are device driver > complaints? > > The other one indicates index corruption, which I might think suggested > a Postgres bug if I hadn't seen the first one. You can fix that index > with REINDEX, but you'd better deal with the underlying issue first > before it corrupts data that's not so easily fixed. > > regards, tom lane Thanks, in fact it was a HW issue. we did a REINDEX and stopped the db so we could fix the HW issues.