Re: Corrupted Data ?

On 08/12/2016 08:10 AM, Ioana Danes wrote:
>
>
> On Fri, Aug 12, 2016 at 10:47 AM, Francisco Olarte
> <folarte@peoplecall.com <mailto:folarte@peoplecall.com>> wrote:
>
>     CCing to the list...
>
> Thanks
>
>
>     On Fri, Aug 12, 2016 at 4:10 PM, Ioana Danes <ioanadanes@gmail.com
>     <mailto:ioanadanes@gmail.com>> wrote:
>     >> given 318220 and 318216 are just a bit away ( 4db08/4db0c ), and it
>     >> repeats sporadically, have you ruled out ( by having page
>     checksums or
>     >> other mechanism ) a potential disk read/write error ?
>     >>
>     >>
>     >> > Also the index is correct on db3 as the record in case (with
>     drawid =
>     >> > 318216) is retrieved if I filter by drawid = 318220
>     >>
>     >> Specially if this happens, you may have some slightly bad disks/ram/
>     >> leading to this kind of problems.
>     >>
>     >
>     > Could be. I also had some issues with an rsync between db3 and
>     drdb a week
>     > ago that did not complete for bigger files (> 200MB) and gave me some
>     > corruption messages. Then the system was revbooted and everything
>     seemed
>     > fine but apparently it is not.
>     > I am planning to drop & create the table from a good backup and if
>     that does
>     > not fix the issue then I will rebuild the server.
>
>     I would check whatever logs you can ( syslog or eventlog, smart log,
>     etc.. ) hunting for disk errors ( sometimes they are reported ). This
>     kind of problems, with programs as tested as postgres and rsync, tend
>     to indicate controller/RAM/disk going bad ( in your case it could be
>     caused by a single bit getting flipped in a sector for the data
>     portion of the table, and not being propagated either because it
>     happened after your sync of drdb or because it was synced from the WAL
>     and not the table, or because it was read from the disk cache ).
>
> I agree, unfortunately I did not find any clues about corruption or any
> anomalies in the logs.
> I will work tonight to rebuild that table and see where I go from there.

The db3 database is on a different machine from all the other databases
you set up, correct?

>
> Thanks,
> ioana
>
>     Francisco Olarte.
>
>


--
Adrian Klaver
adrian.klaver@aklaver.com