Re: streaming replication, "frozen snapshot backup on it" and missing relfile (postgres 9.2.3 on xfs + LVM)

On 14.05.2013 23:47, Benedikt Grundmann wrote:
> The only thing that is *new* is that we took the snapshot from the
> streaming replica.  So again my best guess as of now is that if the
> database crashes while it is in streaming standby a invalid disk state can
> result during during the following startup (in rare and as of now unclear
> circumstances).

A bug is certainly possible. There isn't much detail here to debug with, 
I'm afraid. Can you share the full logs on all three systems? I'm 
particularly interest

> You seem to be quite convinced that it must be LVM can you elaborate why?

Well, you said that there was a file in the original filesystem, but not 
in the snapshot. If you didn't do anything in between, then surely the 
snapshot is broken, if it skipped a file. Or was the file created in the 
original filesystem after the snapshot was taken? You probably left out 
some crucial details on how exactly the snapshot and rsync are 
performed. Can you share the scripts you're using?

Can you reproduce this problem with a new snapshot? Do you still have 
the failed snapshot unchanged?

- Heikki