Re: Concurrency issue in pg_rewind

On 18/09/2020 10:17, Alexander Kukushkin wrote:
> At the same time, pg_rewind due to such "fatal" error leaves PGDATA in
> an inconsistent state with empty pg_control file, this is totally bad
> and easily fixable. We want the specific file to be absent and it is
> already absent, why should it be a fatal error and not warning?

Whenever pg_rewind runs into something unexpected, it fails loudly, so 
that the administrator can re-initialize from a base backup. That's the 
general rule. If a file goes missing while pg_rewind is running, that is 
unexpected. It could be a sign that the server was started concurrently, 
or another pg_rewind was started against it, for example.

I feel that we could make an exception of some sort here, but I'm not 
sure what exactly. I don't feel comfortable just downgrading the 
unexpected ENOENT on unlink() to warning in all cases. Besides, scary 
warnings that you routinely ignore is not good either.

I have a hard time coming up with a general rule and justification 
that's not just "do X because WAL-G does Y". pg_rewind failing because 
WAL-G removed a file unexpectedly is one problem, but another is that 
the restore_command might get confused if a pg_rewind removes a file 
that restore_command needs. This is hard when restore_command does 
things in the background, and there's no communication between the 
background process and pg_rewind.

The general principle is that pg_rewind is equivalent to overwriting the 
target with the source, only faster. Perhaps pg_wal should be an 
exception, and pg_rewind should leave alone any files under pg_wal that 
it doesn't recognize.

- Heikki