BUG #13368: standby cluster immediately promotes after pg_basebackup from previously promoted master

The following bug has been logged on the website:

Bug reference:      13368
Logged by:          Feike Steenbergen
Email address:      feikesteenbergen@gmail.com
PostgreSQL version: 9.4.2
Operating system:   Debian 8.0 x86_64
Description:

We sometimes see a standby server promoting itself to master immediately.

Analysis shows us that the master still has a promote file in the PGDATA
directory. We assume the presence of the promote file (which is copied
by pg_basebackup) is triggering the promotion.

The master itself previously was a standby server. The promotion was done
using pg_ctl promote. Analysis of our logs show that we sent pg_ctl promote
twice to this cluster, this also is reflected in the server log,
"server promoting" shows up twice.

Some testing shows us that in some cases, when pg_ctl promote is called
multiple
times, a promote file is left in the PGDATA directory, even though the
cluster
has been succesfully promoted and is accepting read/write queries.

I tested it in the following manner:

- stop the cluster
- remove promote file if exists
- create recovery.conf containing only "standby_mode = on"
- ensure cluster is ready to accept read only queries
- send pg_ctl promote twice (simultaneously)
- see if the promote file still exists

I ran this test for a while:

111 times no promote file existed afterwards
 41 times a promote file still existed

A short excerpt from the running of the script in a loop:

        20150528120345 Promote file does not exists
        20150528120346 Promote file exists
        20150528120348 Promote file does not exists
        20150528120349 Promote file exists
        20150528120350 Promote file does not exists
        20150528120351 Promote file does not exists
        20150528120352 Promote file does not exists
        20150528120354 Promote file exists

We will try to workaround this issue by ensuring we do not send multiple
promote request using pg_ctl to the same cluster.

We judge this to be a bug, as we would assume the promote file to be
removed
after successful promotion of a cluster.

Version: PostgreSQL 9.4.2 on x86_64-unknown-linux-gnu, compiled by gcc
(Debian 4.9.2-10) 4.9.2, 64-bit