Re: repmgr cannot bring up the standby database after switchover manaully

 

In my recent experience, there was no issue starting the old primary—it came up normally. However, it resulted in a split-brain situation where the old primary continued to accept both read and write operations while still assuming the other two nodes were replicas.

Hi Tayyab,

A split-brain is definitely an unexpected behavior. After issuing a failover or switchover command, always check the exit code to ensure it was successful. If not, you should find in the command output or in the postgresql logs an indication of what went wrong.

Seems that either the previous primary couldn't be shutdown or repmgr failed somehow to change it to a standby. Repmgr sets the node's role by creating the standby.signal file in the data directory. Upon startup, if Postgres finds the signal file, it will assume the standby role (providing the postgresql.conf file has the correct configuration too). I can only theorize here, but maybe repmgr failed to write the signal file in $PGDATA either due to lack of permissions or a network failure.

The exact output would help in figuring out what went wrong.

Regards,

Fernando

El mié, 1 oct 2025 a la(s) 4:04 p.m., Tayyab Fayyaz (tayyab.humayl@gmail.com) escribió:

Hello Fernando,

In my recent experience, there was no issue starting the old primary—it came up normally. However, it resulted in a split-brain situation where the old primary continued to accept both read and write operations while still assuming the other two nodes were replicas.

This issue occurred with the following environment:

• OS version: RHEL 8.10

• Postgres DB version: 14.9

• repmgr version: 5.5.0

Tayyab

On Wed, Oct 1, 2025 at 11:52 AM Fernando Hevia <fhevia@gmail.com> wrote:

I have 2 postgresql servers. One is the primary and another one is the standby. I am trying to setup repmgr to do the switchover manually. Passwordless ssh have been setup for postgres ID on both servers.

I use this command "repmgr standby switchover --log-level=DEBUG --verbose". The standy database is able to promote to be the primary. For the previous primary database, it was shutdown. It was not able to bring up as standby by repmgr.  

In a switchover the primary server is shutdown and restarted as a standby server after the newly promoted primary (former secondary) node has been started.

If the primary did not start, there must have been an issue since this is not the standard behavior for a switchover command.

Have you checked the Postgres log file for the previous primary? You should find the startup failure cause in the log.

Regards,

Fernando

 

El mié, 1 oct 2025 a la(s) 7:30 a.m., Chris Lee (clee.hk@gmail.com) escribió:

Hi Tayyab,

Thanks for your information . I also want to find out whether that is the default behavior,  or I am not configuring repmgr correctly.

Regards,

Chris

On Wed, 1 Oct 2025, 18:12 Imran Khan, <imran.k.23@gmail.com> wrote:

Hi Tayyab,

 Is this a default behavior? We have 4 nodes cluster but never had issue in switchovers. 

Thanks, 

Imran

On Wed, Oct 1, 2025, 1:10 PM Tayyab Fayyaz <tayyab.humayl@gmail.com> wrote:

Hello Chris,

I faced this issue it will not add automatically as standby you have to add it manually.

But I wrote a script which perform to add old primary as standby once it's back online.

Tayyab

On Wed, 1 Oct 2025, 3:02 pm Chris Lee, <clee.hk@gmail.com> wrote:

Hi all,

I have 2 postgresql servers. One is the primary and another one is the standby. I am trying to setup repmgr to do the switchover manually. Passwordless ssh have been setup for postgres ID on both servers.

I use this command "repmgr standby switchover --log-level=DEBUG --verbose". The standy database is able to promote to be the primary. For the previous primary database, it was shutdown. It was not able to bring up as standby by repmgr.  

Does anyone encounter this issue before? Thanks a lot for any suggestions.

Here is my OS and DB versions:

OS version: CentOS Stream release 8
Postgres DB version:  15.12
rempmgr version: 5.5.0

Here is the repmgr conf files:
>>>>>
node_id=1  # Use 2 on standby
node_name='primary'
conninfo='host=centos804 user=repmgr dbname=repmgr password=xxx connect_timeout=15'
use_primary_conninfo_password=true
data_directory='/var/lib/pgsql/15/data'  # Adjust for your setup
pg_bindir='/usr/pgsql-15/bin'
service_start_command = 'sudo systemctl start postgresql-15'
service_stop_command  = 'sudo systemctl stop postgresql-15'
<<<<<

>>>>>
node_id=2  # Use 2 on standby
node_name='standby'
conninfo='host=centos803 user=repmgr dbname=repmgr password=xxx connect_timeout=15'
use_primary_conninfo_password=true
data_directory='/var/lib/pgsql/15/data'  # Adjust for your setup
pg_bindir='/usr/pgsql-15/bin'
service_start_command = 'sudo systemctl start postgresql-15'
service_stop_command  = 'sudo systemctl stop postgresql-15'
<<<<<

Regards,
Chris