Обсуждение: [BUG] standby node can not provide service even it replays all logfiles

On Mon, Oct 21, 2019 at 4:13 AM Thunder <thunder1@126.com> wrote:
> Can we fix this issue like the following patch?
>
> $git diff src/backend/access/transam/xlog.c
> diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
> index 49ae97d4459..0fbdf6fd64a 100644
> --- a/src/backend/access/transam/xlog.c
> +++ b/src/backend/access/transam/xlog.c
> @@ -8365,7 +8365,7 @@ CheckRecoveryConsistency(void)
>          * run? If so, we can tell postmaster that the database is consistent now,
>          * enabling connections.
>          */
> -       if (standbyState == STANDBY_SNAPSHOT_READY &&
> +       if ((standbyState == STANDBY_SNAPSHOT_READY || standbyState == STANDBY_SNAPSHOT_PENDING) &&
>                 !LocalHotStandbyActive &&
>                 reachedConsistency &&
>                 IsUnderPostmaster)

I think that the issue you've encountered is design behavior.  In
other words, it's intended to work that way.

The comments for the code you propose to change say that we can allow
connections once we've got a valid snapshot. So presumably the effect
of your change would be to allow connections even though we don't have
a valid snapshot.

That seems bad.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hello.

At Tue, 22 Oct 2019 20:42:21 +0800 (CST), Thunder  <thunder1@126.com> wrote in 
> Update the patch.
> 
> 1. The STANDBY_SNAPSHOT_PENDING state is set when we replay the first XLOG_RUNNING_XACTS and the sub transaction ids
areoverflow.
 
> 2. When we log XLOG_RUNNING_XACTS in master node, can we assume that all xact IDS < oldestRunningXid are considered
finished?

Unfortunately we can't. Standby needs to know that the *standby's*
oldest active xid exceeds the pendig xmin, not master's. And it is
already processed in ProcArrayApplyRecoveryInfo. We cannot assume that
the oldest xids are not same on the both side in a replication pair.

> 3. If we can assume this, when we replay XLOG_RUNNING_XACTS and change standbyState to STANDBY_SNAPSHOT_PENDING, can
werecord oldestRunningXid to a shared variable, like procArray->oldest_running_xid?
 
> 4. In standby node when call GetSnapshotData if procArray->oldest_running_xid is valid, can we set xmin to be
procArray->oldest_running_xid?
> 
> Appreciate any suggestion to this issue.

At 2019-10-22 01:27:58, "Robert Haas" <robertmhaas@gmail.com> wrote:
>On Mon, Oct 21, 2019 at 4:13 AM Thunder <thunder1@126.com> wrote:
..
> >I think that the issue you've encountered is design behavior.  In
> >other words, it's intended to work that way.
> >
> >The comments for the code you propose to change say that we can allow
> >connections once we've got a valid snapshot. So presumably the effect
> >of your change would be to allow connections even though we don't have
> >a valid snapshot.
> >
> >That seems bad.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

At Wed, 23 Oct 2019 12:51:19 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in 
> Hello.
> 
> At Tue, 22 Oct 2019 20:42:21 +0800 (CST), Thunder  <thunder1@126.com> wrote in 
> > Update the patch.
> > 
> > 1. The STANDBY_SNAPSHOT_PENDING state is set when we replay the first XLOG_RUNNING_XACTS and the sub transaction
idsare overflow.
 
> > 2. When we log XLOG_RUNNING_XACTS in master node, can we assume that all xact IDS < oldestRunningXid are considered
finished?
> 
> Unfortunately we can't. Standby needs to know that the *standby's*
> oldest active xid exceeds the pendig xmin, not master's. And it is
> already processed in ProcArrayApplyRecoveryInfo. We cannot assume that
> the oldest xids are not same on the both side in a replication pair.

Could we send a full xid list after new standby comes in? Or can
START_REPLICATION return it?

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

At Thu, 24 Oct 2019 17:37:52 +0800 (CST), Thunder  <thunder1@126.com> wrote in 
> Thanks for replay.I feel confused about snapshot.
> 
> At 2019-10-23 11:51:19, "Kyotaro Horiguchi" <horikyota.ntt@gmail.com> wrote:
> >Hello.
> >
> >At Tue, 22 Oct 2019 20:42:21 +0800 (CST), Thunder  <thunder1@126.com> wrote in 
> >> Update the patch.
> >> 
> >> 1. The STANDBY_SNAPSHOT_PENDING state is set when we replay the first XLOG_RUNNING_XACTS and the sub transaction
idsare overflow.
 
> >> 2. When we log XLOG_RUNNING_XACTS in master node, can we assume that all xact IDS < oldestRunningXid are
consideredfinished?
 
> >
> >Unfortunately we can't. Standby needs to know that the *standby's*
> >oldest active xid exceeds the pendig xmin, not master's. And it is
> >already processed in ProcArrayApplyRecoveryInfo. We cannot assume that
> 
> >the oldest xids are not same on the both side in a replication pair.
> 
> 
> This issue occurs when master does not commit the transaction which has lots of sub transactions, while we restart or
createa new standby node.
 
> The standby node can not provide service because of this issue.
> Can the standby have any active xid while it can not provide service?

The problem is not xid, but snapshot, information on what xids are not
committed yet on the master. Standby cannot deterine what rows should
be visible without the information. The xid list is maintained using
incoming commit records and vanishes on restart. So the restarted
standby needs non-subxid-overflown XLOG_RUNNING_XACTS to make sure the
xid list is complete.

> >> 3. If we can assume this, when we replay XLOG_RUNNING_XACTS and change standbyState to STANDBY_SNAPSHOT_PENDING,
canwe record oldestRunningXid to a shared variable, like procArray->oldest_running_xid?
 
> >> 4. In standby node when call GetSnapshotData if procArray->oldest_running_xid is valid, can we set xmin to be
procArray->oldest_running_xid?
> >> 
> >> Appreciate any suggestion to this issue.

So, somehow we need to complete the KnownAssignedTransactionIds even
if there's any subxid-overflown transactions. As mentioned upthread,
I think we have at least the following choices.

- Send back the complete xid list for START REPLICATION command from
  walreceiver.

- The first XLOG_RUNNING_XACTS after a standby comes in while
  subxid-overflown transaction lives.

I think the first is better.

Any suggestions?

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

Mmm..

At Tue, 29 Oct 2019 13:57:19 +0900 (JST), Kyotaro Horiguchi <horikyota.ntt@gmail.com> wrote in 
> So, somehow we need to complete the KnownAssignedTransactionIds even
> if there's any subxid-overflown transactions. As mentioned upthread,
> I think we have at least the following choices.
> 
> - Send back the complete xid list for START REPLICATION command from
>   walreceiver.
> 
> - The first XLOG_RUNNING_XACTS after a standby comes in while
>   subxid-overflown transaction lives.
> 
> I think the first is better.
> 
> Any suggestions?

On second thought, for the first choice, currently we don't have a
means to recall snapshot at arbitrary checkpoint time so it would be
hard to do that. But the second choice doesn't seem a good way..

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center