Re: 12.3 replicas falling over during WAL redo

On 2020-Aug-03, Ben Chobot wrote:

> rmgr: Btree       len (rec/tot):     72/    72, tx:   76396065, lsn:
> A0A/AC4204A0, prev A0A/AC420450, desc: INSERT_LEAF off 48, blkref #0: rel
> 16605/16613/60529051 blk 6501
> 
> So then I did:
> 
> /usr/lib/postgresql/12/bin/pg_waldump -p /var/lib/postgresql/12/main/pg_wal/
> 0000000100000A0A000000AB 0000000100000A0A000000AD | grep
> 16605/16613/60529051

Yep. Looking at the ones in block 6501,

> rmgr: Btree       len (rec/tot):     72/    72, tx:   76393394, lsn:
> A0A/AB2C43D0, prev A0A/AB2C4378, desc: INSERT_LEAF off 41, blkref #0: rel
> 16605/16613/60529051 blk 6501

> rmgr: Btree       len (rec/tot):     72/    72, tx:   76396065, lsn:
> A0A/AC4204A0, prev A0A/AC420450, desc: INSERT_LEAF off 48, blkref #0: rel
> 16605/16613/60529051 blk 6501

My question was whether the block has received the update that added the
item in offset 41; that is, is the LSN in the crashed copy of the page
equal to A0A/AB2C43D0?  If it's an older value, then the write above was
lost for some reason.

> pg_waldump: fatal: error in WAL record at A0A/AC5411B0: invalid resource
> manager ID 110 at A0A/AC5411E0
> 
> ...and I have no idea what I'm looking at. I assume/hope the error at the
> end is due to the db shutting down, and nothing to be particularly worried
> about?

Yeah.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services