Обсуждение: Important lag in replication streaming between master and standby
Hello, I’ve got several times this error : "requested WAL segment 000000020000A01A0000004F has already been remove". I think it’sdue to the lag between our master in EU zone and our standby in US zone. How could we configure Postgres to handle this issue? Thx a lot
On Mon, Aug 26, 2019 at 11:21 AM Mai Peng <maily.peng@webedia-group.com> wrote:
Hello,
I’ve got several times this error : "requested WAL segment 000000020000A01A0000004F has already been remove". I think it’s due to the lag between our master in EU zone and our standby in US zone.
How could we configure Postgres to handle this issue?
You can increase wal_keep_segments so that the master keeps more segments around for the standy to stream. This has the risk that you still won't set it high enough and so might not actually solve the problem.
You can use a replication slot, which will force the master to keep the segments until the standby streams them. This has the risk that the pg_wal can grow without bound (if the standby can't keep up, or if it goes offline), and the master will fill up its storage and crash (running out of storage on the pg_wal directory is not handled very gracefully).
You can set up a wal archive, and then set up the replica so it can fetch from that archive as a backstop to streaming. As a bonus, WAL files usually compress very well, and if you compress them during archival you can save a lot of network bandwidth when the standby needs to retrieve them.
Cheers,
Jeff
Hi Mai, It seems that the wal log requested by slave is not available in master’s wal directory. In that case if the DB is archival mode the same wal can be found there and configure slave accordingly so that it can readwal from archive location. Sent from my iPhone > On 26-Aug-2019, at 8:51 PM, Mai Peng <maily.peng@webedia-group.com> wrote: > > Hello, > > I’ve got several times this error : "requested WAL segment 000000020000A01A0000004F has already been remove". I think it’sdue to the lag between our master in EU zone and our standby in US zone. > How could we configure Postgres to handle this issue? > Thx a lot >
On Mon, Aug 26, 2019 at 5:21 PM Mai Peng <maily.peng@webedia-group.com> wrote: > > Hello, > > I’ve got several times this error : "requested WAL segment 000000020000A01A0000004F has already been remove". I think it’sdue to the lag between our master in EU zone and our standby in US zone. > How could we configure Postgres to handle this issue? As others already stated, you can use either wal_keep_segments or replication_slots with advantages and drawbacks on both sides. One thing to point out, also, since you did not provide a lot of information, is that if you have another slave consuming wals, it could remove (archive_cleanup) segments ignoring other slaves. Personally, I would go for a replication slot and see if the lag can be kept under control (i.e., the pg_wal directory does not grow too much). Luca