Обсуждение: Full backup failed

Поиск
Список
Период
Сортировка

Full backup failed

От
Rajesh Kumar
Дата:
Hi,

Backup policy is weekly full backup using pgbackrest. Yesterday backups failed and got below error. Please help

time="2024-02-18T18:23:17Z" level=info msg="crunchy-pgbackrest starts"

2time="2024-02-18T18:23:17Z" level=info msg="debug flag set to false"

3time="2024-02-18T18:23:17Z" level=info msg="backrest backup command requested"

4time="2024-02-18T18:23:17Z" level=info msg="command to execute is [pgbackrest backup --stanza=db --repo=1 --type=full]"

5time="2024-02-18T18:26:59Z" level=info msg="output=[]"

6time="2024-02-18T18:26:59Z" level=info msg="stderr=[ERROR: [082]: WAL segment 0000003F0000050A0000000C was not archived before the 60000ms timeout\n HINT: check the archive_command to ensure that all options are correct (especially --stanza).\n HINT: check the PostgreSQL server log for errors.\n HINT: run the 'start' command if the stanza was previously stopped.\n]"

7time="2024-02-18T18:26:59Z" level=fatal msg="command terminated with exit code 82"

Re: Full backup failed

От
Don Seiler
Дата:
On Sun, Feb 18, 2024 at 10:05 PM Rajesh Kumar <rajeshkumar.dba09@gmail.com> wrote:

6time="2024-02-18T18:26:59Z" level=info msg="stderr=[ERROR: [082]: WAL segment 0000003F0000050A0000000C was not archived before the 60000ms timeout\n HINT: check the archive_command to ensure that all options are correct (especially --stanza).\n HINT: check the PostgreSQL server log for errors.\n HINT: run the 'start' command if the stanza was previously stopped.\n]"

7time="2024-02-18T18:26:59Z" level=fatal msg="command terminated with exit code 82"

What is your database instance's archive_command set to? Is that normally succeeding?

--
Don Seiler
www.seiler.us

Re: Full backup failed

От
Laurenz Albe
Дата:
On Mon, 2024-02-19 at 09:35 +0530, Rajesh Kumar wrote:
> 6time="2024-02-18T18:26:59Z" level=info msg="stderr=[ERROR: [082]: WAL segment 0000003F0000050A0000000C was not
archivedbefore the 60000ms timeout\n HINT: check the archive_command to ensure that all options are correct (especially
--stanza).\nHINT: check the PostgreSQL server log for errors.\n HINT: run the 'start' command if the stanza was
previouslystopped.\n]" 

Look into the PostgreSQL log for messages from the "archive_comand".

Yours,
Laurenz Albe



Re: Full backup failed

От
Rajesh Kumar
Дата:
Archive command is set to pgbackrest --stanza=db archive-push "%p"

There were no such issue since last year.

In postgres logs, don't see anything related to archive. But I see, WARNING,0100, aborting backup due to backend exiting before pg_backup_stop was called....pgbackrest[backup] client backend, 58568621736954596

On Mon, 19 Feb 2024, 12:53 Laurenz Albe, <laurenz.albe@cybertec.at> wrote:
On Mon, 2024-02-19 at 09:35 +0530, Rajesh Kumar wrote:
> 6time="2024-02-18T18:26:59Z" level=info msg="stderr=[ERROR: [082]: WAL segment 0000003F0000050A0000000C was not archived before the 60000ms timeout\n HINT: check the archive_command to ensure that all options are correct (especially --stanza).\n HINT: check the PostgreSQL server log for errors.\n HINT: run the 'start' command if the stanza was previously stopped.\n]"

Look into the PostgreSQL log for messages from the "archive_comand".

Yours,
Laurenz Albe

Re: Full backup failed

От
Michael Banck
Дата:
Hi,

On Mon, Feb 19, 2024 at 01:13:36PM +0530, Rajesh Kumar wrote:
> Archive command is set to pgbackrest --stanza=db archive-push "%p"

That looks right.
 
> There were no such issue since last year.
> 
> In postgres logs, don't see anything related to archive. But I see,
> WARNING,0100, aborting backup due to backend exiting before pg_backup_stop
> was called....pgbackrest[backup] client backend, 58568621736954596

This error is due to the timeout that pgbackrest backend suffered.

I think one reason pgbackrest backup could time out after 60s is WAL
pileup, i.e. it tries to check whether the current WAL archive was
successfully archived in 60s but if there is a huge lag in archiving
(for whatever reason) the current WAL segment might not get archived for
another 60 seconds and the backup fails.

So if you have that data in monitoring, check whether archiving was
maybe lagging behind, or otherwise, check whether there are a lot of
.ready files in the pg_wal/archive_status sub-directory right now and/or
possibly just try the backup again.


Michael



Re: Full backup failed

От
Amitpal Singh
Дата:
You can do one thing delete the stanza and re create it , it will work .

On Mon, Feb 19, 2024, 1:14 PM Rajesh Kumar <rajeshkumar.dba09@gmail.com> wrote:
Archive command is set to pgbackrest --stanza=db archive-push "%p"

There were no such issue since last year.

In postgres logs, don't see anything related to archive. But I see, WARNING,0100, aborting backup due to backend exiting before pg_backup_stop was called....pgbackrest[backup] client backend, 58568621736954596

On Mon, 19 Feb 2024, 12:53 Laurenz Albe, <laurenz.albe@cybertec.at> wrote:
On Mon, 2024-02-19 at 09:35 +0530, Rajesh Kumar wrote:
> 6time="2024-02-18T18:26:59Z" level=info msg="stderr=[ERROR: [082]: WAL segment 0000003F0000050A0000000C was not archived before the 60000ms timeout\n HINT: check the archive_command to ensure that all options are correct (especially --stanza).\n HINT: check the PostgreSQL server log for errors.\n HINT: run the 'start' command if the stanza was previously stopped.\n]"

Look into the PostgreSQL log for messages from the "archive_comand".

Yours,
Laurenz Albe