Обсуждение: PGbackrest very slow!

Поиск
Список
Период
Сортировка

PGbackrest very slow!

От
Pär Mattsson
Дата:
Hi!
I have  3 node postgres cluster with repmgr and pgpool.

Pgbackrest is configed like this
We use pgbackrest with this config
=======================================

[global]

backup-standby=y

repo1-path=/postgresql/backup

repo1-retention-full=2

start-fast=y

 

[PG]

pg1-path=/postgresql/data

pg2-host=node2

pg2-host-user=postgres

pg2-path=/postgresql/data

pg3-host=node3

pg3-host-user=postgres

pg3-path=/postgresql/data

=================================================

I run the backuplike this


/usr/bin/pgbackrest --log-level-console=detail --stanza=PG --type=full backup

 


The backup of 50GB took 40 min.

It is not a network issue. I tested to transfer 2GB between two host and it took 4 sec.


I think backup-standby=y slows down the backup out of all minds.


Br Pär

 

Re: PGbackrest very slow!

От
Ilya Kosmodemiansky
Дата:
Hi Pär,

On Thu, Oct 23, 2025 at 10:59 AM Pär Mattsson <par.x.mattsson@gmail.com> wrote:
>
> Hi!
> I have  3 node postgres cluster with repmgr and pgpool.
>
> Pgbackrest is configed like this
> We use pgbackrest with this config
> =======================================

It is a quite basic config,

> It is not a network issue. I tested to transfer 2GB between two host and it took 4 sec.
>
>
> I think backup-standby=y slows down the backup out of all minds.
>

so if you are sure it is not a network issue, your guess could be
right. You can try to compare with backup from primary and if it is
faster, it could be archiving process on standby which slows you down.
You can try to improve it with more processes [1] and with better zst
compression [2]. Async archiving [3] could also help

>
> Br Pär
>
>

[1] https://pgbackrest.org/configuration.html#section-general/option-process-max
[2] https://pgbackrest.org/configuration.html#section-general/option-compress
[3] https://pgbackrest.org/configuration.html#section-archive/option-archive-async


best regards,
Ilya Kosmodemiansky
CEO, Founder

Data Egret GmbH
Your remote PostgreSQL DBA team
T.: +49 6821 919 3297
ik@dataegret.com



Re: PGbackrest very slow!

От
Scott Ribe
Дата:
> On Oct 23, 2025, at 2:59 AM, Pär Mattsson <par.x.mattsson@gmail.com> wrote:
>
> ...
> The backup of 50GB took 40 min.
> It is not a network issue. I tested to transfer 2GB between two host and it took 4 sec.
>
> I think backup-standby=y slows down the backup out of all minds.
>
> Br Pär

pgbackrest defaults to gzip compression, which is slow--try lz4 and see what happens




Re: PGbackrest very slow!

От
Pär Mattsson
Дата:
Thanks I will test that!
Br Pär

Mvh Pär

Från: Scott Ribe <scott_ribe@elevated-dev.com>
Skickat: Thursday, October 23, 2025 3:14:20 PM
Till: Pär Mattsson <par.x.mattsson@gmail.com>
Kopia: pgsql-admin <pgsql-admin@postgresql.org>
Ämne: Re: PGbackrest very slow!
 
> On Oct 23, 2025, at 2:59 AM, Pär Mattsson <par.x.mattsson@gmail.com> wrote:
>
> ...
> The backup of 50GB took 40 min.
> It is not a network issue. I tested to transfer 2GB between two host and it took 4 sec.
>
> I think backup-standby=y slows down the backup out of all minds.
>
> Br Pär

pgbackrest defaults to gzip compression, which is slow--try lz4 and see what happens

Re: PGbackrest very slow!

От
Ron Johnson
Дата:
On Thu, Oct 23, 2025 at 9:14 AM Scott Ribe <scott_ribe@elevated-dev.com> wrote:
> On Oct 23, 2025, at 2:59 AM, Pär Mattsson <par.x.mattsson@gmail.com> wrote:
>
> ...
> The backup of 50GB took 40 min.
> It is not a network issue. I tested to transfer 2GB between two host and it took 4 sec.
>
> I think backup-standby=y slows down the backup out of all minds.
>
> Br Pär

pgbackrest defaults to gzip compression, which is slow--try lz4 and see what happens

lz4 is faster, but not 1440x faster.

--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!

Re: PGbackrest very slow!

От
Ken Marshall
Дата:
On Thu, Oct 23, 2025 at 10:21:37AM -0400, Ron Johnson wrote:
> On Thu, Oct 23, 2025 at 9:14 AM Scott Ribe <scott_ribe@elevated-dev.com>
> wrote:
> 
> > > On Oct 23, 2025, at 2:59 AM, Pär Mattsson <par.x.mattsson@gmail.com>
> > wrote:
> > >
> > > ...
> > > The backup of 50GB took 40 min.
> > > It is not a network issue. I tested to transfer 2GB between two host and
> > it took 4 sec.
> > >
> > > I think backup-standby=y slows down the backup out of all minds.
> > >
> > > Br Pär
> >
> > pgbackrest defaults to gzip compression, which is slow--try lz4 and see
> > what happens
> >
> 
> lz4 is faster, but not 1440x faster.
> 

But it might be 24x faster, using 2s/GB. :)

--Ken



Re: PGbackrest very slow!

От
adnan salam
Дата:
Hi,

Increase the Parallel processes for pg back rest backup.

Regards,
Adnan


On Thu, 23 Oct 2025 at 19:22, Ron Johnson
<ronljohnsonjr@gmail.com> wrote:
On Thu, Oct 23, 2025 at 9:14 AM Scott Ribe <scott_ribe@elevated-dev.com> wrote:
> On Oct 23, 2025, at 2:59 AM, Pär Mattsson <par.x.mattsson@gmail.com> wrote:
>
> ...
> The backup of 50GB took 40 min.
> It is not a network issue. I tested to transfer 2GB between two host and it took 4 sec.
>
> I think backup-standby=y slows down the backup out of all minds.
>
> Br Pär

pgbackrest defaults to gzip compression, which is slow--try lz4 and see what happens

lz4 is faster, but not 1440x faster.

--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!

Re: PGbackrest very slow!

От
Pär Mattsson
Дата:
Thanks 

Mvh Pär

Från: adnan salam <adnan_pk_1@yahoo.com>
Skickat: Thursday, October 23, 2025 7:54:40 PM
Till: ronljohnsonjr@gmail.com <ronljohnsonjr@gmail.com>; Pgsql-admin <pgsql-admin@lists.postgresql.org>
Ämne: Re: PGbackrest very slow!
 
Hi,

Increase the Parallel processes for pg back rest backup.

Regards,
Adnan


On Thu, 23 Oct 2025 at 19:22, Ron Johnson
<ronljohnsonjr@gmail.com> wrote:
On Thu, Oct 23, 2025 at 9:14 AM Scott Ribe <scott_ribe@elevated-dev.com> wrote:
> On Oct 23, 2025, at 2:59 AM, Pär Mattsson <par.x.mattsson@gmail.com> wrote:
>
> ...
> The backup of 50GB took 40 min.
> It is not a network issue. I tested to transfer 2GB between two host and it took 4 sec.
>
> I think backup-standby=y slows down the backup out of all minds.
>
> Br Pär

pgbackrest defaults to gzip compression, which is slow--try lz4 and see what happens

lz4 is faster, but not 1440x faster.

--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!

Re: PGbackrest very slow!

От
"DINESH NAIR"
Дата:
Hi,

Kindly try changing below mentioned parameters.



Parallel-process=      -- based on the CPU CORE
compress-type=lz4
compress-level=       -- based on the CPU CORE


Thanks & Regards

Dinesh Nair


From: Pär Mattsson <par.x.mattsson@gmail.com>
Sent: Thursday, October 23, 2025 7:21 PM
To: Scott Ribe <scott_ribe@elevated-dev.com>
Cc: pgsql-admin <pgsql-admin@postgresql.org>
Subject: Re: PGbackrest very slow!
 
You don't often get email from par.x.mattsson@gmail.com. Learn why this is important
Caution: This email was sent from an external source. Please verify the sender’s identity before clicking links or opening attachments.
Thanks I will test that!
Br Pär

Mvh Pär

Från: Scott Ribe <scott_ribe@elevated-dev.com>
Skickat: Thursday, October 23, 2025 3:14:20 PM
Till: Pär Mattsson <par.x.mattsson@gmail.com>
Kopia: pgsql-admin <pgsql-admin@postgresql.org>
Ämne: Re: PGbackrest very slow!
 
> On Oct 23, 2025, at 2:59 AM, Pär Mattsson <par.x.mattsson@gmail.com> wrote:
>
> ...
> The backup of 50GB took 40 min.
> It is not a network issue. I tested to transfer 2GB between two host and it took 4 sec.
>
> I think backup-standby=y slows down the backup out of all minds.
>
> Br Pär

pgbackrest defaults to gzip compression, which is slow--try lz4 and see what happens

Re: PGbackrest very slow!

От
Pär Mattsson
Дата:
Hi! 

[global]

repo1-path=/postgresql/backup

repo1-retention-full=2

start-fast=y

process-max=4


When I deleted  backup-standby=y and increased process-max=4 , then the backup took 7 min of 50GB.


I will test all your tips! Thanks alot!

Br Pär


Den fre 24 okt. 2025 kl 04:15 skrev DINESH NAIR <Dinesh_Nair@iitmpravartak.net>:
Hi,

Kindly try changing below mentioned parameters.



Parallel-process=      -- based on the CPU CORE
compress-type=lz4
compress-level=       -- based on the CPU CORE


Thanks & Regards

Dinesh Nair


From: Pär Mattsson <par.x.mattsson@gmail.com>
Sent: Thursday, October 23, 2025 7:21 PM
To: Scott Ribe <scott_ribe@elevated-dev.com>
Cc: pgsql-admin <pgsql-admin@postgresql.org>
Subject: Re: PGbackrest very slow!
 
Caution: This email was sent from an external source. Please verify the sender’s identity before clicking links or opening attachments.
Thanks I will test that!
Br Pär

Mvh Pär

Från: Scott Ribe <scott_ribe@elevated-dev.com>
Skickat: Thursday, October 23, 2025 3:14:20 PM
Till: Pär Mattsson <par.x.mattsson@gmail.com>
Kopia: pgsql-admin <pgsql-admin@postgresql.org>
Ämne: Re: PGbackrest very slow!
 
> On Oct 23, 2025, at 2:59 AM, Pär Mattsson <par.x.mattsson@gmail.com> wrote:
>
> ...
> The backup of 50GB took 40 min.
> It is not a network issue. I tested to transfer 2GB between two host and it took 4 sec.
>
> I think backup-standby=y slows down the backup out of all minds.
>
> Br Pär

pgbackrest defaults to gzip compression, which is slow--try lz4 and see what happens