Re: WIP/PoC for parallel backup

On Tue, Apr 21, 2020 at 5:18 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Apr 21, 2020 at 1:00 PM Asif Rehman <asifr.rehman@gmail.com> wrote:
> >
> > I did some tests a while back, and here are the results. The tests were done to simulate
> > a live database environment using pgbench.
> >
> > machine configuration used for this test:
> > Instance Type:    t2.xlarge
> > Volume Type  :    io1
> > Memory (MiB) :    16384
> > vCPU #           :    4
> > Architecture    :    X86_64
> > IOP                 :    16000
> > Database Size (GB) :    102
> >
> > The setup consist of 3 machines.
> > - one for database instances
> > - one for pg_basebackup client and
> > - one for pgbench with some parallel workers, simulating SELECT loads.
> >
> >                                    basebackup | 4 workers | 8 Workers  | 16 workers
> > Backup Duration(Min):       69.25    |  20.44      | 19.86          | 20.15
> > (pgbench running with 50 parallel client simulating SELECT load)
> >
> > Backup Duration(Min):       154.75   |  49.28     | 45.27         | 20.35
> > (pgbench running with 100 parallel client simulating SELECT load)
> >
>
> Thanks for sharing the results, these show nice speedup!  However, I
> think we should try to find what exactly causes this speed up.  If you
> see the recent discussion on another thread related to this topic,
> Andres, pointed out that he doesn't think that we can gain much by
> having multiple connections[1].  It might be due to some internal
> limitations (like small buffers) [2] due to which we are seeing these
> speedups.  It might help if you can share the perf reports of the
> server-side and pg_basebackup side.
>

Just to be clear, we need perf reports both with and without patch-set.

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com