Re: Allowing parallel pg_restore from pipe

On Wed, Apr 24, 2013 at 03:33:42PM -0400, Andrew Dunstan wrote:
> 
> On 04/23/2013 07:53 PM, Timothy Garnett wrote:
> >Hi All,
> >
> >Currently the -j option to pg_restore, which allows for
> >parallelization in the restore, can only be used if the input file
> >is a regular file and not, for ex., a pipe.  However this is a
> >pretty common occurrence for us (usually in the form of pg_dump |
> >pg_restore to copy an individual database or some tables thereof
> >from one machine to another).  While there's no good way to
> >parallelize the data load steps when reading from a pipe, the
> >index and constraint building can still be parallelized and as
> >they are generally CPU bound on our machines we've found quite a
> >bit of speedup from doing so.
> >
> >Attached is two diffs off of the REL9_2_4 tag that I've been
> >using.  The first is a simple change that serially loads the data
> >section before handing off the remainder of the restore to the
> >existing parallelized restore code (the .ALT. diff).  The second
> >which gets more parallelization but is a bit more of a change uses
> >the existing dependency analysis code to allow index building etc.
> >to occur in parallel with data loading. The data loading tasks are
> >still performed serially in the main thread, but non-data loading
> >tasks are scheduled in parallel as their dependencies are
> >satisfied (with the caveat that the main thread can only dispatch
> >new tasks between data loads).
> >
> >Anyways, the question is if people think this is generally useful.
> >If so I can clean up the preferred choice a bit and rebase it off
> >of master, etc.
> 
> 
> I don't think these are bad ideas at all, and probably worth doing.
> Note that there are some fairly hefty changes affecting this code in
> master, so your rebasing could be tricky.

Is there any progress on this:  doing parallel pg_restore from a pipe?

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + Everyone has their own god. +