Re: parallel mode and parallel contexts

On Fri, Jan 16, 2015 at 5:10 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> New patch attached.  I'm going to take the risk of calling this v1
> (previous versions have been 0.x), since I've now done something about
> the heavyweight locking issue, as well as fixed the message-looping
> bug Amit pointed out.  It doubtless needs more work, but it's starting
> to smell a bit more like a real patch.

Here's a new version.  Andres mentioned previously that he thought it
would be a good idea to commit the addition of
BackgroundWorkerInitializeConnectionByOid() separately, as he's had
cause to want it independently of the rest of this stuff.  It would be
useful for pg_background, too.  So I've broken that out into a
separate patch here (bgworker-by-oid.patch) and will commit that RSN
unless somebody thinks it's a bad idea for some reason.  AFAICS it
should be uncontroversial.

The main patch (parallel-mode-v2.patch) has evolved just a bit more
since the previous version.  It now arranges for all libraries
libraries which we'd dynamically loaded into the original process to
be loaded into the worker as well, which fixes a possible failure when
those libraries define custom GUCs that need to be properly restored.
I'm wondering how much more there really is to do here.  From the
parallel sequential scan discussion, and off-list conversations with
Amit, it's becoming clear to me that there are a bunch of things that
are generic to query planning and execution that this patch doesn't
currently consider, so Amit's handling them in the parallel sequential
scan patch.  It seems likely that some of that stuff should be pulled
out of that patch and formed into some more generic infrastructure.

But I don't favor putting that stuff in this patch, because there are
already useful things you can do with what I've got here.  If you want
to write a parallel version of some SQL-callable function, for
example, you can do that without anything more than this.  You could
probably also parallelize utility commands with just this
infrastructure.  The stuff that may need to be pulled out of Amit's
patch is generic to query planning and execution, but probably not for
other applications of server-side parallelism.  That seems like a
relatively coherent place to draw a line.  So I'm feeling like it
might be just as well to push this much into the tree and move on to
the next set of problems.  Of course, I don't want to preempt anyone's
desire to review and comment on this further, either.

The final patch attached her (parallel-dummy-v2.patch) has been
updated slightly to incorporate some prefetching logic.  It's still
just demo code and is not intended for commit.  I'm not sure whether
the prefetching logic can actually be made to improve performance,
either; if anyone feels like playing with that and reporting results
back, that would be swell.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company