Re: An idea for parallelizing COPY within one backend

Andrew Dunstan wrote:
> Florian G. Pflug wrote:
>>> Would it be possible to determine when the copy is starting that this 
>>> case holds, and not use the parallel parsing idea in those cases?
>>
>> In theory, yes. In pratice, I don't want to be the one who has to 
>> answer to an angry user who just suffered a major drop in COPY 
>> performance after adding an ENUM column to his table.
>>
> I am yet to be convinced that this is even theoretically a good path to 
> follow. Any sufficiently large table could probably be partitioned and 
> then we could use the parallelism that is being discussed for pg_restore 
> without any modification to the backend at all. Similar tricks could be 
> played by an external bulk loader for third party data sources.

That assumes that some specific bulkloader like pg_restore, pgloader
or similar is used to perform the load. Plain libpq-users would either 
need to duplicate the logic these loaders contain, or wouldn't be able 
to take advantage of fast loads.

Plus, I'd see this as a kind of testbed for gently introducing 
parallelism into postgres backends (especially thinking about sorting 
here). CPU gain more and more cores, so in the long run I fear that we 
will have to find ways to utilize more than one of those to execute a 
single query.

But of course the architectural details need to be sorted out before any 
credible judgement about the feasability of this idea can be made...

regards, Florian Pflug