Re: Extending copy_expert

On 10/13/2014 01:49 PM, Andrea Riciputi wrote:
> Hi all,
> thanks for your suggestions but they don’t fit the use-case at hand.
>
> Regarding using unix2dos, it is quite slow when the file become large and here we are talking of several hundreds of
GB.
>
> Using io.open() is a no way either, since the “newline” kwarg only works for “text” files. This means that all the
datacoming from PG (which are python bytes/str objects) must be converted to Python unicode objects and then back again
tobytes. This (useless) decoding/encoding dance nearly doubles execution times. 

Well I am only seeing about a 30% difference, but I am not working with
anything close to the same size dataset.

>
> My point was not to get the result, which is trivial, but to get it efficiently. Enabling psycopg (or even better
Postgresitself) to write the EOL straight away in C is a much more efficient way to get the task done. 
>
> In my opinion it’d be better to push such a feature upstream to PG, but even having it in psycopg could be a good
compromise.Do you have any strong argument against such a feature in psycopg? Do you think it’d be better part of PG
itself?If so, how do you think I can gain support in the pgsql-hackers ml? 

I have been on Postgres related lists for many years now and this is the
first time I have seen this issue brought up. This is probably why you
are not seeing a lot of enthusiasm about a patch. It is seen as a one-of
solution, not generally applicable. I would say your best bet would be
to present is as an extension, either a Psycopg extension/extra or a
Postgres contrib module/extension.

>
> Thanks,
> a.
>
> On 13 Oct 2014, at 15:45, Adrian Klaver <adrian.klaver@aklaver.com> wrote:
>



--
Adrian Klaver
adrian.klaver@aklaver.com