Re: [psycopg] speed concerns with executemany()

On 12/23/16 7:04 PM, Daniele Varrazzo wrote:
> In my knowledge there is no libpq
> function to execute repeatedly a prepared statement with an array of
> arrays of values

I don't think there is, but the community might welcome one. Though, I
think the first question you'll get is "why not just use async commands".

Somewhat related to this, I recently heard a complaint that
Pandas.DataFrame.from_sql() was so incredibly slow on Postgres that
people were moving to storing data in things like Hadoop just for doing
data science modelling. I'm not looking into that right now because it
turns out moving data into plpython is almost as slow as via libpq in
some simple testing (which strikes me as rather absurd), but it'd be
worth looking into improving the libpq case as well. I suspect the way
to do that would be to create a tuple of lists, instead of a list of
tuples (or dicts, or whatever). Single row mode[1] would probably be
helpful too...

1: https://www.postgresql.org/docs/current/static/libpq-single-row-mode.html
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)