Re: [HACKERS] libpq Alternate Row Processor

On 2/3/17 3:53 PM, Kyle Gearhart wrote:
> The guts of pqRowProcessor in libpq does a good bit of work to maintain the internal data structure of a PGresult.
Thereare a few use cases where the caller doesn't need the ability to access the result set row by row, column by
columnusing PQgetvalue.  Think of an ORM that is just going to copy the data from PGresult for each row into its own
structures.
>
> I've got a working proof of concept that allows the caller to attach a callback that pqRowProcessor will call instead
ofgoing thru its own routine.  This eliminates all the copying of data from the PGconn buffer to a PGresult buffer and
thenultimately a series of PQgetvalue calls by the client.  The callback allows the caller to receive each row's data
directlyfrom the PGconn buffer.
 
>
> It would require exposing struct pgDataValue in libpq-fe.h.  The prototype for the callback pointer would be:
> int (*PQrowProcessorCB)(PGresult*, const PGdataValue*, int col_count, void *user_data);
>
> My initial testing shows a significant performance improvement.  I'd like some opinions on this before wiring up a
performanceproof and updating the documentation for a formal patch submission.
 

I just did essentially the same thing for SPI (use a callback to allow 
the caller to handle the tuple instead of shoving it into a tuplestore). 
A simple test in plpython showed a 460% improvement.
-- 
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532)