Re: [HACKERS] Faster methods for getting SPI results

On 3/2/17 8:03 AM, Peter Eisentraut wrote:
> On 12/20/16 23:14, Jim Nasby wrote:
>> I've been looking at the performance of SPI calls within plpython.
>> There's a roughly 1.5x difference from equivalent python code just in
>> pulling data out of the SPI tuplestore. Some of that is due to an
>> inefficiency in how plpython is creating result dictionaries, but fixing
>> that is ultimately a dead-end: if you're dealing with a lot of results
>> in python, you want a tuple of arrays, not an array of tuples.
>
> There is nothing that requires us to materialize the results into an
> actual list of actual rows.  We could wrap the SPI_tuptable into a
> Python object and implement __getitem__ or __iter__ to emulate sequence
> or mapping access.

Would it be possible to have that just pull tuples directly from the 
executor? The overhead of populating the tuplestore just to drain it 
again can become quite significant, and AFAICT it's completely unnecessary.

Unfortunately, I think adding support for that would be even more 
invasive, which is why I haven't attempted it. On the flip side, I 
believe that kind of an interface would be usable by plpgsql, whereas 
the DestReceiver approach is not (AFAICT).
-- 
Jim Nasby, Chief Data Architect, OpenSCG