Re: Incremental results from libpq

On Nov 9, 2005, at 2:09 PM, Scott Lamb wrote:
> Cool. I think I'll get my own interface hashed out in my kludgey  
> way, then look at the broader need if it's a success.

I've done some of this. I've got the start of a new Python<- >PostgreSQL interface at
<http://www.slamb.org/svn/repos/trunk/
 
projects/apypg/>.

Currently, it just "peeks" at connection->result after polling, grabs  
a bunch of results and handles them, then empties the resultset  
kludgily. (If it has a different idea of PGRESULT_DATA_BLOCKSIZE than  
libpq, it'll probably crash.)

Aside from the nastiness of playing with libpq's internal data  
structures, it doesn't handle the case Tom mentioned particularly  
well. It will handle an undetermined fraction of the rows before the  
error, then error out. I think it should handle _all_ the rows it  
gets before the error consistently.

> My first idea, though, is to add a callback interface - "got the  
> RowDescription", "got a DataRow" - and make the storage of stuff  
> tuples in PGresult optional. (Maybe pqAddTuple would just be the  
> default callback.)

Actually, I'm not sure about this anymore. There's a complication -  
if I want to do something fancy while handling a row - a Python  
exception, C++ exception, thread cancellation - I don't think there's  
any good way to do that in a callback structure.

Another complication is the limitation of one active resultset. libpq  
needs to be extended to support cursors other than the unnamed one  
('') in order to handle other statements as it iterates over an  
incremental result set.

-- 
Scott Lamb <http://www.slamb.org/>