Re: [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

On Monday, September 17, 2012 12:52:32 PM Heikki Linnakangas wrote:
> On 17.09.2012 12:07, Andres Freund wrote:
> > On Monday, September 17, 2012 10:30:35 AM Heikki Linnakangas wrote:
> >> The user of the facility doesn't need to be aware of record boundaries,
> >> that's the responsibility of the facility. I thought that's exactly the
> >> point of generalizing this thing, to make it unnecessary for the code
> >> that uses it to be aware of such things.
> > 
> > With the proposed API it seems pretty much a requirement to wait inside
> > the callback.
> 
> Or you can return false from the XLogPageRead() callback if the
> requested page is not available. That will cause ReadRecord() to return
> NULL, and you can retry when more WAL is available.
That requires to build quite a bit of knowledge on the outside:
* you need to transport the information that you need more input via some 
external variable/->private_data
* you need to transport at which RecPtr you needed more data
* you need to signal that youre not dealing with an invalid record after 
returning, given both conditions return NULL
* you need to buffer all incoming data somewhere if it comes from the network 
or similar, because at the next call XLgReadRecord will restart reading from 
the beginning


Sorry, if I sound sceptical! If I had your patch in my hands half a year ago I 
would have been very happy, but after building the more generic version that 
can do all of the above (including a compatible XLogReaderReadOne(state)) its a 
bit hard to do that. Not sure if its just the feeling of possibly having wasted 
the time...

Greetings,

Andres
-- Andres Freund                       http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training &
Services