Обсуждение: Async PQgetResult() question.
Greetings, I'm working with pqlib in asynchronous mode and I have a question about PQgetResult. I have this situation: submit a query via PQsendQuery() flush to the backend with PQflush() set my read descriptor on the socket and enter a select() select returns read_ready on the socket, so call PGconsumeInput() PQisBusy() returns zero, so call PQgetResult() PQgetResult() returns a pointer so do whatever with the result call PQclear() on the result Now what do I do? The docs say that in async mode that PQgetResult() must be called until it returns NULL. But, how do I know that calling PQgetResult() a second, third, fourth, etc. time will not block? When PQisBusy() returns zero, does that mean that PQgetResult() is guaranteed not to block for all results, i.e. until it returns NULL and a new query is issued? Thanks, Matthew
Matthew Hagerty <mhagerty@voyager.net> writes:
> I'm working with pqlib in asynchronous mode and I have a question about
> PQgetResult. I have this situation:
> submit a query via PQsendQuery()
> flush to the backend with PQflush()
I think the flush is not necessary; PQsendQuery should do it.
> set my read descriptor on the socket and enter a select()
> select returns read_ready on the socket, so call PGconsumeInput()
> PQisBusy() returns zero, so call PQgetResult()
> PQgetResult() returns a pointer so do whatever with the result
> call PQclear() on the result
So far so good.
> Now what do I do?
Loop back to the PQisBusy() step. In practice, you are probably doing
this in an event-driven application, and the select() is the place in
the outer loop that blocks for an event. PQconsumeInput followed by
a PQisBusy/PQgetResult/process result loop are just your response
subroutine for an input-ready-on-the-database-connection event.
> The docs say that in async mode that PQgetResult() must
> be called until it returns NULL. But, how do I know that calling
> PQgetResult() a second, third, fourth, etc. time will not block?
When PQisBusy says you can.
> When
> PQisBusy() returns zero, does that mean that PQgetResult() is guaranteed
> not to block for all results, i.e. until it returns NULL and a new query is
> issued?
No, it means *one* result is available (or that a NULL is available, ie,
libpq has detected the end of the query cycle). Its answer will
probably change after you read the result.
regards, tom lane
Thanks for the response Tom! Does anyone ever tell you how much you are appreciated? If not, I will. When I post to pgHackers I know I will get a response (usually) from many knowledgeable people, and for that I thank everyone. But I *always* receive a response from Tom, and for that I am truly and greatly thankful! Now, on with asynchronous query processing! WooHoo! :) At 12:35 PM 7/7/2001 -0400, Tom Lane wrote: >Matthew Hagerty <mhagerty@voyager.net> writes: > > I'm working with pqlib in asynchronous mode and I have a question about > > PQgetResult. I have this situation: > > > submit a query via PQsendQuery() > > flush to the backend with PQflush() > >I think the flush is not necessary; PQsendQuery should do it. The docs warn that PQflush() must be called, here is what is says: "PQflush needs to be called on a non-blocking connection before calling select to determine if a response has arrived. If 0 is returned it ensures that there is no data queued to the backend that has not actually been sent. Only applications that have used PQsetnonblocking have a need for this." Since I use PQsetnonblocking(), I included PQflush(). > > set my read descriptor on the socket and enter a select() > > > select returns read_ready on the socket, so call PGconsumeInput() > > PQisBusy() returns zero, so call PQgetResult() > > PQgetResult() returns a pointer so do whatever with the result > > call PQclear() on the result > >So far so good. > > > Now what do I do? > >Loop back to the PQisBusy() step. In practice, you are probably doing >this in an event-driven application, and the select() is the place in >the outer loop that blocks for an event. PQconsumeInput followed by >a PQisBusy/PQgetResult/process result loop are just your response >subroutine for an input-ready-on-the-database-connection event. This is my primary concern. I'm actually writing a server and I have other events going on. I wrote a small test program and at this point (after reading the first result) I looped back to my select, but the socket never went read-ready again, so the last PQconsumeInput()/PQisBusy()/PQgetResults() was never called to receive the NULL response from PQgetResult(), which is how the docs say I know the query is done. But if I loop back to PQconsumeInput()/PQisBusy(), then I am effectively blocking since I have no way to know that PQconsumeInput() won't block or that the PQisBusy() will ever return zero again. If there is more input to read, i.e. the NULL from PQgetResult(), then how come the socket never goes read-ready again, as in my test program above? Or can I call PQconsumeInput() repeatedly and be guaranteed that it will not block and that I will be able to process the remaining results? > > The docs say that in async mode that PQgetResult() must > > be called until it returns NULL. But, how do I know that calling > > PQgetResult() a second, third, fourth, etc. time will not block? > >When PQisBusy says you can. > > > When > > PQisBusy() returns zero, does that mean that PQgetResult() is guaranteed > > not to block for all results, i.e. until it returns NULL and a new > query is > > issued? > >No, it means *one* result is available (or that a NULL is available, ie, >libpq has detected the end of the query cycle). Its answer will >probably change after you read the result. My main problem is that sockets are slow devices and I have some other disk I/O I have to be doing as well. I know that another call to PQgetResult() will probably return NULL, but if that data has not come from the backend yet then PQgetResult() will block, and I can't sit in a loop calling PQconsumeInput()/PQisBusy() becuse that is just like blocking and I may as well just let PQgetResult() block. It seems that when the query cycle has ended and PQgetResult() would return NULL, the socket should be set read-ready again, but that never happens... Well, at least I did not see it happen. I can send my test code if need be. Thanks, Matthew > regards, tom lane > >---------------------------(end of broadcast)--------------------------- >TIP 6: Have you searched our list archives? > >http://www.postgresql.org/search.mpl
Matthew Hagerty <mhagerty@voyager.net> writes:
> Only applications that have used PQsetnonblocking have a need for this."
> Since I use PQsetnonblocking(), I included PQflush().
Hmm. My opinions about the PQsetnonblocking patch are on record:
it's broken and needs fundamental redesign before it has any chance
of operating reliably. Unless you are sending queries whose text is
many kB (more than your kernel will buffer for one send() call),
I recommend you not use it.
However, that only affects output, not input.
> I wrote a small test program and at this point (after
> reading the first result) I looped back to my select, but the socket never
> went read-ready again, so the last
> PQconsumeInput()/PQisBusy()/PQgetResults() was never called to receive the
> NULL response from PQgetResult(), which is how the docs say I know the
> query is done.
> But if I loop back to PQconsumeInput()/PQisBusy(), then I am effectively
> blocking since I have no way to know that PQconsumeInput() won't block or
> that the PQisBusy() will ever return zero again.
(1) No, you don't need to repeat the PQconsumeInput, unless select still
says read-ready. (You could call it again, but there's no point.)
(2) You should repeat PQisBusy && PQgetResult until one of them fails,
however. What you're missing here is that a single TCP packet might
provide zero, one, or more than one PQgetResult result. You want to
loop until you've gotten all the results you can get from the current
input packet. Then you go back to select(), and eventually you'll see
more backend input and you do another consumeInput and another isBusy/
getResult loop.
(3) PQconsumeInput never blocks. Period. PQgetResult can block, but
it promises not to if an immediately prior PQisBusy returned 0.
regards, tom lane
At 02:13 PM 7/7/2001 -0400, Tom Lane wrote: >Matthew Hagerty <mhagerty@voyager.net> writes: > > Only applications that have used PQsetnonblocking have a need for this." > > > Since I use PQsetnonblocking(), I included PQflush(). > >Hmm. My opinions about the PQsetnonblocking patch are on record: >it's broken and needs fundamental redesign before it has any chance >of operating reliably. Unless you are sending queries whose text is >many kB (more than your kernel will buffer for one send() call), >I recommend you not use it. > >However, that only affects output, not input. If I don't call PQsetnonblocking() will that affect any of the async functions I'm dealing with? I might have insert queries that are rather large and I'm not sure how big my kernel's buffers are (and surely it will be different on other OSes.) > > I wrote a small test program and at this point (after > > reading the first result) I looped back to my select, but the socket never > > went read-ready again, so the last > > PQconsumeInput()/PQisBusy()/PQgetResults() was never called to receive the > > NULL response from PQgetResult(), which is how the docs say I know the > > query is done. > > > But if I loop back to PQconsumeInput()/PQisBusy(), then I am effectively > > blocking since I have no way to know that PQconsumeInput() won't block or > > that the PQisBusy() will ever return zero again. > >(1) No, you don't need to repeat the PQconsumeInput, unless select still >says read-ready. (You could call it again, but there's no point.) > >(2) You should repeat PQisBusy && PQgetResult until one of them fails, >however. What you're missing here is that a single TCP packet might >provide zero, one, or more than one PQgetResult result. You want to >loop until you've gotten all the results you can get from the current >input packet. Then you go back to select(), and eventually you'll see >more backend input and you do another consumeInput and another isBusy/ >getResult loop. Yup, I think that is what I was misunderstanding. I'll modify my loop and see how it goes. >(3) PQconsumeInput never blocks. Period. PQgetResult can block, but >it promises not to if an immediately prior PQisBusy returned 0. > > regards, tom lane Thanks, Matthew
Matthew Hagerty <mhagerty@voyager.net> writes:
> If I don't call PQsetnonblocking() will that affect any of the async
> functions I'm dealing with?
PQsetnonblocking has nothing to do with the
PQconsumeInput/PQisBusy/PQgetResult family of functions. The point of
the latter is to avoid blocking while waiting for input from the
database. The point of PQsetnonblocking is to avoid blocking while
sending stuff to the database.
Now in a TCP environment, the only way send() is going to block is if
you send more stuff than there's currently room for in your kernel's
networking buffers --- which typically are going to be hundreds of K.
I could see needing PQsetnonblocking if you need to avoid blocking
while transmitting COPY IN data to the database ... but for queries
it's harder to credit. Also, unless you are sending more than one
query in a query string, the backend is going to be absorbing the
data as fast as it can anyway; so even if you do block it's only
going to be for a network transit delay, not for database processing.
Personally I've done quite a bit of asynchronous-application coding with
PQconsumeInput &friends, but never felt the need for PQsetnonblocking.
This is why I've not been motivated to try to fix its problems...
regards, tom lane
I said:
> Also, unless you are sending more than one
> query in a query string, the backend is going to be absorbing the
> data as fast as it can anyway; so even if you do block it's only
> going to be for a network transit delay, not for database processing.
Actually, forget the "unless" part -- the backend won't start parsing
the querystring until it's got it all. It just reads the query into
memory as fast as it can, semicolons or no.
regards, tom lane
At 03:46 PM 7/7/2001 -0400, Tom Lane wrote: >Matthew Hagerty <mhagerty@voyager.net> writes: > > If I don't call PQsetnonblocking() will that affect any of the async > > functions I'm dealing with? > >PQsetnonblocking has nothing to do with the >PQconsumeInput/PQisBusy/PQgetResult family of functions. The point of >the latter is to avoid blocking while waiting for input from the >database. The point of PQsetnonblocking is to avoid blocking while >sending stuff to the database. > >Now in a TCP environment, the only way send() is going to block is if >you send more stuff than there's currently room for in your kernel's >networking buffers --- which typically are going to be hundreds of K. >I could see needing PQsetnonblocking if you need to avoid blocking >while transmitting COPY IN data to the database ... but for queries >it's harder to credit. Also, unless you are sending more than one >query in a query string, the backend is going to be absorbing the >data as fast as it can anyway; so even if you do block it's only >going to be for a network transit delay, not for database processing. > >Personally I've done quite a bit of asynchronous-application coding with >PQconsumeInput &friends, but never felt the need for PQsetnonblocking. >This is why I've not been motivated to try to fix its problems... So then how would I code for the exception, i.e. the backend goes down just before or during my call to PQsendQuery()? If I am non-blocking then I can determine that my query did not go (PQsendQuery() or PQflush() returns an error) and attempt to recover. Otherwise, my server could block until PQsendQuery() times out and returns an error. I guess it would depend on how long PQsendQuery() waits to return if there is an error or problem with the backend or the connection to the backend? Matthew
Matthew Hagerty <mhagerty@voyager.net> writes:
> So then how would I code for the exception, i.e. the backend goes down just
> before or during my call to PQsendQuery()? If I am non-blocking then I can
> determine that my query did not go (PQsendQuery() or PQflush() returns an
> error) and attempt to recover.
This is the nasty part of any async client, all right. The case of a
backend crash doesn't bother me particularly: in the first place, you'll
get back a "connection closed" failure quickly, and in the second place,
backend crashes while absorbing query text (as opposed to while
executing a query) are just about unheard of. However, the possibility
of loss of network connectivity is much more dire: it's plausible, and
in most cases you're looking at a very long timeout before the kernel
will decide that the connection is toast and report an error to you.
I'm unconvinced, however, that using PQsetnonblocking improves the
picture very much. Unless the database operations are completely
noncritical to what your app is doing, you're going to be pretty
much dead in the water anyway with a lost connection :-(
In the end you pays your money and you takes your choice. I do
recommend reading my past rants about why PQsetnonblocking is broken
(circa Jan 2000, IIRC) before you put any faith in it. If you end
up deciding that it really is something you gotta have, maybe you'll
be the one to do the legwork to make it reliable.
regards, tom lane
At 11:44 PM 7/7/2001 -0400, Tom Lane wrote: >Matthew Hagerty <mhagerty@voyager.net> writes: > > So then how would I code for the exception, i.e. the backend goes down > just > > before or during my call to PQsendQuery()? If I am non-blocking then I > can > > determine that my query did not go (PQsendQuery() or PQflush() returns an > > error) and attempt to recover. > >This is the nasty part of any async client, all right. The case of a >backend crash doesn't bother me particularly: in the first place, you'll >get back a "connection closed" failure quickly, and in the second place, >backend crashes while absorbing query text (as opposed to while >executing a query) are just about unheard of. However, the possibility >of loss of network connectivity is much more dire: it's plausible, and >in most cases you're looking at a very long timeout before the kernel >will decide that the connection is toast and report an error to you. > >I'm unconvinced, however, that using PQsetnonblocking improves the >picture very much. Unless the database operations are completely >noncritical to what your app is doing, you're going to be pretty >much dead in the water anyway with a lost connection :-( > >In the end you pays your money and you takes your choice. I do >recommend reading my past rants about why PQsetnonblocking is broken >(circa Jan 2000, IIRC) before you put any faith in it. If you end >up deciding that it really is something you gotta have, maybe you'll >be the one to do the legwork to make it reliable. > > regards, tom lane Well, I guess sending a query will have to be my weak link for the moment, heck, that's why we have version releases, right? ;) I'll take your advise and disable PQsetnonblocking for now, but I would like to read your rants and maybe (if I think I can muster the courage), look into fixing PQsetnonblocking. I have never dug around for an IIRC archive before, might you recommend one that contains your rants? Thanks, Matthew
Uh oops! I misread IIRC as (IRC, i.e. Internet Relay Chat or something similar.) It is too early! ;) I'll dig in the archives. Thanks, Matthew At 12:03 PM 7/8/2001 -0400, Matthew Hagerty wrote: >At 11:44 PM 7/7/2001 -0400, Tom Lane wrote: >>Matthew Hagerty <mhagerty@voyager.net> writes: >> > So then how would I code for the exception, i.e. the backend goes down >> just >> > before or during my call to PQsendQuery()? If I am non-blocking then >> I can >> > determine that my query did not go (PQsendQuery() or PQflush() returns an >> > error) and attempt to recover. >> >>This is the nasty part of any async client, all right. The case of a >>backend crash doesn't bother me particularly: in the first place, you'll >>get back a "connection closed" failure quickly, and in the second place, >>backend crashes while absorbing query text (as opposed to while >>executing a query) are just about unheard of. However, the possibility >>of loss of network connectivity is much more dire: it's plausible, and >>in most cases you're looking at a very long timeout before the kernel >>will decide that the connection is toast and report an error to you. >> >>I'm unconvinced, however, that using PQsetnonblocking improves the >>picture very much. Unless the database operations are completely >>noncritical to what your app is doing, you're going to be pretty >>much dead in the water anyway with a lost connection :-( >> >>In the end you pays your money and you takes your choice. I do >>recommend reading my past rants about why PQsetnonblocking is broken >>(circa Jan 2000, IIRC) before you put any faith in it. If you end >>up deciding that it really is something you gotta have, maybe you'll >>be the one to do the legwork to make it reliable. >> >> regards, tom lane > > >Well, I guess sending a query will have to be my weak link for the moment, >heck, that's why we have version releases, right? ;) I'll take your >advise and disable PQsetnonblocking for now, but I would like to read your >rants and maybe (if I think I can muster the courage), look into fixing >PQsetnonblocking. I have never dug around for an IIRC archive before, >might you recommend one that contains your rants? > >Thanks, >Matthew > > >---------------------------(end of broadcast)--------------------------- >TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org