Обсуждение: ECPG thread-safety
As I found out recently, ECPG is not thread-safe. This is a problem for us, as we really need to be able to use ECPG* in a multi-threaded C application. Michael mentioned that Lee is working on fixing these problems, but was not sure when it would be complete. So, my questions are: Lee/anyone, do you have an estimated completion date? If not, can we (myself plus a "real" C programmer) help you? Regards, Philip Yarra. * Yes, we could use libpq, but ECPG minimises our porting from Informix. Rewriting everything to use libpq would be too expensive.
I have put a mailbox file at: ftp://candle.pha.pa.us/pub/postgresql/mypatches where there is a partial implementation of thread-safe ecpg. Maybe you can work with Michael to complete it. --------------------------------------------------------------------------- Philip Yarra wrote: > As I found out recently, ECPG is not thread-safe. This is a problem for us, as > we really need to be able to use ECPG* in a multi-threaded C application. > Michael mentioned that Lee is working on fixing these problems, but was not > sure when it would be complete. > > So, my questions are: > Lee/anyone, do you have an estimated completion date? > If not, can we (myself plus a "real" C programmer) help you? > > Regards, Philip Yarra. > > * Yes, we could use libpq, but ECPG minimises our porting from Informix. > Rewriting everything to use libpq would be too expensive. > > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Fri, 28 Mar 2003 13:00, you wrote: > I have put a mailbox file at: > > ftp://candle.pha.pa.us/pub/postgresql/mypatches > > where there is a partial implementation of thread-safe ecpg. Maybe you > can work with Michael to complete it. Okay, thanks for that. Have grabbed the file, and started reading. I'll also wait to hear from Michael or Lee before going too far on this. Everything I have read so far suggests the issue is related to SQLCA itself... and that would be consistent with what happens in the backend (data is inserted, but threads lock straight after). Can anyone enlighten me further? Thanks and regards, Philip.
Philip, You can find the "patch so far" at: http://services.csl.co.uk/postgresql/ along with a libpq thread-safe patch. The thread referenced by Bruce's email details some of the issues still to be looked at. The main work is integration with the build system (testing for and linking in POSIX threads) and testing. I'm more than happy for you to look at at, I haven't had the time I hoped to complete this. Thanks, Lee. Philip Yarra writes:> As I found out recently, ECPG is not thread-safe. This is a problem for us, as > we really need tobe able to use ECPG* in a multi-threaded C application. > Michael mentioned that Lee is working on fixing these problems,but was not > sure when it would be complete. > > So, my questions are: > Lee/anyone, do you have an estimated completiondate? > If not, can we (myself plus a "real" C programmer) help you?> > Regards, Philip Yarra.> > * Yes, we coulduse libpq, but ECPG minimises our porting from Informix. > Rewriting everything to use libpq would be too expensive.>
I have the libpq patch in my mailbox that just needs configure thread testing code. I will get that in for 7.4. --------------------------------------------------------------------------- Lee Kindness wrote: > Philip, > > You can find the "patch so far" at: > > http://services.csl.co.uk/postgresql/ > > along with a libpq thread-safe patch. > > The thread referenced by Bruce's email details some of the issues > still to be looked at. The main work is integration with the build > system (testing for and linking in POSIX threads) and testing. > > I'm more than happy for you to look at at, I haven't had the time I > hoped to complete this. > > Thanks, Lee. > > Philip Yarra writes: > > As I found out recently, ECPG is not thread-safe. This is a problem for us, as > > we really need to be able to use ECPG* in a multi-threaded C application. > > Michael mentioned that Lee is working on fixing these problems, but was not > > sure when it would be complete. > > > > So, my questions are: > > Lee/anyone, do you have an estimated completion date? > > If not, can we (myself plus a "real" C programmer) help you? > > > > Regards, Philip Yarra. > > > > * Yes, we could use libpq, but ECPG minimises our porting from Informix. > > Rewriting everything to use libpq would be too expensive. > > > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Fri, Mar 28, 2003 at 05:49:04PM +1100, Philip Yarra wrote: > Okay, thanks for that. Have grabbed the file, and started reading. I'll also > wait to hear from Michael or Lee before going too far on this. Don't be surprised if I'm a bit late in answering emails on the list. I'm currently very busy. I'd be interested to get ecpg thread safe. Is this the only problem you're facing with Informix? I'm currently working on other Informix compatibility stuff. And I'd like to get that tested as good as possible. Michael -- Michael Meskes Email: Michael@Fam-Meskes.De ICQ: 179140304 Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!
On Mon, 31 Mar 2003 00:06, Michael Meskes wrote: > Is this the only problem you're facing with Informix? I'm currently > working on other Informix compatibility stuff. And I'd like to get that > tested as good as possible. Yes, I recall reading that you were working on better Informix compatibility - most welcome! Honestly, the changes we needed to make to port from Informix were pretty minor, as I recall: - some of the connection syntax differred - figuring out that OID was not populated in sqlca if not auto-committing (makes sense really) - our original code used non-standard ways to name a PREPAREd statement which threw ECPG (and according the Informix doco shouldn't really have worked for that either!) so I fixed that in both development streams You can probably deduce from this that we are not really using a great deal of functionality in this app - what we have ported so far is a simple logging server. We really won't be porting any more until ECPG is thread-safe. It's a major show-stopper, since all of our apps are built on the same (pthreads) framework. That said, we would love to help test further Informix compatibility stuff - anything that makes our future porting process easier would be welcome! Let us know what you'd like us to test. Diffs are fine. Regards, Philip.
--On Friday, March 28, 2003 10:54:23 -0500 Bruce Momjian <pgman@candle.pha.pa.us> wrote: > > I have the libpq patch in my mailbox that just needs configure thread > testing code. I will get that in for 7.4. I can help with the UnixWare threads stuff. Also, Micheal Meskes now has an account on my UnixWare box. The threads are native on UW, but you need to invoke the cc -Kpthread option. LER > > ------------------------------------------------------------------------- > -- > > Lee Kindness wrote: >> Philip, >> >> You can find the "patch so far" at: >> >> http://services.csl.co.uk/postgresql/ >> >> along with a libpq thread-safe patch. >> >> The thread referenced by Bruce's email details some of the issues >> still to be looked at. The main work is integration with the build >> system (testing for and linking in POSIX threads) and testing. >> >> I'm more than happy for you to look at at, I haven't had the time I >> hoped to complete this. >> >> Thanks, Lee. >> >> Philip Yarra writes: >> > As I found out recently, ECPG is not thread-safe. This is a problem >> > for us, as we really need to be able to use ECPG* in a >> > multi-threaded C application. Michael mentioned that Lee is working >> > on fixing these problems, but was not sure when it would be >> > complete. >> > >> > So, my questions are: >> > Lee/anyone, do you have an estimated completion date? >> > If not, can we (myself plus a "real" C programmer) help you? >> > >> > Regards, Philip Yarra. >> > >> > * Yes, we could use libpq, but ECPG minimises our porting from >> > Informix. Rewriting everything to use libpq would be too expensive. >> > >> > > -- > Bruce Momjian | http://candle.pha.pa.us > pgman@candle.pha.pa.us | (610) 359-1001 > + If your life is a hard drive, | 13 Roberts Road > + Christ can be your backup. | Newtown Square, Pennsylvania > 19073 > > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org > -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: ler@lerctr.org US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
On Mon, Mar 31, 2003 at 11:23:04AM +1100, Philip Yarra wrote: > That said, we would love to help test further Informix compatibility stuff - > anything that makes our future porting process easier would be welcome! Let > us know what you'd like us to test. Diffs are fine. Can you take ecpg from CVS? I will not be able to answer email for about a two week long vacation trip, but everything I've done so far is in CVS. Michael -- Michael Meskes Email: Michael@Fam-Meskes.De ICQ: 179140304 Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!
On Fri, 11 Apr 2003 03:20, Michael Meskes wrote: > Can you take ecpg from CVS? I will not be able to answer email for about > a two week long vacation trip, but everything I've done so far is in > CVS. Hi Michael, I grabbed everything from CVS and tried building, but ran into a few problems. Bison 1.35 and 1.50 both have bugs that prevent successful compilation, so I moved to 1.85. This appears to work okay. Is there some way during configure to check for broken bison versions and complain loudly? Once I compiled the source tree, and got it installed, I ran into further problems as soon as I tried to link against the ecpg libs: $ gcc -I/usr/local/pgsql/include -L/usr/local/pgsql/lib -lecpg test.c /usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESnumeric_from_asc' /usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESdate_to_asc' /usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESinterval_from_asc' /usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESinterval_to_asc' /usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPEStimestamp_from_asc' /usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESdate_from_asc' /usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESnumeric_copy' /usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESinterval_copy' /usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPEStimestamp_to_asc' /usr/local/pgsql/lib/libecpg.so: undefined reference to `PGTYPESnumeric_to_asc' collect2: ld returned 1 exit status Is there something obvious I am doing wrong here? Is there something borked in my build environment? I've attached my test.pgc. Regards, Philip Yarra.
On Tue, 22 Apr 2003 13:45, Philip Yarra wrote: > Bison 1.35 and 1.50 both have bugs that prevent successful compilation, so > I moved to 1.85. Correction, bison 1.875. Sorry about that, folks. Philip.
On Fri, 28 Mar 2003 17:49, Philip Yarra wrote: > On Fri, 28 Mar 2003 13:00, you wrote: > > I have put a mailbox file at: > > > > ftp://candle.pha.pa.us/pub/postgresql/mypatches > > > > where there is a partial implementation of thread-safe ecpg. Maybe you > > can work with Michael to complete it. > > Okay, thanks for that. Have grabbed the file, and started reading. I'll > also wait to hear from Michael or Lee before going too far on this. I've had a few tries at testing the ECPG threadsafe patch, but so far no luck, probably due to my own lack of experience. Can someone help me out with how to test this? So far I have: - run configure - applied Lee's patch to the source tree for 7.3.2 - edited src/Makefile.global, adding -pthread to CFLAGS and and -lpthread to LIBS - make and make install - make our own pthread'd app, linking against newly installed pgsql Then when I run our app with 3 DB threads, I can watch 2 of the threads fail to return from EXEC SQL, which is exactly the same behaviour we used to get. Can anyone point me in the right direction? I'm building on RedHat Linux 7.3 on Intel, using all standard tools for that version (except updated bison from 1.35 to 1.875). So gcc 2.96, gmake 3.79.1, etc. Let me know if more details are required. Regards, Philip Yarra.
Philip Yarra writes:> I've had a few tries at testing the ECPG threadsafe patch, but so> far no luck, probably due to myown lack of experience. Can someone> help me out with how to test this?> > So far I have:> - run configure> - applied Lee'spatch to the source tree for 7.3.2> - edited src/Makefile.global, adding -pthread to CFLAGS and and> -lpthread to LIBS>- make and make install> - make our own pthread'd app, linking against newly installed pgsql> Then when I run our appwith 3 DB threads, I can watch 2 of the threads fai=> l=20> to return from EXEC SQL, which is exactly the same behaviourwe used to get.> > Can anyone point me in the right direction? I'm building on RedHat Linux 7.=> 3=20> on Intel,using all standard tools for that version (except updated bison=> =20> from 1.35 to 1.875). So gcc 2.96, gmake 3.79.1,etc. Let me know if more=20> details are required. Even with this patch you'll still be limited by the underlying libpq. So for example you cannot share a connection between threads without implementing locking to prevent multiple access to that connection. I guess the best way would be to open a named connection for EACH thread. This SHOULD work, but again not tested... L.
I have libpq thread-safe patch in my queue: http://momjian.postgresql.org/cgi-bin/pgpatches Once I have a way to test thread flags for various OS's, I will apply it. Maybe you guys can help with that too. --------------------------------------------------------------------------- Lee Kindness wrote: > Philip Yarra writes: > > I've had a few tries at testing the ECPG threadsafe patch, but so > > far no luck, probably due to my own lack of experience. Can someone > > help me out with how to test this? > > > > So far I have: > > - run configure > > - applied Lee's patch to the source tree for 7.3.2 > > - edited src/Makefile.global, adding -pthread to CFLAGS and and > > -lpthread to LIBS > > - make and make install > > - make our own pthread'd app, linking against newly installed pgsql > > Then when I run our app with 3 DB threads, I can watch 2 of the threads fai= > > l=20 > > to return from EXEC SQL, which is exactly the same behaviour we used to get. > > > > Can anyone point me in the right direction? I'm building on RedHat Linux 7.= > > 3=20 > > on Intel, using all standard tools for that version (except updated bison= > > =20 > > from 1.35 to 1.875). So gcc 2.96, gmake 3.79.1, etc. Let me know if more=20 > > details are required. > > Even with this patch you'll still be limited by the underlying > libpq. So for example you cannot share a connection between threads > without implementing locking to prevent multiple access to that > connection. > > I guess the best way would be to open a named connection for EACH > thread. This SHOULD work, but again not tested... > > L. > > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Wed, 23 Apr 2003 18:47, Lee Kindness wrote: > Even with this patch you'll still be limited by the underlying > libpq. So for example you cannot share a connection between threads > without implementing locking to prevent multiple access to that > connection. Oh yes, I figured that. Each thread in this application opens its own connection, so this ought not to be the issue. Each DB insert actually gets the data into the table, but 2 of the 3 threads stop after that... possibly due to concurrently accessing sections of the sqlca struct? (that's just a guess though) Our code is a bit tricky to get running - I might see if I can create a simple test harness to duplicate the problem and post it to you. Regards, Philip.
On Thu, 24 Apr 2003 04:15, Bruce Momjian wrote: > I have libpq thread-safe patch in my queue: > Once I have a way to test thread flags for various OS's, I will apply > it. Maybe you guys can help with that too. Happy to help. We have various Linux flavours plus Tru64 Unix (OSF1 v4.0) that we can test on. Regards, Philip.
Michael, I have the ecpg thread stuff in the patches queue: http://momjian.postgresql.org/cgi-bin/pgpatches I need you to look at that and integrate it into ecpg soon so we can get some testing before we hit 7.4 beta. As far as the thread flags, I think we are going to just have to use a HAVE_THREADS configure define, and then fill it in for every OS with OS-specific tests. I can do that part if you code with HAVE_THREADS. --------------------------------------------------------------------------- Michael Meskes wrote: > On Fri, Mar 28, 2003 at 05:49:04PM +1100, Philip Yarra wrote: > > Okay, thanks for that. Have grabbed the file, and started reading. I'll also > > wait to hear from Michael or Lee before going too far on this. > > Don't be surprised if I'm a bit late in answering emails on the list. > I'm currently very busy. > > I'd be interested to get ecpg thread safe. > > Is this the only problem you're facing with Informix? I'm currently > working on other Informix compatibility stuff. And I'd like to get that > tested as good as possible. > > Michael > -- > Michael Meskes > Email: Michael@Fam-Meskes.De > ICQ: 179140304 > Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL! > > > ---------------------------(end of broadcast)--------------------------- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/docs/faqs/FAQ.html > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Sun, 15 Jun 2003 09:06 pm, you wrote:
> [Sorry for the late answer. Somehow I didn't notice your mail wainting
> for an answer.]
No problems, I'd kinda forgotten myself.
> On Tue, Apr 22, 2003 at 01:45:40PM +1000, Philip Yarra wrote:
> > Bison 1.35 and 1.50 both have bugs that prevent successful compilation,
> > so I moved to 1.85. This appears to work okay. Is there some way during
> > configure to check for broken bison versions and complain loudly?
>
> Not that I know of.
I wondered if we could do something like:
BISON_VERSION=`bison -V | head -n 1 | awk '{print $4 }'`
during configure, compare the result to known buggy versions, and take some
sort of action based on that? I haven't delved into configure though, so not
sure how to implement "some sort of action".
I think it is fairly important to try to detect these buggy bisons if the same
problem will be exposed in release versions of the source - it cost me quite
a few hours trying to figure out why make was failing. I'd like to save other
people the hassle (and that list would include people running RedHat Linux
7.3, which includes the bison 1.35).
Regards, Philip.
Philip Yarra <philip@utiba.com> writes:
> Bison 1.35 and 1.50 both have bugs that prevent successful compilation,
> so I moved to 1.85. This appears to work okay. Is there some way during
> configure to check for broken bison versions and complain loudly?
Uh, this is done as of a few days ago. Or is the check not working?
regards, tom lane
On Mon, 16 Jun 2003 12:49 pm, Tom Lane wrote: > Philip Yarra <philip@utiba.com> writes: > > Bison [blah blah] > Uh, this is done as of a few days ago. Or is the check not working? > > regards, tom lane Tom, this post relates to problems I had with the CVS build around April 22... Michael's reply was somewhat delayed. So you can probably ignore it. I just did a test build on a machine with bison 1.35, and there were no complaints regarding its version - I assume that a way was found to make older versions of bison work (which is definitely preferable)? Whatever was done, it seems to have done the trick. Regards, Philip.
Philip Yarra <philip@utiba.com> writes:
> I just did a test build on a machine with bison 1.35, and there were no
> complaints regarding its version
You sure? I see in configure's output
...
checking whether it is possible to strip libraries... yes
checking for bison... bison -y
configure: WARNING:
*** The installed version of Bison is too old. PostgreSQL needs
*** Bison version 1.875 or later.
checking for perl... /usr/bin/perl
checking for main in -lbsd... yes
...
which admittedly is easy to miss, but it's there. (The reason it's not
a hard error is that you don't actually need bison unless you don't have
up-to-date copies of the derived files made by bison. This should not
be the case when using a snapshot, only when building from a CVS pull.)
> I assume that a way was found to make
> older versions of bison work
You assume wrong. Not all the .y files need recent bison, but the ones
that do will fail badly. The issues are grammar-size limits for bison
less than about 1.50, and outright bison bugs for versions from there to
1.875. We do not have workarounds.
regards, tom lane
I wonder is we should check for *dev on the version string and make missing bison version a hard error in these cases. --------------------------------------------------------------------------- Tom Lane wrote: > Philip Yarra <philip@utiba.com> writes: > > I just did a test build on a machine with bison 1.35, and there were no > > complaints regarding its version > > You sure? I see in configure's output > > ... > checking whether it is possible to strip libraries... yes > checking for bison... bison -y > configure: WARNING: > *** The installed version of Bison is too old. PostgreSQL needs > *** Bison version 1.875 or later. > checking for perl... /usr/bin/perl > checking for main in -lbsd... yes > ... > > which admittedly is easy to miss, but it's there. (The reason it's not > a hard error is that you don't actually need bison unless you don't have > up-to-date copies of the derived files made by bison. This should not > be the case when using a snapshot, only when building from a CVS pull.) > > > I assume that a way was found to make > > older versions of bison work > > You assume wrong. Not all the .y files need recent bison, but the ones > that do will fail badly. The issues are grammar-size limits for bison > less than about 1.50, and outright bison bugs for versions from there to > 1.875. We do not have workarounds. > > regards, tom lane > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I wonder is we should check for *dev on the version string and make
> missing bison version a hard error in these cases.
No, because that breaks snapshot tarballs.
You could possibly make some argument for going out and actually
checking the existence and timestamp of each bison-derived file that
should be there, and erroring out only if it looks like bison is really
gonna get invoked during the build. But that seems like an
unmaintainable kluge to me.
It's not like the bison-related build failures aren't pretty obvious.
I think the configure-time test is doing its job by producing a
warning. If people don't notice it, well, they'll find out later.
We do assume some level of technical competence for those who build
from CVS pulls ...
regards, tom lane
On Mon, 16 Jun 2003 02:35 pm, Tom Lane wrote: > Philip Yarra <philip@utiba.com> writes: > > I just did a test build on a machine with bison 1.35, and there were no > > complaints regarding its version > > You sure? I see [it] in configure's output Oh, yes, well so it is. Sorry about that. > (The reason it's not > a hard error is that you don't actually need bison unless you don't have > up-to-date copies of the derived files made by bison. This should not > be the case when using a snapshot, only when building from a CVS pull.) Provided preproc.y has changed... `make clean` does not remove the output of bison, I found. When I copied the CVS source from my workstation to the machine with older bison I didn't realise that, hence the confusion. > > I assume that a way was found to make > > older versions of bison work > > You assume wrong. Yep. Part of the learning-curve, I suppose. I'll get there. Regards, Philip.
On Mon, Jun 16, 2003 at 02:04:55PM +1000, Philip Yarra wrote: > I just did a test build on a machine with bison 1.35, and there were no > complaints regarding its version - I assume that a way was found to make > older versions of bison work (which is definitely preferable)? Whatever was > done, it seems to have done the trick. I think we still have preproc.c auto-generated from preproc.y in our archive, don't we? If so, most users wanting to compile ecpg do not need bison at all as they can use this one. Or do I miss something? Michael -- Michael Meskes Email: Michael@Fam-Meskes.De ICQ: 179140304, AIM: michaelmeskes, Jabber: meskes@jabber.org Go SF 49ers! Go Rhein Fire! Use Debian GNU/Linux! Use PostgreSQL!
Michael Meskes <meskes@postgresql.org> writes:
> I think we still have preproc.c auto-generated from preproc.y in our
> archive, don't we?
It's not in CVS (and shouldn't be IMHO); but it is built during tarball
creation (including nightly snapshots). So only people who use CVS
directly need to have bison.
regards, tom lane