Обсуждение: pulling libpqtypes from queue
This patch has been lingering around since Aug 2007. It has matured a lot and now calls libpq home. Unfortunately, ISTM that there is limited support for our proposal. We either pitched to the wrong crowd or pqtypes doesn't have the mass appeal we expected. With that said, we are considering shopping this elsewhere ... ie. pgfoundry. At this point, we feel this has gone in the wrong direction. Our goal was to add some pqtype api calls to libpq (directly or as stubs). We now find ourselves implementing a hook API which is opposite of our proposal. We did try to make it work, but there are many dirty details that make it unworkable and beneath the quality of the rest of the postgresql project. Even though the library size issue was solved, by using stub funcs and dlopen, it appears there are other reasons for rejecting pqtypes. BTW, all we have heard in regards to stub funcs are crickets. pqtypes is bowing out for now :( If the community wants to run with it, we will help in any way we can. We will continue to manage this ourselves; most likely as stub funcs. Keep in mind, we were using pqtypes (in a raw form) for 8 months before submitting it to the community. We thought it would be useful to others and we wanted to give back. We appreciate everyone's willingness to get this working. We respect the fact that with little interest (or good old distaste), everyone was still willing to add api calls to make this work. We appreciate this and think it demonstrates one of the community's strengths. Thanks again for your time and suggestions. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/
Andrew Chernow escribió: > > This patch has been lingering around since Aug 2007. It has matured a > lot and now calls libpq home. Unfortunately, ISTM that there is limited > support for our proposal. We either pitched to the wrong crowd or > pqtypes doesn't have the mass appeal we expected. With that said, we > are considering shopping this elsewhere ... ie. pgfoundry. I expect you intend to get at least the hooks in, right? -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
On Tue, Apr 15, 2008 at 9:36 AM, Alvaro Herrera <alvherre@commandprompt.com> wrote: > Andrew Chernow escribió: > > This patch has been lingering around since Aug 2007. It has matured a > > lot and now calls libpq home. Unfortunately, ISTM that there is limited > > support for our proposal. We either pitched to the wrong crowd or > > pqtypes doesn't have the mass appeal we expected. With that said, we > > are considering shopping this elsewhere ... ie. pgfoundry. > > I expect you intend to get at least the hooks in, right? not likely. Keep in mind, this is not how we really wanted to do things in the first place. We don't think this is the right strategy for integrating libpqtypes with libpq. It over-complicates things and we don't really see a use case outside of libpqtypes. If a reasonable case can be made for putting the hooks in, we will consider it. Can you think of any good reasons for hooking libpq outside of our intentions? PQmakeResult, PQsetValue, and PQresultAlloc OTOH, we think are good functions and stand up on their own merits. At minimum we will fully support their inclusion into 8.4. (we are readying a patch for that). merlin
Alvaro Herrera wrote: > Andrew Chernow escribi?: > > > > This patch has been lingering around since Aug 2007. It has matured a > > lot and now calls libpq home. Unfortunately, ISTM that there is limited > > support for our proposal. We either pitched to the wrong crowd or > > pqtypes doesn't have the mass appeal we expected. With that said, we > > are considering shopping this elsewhere ... ie. pgfoundry. > > I expect you intend to get at least the hooks in, right? No, I don't think so based on what he said. Even to add the hooks to libpq we would need to have more community demand than we have seen, which I think is what he is alluding to in his email. I think they have come upon the right answer, and that is to develop a version on pgfoundry and see if they can get user demand for a more invasive addition to our core code at some point in the future. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, Apr 15, 2008 at 09:48:37AM -0400, Merlin Moncure wrote: > On Tue, Apr 15, 2008 at 9:36 AM, Alvaro Herrera > <alvherre@commandprompt.com> wrote: [...] > > I expect you intend to get at least the hooks in, right? > > not likely. Keep in mind, this is not how we really wanted to do > things in the first place. We don't think this is the right strategy > for integrating libpqtypes with libpq. It over-complicates things and > we don't really see a use case outside of libpqtypes. If a reasonable > case can be made for putting the hooks in, we will consider it. Can > you think of any good reasons for hooking libpq outside of our > intentions? Yes, this one comes to mind: From: sanjay sharma Subject: Submission of Feature Request : RFC- for Implementing Transparent Data Encryptionin Postgres <http://archives.postgresql.org/pgsql-hackers/2008-03/msg01231.php> I know that the original poster wanted to encrypt and decrypt things server-side, but as was noted in the thread this doesn't make that much sense because the decryption keys must be somehow kept around there. But for doing it transparently client-side such libpq hooks might come in handy... Regards - -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFIBMBDBcgs9XrR2kYRAoLzAJ0XX4Xo/ZAoqH/RDEHXg2IuCgnCcwCfdE/z nXz3eP5S2dflpt0GAZULHfU= =ofgk -----END PGP SIGNATURE-----
On Tue, Apr 15, 2008 at 10:48 AM, <tomas@tuxteam.de> wrote: > On Tue, Apr 15, 2008 at 09:48:37AM -0400, Merlin Moncure wrote: > > On Tue, Apr 15, 2008 at 9:36 AM, Alvaro Herrera > > <alvherre@commandprompt.com> wrote: > > > > I expect you intend to get at least the hooks in, right? > > > > not likely. Keep in mind, this is not how we really wanted to do > > things in the first place. We don't think this is the right strategy > > for integrating libpqtypes with libpq. It over-complicates things and > > we don't really see a use case outside of libpqtypes. If a reasonable > > case can be made for putting the hooks in, we will consider it. Can > > you think of any good reasons for hooking libpq outside of our > > intentions? > > Yes, this one comes to mind: > > From: sanjay sharma > Subject: Submission of Feature Request : RFC- for Implementing > Transparent Data Encryption in Postgres > <http://archives.postgresql.org/pgsql-hackers/2008-03/msg01231.php> > > I know that the original poster wanted to encrypt and decrypt things > server-side, but as was noted in the thread this doesn't make that much > sense because the decryption keys must be somehow kept around there. > > But for doing it transparently client-side such libpq hooks might come > in handy... libpqtypes was designed to handle this with our without hooking. (the 'hooking' debate was mainly about exactly how libpq and libpqtypes was going to be separated). libpqtypes had a superclassing concept (not much discussed on the lists) where you could introduce new type handlers that wrapped existing ones and was desgined exactly for things like this. keep an eye on our upcoming pgfoundry project. merlin
Merlin Moncure wrote: > On Tue, Apr 15, 2008 at 10:48 AM, <tomas@tuxteam.de> wrote: >> On Tue, Apr 15, 2008 at 09:48:37AM -0400, Merlin Moncure wrote: >> > On Tue, Apr 15, 2008 at 9:36 AM, Alvaro Herrera >> > <alvherre@commandprompt.com> wrote: >> >> > > I expect you intend to get at least the hooks in, right? >> > >> > not likely. Keep in mind, this is not how we really wanted to do >> > things in the first place. We don't think this is the right strategy >> > for integrating libpqtypes with libpq. It over-complicates things and >> > we don't really see a use case outside of libpqtypes. If a reasonable >> > case can be made for putting the hooks in, we will consider it. Can >> > you think of any good reasons for hooking libpq outside of our >> > intentions? >> >> Yes, this one comes to mind: >> >> From: sanjay sharma >> Subject: Submission of Feature Request : RFC- for Implementing >> Transparent Data Encryption in Postgres >> <http://archives.postgresql.org/pgsql-hackers/2008-03/msg01231.php> >> >> I know that the original poster wanted to encrypt and decrypt things >> server-side, but as was noted in the thread this doesn't make that much >> sense because the decryption keys must be somehow kept around there. >> >> But for doing it transparently client-side such libpq hooks might come >> in handy... > > libpqtypes was designed to handle this with our without hooking. (the > 'hooking' debate was mainly about exactly how libpq and libpqtypes was > going to be separated). > > libpqtypes had a superclassing concept (not much discussed on the > lists) where you could introduce new type handlers that wrapped > existing ones and was desgined exactly for things like this. keep an > eye on our upcoming pgfoundry project. > > merlin > > libpqtypes should be up on pgfoundry shortly. /* register a new type named secure_text. Make it a * sub-class of bytea. Subclass means you can extend * the behaviorof another type. */ PQregisterTypeHandler(conn, "secure_text=bytea", put_secure_text, get_secure_text); int put_secure_text(PGtypeArgs *args) { char secure_buf[1024]; char *text = va_arg(args->ap, char *); size_t len = encrypt(secure_buf, text, strlen(text)); // tell bytea super class to put resulting bytea value return args->super(args, len, secure_buf); } int get_secure_text(PGtypeArgs *args) { size_t len; char *secure_text; size_t user_buflen = va_arg(args->ap, size_t); char *user_buf = va_arg(args->ap, char*); // ask the super class, bytea, to get the bytea value if(args->super(args, &len, &secure_text) == -1) return -1; decrypt(user_buf, user_buflen, secure_text, len); return 0; } /* put a secure_text into a pgparam, which will encrypt it * and send it to the server as a bytea. The "*" is a * pointerflag telling pqtypes to not make an internal copy * of the string (keep a direct pointer). */ PQputf(param, "%secure_text*", "encrypt me"); /* put it in the db, the last argument is resultFormat */ PQparamExec(conn, param, "INSERT INTO t VALUES ($1)", 1); /* get a secure_text from a result, which will decrypt * the bytea field and return text to user. */ char text[1024]; PQgetf(res, tup_num, "%secure_text", field_num, sizeof(text), text); You could make the variable arguments for %secure_text include an encryption key if you want. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/
On Tue, Apr 15, 2008 at 11:24:43AM -0400, Andrew Chernow wrote: > >libpqtypes had a superclassing concept (not much discussed on the > >lists) where you could introduce new type handlers that wrapped > >existing ones and was desgined exactly for things like this. keep an > >eye on our upcoming pgfoundry project. > /* register a new type named secure_text. Make it a > * sub-class of bytea. Subclass means you can extend > * the behavior of another type. > */ All I can say is, way cool. Pity you couldn't get the necessary support but if you can put it up in a way so it can be easily packaged by distributors that'd be cool. I can think of some projects that could really benefit from a feature like this. Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Please line up in a tree and maintain the heap invariant while > boarding. Thank you for flying nlogn airlines.
Martijn van Oosterhout wrote:
> On Tue, Apr 15, 2008 at 11:24:43AM -0400, Andrew Chernow wrote:
>>> libpqtypes had a superclassing concept (not much discussed on the
>>> lists) where you could introduce new type handlers that wrapped
>>> existing ones and was desgined exactly for things like this. keep an
>>> eye on our upcoming pgfoundry project.
>> /* register a new type named secure_text. Make it a
>> * sub-class of bytea. Subclass means you can extend
>> * the behavior of another type.
>> */
>
> All I can say is, way cool. Pity you couldn't get the necessary support
> but if you can put it up in a way so it can be easily packaged by
> distributors that'd be cool. I can think of some projects that could
> really benefit from a feature like this.
>
> Have a nice day,
We are working on a version to put on pgfoundry. This version will add
stub functions to libpq and have the meat of the pqtypes in separate
shared library: loadable via
PQtypesLoad(){ dlopen("libpqtypes.so"); dlsym("pqt_paramExec"); dlsym("pqt_paramCreate"); // etc...
}
Maybe we'll get enough support in the future to get our stubs into core. Until then, you'll have to patch libpq. We
planto provide binary
patches for the major platforms. You won't have to patch the guts of
pqtypes though because that is dynamically loaded behind PQtypesLoad ...
you just need libpqtypes.[so|dll] in your libpath.
--
Andrew Chernow
eSilo, LLC
every bit counts
http://www.esilo.com/
Andrew Chernow escribió: > Maybe we'll get enough support in the future to get our stubs into core. > Until then, you'll have to patch libpq. We plan to provide binary > patches for the major platforms. You won't have to patch the guts of > pqtypes though because that is dynamically loaded behind PQtypesLoad ... > you just need libpqtypes.[so|dll] in your libpath. Requiring your users to patch libpq is not going to gain libpqtypes much acceptance. It would be far better if it can be enabled by just calling some additional function in the application code (after requiring the app to link with pqtypes of course). This additional function would set up pqtypes for later use. I don't really see the problem you got with the hooks. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On Tue, Apr 15, 2008 at 11:51 AM, Alvaro Herrera <alvherre@commandprompt.com> wrote: > Andrew Chernow escribió: > > Maybe we'll get enough support in the future to get our stubs into core. > > Until then, you'll have to patch libpq. We plan to provide binary > > patches for the major platforms. You won't have to patch the guts of > > pqtypes though because that is dynamically loaded behind PQtypesLoad ... > > you just need libpqtypes.[so|dll] in your libpath. > > Requiring your users to patch libpq is not going to gain libpqtypes much > acceptance. It would be far better if it can be enabled by just calling > some additional function in the application code (after requiring the > app to link with pqtypes of course). This additional function would > set up pqtypes for later use. > > I don't really see the problem you got with the hooks. You are quite right that it will be easier for people to try the thing out without having use a custom libpq. So, we will submit an updated libpq hooks patch to -patches. If you think it's possible for this to get through, we can pursue it (we were starting to become skeptical).We are flexible on this. For posterity, here are our objections to hooking libpq: *) Is there any other plausible scenario of another use for hooking into libpq events? (this is rhetorical...we don't think there is.) We think that there is a better way to integrate libpqtypes with libpq so maybe the abstraction is unnecessary. *) keeping PQparamExec & friends outside of libpq makes error handling a little awkward...we expect to use TLS errors in libpqtypes (which, quite frankly, I wish libpq used) but think it would be cleaner to handle errors in consistent fashion with libpq...libpqtypes adds PQseterror, PQgeterror. *) We especially don't like having to explicitly install into every PGconn (PQaddObjectHooks). So, an app that wants to be ported to using PQgetf for example, needs to locate and inject code into all places connections are made, rather than just inject the call. We would rather have things just 'work'. *) In the event pqtypes becomes popular, will it remain a hooked library forever? If not (a tighter integration that we are advocating takes place), then we are stuck with the 'hook' api functions forever, unless this happens before 8.4 gets out. merlin
Merlin Moncure wrote: > > For posterity, here are our objections to hooking libpq: > *) Is there any other plausible scenario of another use for hooking > into libpq events? (this is rhetorical...we don't think there is.) We > think that there is a better way to integrate libpqtypes with libpq so > maybe the abstraction is unnecessary. > *) keeping PQparamExec & friends outside of libpq makes error handling > a little awkward...we expect to use TLS errors in libpqtypes (which, > quite frankly, I wish libpq used) but think it would be cleaner to > handle errors in consistent fashion with libpq...libpqtypes adds > PQseterror, PQgeterror. > *) We especially don't like having to explicitly install into every > PGconn (PQaddObjectHooks). So, an app that wants to be ported to > using PQgetf for example, needs to locate and inject code into all > places connections are made, rather than just inject the call. We > would rather have things just 'work'. > *) In the event pqtypes becomes popular, will it remain a hooked > library forever? If not (a tighter integration that we are advocating > takes place), then we are stuck with the 'hook' api functions forever, > unless this happens before 8.4 gets out. > > merlin > > Installing it per-conn doesn't get you anything. pqtypes has already been linked in. If you use PQexec and PQgetvalue, the pqtypes code pretty much does nothing. So, a per-conn install seems redundant. You are installing the same function pointers under the same name over and over. If you link with, it should just be available. #include <libpqtypes.h> // library-wide init PQtypesInit(void); // libpqtypes is ready for use on any conn That is what we would prefer. We tried to do it with a global array and a lock, but that has its own problems (namely, all the locking). -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/
Andrew Chernow <ac@esilo.com> writes:
> Installing it per-conn doesn't get you anything. pqtypes has already
> been linked in. If you use PQexec and PQgetvalue, the pqtypes code
> pretty much does nothing. So, a per-conn install seems redundant. You
> are installing the same function pointers under the same name over and
> over. If you link with, it should just be available.
I don't really agree with that argument; it's not impossible that you'd
want it on some connections and not others. The server version for
instance could affect the choice.
Per-conn install also gets you out of a bunch of thread safety issues
(because we've already decreed that it's the app's problem to ensure
that only one thread touches a PGconn at a time).
regards, tom lane
Tom Lane wrote: > Andrew Chernow <ac@esilo.com> writes: >> Installing it per-conn doesn't get you anything. pqtypes has already >> been linked in. If you use PQexec and PQgetvalue, the pqtypes code >> pretty much does nothing. So, a per-conn install seems redundant. You >> are installing the same function pointers under the same name over and >> over. If you link with, it should just be available. > > I don't really agree with that argument; it's not impossible that you'd > want it on some connections and not others. The server version for > instance could affect the choice. > > Per-conn install also gets you out of a bunch of thread safety issues > (because we've already decreed that it's the app's problem to ensure > that only one thread touches a PGconn at a time). > > regards, tom lane > > AFAICS, thread-safety is the big problem. I didn't really like the locking code either -- I was grimacing while typing the code. Server version is handled by the fact that state is not global, only the hook callbacks are. If you don't use a pqtypes function, there is no memory or performance overhead (the real overhead was added at link time). An application has to toggle based on PQserverVersion, to install or not install the pqtypes objhooks. Although, more toggling is needed to choose PQgetvalue over PQgetf for instance. But, the community has held their ground on using hooks. We tried to sell our stubs idea but no one was buying. So, per-conn hooks it is. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/
On Tue, Apr 15, 2008 at 3:13 PM, Andrew Chernow <ac@esilo.com> wrote: > Tom Lane wrote: > > Andrew Chernow <ac@esilo.com> writes: > > > > > Installing it per-conn doesn't get you anything. pqtypes has already > been linked in. If you use PQexec and PQgetvalue, the pqtypes code pretty > much does nothing. So, a per-conn install seems redundant. You are > installing the same function pointers under the same name over and over. If > you link with, it should just be available. > > > > > > > I don't really agree with that argument; it's not impossible that you'd > > want it on some connections and not others. The server version for > > instance could affect the choice. > > > > Per-conn install also gets you out of a bunch of thread safety issues > > (because we've already decreed that it's the app's problem to ensure > > that only one thread touches a PGconn at a time). > AFAICS, thread-safety is the big problem. I didn't really like the locking Maybe if there was PQinitGlobalHooks so that all PGconn structs created would inherit the hooks automatically...this allows per conn initialization (if desired) and global initialization which is often easier. As I understand this, no locking is required, except the init function needs to be called before any real libpq code takes place (in threaded environments). merlin
Merlin Moncure wrote: > > Maybe if there was PQinitGlobalHooks so that all PGconn structs > created would inherit the hooks automatically...this allows per conn > initialization (if desired) and global initialization which is often > easier. As I understand this, no locking is required, except the init > function needs to be called before any real libpq code takes place (in > threaded environments). > > merlin > > That's interesting. I don't see much need for per-conn hooks in regards to pqtypes, but I can definately see a reason for other object hook implementations. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tue, Apr 15, 2008 at 10:57:37AM -0400, Merlin Moncure wrote: > On Tue, Apr 15, 2008 at 10:48 AM, <tomas@tuxteam.de> wrote: > > On Tue, Apr 15, 2008 at 09:48:37AM -0400, Merlin Moncure wrote: > > > On Tue, Apr 15, 2008 at 9:36 AM, Alvaro Herrera > > > <alvherre@commandprompt.com> wrote: > > > > > > I expect you intend to get at least the hooks in, right? [...] > libpqtypes was designed to handle this with our without hooking. (the > 'hooking' debate was mainly about exactly how libpq and libpqtypes was > going to be separated). > > libpqtypes had a superclassing concept (not much discussed on the > lists) where you could introduce new type handlers that wrapped > existing ones and was desgined exactly for things like this. That sounds cool. So in a way you do have the hooks. That would be the libpqtypes without the types, no? > keep an > eye on our upcoming pgfoundry project. I sure will. Thanks - -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFIBaFWBcgs9XrR2kYRAgHNAJ9L59j9eHw0EkZ5Imzg5txAX1lF4ACcCbgS 6bgw0sejoFlCbHXvV55kimo= =QuvI -----END PGP SIGNATURE-----
tomas@tuxteam.de wrote: >>> >>> > > I expect you intend to get at least the hooks in, right? > > [...] > >> libpqtypes was designed to handle this with our without hooking. (the >> 'hooking' debate was mainly about exactly how libpq and libpqtypes was >> going to be separated). >> >> libpqtypes had a superclassing concept (not much discussed on the >> lists) where you could introduce new type handlers that wrapped >> existing ones and was desgined exactly for things like this. > > That sounds cool. So in a way you do have the hooks. > A patch has been submited for supporting libpq object hooks, which allows one to associate private storage with a PGconn or PGresult object. The hook is called when a conn or result is created or destroyed (PQreset for conn as well). http://archives.postgresql.org/pgsql-patches/2008-04/msg00309.php For libpqtypes, this means it can operate outside of libpq. libpqtypes was initially designed as a direct extension to libpq (internal code), but the community prefered using a generic hook interface that allowed libpqtypes to work externally. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/