Re: bytea size limit?

Okay, I fixed the implementation of the BYTEA support in the driver
based on your guidelines and it is very happy now. I want to clean
up a few things before I do that, but what is the process for
submitting a patch to the dev team?

Sunday, April 11, 2004, 11:14:01 PM, you wrote:

OJ> Michael Privat wrote:
>> Mmmh, well thanks guys. Only problem with changing to LOs is that I
>> already have data in production so changing the DB schema will be a
>> little complicated. I guess I could also contribute to the driver to
>> provide a streaming implementation. Do you know where that char[] is
>> in the code?

OJ> The char[] arises from the fact that the driver turns each parameter
OJ> into a string representation at the time setBinaryStream etc. is called.
OJ> It's this string representation that is big. Take a look at
OJ> AbstractJdbc1Statement.setBytes() and the call to PGbytea.toPGString().

OJ> To fix this the Right Way involves:

OJ>   - changing setBytes() and setBinaryStream() to store the parameter
OJ> value for later use, not turn it into a string immediately.
OJ>   - using the V3 extended query protocol (this requires a number of
OJ> other driver changes, as at a minimum the driver will need to split up
OJ> queries that contain multiple statements) to allow use of a separate
OJ> Bind message.
OJ>   - using a binary-format parameter in the Bind message to represent the
OJ> bytea field, and streaming from the byte[]/InputStream to the socket
OJ> directly when writing the parameter.

OJ> This is not a trivial piece of work unfortunately.

OJ> There may be a way to do a temporary fix that provides streaming without
OJ> the other work. For V2, it may be possible to stream while still using a
OJ> string representation of the bytea, as V2 queries are null-terminated
OJ> and we don't need to know the length in advance. I haven't looked into
OJ> this in detail.

OJ> It may be possible to do this for V3 too, as we can in theory predict
OJ> the length of the stringized parameter (necessary as V3 messages are
OJ> preceeded by a total message length field): the bytea string format is
OJ> predictable, and the driver is using a predetermined encoding
OJ> (unicode/utf8).

OJ> It'd be ugly, though..

OJ> -O