Re: Null Characters in Strings, Version 9.3.1

On 02/19/2014 02:09 AM, Hiroshi Inoue wrote:
> (2014/02/19 5:30), Heikki Linnakangas wrote:
>> On 02/18/2014 09:39 PM, Nils Gösche wrote:
>>> If I run this with version 9.2.1 of the driver (I am using 32 Bit
>>> here), it will go to the else clause and print the value of the text
>>> column. However, if I run this with the latest version, I get the
>>> following output:
>>
>> Hmm. If you could pinpoint it to the exact commit that changed the
>> behavior, that would help too.
>
> Maybe it's caused by the commit 3666c87c1440862bde2e6b8f43ee585deed70d31
> Author: Hiroshi Inoue <inoue@tpf.co.p>
> Date:   Thu Aug 22 20:40:01 2013 +0900
>
>       When LF->CR+LF conversion causes an buffer truncation, supress the
> conversion (in case of unicode).
>
> Nils, please try to turn off *LF <-> CR/LF conversion* option.
> Anyway I can't think of the way to cure the problem other than putting
>    back the commit.

Hmm, what was the point of that commit in the first place? I don't think
it was a good idea. If your application needs CR+LF, it's not going to
be happy if you sometimes don't do the conversion. Better to truncate.

While looking into this, I found the conversion functions difficult to
debug, because it's defined by the def_utf2ucs macro. It would be much
more natural to have one regular function with an argument to indicate
if it should do error-checking. It used to be a regular function, but
commit ca1daf08f0eef73222abcd8aafe96b8e29afff91 turned it into a macro.
Not sure why; premature optimization perhaps?

I propose the attached patch. It turns the utf8_to_ucs2_lf function back
into a regular function, and reverts the troublesome commit
3666c87c1440862bde2e6b8f43ee585deed70d31. It also adds a regression test
case similar to Nils'. Barring objections, I'll commit this.

- Heikki