Re: Null Characters in Strings, Version 9.3.1

(2014/02/19 22:12), Heikki Linnakangas wrote:
> On 02/19/2014 02:09 AM, Hiroshi Inoue wrote:
>> (2014/02/19 5:30), Heikki Linnakangas wrote:
>>> On 02/18/2014 09:39 PM, Nils Gösche wrote:
>>>> If I run this with version 9.2.1 of the driver (I am using 32 Bit
>>>> here), it will go to the else clause and print the value of the text
>>>> column. However, if I run this with the latest version, I get the
>>>> following output:
>>>
>>> Hmm. If you could pinpoint it to the exact commit that changed the
>>> behavior, that would help too.
>>
>> Maybe it's caused by the commit 3666c87c1440862bde2e6b8f43ee585deed70d31
>> Author: Hiroshi Inoue <inoue@tpf.co.p>
>> Date:   Thu Aug 22 20:40:01 2013 +0900
>>
>>       When LF->CR+LF conversion causes an buffer truncation, supress the
>> conversion (in case of unicode).
>>
>> Nils, please try to turn off *LF <-> CR/LF conversion* option.
>> Anyway I can't think of the way to cure the problem other than putting
>>    back the commit.
>
> Hmm, what was the point of that commit in the first place? I don't think
> it was a good idea. If your application needs CR+LF, it's not going to
> be happy if you sometimes don't do the conversion. Better to truncate.

The commit was for e.g. columns bound by SQLBindCols. In such cases
truncated data can't be seen any more. I forgot about SQLGetData()
when applying the commit.
I already put back the commit.

> While looking into this, I found the conversion functions difficult to
> debug, because it's defined by the def_utf2ucs macro. It would be much
> more natural to have one regular function with an argument to indicate
> if it should do error-checking. It used to be a regular function, but
> commit ca1daf08f0eef73222abcd8aafe96b8e29afff91 turned it into a macro.
> Not sure why; premature optimization perhaps?

I didn't want to open error-checking on the face because the use cases
  are considerably depended on luck.

> I propose the attached patch. It turns the utf8_to_ucs2_lf function back
> into a regular function, and reverts the troublesome commit
> 3666c87c1440862bde2e6b8f43ee585deed70d31. It also adds a regression test
> case similar to Nils'. Barring objections, I'll commit this.

I don't object to the patch.

regards,
Hiroshi Inoue