Re: [HACKERS] Unicode combining characters

* Tatsuo Ishii <t-ishii@sra.co.jp> [011009 18:38]:
> > - corrects a bit the UTF-8 code from Tatsuo to allow Unicode 3.1
> >   characters (characters with values >= 0x10000, which are encoded on
> >   four bytes).
>
> After applying your patches, do the 4-bytes UTF-8 convert to UCS-2 (2
> bytes) or UCS-4 (4 bytes) in pg_utf2wchar_with_len()? If it were 4
> bytes, we are in trouble. Current regex implementaion does not handle
> 4 byte width charsets.

*sigh* yes, it does encode to four bytes :(

Three solutions then :

1) we support these supplementary characters, knowing that they won't
   work with regexes,

2) I back out the change, but then anyone using these characters will
   get something weird, since the decoding would be faulty (they would
   be handled as 3 bytes UTF-8 chars, and then the fourth byte would
   become a "faulty char"... not very good, as the 3-byte version is
   still not a valid UTF-8 code !),

3) we fix the regex engine within the next 24 hours, before the beta
   deadline is activated :/

I must say that I doubt that anyone will use these characters in the
next few months : these are mostly chinese extended characters, with
old italic, deseret, and gothic scripts, and bysantine and western
musical symbols, as well as the mathematical alphanumerical symbols.

I would prefer solution 1), as I think it is better to allow these
characters, even with a temporary restriction on the regex, than to
fail completely on them. As for solution 3), we may still work at it
in the next few months :) [I haven't even looked at the regex engine
yet, so I don't know the implications of what I have just said !]

What do you think ?

Patrice

--
Patrice Hédé
email: patrice hede à islande org
www  : http://www.islande.org/