Re: UTF8 with BOM support in psql

> -----Original Message-----
> From: Andrew Dunstan [mailto:andrew@dunslane.net]
> Sent: Tuesday, November 17, 2009 9:15 AM
> To: Peter Eisentraut
> Cc: Chuck McDevitt; Itagaki Takahiro; pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] UTF8 with BOM support in psql
>
>
>
> Peter Eisentraut wrote:
> > On tis, 2009-11-17 at 00:59 -0800, Chuck McDevitt wrote:
> >
> >> Or is there a plan to read and convert the UTF-16 or UTF-32 to UTF-8,
> >> so psql and PostgreSQL understand it?
> >> (BTW, that would actually be nice on Windows, where UTF-16 is
> common).
> >>
> >
> > Well, someone could implement UTF-16 or UTF-whatever as client
> encoding.
> > But I have not heard of any concrete proposals about that.
> >
> >
>
> Doesn't the nul byte problem make that seriously hard?
>

Not really... You can't treat UTF-16 the same way you do UTF-8, but we are talking about it being a client_encoding,
nota server_encoding.  So, it's only the routines that look at the strings pre-conversion, and the conversion routines
themselves,that need to understand UTF-16 strings are 16-bits at a time, and end with a 16 bit 0x0000. 
Obviously, it's more work than handing another 8-bit client_encoding, but doesn't seem insurmountable.
And given the 1:1 mapping from UTF-16 to UTF-8, you don't have any new issues due to characters that can't be
converted.