Re: Reporting UnicodeEncodeError info on arbitrary data sent to PG with psycopg3

Am Thu, Feb 15, 2024 at 11:45:15PM -0600 schrieb Karl O. Pinc:

>   Today there is no substitute for knowing the encoding of the
> text your application obtains from the outside world.
> This can be highly system dependent because when reading
> files open()-ed as text, Python decodes (into UTF-8) the bytes read.

Not quite. Python assumes the bytes in the file *are* encoded
by whatever encoding is passed to open(), including, if so
UTF-8). It then decodes said bytes into *unicode code
points*. If we want them back as UTF-8 we need to encode them
as such.

> By default decoding from the system locale's character encoding.
> And when writing files open()-ed as text Python encodes (from UTF-8)

again, from unicode, that is:

    https://docs.python.org/3/howto/unicode.html

>   No matter how you get your data, to put your data into
> the database as text, its bytes must first have their external
> encoding decoded to UTF-8.  Because Python strings are
> UTF-8.

unicode code points, but, yeah

>   Once in Python, psycopg converts the UTF-8 text to the database

unicode

> It's important to get the encoding right so I think it'd be
> good to talk about it.

+1

Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B