Re: [HACKERS] Logical Replication and Character encoding

Hello.

This version makes subscriber automatically set its database
encoding to clinet_encoding (if not explicitly set). And changed
the behavior when pg_server_to_client returns NULL to ERROR from
sending original string.

At Wed, 1 Feb 2017 08:39:41 +0000, "Shinoda, Noriyoshi" <noriyoshi.shinoda@hpe.com> wrote in
<AT5PR84MB0084A18D3BF1D93B862E95E4EE4D0@AT5PR84MB0084.NAMPRD84.PROD.OUTLOOK.COM>
> Thank you for creating patches.
> I strongly hope that your patch will be merged into the new
> version.  Since all databases are not yet based on UTF - 8, I
> think conversion of character codes is still necessary.

Thanks.

> -----Original Message-----
> From: Kyotaro HORIGUCHI [mailto:horiguchi.kyotaro@lab.ntt.co.jp] 
> Sent: Wednesday, February 01, 2017 3:31 PM
> To: Shinoda, Noriyoshi <noriyoshi.shinoda@hpe.com>
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Logical Replication and Character encoding
> 
> At Wed, 01 Feb 2017 12:13:04 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp> wrote
in<20170201.121304.267734380.horiguchi.kyotaro@lab.ntt.co.jp>
 
> > > >  I tried a committed Logical Replication environment. I found  
> > > > that replication between databases of different encodings did  not 
> > > > convert encodings in character type columns. Is this  behavior 
> > > > correct?
> > > 
> > > The output plugin for subscription is pgoutput and it currently 
> > > doesn't consider encoding but would easiliy be added if desired 
> > > encoding is informed.
> > > 
> > > The easiest (but somewhat seems fragile) way I can guess is,
> > > 
> > > - Subscriber connects with client_encoding specification and the
> > >   output plugin pgoutput decide whether it accepts the encoding
> > >   or not. If the subscriber doesn't, pgoutput send data without
> > >   conversion.
> > > 
> > > The attached small patch does this and works with the following 
> > > CREATE SUBSCRIPTION.
> > 
> > Oops. It forgets to care conversion failure. It is amended in the 
> > attached patch.
> > 
> > > CREATE SUBSCRIPTION sub1 CONNECTION 'host=/tmp port=5432 
> > > dbname=postgres client_encoding=EUC_JP' PUBLICATION pub1;
> > > 
> > > 
> > > Also we may have explicit negotiation on, for example, 
> > > CREATE_REPLICATION_SLOT.
> > > 
> > >  'CREATE_REPLICATION_SLOT sub1 LOGICAL pgoutput ENCODING EUC_JP'
> > > 
> > > Or output plugin may take options.
> > > 
> > >  'CREATE_REPLICATION_SLOT sub1 LOGICAL pgoutput OPTIONS(encoding EUC_JP)'
> > > 
> > > 
> > > Any opinions?
> 
> This patch chokes replication when the publisher finds an inconvertible character in a tuple to be sent. For the
case,dropping-then-recreating subscription is necessary to go forward.
 

I think this behavior is right and inevitable in order to protect
subscribers from broken strings.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center