Re: tsearch2: enable non ascii stop words with C locale

> > Currently tsearch2 does not accept non ascii stop words if locale is
> > C. Included patches should fix the problem. Patches against PostgreSQL
> > 8.2.3.
> 
> I'm not sure about correctness of patch's description.
> 
> First, p_islatin() function is used only in words/lexemes parser, not stop-word 
> code.

I know. My guess is the parser does not read the stop word file at
least with default configuration.

> Second, p_islatin() function is used for catching lexemes like URL or HTML 
> entities, so, it's important to define real latin characters. And it works 
> right: it calls p_isalpha (already patched for your case),  then it calls 
> p_isascii which should be correct for any encodings with C-locale.

original p_islatin is defined as follows:

static int
p_islatin(TParser * prs)
{return (p_isalpha(prs) && p_isascii(prs)) ? 1 : 0;
}

So if a character is not ASCII, it returns 0 even if p_isalpha returns
1. Is this what you expect?

> Third (and last):
> contrib_regression=# show server_encoding;
>   server_encoding
> -----------------
>   UTF8
> contrib_regression=# show lc_ctype;
>   lc_ctype
> ----------
>   C
> contrib_regression=# select lexize('ru_stem_utf8', RUSSIAN_STOP_WORD);
>   lexize
> --------
>   {}
> 
> Russian characters with UTF8 take two bytes.

In our case, we added JAPANESE_STOP_WORD into english.stop then:

select to_tsvector(JAPANESE_STOP_WORD)

which returns words even they are in JAPANESE_STOP_WORD.

And with the patches the problem was solved.
--
Tatsuo Ishii
SRA OSS, Inc. Japan