Re: Latin vs non-Latin words in text search parsing

"Heikki Linnakangas" <heikki@enterprisedb.com> writes:
> Alvaro Herrera wrote:
>> lword        Entirely letters per iswalpha, with at least one ASCII
>> nlword        Entirely letters per iswalpha
>> word        Entirely alphanumeric per iswalnum, but not nlword

> I don't like this categorization much more than the original. The
> distinction between lword and nlword is useless for most European
> languages.

Right.  That's not an objection in itself, since you can just add the
same dictionary mappings to both token types, but the question is when
would such a distinction actually be useful?  AFAICS the only case where
it'd make sense to put different mappings on lword and nlword with the
above definitions is when dealing with Russian or similar languages,
where the entire alphabet is non-ASCII.  However, my proposal (pure
ASCII vs not pure ASCII) seems to work just as well for that case as
this proposal does.

> ... I'm also not clear what the use case for
> the distinction between words with digits or not is. I don't think
> there's any natural languages where a word can contain digits, so it
> must be a computer-oriented thing as well.

Well, that's exactly why we *should* distinguish words-with-digits;
it's unlikely that any standard dictionary will do sane things with
them, so if you want to index them they need to go down a different
dictionary chain.

A more drastic change would be to not treat a string like "beta1"
as a single token at all, so that the alphanumeric-word category
would go away entirely.  However I'm disinclined to tinker with
the parser that much.  It's seen enough use in the contrib module
that I'm prepared to grant that the design is generally useful.
I'm just worried that the subcategories of "word" need a bit of
adjustment for languages other than Russian and English.
        regards, tom lane