Re: [BUGS] BUG #3975: tsearch2 index should not bomb out of 1Mb limit

Bruce Momjian <bruce@momjian.us> writes:
> Tom Lane wrote:
>> I don't think that follows.  A tsearch index is lossy anyway, so there's

> Uh, the index is lossy but I thought it was lossy in a way that just
> required additional heap accesses, not lossy in that it doesn't index
> everything.

Sure it's lossy.  It doesn't index stopwords, and it doesn't index the
difference between various forms of a word (when the dictionaries reduce
them to a common root).

> I am concerned a 1mb limit is too low though.  Exactly why can't we have
> a higher limit?  Is positional information that significant?

That's pretty much exactly the point: it's not very significant, and it
doesn't justify a total inability to index large documents.

One thing we could do is index words that are past the limit but not
store a position, or perhaps have the convention that the maximum
position value means "somewhere past here".

            regards, tom lane