Re: BUG #19018: high memory usage and "stack depth limit exceeded", with GiST index on ltree

Hi,

Dilip, Sergei, thank you for your feedback. After researching it more
I think the fix is incorrect. It seems that we shouldn't limit index
tuple size of leaf pages on the gist core level. Some operator classes
can handle page size tuples on the leaf level and it doesn't break
split (like trgm, hstore) because representation of leaf and not leaf
index tuples are different. So you can have almost page sized index
tuples on the leaf level and all inner level tuples will still be
small enough to provide valid splitting.
It means if we want to introduce some limitation for leaf tuples, we
should do it in operator class implementation.

For inner level tuples it's probably a good idea to introduce some
limitation in the gist core, because no matter what opclass do you
use, it doesn't make sense to have page size tuples.


Speaking about the bug, I still think the main cause is a lack of
index tuple size limitation, but we need to fix it in the operator
class, not gist core. ltree inner page index tuple has next structure:

 *  Non-Leaf
 *            (len)(flag)(sign)(left_ltree)(right_ltree)
 *     ALLTRUE: (len)(flag)(left_ltree)(right_ltree)

Sign is a signature, it has a fixed size that you can set up while
index creation (param sigsize). left_ltree, right_ltree are keys that
we build index on.
So if we want to have internal tuples that less than
GISTMaxIndexTupleSize (~1/4 page size) than we should limit leaf index
tuples keys, and max size for such keys should be:

max_size  = (GISTMaxIndexKeySize - sigsize) / 2

For default signature size and default page size maximum key size is
1008. It works only for single column indexes, for multicolumn we need
a more strict limit, but it seems all operator classes are quite
carefree about it, so probably if we fix at least a single column case
it would be good enough.

There is a new patch version with implementation of the above idea.
Bug reproducer fails on the new check. Sergei's reproducer fails as
well (most of the inserts in it now fail on the new check). I found an
ereport about exceeding size limitation in gist code, so I use the
same error message here too, it's more user friendly than the original
message in v1.

Unfortunately, there is another point that should be addressed. Upper
limit for signature size is 2024, which results in max_size = 0 which
means we can't insert any tuples with it. There is a regression test
that fails because of it. I think the upper limit for signature should
be decreased. But I don't know what to do with existing indexes that
could have big signatures.

What do you think?


Best regards,
Arseniy Mukhin