[HACKERS] Protect syscache from bloating with negative cache entries

Hello, recently one of my customer stumbled over an immoderate
catcache bloat.

This is a known issue living on the Todo page in the PostgreSQL
wiki.

https://wiki.postgresql.org/wiki/Todo#Cache_Usage
> Fix memory leak caused by negative catcache entries

https://www.postgresql.org/message-id/51C0A1FF.2050404@vmware.com

This patch addresses the two cases of syscache bloat by using
invalidation callback mechanism.


Overview of the patch

The bloat is caused by negative cache entries in catcaches. They
are crucial for performance but it is a problem that there's no
way to remove them. They last for the backends' lifetime.

The first patch provides a means to flush catcache negative
entries, then defines a relcache invalidation callback to flush
negative entries in syscaches for pg_statistic(STATRELATTINH) and
pg_attributes(ATTNAME, ATTNUM).  The second patch implements a
syscache invalidation callback so that deletion of a schema
causes a flush for pg_class (RELNAMENSP).

Both of the aboves are not hard-coded and defined in cacheinfo
using additional four members.


Remaining problems

Still, catcache can bloat by repeatedly accessing non-existent
table with unique names in a permanently-living schema but it
seems a bit too artificial (or malicious). Since such negative
entries don't have a trigger to remove, caps are needed to
prevent them from bloating syscaches, but the limits are hardly
seem reasonably determinable.


Defects or disadvantages

This patch scans over whole the target catcache to find negative
entries to remove and it might take a (comparably) long time on a
catcache with so many entries. By the second patch, unrelated
negative caches may be involved in flushing since they are keyd
by hashvalue, not by the exact key values.



The attached files are the following.

1. 0001-Cleanup-negative-cache-of-pg_statistic-when-dropping.patch  Negative entry flushing by relcache invalidation
using relcache invalidation callback.
 

2. 0002-Cleanup-negative-cache-of-pg_class-when-dropping-a-s.patch  Negative entry flushing by catcache invalidation
using catcache invalidation callback.
 

3. gen.pl  a test script for STATRELATTINH bloating.

4. gen2.pl  a test script for RELNAMENSP bloating.

3 and 4 are used as the following,

./gen.pl | psql postgres > /dev/null



regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center