Re: Some other CLOBBER_CACHE_ALWAYS culprits

Hi,

On 2021-05-11 12:03:33 -0400, Tom Lane wrote:
> In some recent threads I complained about how CLOBBER_CACHE_ALWAYS
> test runs have gotten markedly slower over the past couple of release
> cycles [1][2][3].

I wonder if the best way to attack this in a more fundamental manner would be
to handle nested invalidations different than we do today. Not just for
CCA/CCR performance, but also to make invalidations easier to understand in
general.

Right now, for CCA, we'll often invalidate all the caches dozens of times for
a single syscache lookup. Often rebuilding a lot of the entries over and over
again, even when they are not accessed during the lookup (because relcache
eagerly rebuilds cache entries). Of course that's terribly expensive. It's
something like O(lookups * cache accesses during lookup * total cache entries)
I think?

IMO the problem largely stems from eagerly rebuilding *all* relcache entries
during invalidation processing. Something triggers
InvalidateSystemCaches(). That in turn triggers RelationBuildDesc() for all
relations, which triggers a lot of syscache lookups, which trigger a lot of
relcache lookups, ...  And that's just during the
InvalidateSystemCaches(). Most subsequent syscache lookup will be a cache miss
too (unless accessed during the relcache rebuilds) - each syscache miss will
trigger a few system relations to be locked, triggering separate
InvalidateSystemCaches().

If we split cache invalidation into separate invalidation and cache-rebuild
phases, we'd likely be a lot better off, I think, by being able to avoid the
repeated rebuilds of cache entries that are never accessed during
invalidation.

I'd prototyped a relcache version of this in
https://postgr.es/m/20180829083730.n645apqhb2gyih3g%40alap3.anarazel.de but it
seems like it might be possible to generalize?


> so we have clearly made CCA runs a whole lot worse since v12.
> (Non-CCA buildfarm members show runtimes that are about the same
> across all three branches.)
>
> I've reproduced (some of) these results on my shiny new M1 mini,
> which is a tad faster than hyrax's host: it can do the test on HEAD
> (049e1e2ed) in 15.413s.  (Note: this, and the numbers following, are
> median-of-3-runs; the run variance is enough that I wouldn't trust
> them to less than a tenth of a second.)  The run time at 615cebc94
> (v12 branchoff point) is 11.861s.  Bisecting, I found that there were
> three commits that accounted for almost all of the slowdown since v12:
>
> 0d861bbb7  Add deduplication to nbtree
> 11.836s -> 12.339s
> (that's runtime on the preceding commit -> runtime on this commit)

Hm. The most likely explanation seems to be that this shows that index
accesses without using deduplication are slightly more expensive due to the
change? System tables never use the duplication stuff (cf
_bt_allequalimage())...

Greetings,

Andres Freund