I plan to do testing using my own testing harness after changing it to insert a lot of dummy tuples (ones with negative values in the pseudo-pk column, which are never queried by the core part of the harness) and deleting them at random intervals. I think that none of pgbench's built in tests are likely to give the bucket splitting and squeezing code very much exercise.
I've implemented this, by adding lines 197 through 202 to the count.pl script. (I'm reattaching the test case)
Within a few minutes of testing, I start getting Errors like these:
29236 UPDATE XX000 2016-09-11 17:21:25.893 PDT:ERROR: buffer 2762 is not owned by resource owner Portal
29236 UPDATE XX000 2016-09-11 17:21:25.893 PDT:STATEMENT: update foo set count=count+1 where index=$1
In one test, I also got an error from my test harness itself indicating tuples are transiently missing from the index, starting an hour into a test:
child abnormal exit update did not update 1 row: key 9555 updated 0E0 at count.pl line 194.\n at count.pl line 208.
child abnormal exit update did not update 1 row: key 8870 updated 0E0 at count.pl line 194.\n at count.pl line 208.
child abnormal exit update did not update 1 row: key 8453 updated 0E0 at count.pl line 194.\n at count.pl line 208.
Those key values should always find exactly one row to update.
If the tuples were permanently missing from the index, I would keep getting errors on the same key values very frequently. But I don't get that, the errors remain infrequent and are on different value each time, so I think the tuples are in the index but the scan somehow misses them, either while the bucket is being split or while it is being squeezed.
This on a build without enable-asserts.
Any ideas on how best to go about investigating this?