pgsql: Fix edge-case crashes and misestimation in range containment sel

Fix edge-case crashes and misestimation in range containment selectivity.

When estimating the selectivity of "range_var <@ range_constant" or
"range_var @> range_constant", if the upper (or respectively lower)
bound of the range_constant was above the last bin of the range_var's
histogram, the code would access uninitialized memory and potentially
crash (though it seems the probability of a crash is quite low).
Handle the endpoint cases explicitly to fix that.

While at it, be more paranoid about the possibility of getting NaN
or other silly results from the range type's subdiff function.
And improve some comments.

Ordinarily we'd probably add a regression test case demonstrating
the bug in unpatched code.  But it's too hard to get it to crash
reliably because of the uninitialized-memory dependence, so skip that.

Per bug #16122 from Adam Scott.  It's been broken from the beginning,
apparently, so backpatch to all supported branches.

Diagnosis by Michael Paquier, patch by Andrey Borodin and Tom Lane.

Discussion: https://postgr.es/m/16122-eb35bc248c806c15@postgresql.org

Branch
------
REL_12_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/70c17a81278c7d727ae5096fbb561bdd58eaadd0

Modified Files
--------------
src/backend/utils/adt/rangetypes_selfuncs.c | 98 ++++++++++++++++++++---------
1 file changed, 70 insertions(+), 28 deletions(-)