pgsql: Fix edge-case crashes and misestimation in range containment sel

Fix edge-case crashes and misestimation in range containment selectivity.

When estimating the selectivity of "range_var <@ range_constant" or
"range_var @> range_constant", if the upper (or respectively lower)
bound of the range_constant was above the last bin of the range_var's
histogram, the code would access uninitialized memory and potentially
crash (though it seems the probability of a crash is quite low).
Handle the endpoint cases explicitly to fix that.

While at it, be more paranoid about the possibility of getting NaN
or other silly results from the range type's subdiff function.
And improve some comments.

Ordinarily we'd probably add a regression test case demonstrating
the bug in unpatched code.  But it's too hard to get it to crash
reliably because of the uninitialized-memory dependence, so skip that.

Per bug #16122 from Adam Scott.  It's been broken from the beginning,
apparently, so backpatch to all supported branches.

Diagnosis by Michael Paquier, patch by Andrey Borodin and Tom Lane.

Discussion: https://postgr.es/m/16122-eb35bc248c806c15@postgresql.org

Branch
------
REL9_6_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/2dd10477cbfcf4d54725e9fd2cb3af84aa644fe5

Modified Files
--------------
src/backend/utils/adt/rangetypes_selfuncs.c | 100 ++++++++++++++++++++--------
1 file changed, 72 insertions(+), 28 deletions(-)