pgsql: Fix possible Assert failure in cost_memoize_rescan

Fix possible Assert failure in cost_memoize_rescan

In cost_memoize_rescan(), when calculating the hit_ratio using the calls
and ndistinct estimations, if the value that was set in
MemoizePath.calls had not been processed through clamp_row_est(), then it
was possible that it was set to some non-integer value which could result
in ndistinct being 1 higher than calls due to estimate_num_groups()
performing clamp_row_est() on its input_rows.  This could result in
hit_ratio values slightly below 0.0, which would cause an Assert failure.

The value of MemoizePath.calls comes from the final parameter in the
create_memoize_path() function, of which we only have one true caller of.
That caller passes outer_path->rows.  All the core code I looked at
always seems to call clamp_row_est() on the Path.rows, so there might
have been no issues with any core Paths causing troubles here.  The bug
report was about a CustomPath with a non-clamped row estimated.

The misbehavior as a result of this seems to be mostly limited to the
Assert() failing.  Aside from that, it seems the Memoize costs would
just come out slightly higher than they should have, which is likely
fairly harmless.

Reported-by: Kohei KaiGai <kaigai@heterodb.com>
Discussion: https://postgr.es/m/CAOP8fzZnTU+N64UYJYogb1hN-5hFP+PwTb3m_cnGAD7EsQwrKw@mail.gmail.com
Reviewed-by: Richard Guo
Backpatch-through: 14, where Memoize was introduced

Branch
------
REL_16_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/6143c9c03f7f43a1a16e2587c6f12aa7d81558c8

Modified Files
--------------
src/backend/optimizer/util/pathnode.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)