Re: WIP: rewrite numeric division

"Tom Lane" <tgl@sss.pgh.pa.us> writes:

> Yeah, my proposed patch is schoolbook division.  I don't think I'd trust
> Newton's method to give anything but a close approximation, which is
> what we have in HEAD already.

Well unless we start storing rational numbers it'll always be a limited to a
finite number of decimals. The key is whether we can guarantee a lower bound
on the number of accurate decimal places. As long as we can then we can keep
going until the decimal places we're going to return are all accurate.

So your complaint about the existing code boils down to not having any
rigorous way to know when to stop. I don't think Newton's method has that
problem, at least not for simple polynomials. Any digits which don't change
from one iteration to the next are definitely correct.

The problem with Newton's method is that it needs a fast multiplication
algorithm. Looking at it now though it looks like our multiplication algorithm
is similar to the old division algorithm which looks like an optimized
schoolbook multiplication.

It looks like multiplication can also generate incorrect results. Because it
rounds to rscale and then apply_typmod will round again. So a number like 2.49
could conceivably round up to 3 if the two roundings happen to hit at the
wrong place.

--
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com