Re: Bug in numeric multiplication

On 18 November 2015 at 22:19, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> A bit of progress on this: I've found a test case, namely
>
> select sqrt(999999...
>

Wow.

> Still, this proves that we are onto a real problem.
>

Agreed. I had a go at turning that example into something using
log(base, num) so that the result would be visible in a pure SQL test,
but I didn't have any luck.

>> Another issue here is that with outercarry added into the qdigit
>> computation, it's no longer clear what the bounds of qdigit itself are,
>
> I concluded that that particular issue is a red herring: qdigit should
> always be a fairly accurate estimate of the next output digit, so it
> cannot fall very far outside the range 0..NBASE-1.  Testing confirms that
> the range seen in the regression tests is exactly -1 to 10000, which is
> what you'd expect since there can be some roundoff error.
>

Right, I came to the same conclusion.

> Also, after further thought I've been able to construct an argument why
> outercarry stays bounded.  See what you think of the comments in the
> attached patch, which incorporates your ideas about postponing the div[qi]
> calculation.

I think the logic is sound, but I worry that that comment is going to
be very difficult to follow.

I had a go at trying to find a simpler approach and came up with the
attached, which computes the value intended for div[qi+1] near the top
of the loop (using doubles) so that it can detect if it will overflow,
and if so it enters the normalisation block. It still needs some work
to prove that the normalised value for fnextdigit won't overflow, but
it feels like a promising, easier-to-follow approach to me. What do
you think?

Regards,
Dean