Re: rounding problems

Andy Anderson wrote:
>> On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
>>> My problem is we calculate resistance of parts in a Foxpro app that we
>>> want to move because we want to bring all the custom apps into one
>>> framework and single database.
>>>
>>> Take this calculation  (0.05/30000* 1.0025) which is used to calculate
>>> parts resistance and Tolerance. (its Ohms Law)  The value returned
>>> from
>>> C++ = .0000016708 which is wrong it should be .00000167418.
>>
>> Why are you so sure about the FoxPro result?  I've just checked a few
>> calculators and get results consistent with your C++ version.
>>
>>   Justin C: 0.0000016708
>>   J FoxPro: 0.00000167418
>>       My C: 0.000001670833
>>      bc[1]: 0.0000016708333333333333333333333333333332
>>      PG[2]: 0.0000016708333333333333336675
>>  Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)
>>
>> Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
>> the math, and as they all agree I'm thinking FoxPro is incorrect!  Next
>> I tried doing it accurately (in Haskell if it makes any difference) and
>> get an answer of 401/240000000 out, which would agree with everything
>> but FoxPro.  If I calculate the ratio back out for FoxPro I get
>> 401/239520242 which is a little way out.
>
> I'll add my Casio scientific calculator to the list, which also gives
> the non-FoxPro result.
>
> We can also write this "exactly" as:
>
> (0.05 * 1.0025 / 3) * 10^-5
> = (5 * 10025 / 3) * 10^-11
> = (50,125 / 3) * 0.00000000001
> = (16,708 + 1/3) * 0.00000000001
>
> which you can verify in your head, so the correct answer is clear.
>
> Is it possible that the FoxPro calculation isn't what you describe? If
> you use the tolerance 1.0045, a single digit off, then you get
> 0.00000167417, very close to your result.
>
> -- Andy

I had typo in my calculator and did not catch it OOPS.