Re: Precision loss casting float to numeric
От | Chapman Flack |
---|---|
Тема | Re: Precision loss casting float to numeric |
Дата | |
Msg-id | 5A93ABA2.2050908@anastigmatix.net обсуждение исходный текст |
Ответ на | Precision loss casting float to numeric (Chapman Flack <chap@anastigmatix.net>) |
Ответы |
Re: Precision loss casting float to numeric
|
Список | pgsql-hackers |
Here are two patches. The 0001-*.patch simply adds a regression test to numeric.sql that bits aren't lost casting from float[48] to numeric. Naturally, it fails at this stage. The 0002-*.patch is a proof-of-concept patching float4_numeric and float8_numeric in the trivial way (just using FLT_DECIMAL_DIG and DBL_DECIMAL_DIG in place of FLT_DIG and DBL_DIG). It makes the new regression test pass. (It will only work under a compiler that has __FLT_DECIMAL_DIG__ and __DBL_DECIMAL_DIG__ available, and I used those internal versions to avoid mucking with build tooling to change the target C standard, which I assume wouldn't be welcome anyway. This patch is just POC and not proposed as anything else.) It changes the output of an existing numeric test and a few numeric aggregate tests. I have adjusted the numeric test: its purpose is to check the direction of numeric rounding of ties, but the value to be rounded was computed in float8 and then cast to numeric, and because negative powers of ten aren't tidy in binary, it can turn out that the float8 computation produces a correctly-rounded-53-bit-result that is on the nearer-to-zero side of a tie, and now that that result is correctly cast, the resulting numeric doesn't round in the away-from-zero direction. I changed that test because I concluded it wasn't meant to test float8-to-numeric casting, but only the rounding of tied numerics, so I just made the inner expression be typed numeric, and the expected output is unchanged. The three aggregate tests with changed output are working from a table of float4 values, and my assumption is they are now simply producing the correct aggregate results given the full precision of the input values, but I haven't confirmed that yet, so this patch leaves those three failing for now. -Chap
Вложения
В списке pgsql-hackers по дате отправления: