[vtk-developers] precision problem with vtkAxisActor2D on 32 bits linux

[Francois Bertel]

Hi,

What the results of the attached program with -O0 versus -O3? Can you provide the exact gcc versions (gcc --version)?

I put the result of my machine in the commented lines of the source code.

It might be specific to x86 GNU/Linux with gcc, where the default mode is extended precision (with the aim of providing increased accuracy compare to double precision). The FPU is working in extended precision
(intermediate result on the FPU are in extended precision) but values store in memory are in double precision. Depending on the level of optimization, the FPU mode can be set to double precision.

http://www.network-theory.co.uk/docs/gccintro/gccintro_70.html

http://en.wikipedia.org/wiki/Extended_precision

Daniel.AGUILERA at CEA.FR wrote:
> Hi all,
> 
>  
> 
> I found a problem of precision in vtkAxisActor2D.cxx on Linux 32 bits 
> with gcc 3.4 or 4.1 with –O3 option :
> 
>  
> 
> At line 782 there is:
> 
>    If (fabs(sRange[0]/root – floor(sRange[0]/root)) < 0.01)
> 
>  
> 
> root is computed in another function with root=pow(10.0,rootPower)
> 
>  
> 
> if rootPower = -1 and sRange[0] = 1 we should have root = 0.1 and so 
> floor(sRange[0]/root) = 10
> 
> It’s true without –O3 option but false with it (in that case the result 
> is 9)
> 
>  
> 
> This configuration is easy to check, just create a vtkXYPlotActor with Y 
> range = [1, 6] and you will have an Y axis in range [0.9, 5.9]
> 
>  
> 
>  
> 
> One way to avoid this problem of optimisation is to replace the line 782 by:
> 
>  
> 
> volatile double val = sRange[0]/root;
> 
> if (fabs(val – floor(val)) < 0.01)
> 
>  
> 
> This solution works on several linux distros I tried (32 and 64 bits, 
> x86, ia64, GNU or Intel compilers).
> 
>  
> 
> Best regards,
> 
> Daniel Aguilera
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> vtk-developers mailing list
> vtk-developers at vtk.org
> http://www.vtk.org/mailman/listinfo/vtk-developers


-- 
François Bertel, PhD  | Kitware Inc. Suite 204
1 (518) 371 3971 x113 | 28 Corporate Drive
                       | Clifton Park NY 12065, USA
-------------- next part --------------
A non-text attachment was scrubbed...
Name: precision.cxx
Type: text/x-c++src
Size: 1421 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20080321/9ac45eab/attachment-0001.cxx>