[vtk-developers] Comparing doubles with floats: precision issues

[David Gobbi]

Hi All,

This is one of those picky math questions that deals with numerical
precision.  Let's say that one has a data set with scalar type "float", and
wants to select values within a range (minval, maxval) where minval, maxval
are of type "double":

    if (fval >= minval && fval <= maxval) { ... }

Now let's say you don't want "fval" to be converted to double, because
floats are faster than doubles on your hardware:

   float fminval = static_cast<float>(minval);
   float fmaxval = static_cast<float>(maxval);
   ...
   if (fval >= fminval && fval <= fmaxval) { ... }

Unfortunately, there are some cases where fval <= fmaxval even though fval
> maxval.  Reducing the precision of the range has invalidated the check.
In order to fix things, you must choose fminval to be the value closest to
but not more than minval, and choose fmaxval to be the value closest to but
not less than maxval.

    float fminval = NearestFloatNotGreaterThan(minval);
    float fmaxval = NearestFloatNotLessThan(maxval);

With these, (fval >= fminval && fval <= fmaxval) gives the same result as
(fval >= minval && val <= fmaxval) and all is right with the world.

So my question is, have any other devs created a solution for this issue,
either for VTK or for related code?  I'm considering a solution based on
the C++11 function std::nextafter(), as described on this stackoverflow
page:
https://stackoverflow.com/questions/15294046/round-a-double-to-the-closest-and-greater-float

 - David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20170612/1f2c4385/attachment.html>