[vtk-developers] VTK image error 0 despite differences - false pass

[Marcus D. Hanwell]

On Fri, May 4, 2018 at 2:08 PM, Marcus D. Hanwell <
marcus.hanwell at kitware.com> wrote:

> On Fri, May 4, 2018 at 1:59 PM, Marcus D. Hanwell <
> marcus.hanwell at kitware.com> wrote:
>
>> On Fri, May 4, 2018 at 1:24 PM, Marcus D. Hanwell <
>> marcus.hanwell at kitware.com> wrote:
>>
>>> On Fri, May 4, 2018 at 9:34 AM, Marcus D. Hanwell <
>>> marcus.hanwell at kitware.com> wrote:
>>>
>>>> A while ago I opened issue https://gitlab.kitware.com/vtk
>>>> /vtk/issues/17201 that talked about a bug on Windows where setting the
>>>> precision to 0 resulted in 6 precision on Windows. There was a test, but it
>>>> was passing even on Windows, this week we had a new employee start and I
>>>> thought it would be a good first bug to create a failing test, and then
>>>> make it pass by addressing the issue.
>>>>
>>>> I thought that TestAxes was a little crowded, and that had perhaps been
>>>> enough to throw the image diff off. So he wrote a test that just had this
>>>> axis in, made a baseline for what was expected, and then changed it so that
>>>> we could see the test fail. Changing to 6 or 8 is still showing passing
>>>> test:
>>>>
>>>>
>>>>
>>>> This looks like a good difference to me (not sure if we are allowed
>>>> attachments, but I tried). The ImageError is being shown as 0. We could
>>>> make the text bigger, but there are a few chart tests where we use text of
>>>> this size and I thought they would fail with extra numbers like this.
>>>>
>>>
>>> Digging into this for anyone following, I can change vtkTesting.cxx to
>>> use 'rtId->SetAllowShift(0);' which removes the single pixel shift logic,
>>> and I get a diff showing my extra digits, as I would expect. We likely have
>>> this on for a reason, and the text is pretty tiny, so I could see a nearest
>>> neighbor search turning up no difference for every character added.
>>>
>>
>> Lots of tests fail when this is turned off, we can actually make the text
>> larger and see the expected failure in the axis test. We could redo the
>> TestAxes to use a larger font, or perhaps add an option to disable the
>> pixel shift on a per test basis (I didn't see anything exposing that in the
>> test driver.
>>
>
> For anyone interested, a merge request with dashboard results looking at
> the impact of not allowing shifts in image diffs globally,
>
> https://gitlab.kitware.com/vtk/vtk/merge_requests/4261
>

For anyone following along, I wrote up  a blog post summarizing the issue,
linking to this thread, and the resolution,

https://blog.kitware.com/image-comparison-tests-vtk/

It turned up a number of issues that had been lurking with passing tests
that should have been failing. You may want to be aware if you reuse this
mechanism too.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://public.kitware.com/pipermail/vtk-developers/attachments/20180608/73d7c26f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: TestAxes2.png
Type: image/png
Size: 1770 bytes
Desc: not available
URL: <https://public.kitware.com/pipermail/vtk-developers/attachments/20180608/73d7c26f/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: TestAxes22.png
Type: image/png
Size: 1876 bytes
Desc: not available
URL: <https://public.kitware.com/pipermail/vtk-developers/attachments/20180608/73d7c26f/attachment-0003.png>