[Insight-developers] ctest problem

[Bill Lorensen]

Dirk,

Definitely smells like a bug in the test driver. I'm surprised that
the test driver even checked the images since an exception was thrown.

1) Please assign a bug to me with an example if you can.

2) Please address issues like this to the developers' list and not an
individual.

Regards,

Bill


On Thu, Nov 17, 2011 at 12:39 PM, Matt McCormick
<matt.mccormick at kitware.com> wrote:
> Hi Dirk,
>
> Sounds like you have identified a subtle bug.  The logic to deal with
> the bug probably belongs in ITK's test driver instead of ctest itself.
>  Like you mention, the fix is likely to delete the output arguments to
> --compare before the test.  In general, it is hard to identify what
> will be an output and what will be an input.
>
> I am CC'ing Bill Lorenson, who wrote the excellent Core/TestKernel.
> He may have more ideas.
>
> Thanks,
> Matt
>
> On Thu, Nov 17, 2011 at 12:25 PM, Padfield, Dirk R (GE Global
> Research) <padfield at research.ge.com> wrote:
>> Hi Matt,
>>
>> An aspect of ctest was confounding me until I figured out what the problem was, and I wanted to mention it to you in case you think it should be fixed.  Here is what was happening:
>>
>> 1) I ran ctest on a particular test, the test resulted in an exception, so ctest reported that all of the tests failed.  This was expected.
>> 2) I changed some code in the test to correct the exception, ran ctest again, and ctest reported that all of the test passed.  Also expected.
>> 3) I changed the code back to the original, ran ctest, and ctest reported that all of the tests were still passing!  This didn't make any sense.
>>
>> When I looked at the LastTest.log file, I saw that it was full of exceptions, as expected.  The reason ctest was reporting that all of the tests passed was that, since the tests were giving exceptions, no new images were being written out (and overwriting the old ones), so the old images that had passed were still there.  If I deleted those images and ran step 3 again, ctest reported that the tests failed, as expected.
>>
>> So, this problem could be fixed by having ctest delete all of the images first before running.  Alternatively, ctest could be made smart enough so that it reports a failure if there is an exception rather than reporting a success because the old image still passes.  Otherwise, this results in strange behavior which is not obvious at first.
>>
>> What do you think?
>>
>> Thanks,
>> Dirk
>>
>



-- 
Unpaid intern in BillsBasement at noware dot com