[vtk-developers] cdash/gerrit emails about failing tests...

[Kyle Lutz]

On Mon, Jan 28, 2013 at 3:41 PM, Marcus D. Hanwell
<marcus.hanwell at kitware.com> wrote:
> On Mon, Jan 28, 2013 at 11:43 AM, Sean McBride <sean at rogue-research.com> wrote:
>> On Mon, 28 Jan 2013 10:44:08 -0500, David Cole said:
>>
>>>We could write tests that pass.
>>>
>>>Just sayin'

+1 * inf

>> That'd be nice too, but, practically speaking, given the state of the VTK dashboard over the years, I thought my suggestion might be easier, and of course not mutually exclusive to yours.
>
> It would be easier to make them green than try to devise some
> "baseline of failing tests".

I disagree. It took me about 20 minutes to create a list of tests that
fail on my machine and exclude them in my dashboard script. For some
of the rendering test failures I have no idea where to even start. And
then fixing each one and pushing it through gerrit would most
definitely take much more time.

> This is on my list of things to get to,
> but I have gotten behind on some of these tasks. Patches are certainly
> welcome, and if the test can't reliably pass it would be great to
> rethink the test than devise complex solutions to ignore it.
>
> If anyone has input on the tests that are failing (or patches to fix
> them) that would be great. I will coordinate with Dave DeMarle and
> others on setting aside some time for greening.

I've attached the list of failing tests on my machine (Ubuntu 12.10,
GCC-4.7) along with a bit of text about why they failed. It's a simple
cmake file which creates a variable named FAILING_TESTS which can then
be passed to ctest_test() as follows:

# loads the FAILING_TESTS variable
include(vtk_failing_tests.cmake)

# build regex for failing tests
set(FAILING_TESTS_STRING "")
string(REPLACE ";" "|" FAILING_TESTS_STRING "${FAILING_TESTS}")

# run tests
ctest_test(APPEND EXCLUDE "${FAILING_TESTS_STRING}")

As these tests get fixed and start passing I will remove them from the
list. Hopefully, over time, the the list can be removed entirely.
Until then, this allows for a "green" dashboard that is *much* more
useful for identifying regressions.

-kyle
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vtk_failing_tests.cmake
Type: application/octet-stream
Size: 11015 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20130130/45cdab9f/attachment-0001.obj>