[vtkusers] ctest and ctest -D Experimental give different results with VTK_DEBUG_LEAKS ON

[Nicolas Rannou]

Hello,

I experienced a strange behaviour using ctest and I was wondering if  
it is a bug or the normal behaviour.

1) Build VTK with VTK_DEBUG_LEAKS to ON
2) Create a vtk leak  in the test.
3) Run ctest on my machine, it tells me the test passed (doesn't catch  
vtk leaks).

$ ctest -R 3d

Test project /Users/nr52/workspace/gofigure2
     Start  2: imageview3dTest
1/6 Test  #2: imageview3dTest ..................   Passed    0.76 sec
     Start  3: vtkviewimage3dtest
2/6 Test  #3: vtkviewimage3dtest ...............   Passed    0.33 sec
     Start 31: qgoimageview3dTest
3/6 Test #31: qgoimageview3dTest ...............   Passed    2.66 sec
     Start 37: qgotabimageview3dTest
4/6 Test #37: qgotabimageview3dTest ............   Passed    2.47 sec
     Start 38: qgotabimageview3dwtTest
5/6 Test #38: qgotabimageview3dwtTest ..........   Passed    3.80 sec
     Start 50: qgotabimageview3dwtTest2
6/6 Test #50: qgotabimageview3dwtTest2 .........   Passed    3.66 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) =  13.70 sec

4) Run ctest -D Experimental to submit the test on the dashboard, it  
says the test is failing and reports the vtk leaks if you check on the  
dashboard. (http://my.cdash.org/viewTest.php?onlyfailed&buildid=77416)

$ ctest -D Experimental -R 3d

    Site: starfish.megason-lab.hms
    Build name: OSX-10.5-gcc4.0.1
Create new tag: 20100625-1503 - Experimental
Configure project
    Each . represents 1024 bytes of output
     . Size of output: 0K
Build project
    Each symbol represents 1024 bytes of output.
    '!' represents an error and '*' a warning.
     .*..... Size of output: 6K
    0 Compiler errors
    1 Compiler warnings
Test project /Users/nr52/workspace/gofigure2
     Start  2: imageview3dTest
1/6 Test  #2: imageview3dTest ..................   Passed    0.85 sec
     Start  3: vtkviewimage3dtest
2/6 Test  #3: vtkviewimage3dtest ...............   Passed    0.29 sec
     Start 31: qgoimageview3dTest
3/6 Test #31: qgoimageview3dTest ...............***Failed    2.60 sec
     Start 37: qgotabimageview3dTest
4/6 Test #37: qgotabimageview3dTest ............   Passed    2.08 sec
     Start 38: qgotabimageview3dwtTest
5/6 Test #38: qgotabimageview3dwtTest ..........   Passed    3.27 sec
     Start 50: qgotabimageview3dwtTest2
6/6 Test #50: qgotabimageview3dwtTest2 .........***Failed    3.21 sec

67% tests passed, 2 tests failed out of 6

Total Test time (real) =  12.32 sec

The following tests FAILED:
	 31 - qgoimageview3dTest (Failed)
	 50 - qgotabimageview3dwtTest2 (Failed)
Performing coverage
  Cannot find any coverage files. Ignoring Coverage request.
Submit files (using http)
    Using HTTP submit method
    Drop site:http://my.cdash.org/submit.php?project=GoFigure2
    Uploaded: /Users/nr52/workspace/gofigure2/Testing/20100625-1503/ 
Build.xml
    Uploaded: /Users/nr52/workspace/gofigure2/Testing/20100625-1503/ 
Configure.xml
    Uploaded: /Users/nr52/workspace/gofigure2/Testing/20100625-1503/ 
Test.xml
    Submission successful
Errors while running CTest


Is it how it is supposed to work?

Thanks,

     Nicolas