[vtk-developers] Testing VTK -- HELP

[Ken Martin]

Hello Folk,

In moving to the new tree I have been trying to get the new dashboard
to provide some testing to make sure the new tree works, and will
continue to work. I could use some help increasing the coverage on the
new tree. I have managed to get the coverage to about 43% by taking
many of the tests Bill Lorensen wrote and modifying/moving them to the
new tree. But I could use a hand here. If you add a new class to VTK
could you please add a new test as well. Also specifically I could use
a hand in

1) ImageReslice - there are a bunch of tests here but I haven't
converted them to VTK
2) Volume Rendering - I think Lisa will finish this up
3) Parallel directory - Berk ? Charles ? Amy ?
4) *Stencil* - there are some new stencil classes that could use a
test
5) Hybrid - there are a fair number of undercovered classes in Hybrid.

Any volunteers? I'll continue to work on Graphics and Rendering for a
little while longer. I have included a document on how to do testing
below. I will also check this into VTK so that people can update it
and add to it if I wasn't clear. The current coverage can be found on
the dashboard off of  http://public.kitware.com

Thanks
Ken




How to run and create tests....

In creating tests there are a few goals to strive for.

1) Increase coverage of the code to help detect defects

2) Don't mess up the tester's directories. If a test wants to write
out a
   file, first make sure the directory is writeable, and if it is,
then write
   out the files, and then delete them. So that the directory is left
in its
   original state.

3) Don't require excessive disk space or CPU time. Most tests that run
on a
   512 cubed volume will work just as well on a 64 cubed volume. The
64 cubed
   volume will require less disk space for the data, less CPU time to
   process, and much less CPU time to purify. The same applies to
valid
   images. Typically there isn't any need for a 600 by 600 pixel valid
   image. A 300 by 300 image will work fine and require less storage
and less
   CPU time to compute the image difference.

4) Try to use existing data files. If you can create a test using a
data file
   that is already present, use that data file. Don't add a new data
file
   unless it is critical to test the functionality you want tested. If
you do
   need to add a new data file consider its size. If it is a large
data file
   can you produce a lower resolution version of it and check that in
instead?

The current status is that the new tree has about 100 tests and is
achieving
about 43 percent coverage. These tests require about three minutes to
run on
a 1 GHz Pentium and currently require about ten megabytes of disk
space for
the data files and valid images.

How to create a new test
---------------------------

The easiest approach is to look at an existing test and use that as a
starting point. The tests are kept in Testing subdirectories under
each main
package. (e.g. VTK/Graphics/Testing) In Testing there will be
subdirectories
for each type of test (Cxx Tcl Python Java). For C++ tests you need to
write
the test and then add it to the CMakeLists.txt file in Testing/Cxx.
For
Tclexamples you would write a Tcl script, and then add it to the
CMakeLists.txt file in Testing/Tcl. Just follow the examples already
in the
tree. Most tests take a few common arguments:

  -D /path/to/VTKData
  -V relative/path/to/valid/image

Tcl scripts also accept:

  -A /path/to/vtktcl/package

Typically these arguments are optional and the test will try to run
without
them. If the -V option is provided the test will try to perform image
differences between the valid image and the rendered image. If the
valid
image specified with -V doesn't exist it will create it.

Running a Tcl example
-----------------------

The most common way to run a Tcl example is to first set TCLLIBPATH to
/path/to/VTK/Wrapping/Tcl so that the examples can find the vtktcl
package. OnUNIX you can do:

setenv TCLLIBPATH /home/you/VTK/Wrapping/Tcl

or on windows set TCLLIBPATH to c:/your/VTK/Wrapping/Tcl note that
UNIX style slashes are used on windows as well. Once TCLLIBPATH is set
you can run the example as

cd VTK/Graphics/Testing/Tcl
/path/to/VTK/bin/vtk yourScript.tcl

It will look for VTKData to be at the same level as VTK on your disk.
If it
isn't you will need to provide the -D argument.

/path/to/VTK/bin/vtk yourScript.tcl -D /path/to/VTKData

to run the example as a regression test you should make the example
the first
argument to VTK/Common/Testing/Tcl/rtImageTest.tcl. For example

/path/to/VTK/bin/vtk /path/to/VTK/Common/Testing/Tcl/rtImageTest.tcl
yourScript.tcl -D /path/to/VTKData -V Baseline/Graphics/yourScript.png

That is a long command line but normally Dart will take care of it for
you.


Producing a dashboard
-------------------------

If you want to test your build of VTK you can produce a dashboard. To
do this
you will need to have Dart installed on your system. Dart can be
optained
currently using cvs as
:pserver:anonymous at public.kitware.com:/cvsroot/Dart
login with a password of dart. The when you run CMake, make sure
DART_ROOT is
set in your cache and BUILD_TESTING is on. Then you can do

make Experimental

on UNXI to perform a build, run the tests, and submit the result to
the dashboard. If you just want to run the tests you can do

make ExperimentalTests

on windows there are targets in the workspace for Experiemental and
ExperimentalTests