[vtk-developers] Testing VTK -- HELP

[Bill Lorensen]

This is an excellent time for all developers to pitch in and up the coverage of the testing. The old tree is only at 70% which is not very good. We have let things slip over the past year or so. vtk is able to remain dynamic and robust because of the testing we have in place. But we need more. I think we should be able to get the coverage to 80%! The new testing infrastructure fa ciliates multi-site testing. And it's all OPEN SOURCE. No more excuses!

Bill

At 09:09 AM 8/10/01 -0400, Ken Martin wrote:
>Hello Folk,
>
>In moving to the new tree I have been trying to get the new dashboard
>to provide some testing to make sure the new tree works, and will
>continue to work. I could use some help increasing the coverage on the
>new tree. I have managed to get the coverage to about 43% by taking
>many of the tests Bill Lorensen wrote and modifying/moving them to the
>new tree. But I could use a hand here. If you add a new class to VTK
>could you please add a new test as well. Also specifically I could use
>a hand in
>
>1) ImageReslice - there are a bunch of tests here but I haven't
>converted them to VTK
>2) Volume Rendering - I think Lisa will finish this up
>3) Parallel directory - Berk ? Charles ? Amy ?
>4) *Stencil* - there are some new stencil classes that could use a
>test
>5) Hybrid - there are a fair number of undercovered classes in Hybrid.
>
>Any volunteers? I'll continue to work on Graphics and Rendering for a
>little while longer. I have included a document on how to do testing
>below. I will also check this into VTK so that people can update it
>and add to it if I wasn't clear. The current coverage can be found on
>the dashboard off of  http://public.kitware.com
>
>Thanks
>Ken
>
>
>
>
>How to run and create tests....
>
>In creating tests there are a few goals to strive for.
>
>1) Increase coverage of the code to help detect defects
>
>2) Don't mess up the tester's directories. If a test wants to write
>out a
>    file, first make sure the directory is writeable, and if it is,
>then write
>    out the files, and then delete them. So that the directory is left
>in its
>    original state.
>
>3) Don't require excessive disk space or CPU time. Most tests that run
>on a
>    512 cubed volume will work just as well on a 64 cubed volume. The
>64 cubed
>    volume will require less disk space for the data, less CPU time to
>    process, and much less CPU time to purify. The same applies to
>valid
>    images. Typically there isn't any need for a 600 by 600 pixel valid
>    image. A 300 by 300 image will work fine and require less storage
>and less
>    CPU time to compute the image difference.
>
>4) Try to use existing data files. If you can create a test using a
>data file
>    that is already present, use that data file. Don't add a new data
>file
>    unless it is critical to test the functionality you want tested. If
>you do
>    need to add a new data file consider its size. If it is a large
>data file
>    can you produce a lower resolution version of it and check that in
>instead?
>
>The current status is that the new tree has about 100 tests and is
>achieving
>about 43 percent coverage. These tests require about three minutes to
>run on
>a 1 GHz Pentium and currently require about ten megabytes of disk
>space for
>the data files and valid images.
>
>How to create a new test
>---------------------------
>
>The easiest approach is to look at an existing test and use that as a
>starting point. The tests are kept in Testing subdirectories under
>each main
>package. (e.g. VTK/Graphics/Testing) In Testing there will be
>subdirectories
>for each type of test (Cxx Tcl Python Java). For C++ tests you need to
>write
>the test and then add it to the CMakeLists.txt file in Testing/Cxx.
>For
>Tclexamples you would write a Tcl script, and then add it to the
>CMakeLists.txt file in Testing/Tcl. Just follow the examples already
>in the
>tree. Most tests take a few common arguments:
>
>   -D /path/to/VTKData
>   -V relative/path/to/valid/image
>
>Tcl scripts also accept:
>
>   -A /path/to/vtktcl/package
>
>Typically these arguments are optional and the test will try to run
>without
>them. If the -V option is provided the test will try to perform image
>differences between the valid image and the rendered image. If the
>valid
>image specified with -V doesn't exist it will create it.
>
>Running a Tcl example
>-----------------------
>
>The most common way to run a Tcl example is to first set TCLLIBPATH to
>/path/to/VTK/Wrapping/Tcl so that the examples can find the vtktcl
>package. OnUNIX you can do:
>
>setenv TCLLIBPATH /home/you/VTK/Wrapping/Tcl
>
>or on windows set TCLLIBPATH to c:/your/VTK/Wrapping/Tcl note that
>UNIX style slashes are used on windows as well. Once TCLLIBPATH is set
>you can run the example as
>
>cd VTK/Graphics/Testing/Tcl
>/path/to/VTK/bin/vtk yourScript.tcl
>
>It will look for VTKData to be at the same level as VTK on your disk.
>If it
>isn't you will need to provide the -D argument.
>
>/path/to/VTK/bin/vtk yourScript.tcl -D /path/to/VTKData
>
>to run the example as a regression test you should make the example
>the first
>argument to VTK/Common/Testing/Tcl/rtImageTest.tcl. For example
>
>/path/to/VTK/bin/vtk /path/to/VTK/Common/Testing/Tcl/rtImageTest.tcl
>yourScript.tcl -D /path/to/VTKData -V Baseline/Graphics/yourScript.png
>
>That is a long command line but normally Dart will take care of it for
>you.
>
>
>Producing a dashboard
>-------------------------
>
>If you want to test your build of VTK you can produce a dashboard. To
>do this
>you will need to have Dart installed on your system. Dart can be
>optained
>currently using cvs as
>:pserver:anonymous at public.kitware.com:/cvsroot/Dart
>login with a password of dart. The when you run CMake, make sure
>DART_ROOT is
>set in your cache and BUILD_TESTING is on. Then you can do
>
>make Experimental
>
>on UNXI to perform a build, run the tests, and submit the result to
>the dashboard. If you just want to run the tests you can do
>
>make ExperimentalTests
>
>on windows there are targets in the workspace for Experiemental and
>ExperimentalTests
>
>
>
>
>
>
>
>
>
>_______________________________________________
>vtk-developers mailing list
>vtk-developers at public.kitware.com
>http://public.kitware.com/mailman/listinfo/vtk-developers