[vtk-developers] Quality testing new VTK tree: discussion at Kitware
Will Schroeder
will.schroeder at kitware.com
Sat Aug 25 08:38:58 EDT 2001
Hi Folks-
Yesterday the GE folks and Kitware had a meeting discussing some of the
quality issues with the new VTK tree. We also talked some about improving
communication with the VTK community. I've included a summary of some of
the discussion items. If you have any questions, comments, suggestions,
etc. please send them to this list.
One thing not in the summary is concerns about how to propose changes, how
to reach agreement on changes, and how to schedule changes so that
everybody in the community is aware of what's going on. The last that we
left it we talked about creating an entry in the VTK FAQ. Other suggestions
are welcome.
Will
------------------------------
VTK Quality: August 24, 2001
Bill Lorensen, Jim Miller GE CRD
Ken Martin, Berk Geveci, Will Schroeder, Brad King, Bill Hoffman Kitware
=======================================================================
Issues / Topics
===============
- 1. 0 Defects
tough to maintain something unless there are ~defects
got to get down to zero on all measured categories
- 2. Force a version every night
Jim Miller to send Ken cron job
- 3. Check mailing addresses
- 4. Data
sizes
structure of data tree
- 5. Examples vs. tests
- 6. pkgIndex
- 7. CMake stability (syntactic stability)
- 8. valid images (alternate images)
why they were added needs to be placed into CVS log/commit message
remove instability
print out which image it fails against
- 9. rtImage regression test script
-10. Running DART vs. make test vs. passing multiple arguments
-11. Running DART - making it easier to use
-12. Adding valid images
valid image generators
location of *.diff.png images (DART Temporary directory ?)
-13. Retention of build results
-14. Coverage - how to improve it
-15. Purify - invalid pointer read (IPR)
-16. DART: XSL performance/memory
-17. DART: Continuous build
-18. DART Time
-19. Miscellaneous DART / Testing process
-20. Data directory: go through data and make a selection
-21. develop XML to represent trending information
-22. PrintSelf test
-23. .dart file to configure testing
-24. MTime testing scripts/where to place tests that test all of vtk
-25. Files that are being covered, what to do?
+ chair display, etc.
-26. Tcl bindings: loading everything; create separate packages?
(e.g., package require vtkCommon)
-27. Deprecating classes: put message in constructor (Lorenson)
Resolution / Solution
=====================
1. Focus on
- coverage (refer to multiple/non-linear threshold)
- purify
2. Jim Miller will send the cron job to Ken
3. Send mail to both addresses work/home
4. Data:
Three levels of examples
- Examples without data
- VTKData (testing data) (threshold ~500K)
- VTKBigData (creation of this directory) (create VTK_BIG_DATA_ROOT ?)
5. Examples will be tested
mini-applications [info comlplete]
support -t (driver script)
examples should be well commented and named
6. pkgIndex - create a testing package
common files in Wrapping/Tcl
7. CMake stability: (stability of commands between releases)
e.g., 0.92 to 0.94 will work
8. valid images (alternate images)
why they were added needs to be placed into CVS log/commit message
remove instability
print out which image it fails against
9. rtImage regression test script will be modified
10. CTest prints out one line (name, time, passed/failed, image difference)
11. Make sure that tests are added work (Ken adding ctest on Windows)
12. - Valid images should be named: .py.png, .tcl.png, .cxx.png
in the Baseline/ directory
- Baseline will also have a .txt for writing stuff
- add an option to rtImageTest (add -f option (Ken))
- support interactive .cxx tests for test writers (Berk)
- -kb will be added to cvs modules file for .png (Bill Hoffman)
- Don't create C++ tests that can be done in Tcl
- Tests will be supported in any language
- .diff.png images are placed into VTKBin/Testing/Temporary (Ken/Berk)
13. Retention of build results:
- buy 60 gbyte disk and keep buying hard disks (Kitware)
- need the build logs and they are kept on the client (Jim Miller)
- expand regular expession to extract pertinent information from build log
(everybody)
- regular expressions currently go into two spots;
duplication of code (coalesce) - look at emacs process (Bill Lorenson)
14. - Look into non-linear thresholds that vary depending on file size
- Use more than one threshold
15. Purify add IPR - Jim Miller
16. XSL performance (Jim/Bill)
- limit one rollup at a time
- look at how coverage is done (time bottleneck)
- revisit C++ XSL alternative
17. DART: Continuous build
- User list (to send email) (Jim/Bill/Berk) server does the email
- Continuous has to be labeled as continuous (to differentiate it
from experimental builds)
- Do the continuous on the HP?
18. DART Time: date is always encoded in GMT; move rollup time to 11:00pm
Every submission will have a change log (Jim Miller)
19. Miscellaneous DART / Testing process
- Union of tests (versus alphabetical) (Jim Miller)
- No reporting of catastrophic failure; for example, dashboard results
expected but not reported (list of expected contributors
when it doesn't show up) (lower priority)
20. Please limit to <500K; use an existing file if possible (meaning adapting
tests)
21. develop XML to represent trending information (Bill/Jim)
22. Needs to be modified so that it prints out only the stuff that fails
(print to stderr - don't want to purify)
23. .dart file to ignore certain tests
- dealing with untested files (.h files in particular)
24. MTime testing scripts/general vtk tests should be placed into
Common/Testing/Tcl even though the test may test objects in other
directories
25. Remove files that are not being built
+ chair display, etc.
26. Tcl bindings: create separate packages (Jim)
package require vtktcl loads everything that compiled
look into creating example/loading
create separate packages
27. Deprecating classes: put message in constructor (Lorenson)
Continue to use depracation macros
More information about the vtk-developers
mailing list