[vtk-developers] Rudimentary testing of vtkpython [long].

[Prabhu Ramachandran]

hi,

>>>>> "Bill" == William E Lorensen <Lorensen> writes:

    Bill> Prabhu, This sort of black box testing is great! We do some
    Bill> now, but not enough. We need to figure out how to roll this
    Bill> into the nightly testing. Perhaps you could do a submit from
    Bill> your location.

Glad to know you are interested in this.  First let me tell you what
the parser does now.

  (1) Given a vtk object it figures out the following:

      (a) Find all the methods that are created using the BooleanMacro
      i.e. all the XYZOn/Off kind of methods.

      (b) Find all the methods that are of the SetXXXToYYY form and
      tie them all together - so you can use a radiobox.

      (c) Find all the GetXXX/SetXXX methods that have values that are
      floats/ints/strings/tuples that are possibly editable via a gui.
      This eliminates all methods that Get/Set another vtk object.

      (d) Find all the pure Get methods - like GetBounds() or
      GetXXXAsString().

  In order to do the above it has to access every single method and
  figure out the methods so just determining these methods is itself a
  somewhat comprehensive test of a single VTK object in isolation.

  (2) Since this is a generic tool that all sorts of users and
  applications use I need to throw in backward compatibility etc.  In
  December 2000 David added some cool features to the python wrappers
  making it very easy to get the methods of a vtk object etc.  My
  current code has to work for those who still use an old release.

  (3) There are also a lot of bugs, some that caused Segfaults to
  occur or other problems like the infinite loop case with the
  DataReaders infinite loops that I managed to fix.  I have to trap
  these too and ensure my app doesnt crash!  So there are lots of
  silly checks in the code currently that allow me to handle this
  properly.

There are several issues and questions about the testing itself:

  (0) For now I'll aim for simple, fast tests - no dumping images and
  doing ImageDiffs etc.  I like the continuous build page that shows
  how each cvs commit affects the build.  It would be nice if we could
  add a 2 minute test that would atleast test if the python wrappers
  are useable.  Something like last night's problem with the
  vtkFieldDataToAttributeDataFilter problem would have been detected
  immediately.

  (1) What exactly are you looking for in the tests?  Here is a list
  that I would think useful:
  
      (a) Check if we can import vtkpython. Trivial.

      (b) Check if we can instantiate the 500+ classes - again
      trivial.

      (c) Check if we can parse the vtk object as done above.  This
      usually detects all the segfault problems. 

      (d) Do a val = GetXXX(), followed by a SetXXX(val) and check if
      the next GetXXX() is the same.

      (e) For boolean vars do an On and see if the value is 1?  I
      dont know if this is necessary since it is all done anyway by a
      macro.

      (f) Check if the clamping is working okay by using the
      MinValue/MaxValue methods.  Again may not make sense wasting
      time on this?

      (g) Checking for incorrect obsolete messages?

  (2) What kind of output do you want?  I can combine a the python
  code with a shell script that checks for segfaults and automatically
  identifies all the bad classes.  Or even have a driver python
  program BTW are you folks interested in fixing the Segfault problems
  or are they a waste of time?

  (3) The incomplete obsolete messages can be done by searching for
  patterns in the error logs.

  (4) Do you want me to check if the PrintSelf is also okay?  I can
  print the object and see if all variables are covered?


For one thing I will remove all the silly checks in the current code
and check aggressively for bugs and wont try to work around them.
Also, I will only make it work with the latest tree.  So at any point
it will work with the current CVS tree.  Getting it to work with older
releases is a pain.

Let me know what you think and need.

thanks,
prabhu