KWWidgets/GUI Testing/Squish/SquishForKWWidgets

Goals

• Create a CMake module to find Squish on the system (paths to Squish server and Squish runner).
• Create CMake macros that will run a Squish test by invoking the squish server and client runner, and parsing its output,
• Invoke the Squish module when configuring/building KWWidgets,

Files

The following files were added to the KWWidgets/CMake directory. Most of them were adapted from Brad Davis' scripts.

• FindSquish.cmake: locates Squish and sets SQUISH_FOUND.

Warning: The SQUISH_ADD_TEST macro defined here should not be used by KWWidgets since KWWidgets has some special requirements to run Squish tests. Instead, the KWWidgets_ADD_Squish_TEST macro, defined in KWWidgetsTestingMacro.cmake, should be used to add squish tests to CTest.

• KWWidgetsSquishTestScript.cmake: implements a Squish testing sequence (start Squish server, run test, stop Squish server, etc.).
• SquishRunTestCase.sh: The shell script launched by KWWidgetsSquishTestScript.cmake to run the testing sequence.

The KWWidgets_ADD_Squish_TEST macro is defined in KWWidgetsTestingMacro.cmake:

# KWWidgets_ADD_Squish_TEST

macro(KWWidgets_ADD_Squish_TEST 
    test_name
    squish_AUT_full_path
    squish_test_case_path
    aut_env
    aut_path_script
    aut_args
    )

  set(kwwSquishShellScript "${KWWidgets_CMAKE_DIR}/SquishRunTestCase.sh")

  # Can only handle "Tk"
  set(squish_script_wrapper "Tk")

  ADD_TEST(${test_name}
    ${CMAKE_COMMAND} -V -VV --test
    "-Dsquish_aut:STRING=${squish_AUT_full_path}"
    "-Dsquish_server_executable:STRING=${SQUISH_SERVER_EXECUTABLE}"
    "-Dsquish_client_executable:STRING=${SQUISH_CLIENT_EXECUTABLE}"
    "-Dsquish_libqtdir:STRING=${QT_LIBRARY_DIR}"
    "-Dsquish_test_case:STRING=${squish_test_case_path}"
    "-Dsquish_env_vars:STRING=${aut_env}"
    "-Dsquish_wrapper:STRING=${squish_script_wrapper}"
    "-Dsquish_aut_script:STRING=${aut_path_script}"
    "-Dsquish_shell_script:STRING=${kwwSquishShellScript}"
    "-Dsquish_aut_args:STRING=${aut_args}"
    -P "${KWWidgets_CMAKE_DIR}/KWWidgetsSquishTestScript.cmake"
    ${ARGN})

endmacro(KWWidgets_ADD_Squish_TEST)

Usage

The following CMakeLists.txt snippet demonstrates how to add a Squish test to one of the KWWidgets C++ example: Examples/Cxx/Callbacks. The KWWidgets_USE_Squish option should be turned to ON for KWWidgets to add the corresponding tests; make sure Squish is found by pointing KWWidgets to the appropriate SQUISH_INSTALL_DIR directory. The test itself was created from Squish, and saved to Examples/Cxx/Callbacks/Testing/Squish/suite_KWWCallbacksExample/tst_ScrollScale.

find_package(Squish)
[...]
if(BUILD_TESTING AND KWWidgets_BUILD_TESTING)
  include("${KWWidgets_CMAKE_DIR}/KWWidgetsTestingMacros.cmake")
  [...]

  if(KWWidgets_USE_Squish AND SQUISH_FOUND)
    set(Squish_TST_NAME 
  "${KWWidgets_SOURCE_DIR}/Examples/Cxx/${PROJECT_NAME}/Testing/Squish/suite_KWWCallbacksExample/tst_ScrollScale")
    kwwidgets_add_squish_test_from_c_example(
      KWWidgets-${PROJECT_NAME}Squish ${EXE_NAME} "${Squish_TST_NAME}")    
  endif(KWWidgets_USE_Squish AND SQUISH_FOUND)

endif(BUILD_TESTING AND KWWidgets_BUILD_TESTING)

The following command will list all the Squish-enabled tests available in KWWidgets:

barre@ubuntu804desktop:~/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug$ ctest -N -R Squish
Start processing tests
Test project /home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug
  7/ 24 Testing KWWidgets-CallbacksSquish     
  9/ 24 Testing KWWidgets-HelloWorldSquish    
 11/ 24 Testing KWWidgets-WidgetsTourSquish   
 13/ 24 Testing KWWidgets-WindowWithPanelsSqui
 15/ 24 Testing KWWidgets-WizardDialogSquish  
 17/ 24 Testing KWWidgets-MedicalImageViewerSq
 19/ 24 Testing KWWidgets-PolygonalObjectViewe

And here is how one would launch the Squish test for the Callbacks example:

barre@ubuntu804desktop:~/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug$ ctest -R CallbacksSquish -V
[...]
/home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug/Examples/Cxx/Callbacks
  7/ 24 Testing KWWidgets-CallbacksSquish     
Test command: /usr/local/bin/cmake -V -VV --test ...
/home/barre/src/KWWidgets-HEAD/CMake/KWWidgetsSquishTestScript.cmake
Test timeout computed to be: 1500
-- squish_aut='/home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug/bin/KWCallbacksExample'
-- squish_aut_args=''
-- squish_server_executable='/home/barre/build/squish-20081218-tk-src/bin/squishserver'
-- squish_client_executable='/home/barre/build/squish-20081218-tk-src/bin/squishrunner'
-- squish_libqtdir =''
-- squish_test_case='/home/barre/src/KWWidgets-...Testing/Squish/suite_KWWCallbacksExample/tst_ScrollScale'
-- squish_wrapper='Tk'
-- squish_env_vars=''
-- squish_aut_script='/home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug/bin/KWWidgetsSetupPaths.sh'
-- The_sh_torun='/home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug/CMake/SquishRunTestCase.sh'
Starting the squish server.../home/barre/build/squish-20081218-tk-src/bin/squishserver --daemon
Running the test case.../home/barre/build/squish-20081218-tk-src/bin/squishrunner --testcase
/home/barre/src/KWWidgets-HEAD/Examples/Cxx/Callbacks/Testing/Squish/suite_KWWCallbacksExample/tst_ScrollScale 
--wrapper Tk --aut /home/barre/build/KWWidgets-HEAD-VTK-5-2-tcl85-debug/bin/KWCallbacksExample 
* This is an EVALUATION version of Squish. *
RUNNERID: 151590003
AUTID: 151590004
AUTHOST: 127.0.0.1
AUTPORT: 4322
RUNNERID: 151590003
application registered  151590004
Initializing AUT context ...2009-01-02T16:15:37	START     	
Start 'tst_ScrollScale'       	Test 'tst_ScrollScale' started
... done.
2009-01-02T16:15:39	PASS      	/mnt/hgfs/src/kitware/KWWidgets/KWWidgets-HEAD/...
 VP1: Object property comparison of
':KWCallbacksExample.vtkKWWindow0.vtkKWFrame3...vtkKWMyWidget0.vtkKWScale0.getvalue' 
passed'32.0' and '32.0' are equal (VP1)
2009-01-02T16:15:41	END       	End 'tst_ScrollScale'         	End of test 'tst_ScrollScale'
*******************************************************
Summary:
Number of Test Cases:	   1
Number of Tests:	   1
Number of Errors:	   0
Number of Fatals:	   0
Number of Fails:	   0
Number of Passes:	   1
Number of Expected Fails:	   0
Number of Unexpected Passes:	   0
Number of Warnings:	   0
*******************************************************
Stopping the squish server.../home/barre/build/squish-20081218-tk-src/bin/squishserver --stop
Attempting to shut down server at port 4322
Done
-- Process completed
   Passed
[...]
The following tests passed:
	KWWidgets-CallbacksSquish

100% tests passed, 0 tests failed out of 1 

Known Issues

Squish

• Squish does not handle the vtkKWMenuButton widget properly. The replay sequence is sending too many mouse motion events at the application and it takes a long time to playback the corresponding "picking a menu entry" action.

Note: FrogLogic is aware of this issue, and they will try to address it in the next Squish release. In the meantime, one workaround is to manually remove most of the motionEvents (except the last one) from the squish test script.

• Similar to the vtkKWMenuButton, too many mouse move events are recorded and played back when performing a mouse "drag" over a render window.
• Squish sends events to the AUT (Application Under Testing) asynchronously during playback; it does not know if the events are actually processed or not, i.e. how long it will actually take the application to react to each event on a computer different than the one that was used to create the test itself.
• Events from secondary toplevel dialogs are not recorded yet. For example, in WidgetsTourExample, select MultiColumnList from the left panel and double click on the color button: this will launch a color-picker dialog, which is KWWidgets's own color dialog. Squish does not seem to be able to record any of the mouse event coming from this toplevel dialog.
• Squish does not record/track the file name from a file dialog, since it just records the mouse events without knowledge of the semantics of the widget. The same is true for the color-picker-dialog, as reported previously.

KWWidgets

• In order to validate tests that operate on image data, a KWWidgets application needs to be able to grab a screenshot of its VTK render windows, so that baseline images can be created and compared at the end of a Squish test, for validation purposes. This shouldn't be too hard to add since most of VTK is tested that way.