CDash:Testing: Difference between revisions

From KitwarePublic
Jump to navigationJump to search
No edit summary
 
(21 intermediate revisions by 4 users not shown)
Line 1: Line 1:
CDash integrates a self-testing framework based on [http://www.simpletest.org Simple Test]. The self testing code is located in the ''testing'' directory.
CDash integrates a self-testing framework based on [http://www.simpletest.org Simple Test]. The self testing code is located in the ''CDash/testing'' directory.


= Submitting a CDash Dashboard to the CDash dashboard =
CDash testing requires the use of CMake and CTest.  You can download the CMake suite of tools from [http://www.cmake.org/cmake/resources/software.html www.cmake.org].  CDash testing works much the same as any other CMake based project.


No it's not a mistake in the title, self-testing involves submitting a CDash submission to CDash.
= Checkout the CDash source code =


=== Configuring the testing ===
See [[CDash:Installation#Downloading_CDash|Downloading CDash]] for instructions on obtaining a copy of the CDash source code from GitHub.  During this step you should decide whether you want to test the latest release of CDash or the current development version.
Here are the steps on how to configure the testing:


* Edit CDash/cdash/config.php and change the $CDASH_DB_NAME to be 'cdash4simpletest'.
No matter which version of CDash you choose to test, make sure to check out the source code into your web server's htdocs folder.
Because the testing of CDash creates and drops the database you need to make sure this database is not being use for any other purposes than self testing. Also, make sure that the $CDASH_DB_LOGIN has enough privileges to create/drop databases on your SQL server.
* edit the CDash/testing/config.test.php
* '''urlwebsite''' is the URL to access your current CDash. Since simpletest emulates a web browser you need to make sure that the urlwebsite is accessible locally.
'urlwebsite'      => 'http://localhost/CDashTesting',
* '''outputdirectory''' is a directory where the testing output can be stored on the system. Make sure that the current user can write to that directory
'outputdirectory'  => '/tmp'
* '''type''' is the type of submission for CDash: Nightly, Continuous or Experimental
'type'            => 'Nightly'
* '''site''' is the name of the site submitting to CDash
'site'            => 'yellowstone.kitware'
* '''buildname''' is name of the build
'buildname'        => 'CDash-SVN-MySQL'
* '''cdash''' is the URL to the current dashboard. Note that the project should be named CDash, i.e. http://www.cdash.org/CDash/submit.php?project=CDash
'cdash'            => 'http://www.cdash.org/CDash',
* '''svnroot''' is path to your testing installation of CDash. The self testing does a svn update to check what file have changed.
'svnroot'          => '/var/www/CDashTesting'


=== Starting the testing ===
= Setup CDash for testing =
Then launch the test by doing
 
   cd CDash/testing
== Create a build directory ==
   php5 alltests.php
 
cd /path/to/CDash
mkdir build
 
==Run CMake from the build directory ==
 
cd build
ccmake ..
 
== Define CMake variables ==
 
CDASH_DB_HOST
CDASH_DB_LOGIN
CDASH_DB_PASS
CDASH_DB_TYPE
*These four parameters tell CDash how to connect to your local database server.
 
CDASH_SERVER
*CDASH_SERVER should indicate how to connect to your local system via a web browser.  This is typically the system's host name, or simply localhost.
 
CDASH_USE_APACHE2
*Setting this to ON will enable tests specific to the Apache web server.
 
CDASH_USE_SELENIUM
PHPUNIT_EXE
*Setting these options will enable testing of javascript based functionality of CDash.  Requires [http://seleniumhq.org/ selenium] and [http://www.phpunit.de/ PHPUnit].  More on this below.
 
CMake_SOURCE_DIR
*Point this to a local checkout of CMake source code if you'd like to import some tests from CMake into CDash.  This will cause CDash to perform a more thorough end-to-end test of the CMake/CTest/CDash suite of tools.
 
PHP_EXE
*CDash testing requires a command-line version of PHP to be installed on the system.
 
= Test CDash =
 
This is as simple as entering your CDash build directory and running CTest
   cd /path/to/CDashTesting/build
  ctest -VV
 
Note this will create '''CDashTesting/cdash/config.local.php''' and '''CDashTesting/testing/config.test.local.php''' files in your source tree, so svn will show them as ? files.
 
'''For PostGreSQL, make sure you create an empty database named 'host' so that CDash can create the cdash4simpletest database'''
 
= Writing Additional CDash Tests =
 
If you add any new tests, please follow the structure of one of the existing ones.
 
For example, at the top of each test_*.php file in the tests directory, you'll see something like this (example from test_banner.php):
<source lang="php">
<?php
//
// After including cdash_test_case.php, subsequent require_once calls are
// relative to the top of the CDash source tree
//
require_once(dirname(__FILE__).'/cdash_test_case.php');
 
require_once('cdash/common.php');
require_once('cdash/pdo.php');
require_once('models/banner.php');
 
class BannerTestCase extends KWWebTestCase
{
  function __construct()
    {
    parent::__construct();
    }
 
   function testBanner()
    {
    ...
    }
}
?>
</source>
 
Please note the following:
* do not use "include" -- use "require_once"
* the first require_once uses dirname(__FILE__) meaning "same dir as this php file"
* the first require_once includes the cdash_test_case.php file
* subsequent require_once calls refer to files from the root of the CDash source tree (include path set properly by the cdash_test_case.php file)
* the constructor of the test case is '''empty''' -- it just calls the parent -- all the common stuff that used to be in the individual test cases has now been refactored to the base class -- see kwtest/kw_web_tester.php if you need to add to that code
 
 
* similarly for the selenium tests... except with cdash_selenium_test_case.php
* setUp for the individual selenium test cases should simply call $this->browserSetUp (defined once in the parent class)
 
= Advanced Topics =
 
[[CDash:Testing/Coverage|Code coverage]]
 
[[CDash:Testing/Selenium|Testing javascript functionality with Selenium]]
 
[[CDash:Testing/Client Management|Client management]]
 
[[CDash:Testing/Nightly|Setting up nightly Builds]]

Latest revision as of 20:38, 27 February 2017

CDash integrates a self-testing framework based on Simple Test. The self testing code is located in the CDash/testing directory.

CDash testing requires the use of CMake and CTest. You can download the CMake suite of tools from www.cmake.org. CDash testing works much the same as any other CMake based project.

Checkout the CDash source code

See Downloading CDash for instructions on obtaining a copy of the CDash source code from GitHub. During this step you should decide whether you want to test the latest release of CDash or the current development version.

No matter which version of CDash you choose to test, make sure to check out the source code into your web server's htdocs folder.

Setup CDash for testing

Create a build directory

cd /path/to/CDash
mkdir build

Run CMake from the build directory

cd build
ccmake ..

Define CMake variables

CDASH_DB_HOST
CDASH_DB_LOGIN
CDASH_DB_PASS
CDASH_DB_TYPE
  • These four parameters tell CDash how to connect to your local database server.
CDASH_SERVER
  • CDASH_SERVER should indicate how to connect to your local system via a web browser. This is typically the system's host name, or simply localhost.
CDASH_USE_APACHE2
  • Setting this to ON will enable tests specific to the Apache web server.
CDASH_USE_SELENIUM
PHPUNIT_EXE
  • Setting these options will enable testing of javascript based functionality of CDash. Requires selenium and PHPUnit. More on this below.
CMake_SOURCE_DIR
  • Point this to a local checkout of CMake source code if you'd like to import some tests from CMake into CDash. This will cause CDash to perform a more thorough end-to-end test of the CMake/CTest/CDash suite of tools.
PHP_EXE
  • CDash testing requires a command-line version of PHP to be installed on the system.

Test CDash

This is as simple as entering your CDash build directory and running CTest

 cd /path/to/CDashTesting/build
 ctest -VV

Note this will create CDashTesting/cdash/config.local.php and CDashTesting/testing/config.test.local.php files in your source tree, so svn will show them as ? files.

For PostGreSQL, make sure you create an empty database named 'host' so that CDash can create the cdash4simpletest database

Writing Additional CDash Tests

If you add any new tests, please follow the structure of one of the existing ones.

For example, at the top of each test_*.php file in the tests directory, you'll see something like this (example from test_banner.php): <source lang="php"> <?php // // After including cdash_test_case.php, subsequent require_once calls are // relative to the top of the CDash source tree // require_once(dirname(__FILE__).'/cdash_test_case.php');

require_once('cdash/common.php'); require_once('cdash/pdo.php'); require_once('models/banner.php');

class BannerTestCase extends KWWebTestCase {

 function __construct()
   {
   parent::__construct();
   }
 function testBanner()
   {
   ...
   }

} ?> </source>

Please note the following:

  • do not use "include" -- use "require_once"
  • the first require_once uses dirname(__FILE__) meaning "same dir as this php file"
  • the first require_once includes the cdash_test_case.php file
  • subsequent require_once calls refer to files from the root of the CDash source tree (include path set properly by the cdash_test_case.php file)
  • the constructor of the test case is empty -- it just calls the parent -- all the common stuff that used to be in the individual test cases has now been refactored to the base class -- see kwtest/kw_web_tester.php if you need to add to that code


  • similarly for the selenium tests... except with cdash_selenium_test_case.php
  • setUp for the individual selenium test cases should simply call $this->browserSetUp (defined once in the parent class)

Advanced Topics

Code coverage

Testing javascript functionality with Selenium

Client management

Setting up nightly Builds