Difference between revisions of "CDash:Testing"

From KitwarePublic
Jump to navigationJump to search
 
(7 intermediate revisions by 3 users not shown)
Line 1: Line 1:
CDash integrates a self-testing framework based on [http://www.simpletest.org Simple Test]. The self testing code is located in the ''CDash/testing'' directory.
CDash integrates a self-testing framework based on [http://www.simpletest.org Simple Test]. The self testing code is located in the ''CDash/testing'' directory.


= Submitting a CDash Dashboard to the CDash dashboard =
CDash testing requires the use of CMake and CTest.  You can download the CMake suite of tools from [http://www.cmake.org/cmake/resources/software.html www.cmake.org].  CDash testing works much the same as any other CMake based project.


Don't worry, it's not a mistake in the title of this section, self-testing involves submitting a CDash submission to CDash.
= Checkout the CDash source code =


=== Testing CDash with CTest ===
See [[CDash:Installation#Downloading_CDash|Downloading CDash]] for instructions on obtaining a copy of the CDash source code from GitHubDuring this step you should decide whether you want to test the latest release of CDash or the current development version.
CDash can be tested with CTest from cmake (www.cmake.org.)For this style of testing, you treat CDash much like any other CMake based project.


* Checkout the CDash source code from the svn repository to a directory named "CDashTesting" in your web server's htdocs folder.
No matter which version of CDash you choose to test, make sure to check out the source code into your web server's htdocs folder.


* assume that the full path to CDash is /path/to/source/CDashTesting
= Setup CDash for testing =


* Create a build directory
== Create a build directory ==
  cd /path/to/source/CDashTesting
  mkdir build


* Run cmake on on CDashTesting
cd /path/to/CDash
  cd build
mkdir build
  cmake ..


  There are some CMake cache options that configure server names, the default is localhost.
==Run CMake from the build directory ==


* For coverage, you need to have xdebug installed as a module in your php installation. Once that is done, you need to edit the php.ini file, and add these two lines and restart apache:
cd build
  auto_prepend_file =/path/to/source/CDashTesting/prepend_coverage.php
ccmake ..
  auto_append_file =/path/to/source/CDashTesting/build/append_coverage.php


For coverage to work, a CMake next newer than 5/25/2010 at 4:00 pm will be needed.
== Define CMake variables ==


* Note this will create '''CDashTesting/cdash/config.local.php''' and '''CDashTesting/testing/config.test.local.php''' files in your source tree, so svn will show them as ? files.
CDASH_DB_HOST
CDASH_DB_LOGIN
CDASH_DB_PASS
CDASH_DB_TYPE
*These four parameters tell CDash how to connect to your local database server.


* Run ctest on the build tree, like any other ctest project. For example, this will run an experimental coverage build:
CDASH_SERVER
  cd /path/to/source/CDashTesting/build
*CDASH_SERVER should indicate how to connect to your local system via a web browser.  This is typically the system's host name, or simply localhost.
  ctest -D ExperimentalStart -D ExperimentalConfigure -D ExperimentalTest -D ExperimentalCoverage -D ExperimentalSubmit


=== Configuring the testing ===
CDASH_USE_APACHE2
Here are the steps on how to configure the testing without ctest:
*Setting this to ON will enable tests specific to the Apache web server.


* Checkout the CDash source code from the svn repository to a directory named "CDashTesting" in your web server's htdocs folder.
CDASH_USE_SELENIUM
PHPUNIT_EXE
*Setting these options will enable testing of javascript based functionality of CDash.  Requires [http://seleniumhq.org/ selenium] and [http://www.phpunit.de/ PHPUnit].  More on this below.


* Add a file named '''CDashTesting/cdash/config.local.php''' containing:
CMake_SOURCE_DIR
*Point this to a local checkout of CMake source code if you'd like to import some tests from CMake into CDash.  This will cause CDash to perform a more thorough end-to-end test of the CMake/CTest/CDash suite of tools.


<pre>
PHP_EXE
<?php
*CDash testing requires a command-line version of PHP to be installed on the system.
$CDASH_DB_NAME = 'cdash4simpletest';
 
$CDASH_TESTING_MODE = true;
= Test CDash =
?>
 
</pre>
This is as simple as entering your CDash build directory and running CTest
  cd /path/to/CDashTesting/build
  ctest -VV
 
Note this will create '''CDashTesting/cdash/config.local.php''' and '''CDashTesting/testing/config.test.local.php''' files in your source tree, so svn will show them as ? files.


This file is automatically included by CDash's '''config.php''', but only if it exists.
'''For PostGreSQL, make sure you create an empty database named 'host' so that CDash can create the cdash4simpletest database'''


The database has to be named exactly 'cdash4simpletest' for the tests to run. Part of the test suite involves destroying the existing database of that name and re-creating it from scratch. Because the testing of CDash drops and creates the database you need to make sure this database is not being used for any other purpose than self testing. Also, make sure that the $CDASH_DB_LOGIN has enough privileges to create/drop databases on your SQL server.
= Writing Additional CDash Tests =


* Add a file named '''CDashTesting/testing/config.test.local.php''' containing (example values only... adjust to suit your localhost web server):
If you add any new tests, please follow the structure of one of the existing ones.


<pre>
For example, at the top of each test_*.php file in the tests directory, you'll see something like this (example from test_banner.php):
<source lang="php">
<?php
<?php
$configure = array(
//
  'urlwebsite'      => 'http://localhost/CDashTesting',
// After including cdash_test_case.php, subsequent require_once calls are
  'outputdirectory'  => 'C:/tmp',  // or /tmp or ~/tmp on a Linux/Mac server
// relative to the top of the CDash source tree
  'type'             => 'Nightly',
//
  'site'            => 'computername.companyname',
require_once(dirname(__FILE__).'/cdash_test_case.php');
  'buildname'       => 'SVN-Win32-xampp',
 
  'cdash'            => 'http://www.cdash.org/CDash',
require_once('cdash/common.php');
   'svnroot'          => 'C:/xampp/htdocs/CDashTesting'
require_once('cdash/pdo.php');
   );
require_once('models/banner.php');
 
class BannerTestCase extends KWWebTestCase
{
   function __construct()
    {
    parent::__construct();
    }
 
   function testBanner()
    {
    ...
    }
}
?>
?>
</pre>
</source>
 
Please note the following:
* do not use "include" -- use "require_once"
* the first require_once uses dirname(__FILE__) meaning "same dir as this php file"
* the first require_once includes the cdash_test_case.php file
* subsequent require_once calls refer to files from the root of the CDash source tree (include path set properly by the cdash_test_case.php file)
* the constructor of the test case is '''empty''' -- it just calls the parent -- all the common stuff that used to be in the individual test cases has now been refactored to the base class -- see kwtest/kw_web_tester.php if you need to add to that code
 
 
* similarly for the selenium tests... except with cdash_selenium_test_case.php
* setUp for the individual selenium test cases should simply call $this->browserSetUp (defined once in the parent class)


This file is automatically included by CDash's '''config.test.php''', but only if it exists.
= Advanced Topics =


* '''urlwebsite''' is the URL to access the tested installation of CDash. Since simpletest emulates a web browser you need to make sure that the urlwebsite is accessible locally. That is, you should be able to visit this URL in a web browser on the testing machine and see the main CDash page.
[[CDash:Testing/Coverage|Code coverage]]
* '''outputdirectory''' is a directory where the testing output can be stored on the system. Make sure that the current user can write to that directory.
* '''type''' is the type of submission for CDash: Nightly, Continuous or Experimental.
* '''site''' is the name of the site submitting to CDash.
* '''buildname''' is name of the build.
* '''cdash''' is the URL to the current dashboard. Note that the project should be named CDash, i.e. http://www.cdash.org/CDash/submit.php?project=CDash
* '''svnroot''' is the file system path to your testing installation of CDash. The self testing does an svn update to check what files have changed.


=== Starting the testing ===
[[CDash:Testing/Selenium|Testing javascript functionality with Selenium]]
Then to launch the tests and submit a dashboard:
  cd CDashTesting/testing
  php5 alltests.php


For xampp installations on Windows, run:
[[CDash:Testing/Client Management|Client management]]
  cd CDashTesting\testing
  C:\xampp\php\php.exe alltests.php


To run the tests in your browser (without submitting a dashboard) visit the URL:
[[CDash:Testing/Nightly|Setting up nightly Builds]]
  http://localhost/CDashTesting/testing/alltests.php

Latest revision as of 20:38, 27 February 2017

CDash integrates a self-testing framework based on Simple Test. The self testing code is located in the CDash/testing directory.

CDash testing requires the use of CMake and CTest. You can download the CMake suite of tools from www.cmake.org. CDash testing works much the same as any other CMake based project.

Checkout the CDash source code

See Downloading CDash for instructions on obtaining a copy of the CDash source code from GitHub. During this step you should decide whether you want to test the latest release of CDash or the current development version.

No matter which version of CDash you choose to test, make sure to check out the source code into your web server's htdocs folder.

Setup CDash for testing

Create a build directory

cd /path/to/CDash
mkdir build

Run CMake from the build directory

cd build
ccmake ..

Define CMake variables

CDASH_DB_HOST
CDASH_DB_LOGIN
CDASH_DB_PASS
CDASH_DB_TYPE
  • These four parameters tell CDash how to connect to your local database server.
CDASH_SERVER
  • CDASH_SERVER should indicate how to connect to your local system via a web browser. This is typically the system's host name, or simply localhost.
CDASH_USE_APACHE2
  • Setting this to ON will enable tests specific to the Apache web server.
CDASH_USE_SELENIUM
PHPUNIT_EXE
  • Setting these options will enable testing of javascript based functionality of CDash. Requires selenium and PHPUnit. More on this below.
CMake_SOURCE_DIR
  • Point this to a local checkout of CMake source code if you'd like to import some tests from CMake into CDash. This will cause CDash to perform a more thorough end-to-end test of the CMake/CTest/CDash suite of tools.
PHP_EXE
  • CDash testing requires a command-line version of PHP to be installed on the system.

Test CDash

This is as simple as entering your CDash build directory and running CTest

 cd /path/to/CDashTesting/build
 ctest -VV

Note this will create CDashTesting/cdash/config.local.php and CDashTesting/testing/config.test.local.php files in your source tree, so svn will show them as ? files.

For PostGreSQL, make sure you create an empty database named 'host' so that CDash can create the cdash4simpletest database

Writing Additional CDash Tests

If you add any new tests, please follow the structure of one of the existing ones.

For example, at the top of each test_*.php file in the tests directory, you'll see something like this (example from test_banner.php):

<?php
//
// After including cdash_test_case.php, subsequent require_once calls are
// relative to the top of the CDash source tree
//
require_once(dirname(__FILE__).'/cdash_test_case.php');

require_once('cdash/common.php');
require_once('cdash/pdo.php');
require_once('models/banner.php');

class BannerTestCase extends KWWebTestCase
{
  function __construct()
    {
    parent::__construct();
    }

  function testBanner()
    {
    ...
    }
}
?>

Please note the following:

  • do not use "include" -- use "require_once"
  • the first require_once uses dirname(__FILE__) meaning "same dir as this php file"
  • the first require_once includes the cdash_test_case.php file
  • subsequent require_once calls refer to files from the root of the CDash source tree (include path set properly by the cdash_test_case.php file)
  • the constructor of the test case is empty -- it just calls the parent -- all the common stuff that used to be in the individual test cases has now been refactored to the base class -- see kwtest/kw_web_tester.php if you need to add to that code


  • similarly for the selenium tests... except with cdash_selenium_test_case.php
  • setUp for the individual selenium test cases should simply call $this->browserSetUp (defined once in the parent class)

Advanced Topics

Code coverage

Testing javascript functionality with Selenium

Client management

Setting up nightly Builds