Defining Tests

Tests are specified to Dart using the ADD_TEST() command.

ADD_TEST(TestIdentifier ExecutableName [Arguments])

Each test must have a unique "TestIdentifier". The TestIdentifier is a string presented on the Dartboard. The ExecutableName can be a program built by your system project or may be any other executable on our computer (for instance Tcl, Python, etc.). Tests may have additional arguments.

The ADD_TEST() command provides a lot of flexibility. Each test could be a separate executable or each test could use the same executable but provide different arguments. The only restriction is the TestIdentifiers be unique.

If using CMake, the ADD_TEST() command is put in a CMakeList.txt file in the project source tree. If not using CMake, the ADD_TEST() command is put in a DartTestfile.txt file in the project build tree. (Under CMake, the ADD_TEST() commands in CMakeLists.txt files in the source tree are automatically written into DartTestfile.txt files in the build tree).

Test Status

The exit status of a test is used to determine whether a test passes or fails.

Test Results

A test's standard output and standard error are captured and presented on the DartBoard. More elaborate presentations can created by embedding formating commands for Dart in the standard output of your test. There are two formatting commands, DartMeasurement and DartMeasurementFile, which can be placed like XML tags in the standard output of a test

<DartMeasurement name="foo" type="numeric/double" 
                 encoding="none" compression="none">
    data
</DartMeasurement>

<DartMeasurementFile name="foo" type="image/jpeg">
    filename
</DartMeasrementFile>

• DartMeasurement - is used to display standard data types like numbers and strings.
• DartMeasurementFile - is used to display bulk data items like images. The data in the specified file will be compressed and encoded and placed in the Test.xml file. Once on the Dart server, the file will be extracted, decoded, and uncompressed.

The "type" can be one of the following:

• text/plain
• text/string
• text/html
• text/xml
• link/url
• link/image
• numeric/integer
• numeric/float
• numeric/double
• numeric/boolean
• image/png
• image/jpeg

The "encoding" can be:

• none (default, does not have to be specified)
• base64

The "compression" can be:

• none (default, does not have to be specified)
• gzip

The "encoding" and "compression" attributes specify how the data within the tag is encoded or compressed. In most cases, the encoding and compression will be "none".

The mechanims used to extract DartMeasurement and DartMeasurementFile tags from the standard output of a test places a few restrictions on the contents of the tags. The contents of a DartMeasurement tag cannot contain a "<" character.

DartMeasurements will be presented in a table. One column will present the "name"'s of the measurements, a second column will present the "values" (or image). After the DartMeasurements, the test's standard out (with all DartMeasurement tags stripped out) and standard error buffers will be displayed.

Along with DartMeasurements specified by the test, a standard set of DartMeasurements are also displayed

• Execution Time - Amount of time it took Dart to run the test. This includes the time start the executable.
• Completion Status - How the test exited. This can be "Completed" or "TimeOut".
• Exit Code - Displayed only if a test failed. This is either "CHILDSTATUS" indicating the test failed and outputted an non-zero exit value or "CHILDKILLED" indicating the test was aborted.
• Exit Explanation - Displayed only if the exit code is "CHILDKILLED". This measurement will convey whether the program crashed (segmentation violation) or was killed by the user.
• Exit Value - The exit value returned by the test. If a test completed, this will be numeric value return by the test's call to exit(). If the test crashed, this will be a code indicating the type of runtime error (for instance SIGSEGV).