ITK/Policy and Procedure for Adding Dashboards

From KitwarePublic
Jump to navigationJump to search

This page is under construction

The Nightly and Continuous build/test of ITK is essential to its survival.


The ITK Dashboard presents dashboard submissions from the ITK community. The dashboards represent unique configurations of hardware platforms, operating systems, compilers, compiler options and ITK options.


Policy

Each dashboard shall present a unique combination of hardware/compiler/OS, optional features and compiler options. The dashboard submitter will monitor the ITK dashboard on a routine basis to make sure that the dashboard is reporting as a Nightly or Continuous build.

Terminology

Dashboard Types

You will see several types of builds reported on the dashboard. They are as follows:

  • Nightly - These are submissions that users have volunteered to submit on a nightly basis.
  • Continuous - These are dedicated machines that build and test the software every 1 to 3 hours. These submissions help catch problems as soon as possible.
  • Experimental - This type of build is typically for when a developer has made a change to the code and wants to 'prove' that he has done everything in his ability to test it before committing it to the repository.
  • Nightly Expected - These are identical to 'nightly' submissions, but they are 'expected' in the sense that the user has promised that this build will be submitted every night. This ensures that there is at least one of each reasonable system configuration tested every day.
  • Nightly Applications -
  • Nightly Releases -
  • Nightly External -
  • Coverage-
  • Style -
  • Dynamic Analysis -

The average user would want to either submit a Nightly or Experimental build.

Build Types

Another option that must be set before submitting a dashboard is the build type. The choices are as follows:

  • Debug - Compile and build with debugging information enabled
  • RelWithDebInfo -
  • MinSizeRel -
  • Release -

Procedure

The recommended method for dashboard submission is using ctest scripts.

Preparing a Dashboard Submission

  1. Use a CMake GUI (CMakeSetup on Windows or ccmake on UNIX) or the wizard mode (cmake -i) to edit cache entries.
  2. Perform a manual build.
  3. Run the tests.
  4. As of ITK 3.6, the configure process creates a file ${ITK_BINARY_DIR}/CMake/itkSampleBuildTest.cmake that contains ctest commands to build, test and submit a configuration that is the same as your configured build.
  5. Copy the itkSampleBuildTest.cmake file to another directory and rename it.
  6. Edit this file to select the type of submission to Nightly or Continuous
  7. Select the build type, either Debug, Release, RelWithDebInfo or MinSizeRel
  8. By default, the script will
    1. For nightly submissions, start with an empty binary directory. If your platform takes an unreasonable time to build from a clean directory, you can change CTEST_START_WITH_EMPTY_DIRECTORY to FALSE.
    2. For continuous submissions, start with an empty binary directory the first time it runs each day. Here again, if your platform takes a long time to build from a clean binary directory, you can change CTEST_START_WITH_EMPTY_DIRECTORY_ONCE to FALSE.
  9. Test your ctest script
    1. ctest -V -S your_script_file
    2. -V is the verbose output flag, and -S means read the configuration from a Script.

Scheduling a Dashboard Submission

Dashboard submissions can be automatically run with Cron or Scheduler.

Advanced Features

Memory Checker

Coverage

Code coverage can be performed with

  • gcov under Linux
  • BullsEye under Linux and Windows

These two applications compute code coverage in different ways

  • gcov computes line coverage
  • BullsEye computes branch coverage
Configuring GCOV Coverage
  • Make sure that the program gcov is installed in your system and that its version matches the version of the gcc compiler that you are using for building ITK. For example
    • gcov --version
    • gcc --version
  • Run ccmake in the binary directory where you build ITK
    • Add to CMAKE_CXX_FLAGS the flags: -g -O0 -fprofile-arcs -ftest-coverage
    • Add to CMAKE_C_FLAGS the flags: -g -O0 -fprofile-arcs -ftest-coverage
    • If using gcc4x, then the coverage flags must also be passed to the linker,
    • Make sure that the COVERAGE_COMMAND, CMake variable is pointing to the gcov executable that you want to use (the one that matches the version of gcc that you are using for this ITK build).
    • Type "c" for Configuring
    • Type "g" for Generating the Makefiles
  • At this point your build is configured for reporting code coverage
  • Test it by submitting an Experimental build using any of the two following commands:
    • make Experimental
    • ctest -D Experimental
  • The coverage submission should appear close to the bottom of the ITK Dashboard


To make this part of a Nightly or Continuous build, it is convenient to incorporate these settings into the CTest script that you are using for driving the build. The typical way of incorporating these option is to insert the following lines 

  COVERAGE_COMMAND:FILEPATH=/usr/bin/gcov-3.4
  CMAKE_CXX_FLAGS:STRING=-g -O0  -fprofile-arcs -ftest-coverage
  CMAKE_C_FLAGS:STRING=-g -O0  -fprofile-arcs -ftest-coverage
  CMAKE_EXE_LINKER_FLAGS:STRING=-g -O0  -fprofile-arcs -ftest-coverage

into the block: 

   SET (CTEST_INITIAL_CACHE "
   ...
   )


Configuring BullsEye Coverage

See instructions at

http://www.itk.org/Wiki/CTest:Coverage#C.2FC.2B.2B_Coverage_with_Bullseye

Style

OS Compiler Debug/Release Static/Shared 32/64 bit Wrapping

Retrieved from "https://public.kitware.com/Wiki/index.php?title=ITK/Policy_and_Procedure_for_Adding_Dashboards&oldid=16909"