ITK/Policy and Procedure for Adding Dashboards: Difference between revisions

From KitwarePublic
< ITK
Jump to navigationJump to search
No edit summary
 
(11 intermediate revisions by 3 users not shown)
Line 4: Line 4:




The [http://www.cdash.org/CDash/index.php?project=Insight ITK Dashboard] presents dashboard submissions from the ITK community. The dashboards represent unique configurations of hardware platforms, operating systems, compilers, compiler options and ITK options.
The [http://open.cdash.org/index.php?project=Insight ITK Dashboard] presents dashboard submissions from the ITK community. The dashboards represent unique configurations of hardware platforms, operating systems, compilers, compiler options and ITK options.




Line 16: Line 16:
* Nightly - These are submissions that users have volunteered to submit on a nightly basis.
* Nightly - These are submissions that users have volunteered to submit on a nightly basis.
* Continuous - These are dedicated machines that build and test the software every 1 to 3 hours. These submissions help catch problems as soon as possible.
* Continuous - These are dedicated machines that build and test the software every 1 to 3 hours. These submissions help catch problems as soon as possible.
* Experimental - This type of build is typically for when a developer has made a change to the code and wants to 'prove' that he has done everything in his ability to test it before committing it to the repository.
* Experimental - This type of build is typically for when a developer has made a change to the code and wants to 'prove' that he has done everything in his ability to test it before committing it to the repository. CDash@home builds also show up as Experimental.
* Nightly Expected - These are identical to 'nightly' submissions, but they are 'expected' in the sense that the user has promised that this build will be submitted every night. This ensures that there is at least one of each reasonable system configuration tested every day.
* Nightly Expected - These are identical to 'nightly' submissions, but they are 'expected' in the sense that the user has promised that this build will be submitted every night. This ensures that there is at least one of each reasonable system configuration tested every day.
* Nightly Applications - This is a Nightly build of the InsightApplications module
* Nightly Applications - This is a Nightly build of the InsightApplications module
Line 23: Line 23:
* Coverage- These are reports of code coverage. They are typically generated with "gcov", although they could be generated with BullsEye as well.
* Coverage- These are reports of code coverage. They are typically generated with "gcov", although they could be generated with BullsEye as well.
* Style - These are builds that run KWStyle on the ITK source code in order to detect potential coding style violations.
* Style - These are builds that run KWStyle on the ITK source code in order to detect potential coding style violations.
* Dynamic Analysis - These are builds in which the test are run through a memory checker, typically Valgrind, in order to detect uninitialized variables, memory leaks and other run-time problems.
* Dynamic Analysis - These are builds in which the tests are run through a memory checker, typically Valgrind, in order to detect uninitialized variables, memory leaks and other run-time problems.


The average user would want to either submit a Nightly or Experimental build.
The average user would want to either submit a Nightly or Experimental build.
Line 32: Line 32:
Nightly and Continuous builds will perform the following sequence of steps:
Nightly and Continuous builds will perform the following sequence of steps:


# CVS update
# Update (git fetch, git reset)
# CMake configuration
# CMake configuration
# Build (compile and link)
# Build (compile and link)
Line 38: Line 38:
# Submit results to the Dashboard
# Submit results to the Dashboard


Experimental build, on the other hand, will skip the CVS update step, and start with the configuration step.
Experimental builds, on the other hand, will skip the update step, and start with the configuration step.


=== Build Types (Compilation Modes)  ===
=== Build Types (Compilation Modes)  ===
Line 45: Line 45:
* RelWithDebInfo - Compile with optimization options and debug sysmbols.
* RelWithDebInfo - Compile with optimization options and debug sysmbols.
* MinSizeRel -  
* MinSizeRel -  
* Release -Enable optimization (e.g. -O3 in gcc), disable debugging symbols.
* Release - Enable optimization (e.g. -O3 in gcc), disable debugging symbols.


== Procedure ==
== Procedure ==
The recommended method for dashboard submission is using [http://www.cmake.org/Wiki/CMake_Scripting_Of_CTest ctest scripts].
The recommended method for dashboard submission is using [http://www.cmake.org/Wiki/CMake_Scripting_Of_CTest ctest scripts]. Specifically, a small script that includes itk_common.cmake to do the bulk of the work. See [http://www.itk.org/Wiki/ITK/Git/Dashboard here for itk_common.cmake details]. (See also the commented notes at the top of that script.)


=== Preparing a Dashboard Submission ===
=== Preparing a Dashboard Submission ===
# Use a CMake GUI (CMakeSetup on Windows or ccmake on UNIX) or the wizard mode (cmake -i) to edit cache entries.
 
# Perform a manual build.
==== New, easier way ====
# Run the tests. (The simplest way is to go to the binary directory and type: ctest )
# Follow [http://www.itk.org/Wiki/ITK/Git/Dashboard the directions here] to get the latest itk_common.cmake.
# As of ITK 3.6, the configure process creates a file ''${ITK_BINARY_DIR}/CMake/itkSampleBuildTest.cmake'' that contains ctest commands to build, test and submit a configuration that is the same as your configured build.
# Write a small ctest script according to the instructions at the top of itk_common.cmake.
# Copy the ''itkSampleBuildTest.cmake'' file to another directory and rename it.
# Edit this file to select the type of submission to ''Nightly'' or ''Continuous''
# Select the build type, either ''Debug'', ''Release'', ''RelWithDebInfo'' or ''MinSizeRel''
# By default, the script will
## For nightly submissions, start with an empty binary directory. If your platform takes an unreasonable time to build from a clean directory, you can change '''CTEST_START_WITH_EMPTY_DIRECTORY''' to '''FALSE'''.
## For continuous submissions, start with an empty binary directory the first time it runs each day. Here again, if your platform takes a long time to build from a clean binary directory, you can change '''CTEST_START_WITH_EMPTY_DIRECTORY_ONCE''' to '''FALSE'''.
#Specify your "site name" by specifying the "SITE:STRING" in the CTEST_INITIAL_CACHE section of the itkSampleBuildTest.cmake file. Your site name should be descriptive enough so we can reach you in case something fails on your dashboard.
#specify your "build name" by specifying the BUILDNAME:STRING" in the CTEST_INITIAL_CACHE section of the itkSampleBuildTest.cmake file. Your build name should let us know the relevant details of your system, such as operating system and compiler.
# Test your ctest script
## ctest -V -S ''your_script_file''
## ctest -V -S ''your_script_file''
##-V is the verbose output flag, and -S means read the configuration from a Script.
# See also, the [http://insightsoftwareconsortium.github.io/ITKBarCamp-doc/SoftwareQuality/DashboardSubmission/index.html ITK Bar Camp article].
##If the tests finish before 21:00, the build will be submitted to the current days dashboard. Else, it will be submitted for tomorrows dashboard.


=== Scheduling a Dashboard Submission ===
=== Scheduling a Dashboard Submission ===
# Dashboard submissions can be automatically run with [http://www.cmake.org/Wiki/CMake_Scripting_Of_CTest#Setting_Up_Cron.2FScheduler Cron or Scheduler].
# Dashboard submissions can be automatically run with [http://www.cmake.org/Wiki/CMake_Scripting_Of_CTest#Setting_Up_Cron.2FScheduler Cron or Scheduler].
# If the cronjob finishes before 21:00, the build will get submitted to the current days dashboard. The ideal case is for the build to finish AFTER 21:00 so that it is seen on the next days dashboard.
# The ITK Nightly start time (from CTestConfig.cmake) is 01:00 UTC
# Builds should be scheduled such that they start '''after''' the 01:00 UTC Nightly start time (that's 8:00 pm Eastern time in the winter, 9:00 pm Eastern time in the summer, so start builds after 9:00 pm Eastern in your local time and it will always be after the nightly start time all year round...) Dashboard best practice: [http://www.kitware.com/blog/home/post/77 avoid problems when moving] to or from [http://en.wikipedia.org/wiki/Daylight_saving_time daylight saving time]


=== Advanced Features ===
=== Advanced Features ===
==== Run as a specific user ====
It is a good idea to create a specific user with limited rights to run the nightly test. For example a user called '''dashboard'''.
  adduser dashboard
  su dashboard
And follow the procedure mentioned above to set up the nighly test. Quota limitation for disk space and limited directory traversal can be applied to this user.
==== Memory Checker ====
==== Memory Checker ====
==== Coverage ====
==== Coverage ====

Latest revision as of 01:51, 23 May 2014

This page is under construction

The Nightly and Continuous build/test of ITK is essential to its survival.


The ITK Dashboard presents dashboard submissions from the ITK community. The dashboards represent unique configurations of hardware platforms, operating systems, compilers, compiler options and ITK options.


Policy

Each dashboard shall present a unique combination of hardware/compiler/OS, optional features and compiler options. The dashboard submitter will monitor the ITK dashboard on a routine basis to make sure that the dashboard is reporting as a Nightly or Continuous build.

Terminology

Build Submission Types

You will see several groups of builds reported on the dashboard. They are as follows:

  • Nightly - These are submissions that users have volunteered to submit on a nightly basis.
  • Continuous - These are dedicated machines that build and test the software every 1 to 3 hours. These submissions help catch problems as soon as possible.
  • Experimental - This type of build is typically for when a developer has made a change to the code and wants to 'prove' that he has done everything in his ability to test it before committing it to the repository. CDash@home builds also show up as Experimental.
  • Nightly Expected - These are identical to 'nightly' submissions, but they are 'expected' in the sense that the user has promised that this build will be submitted every night. This ensures that there is at least one of each reasonable system configuration tested every day.
  • Nightly Applications - This is a Nightly build of the InsightApplications module
  • Nightly Releases - This is a Nightly build of one official release of ITK, for example, a Nightly build of the ITK 3.12 branch.
  • Nightly External - This are Nightly builds of external projects that use ITK. They are used as early warnings for detecting if any changes in ITK compromise its backward compatibility
  • Coverage- These are reports of code coverage. They are typically generated with "gcov", although they could be generated with BullsEye as well.
  • Style - These are builds that run KWStyle on the ITK source code in order to detect potential coding style violations.
  • Dynamic Analysis - These are builds in which the tests are run through a memory checker, typically Valgrind, in order to detect uninitialized variables, memory leaks and other run-time problems.

The average user would want to either submit a Nightly or Experimental build.


Note:

Nightly and Continuous builds will perform the following sequence of steps:

  1. Update (git fetch, git reset)
  2. CMake configuration
  3. Build (compile and link)
  4. Run tests
  5. Submit results to the Dashboard

Experimental builds, on the other hand, will skip the update step, and start with the configuration step.

Build Types (Compilation Modes)

Another option that must be set before submitting a dashboard is the build type. The choices are as follows:

  • Debug - Compile and build with debugging information enabled (e.g. -g in gcc),
  • RelWithDebInfo - Compile with optimization options and debug sysmbols.
  • MinSizeRel -
  • Release - Enable optimization (e.g. -O3 in gcc), disable debugging symbols.

Procedure

The recommended method for dashboard submission is using ctest scripts. Specifically, a small script that includes itk_common.cmake to do the bulk of the work. See here for itk_common.cmake details. (See also the commented notes at the top of that script.)

Preparing a Dashboard Submission

New, easier way

  1. Follow the directions here to get the latest itk_common.cmake.
  2. Write a small ctest script according to the instructions at the top of itk_common.cmake.
    1. ctest -V -S your_script_file
  3. See also, the ITK Bar Camp article.

Scheduling a Dashboard Submission

  1. Dashboard submissions can be automatically run with Cron or Scheduler.
  2. The ITK Nightly start time (from CTestConfig.cmake) is 01:00 UTC
  3. Builds should be scheduled such that they start after the 01:00 UTC Nightly start time (that's 8:00 pm Eastern time in the winter, 9:00 pm Eastern time in the summer, so start builds after 9:00 pm Eastern in your local time and it will always be after the nightly start time all year round...) Dashboard best practice: avoid problems when moving to or from daylight saving time

Advanced Features

Run as a specific user

It is a good idea to create a specific user with limited rights to run the nightly test. For example a user called dashboard. 

  adduser dashboard
  su dashboard

And follow the procedure mentioned above to set up the nighly test. Quota limitation for disk space and limited directory traversal can be applied to this user.

Memory Checker

Coverage

Code coverage can be performed with

  • gcov under Linux
  • BullsEye under Linux and Windows

These two applications compute code coverage in different ways

  • gcov computes line coverage
  • BullsEye computes branch coverage
Configuring GCOV Coverage
  • Make sure that the program gcov is installed in your system and that its version matches the version of the gcc compiler that you are using for building ITK. For example
    • gcov --version
    • gcc --version
  • Run ccmake in the binary directory where you build ITK
    • Add to CMAKE_CXX_FLAGS the flags: -g -O0 -fprofile-arcs -ftest-coverage
    • Add to CMAKE_C_FLAGS the flags: -g -O0 -fprofile-arcs -ftest-coverage
    • If using gcc4x, then the coverage flags must also be passed to the linker,
    • Make sure that the COVERAGE_COMMAND, CMake variable is pointing to the gcov executable that you want to use (the one that matches the version of gcc that you are using for this ITK build).
    • Type "c" for Configuring
    • Type "g" for Generating the Makefiles
  • At this point your build is configured for reporting code coverage
  • Test it by submitting an Experimental build using any of the two following commands:
    • make Experimental
    • ctest -D Experimental
  • The coverage submission should appear close to the bottom of the ITK Dashboard


To make this part of a Nightly or Continuous build, it is convenient to incorporate these settings into the CTest script that you are using for driving the build. The typical way of incorporating these option is to insert the following lines 

  COVERAGE_COMMAND:FILEPATH=/usr/bin/gcov-3.4
  CMAKE_CXX_FLAGS:STRING=-g -O0  -fprofile-arcs -ftest-coverage
  CMAKE_C_FLAGS:STRING=-g -O0  -fprofile-arcs -ftest-coverage
  CMAKE_EXE_LINKER_FLAGS:STRING=-g -O0  -fprofile-arcs -ftest-coverage

into the block: 

   SET (CTEST_INITIAL_CACHE "
   ...
   )


Configuring BullsEye Coverage

See instructions at

http://www.itk.org/Wiki/CTest:Coverage#C.2FC.2B.2B_Coverage_with_Bullseye

Style

OS Compiler Debug/Release Static/Shared 32/64 bit Wrapping

Retrieved from "https://public.kitware.com/Wiki/index.php?title=ITK/Policy_and_Procedure_for_Adding_Dashboards&oldid=56358"