MantisBT - CDash
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0008643CDashpublic2009-02-28 10:492009-03-30 08:17
ReporterRoscoe A. Bartlett 
Assigned ToJulien Jomier 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version1.4 
Summary0008643: CDash Dev: Email summary lines sent out by CDash not very good for Trilinos
Description

Now that I just started getting failure email notifications from CDash last night, I am seeing that the subject line is not very good. I have experimented a lot with subject lines for such emails in our testing work and I find the the following bits of information are very useful.

1) FAILED!
2) Identify the build case
3) How many builds/tests failed

You should bias the most important info at the very front of the subject line.

Here is the current subject line (for the email shown below):

    CDash [Trilinos] - s909348.sandia.gov - Darwin-Nightly-MPI_RELEASE - Nightly - 2009-02-28T00:37:17 MST

This is way too long and for Trilinos does not provide very useful information. What we would like to see is something like:

    FAILED (t=1): Trilinos/Thyra - Darwin-Nightly-MPI_RELEASE - s909348.sandia.gov - Nightly - 2009-02-28T00:37:17 MST

This way, if my email client (Outlook in my case) cuts off the end of the subject I can still see:

    FAILED (t=1): Trilinos/Thyra - Darwin-Nightly-MPI_RELEASE ...

The is enough information tells me that one of the tests failed (t=1) for Thyra on the Mac (Darwin) for MPI.

If there is a build failure followed by say 5 test failes then I would like to see:

    FAILED (b=1, t=5): Trilinos/Thyra - Darwin-Nightly-MPI_RELEASE ...

The current emails subjects are not very useful for the Trilinos project. You have to click on the email and open it up to see any really useful information. The key to Trilinos is the package subdivision and everything should be marked with the Trilinos package name.

Also, does CDash never send out emails for failed "Experimental" builds? I set my account to "Email me when any builds are breaking the dashboard" but it did not send me emails on failed experimental submissions. Does "any builds" mean everything except for "Experimental"?




-----Original Message-----
From: CDash [mailto:bmpersc@sandia.gov]
Sent: Saturday, February 28, 2009 12:51 AM
To: Bartlett, Roscoe A
Subject: CDash [Trilinos] - s909348.sandia.gov - Darwin-Nightly-MPI_RELEASE - Nightly - 2009-02-28T00:37:17 MST

A submission to CDash for the project Trilinos has failing tests.
You have been identified as one of the authors who have checked in changes that are part of this submission or you are listed in the default contact list.

Details on the submission can be found at http://trilinos-dev.sandia.gov/cdash/buildSummary.php?buildid=1520 [^]

Project: Trilinos
Site: s909348.sandia.gov
BuildName: Darwin-Nightly-MPI_RELEASE
Type: Nightly
Tests failing: 1


*Tests failing*
Thyra_test_linear_op_with_solve_n4_MPI_1(http://trilinos-dev.sandia.gov/cdash/testDetails.php?test=14852&build=1520 [^])


-CDash on trilinos-dev.sandia.gov
Steps To Reproduce
Additional Information
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2009-02-28 10:49Roscoe A. BartlettNew Issue
2009-02-28 10:57Julien JomierStatusnew => assigned
2009-02-28 10:57Julien JomierAssigned To => Julien Jomier
2009-02-28 10:59Julien JomierNote Added: 0015469
2009-03-06 17:57Roscoe A. BartlettNote Added: 0015594
2009-03-09 19:51Julien JomierNote Added: 0015614
2009-03-09 19:51Julien JomierStatusassigned => resolved
2009-03-09 19:51Julien JomierFixed in Version => 1.4
2009-03-09 19:51Julien JomierResolutionopen => fixed
2009-03-30 08:17Julien JomierStatusresolved => closed

Notes
(0015469)
Julien Jomier   
2009-02-28 10:59   
CDash should send emails even for experimental builds if they have errors.
(0015594)
Roscoe A. Bartlett   
2009-03-06 17:57   
Responding to "CDash should send emails even for experimental builds if they have errors" I think that this should at least be an option to turn off or on. I don't want to get an email about an error in one of my packages if the person who runs the build is going to fix it anyway before they check in. Can we add some radio option boxes like:

  Receive email notifications for:

   [x] Nightly
   [x] Continuous
   [ ] Experimental
   [x] Coverage
   [x] Dynamic Analysis

Above I did not check the 'Experimental' because I don't want to see these.



Also, below I append some info I sent out in an email about the format of the email summary lines:


-----Original Message-----
From: Bartlett, Roscoe A
Sent: Friday, March 06, 2009 3:51 PM
To: 'Julien Jomier'
Cc: Bill Hoffman; David Cole; Perschbacher, Brent M; Willenbring, James M
Subject: RE: What is Cdash logic for sending out failure emails or not?

Julien,

Like I said, fixing this 'Previous', 'Next' behavior is not a high priority, especially since the side-effect is that we get emails on every failure.

At this point, the next highest value item is to get the failure email summary message customized for each Trilinos package and mentioned at:

    https://www.kitware.com/cmake-cdash-trilinos [^]

The format suggested in http://public.kitware.com/Bug/view.php?id=8643 [^] is to have something like:

  FAILED (b=1, t=5): Trilinos/Thyra - Darwin-Nightly-MPI_RELEASE - s909348.sandia.gov - Nightly - 2009-02-28T00:37:17 MST

Basically the idea is to sort the most important info to the left since most email client configurations will cut off longer summary messages by default.

The idea of (b=?) and (t=?) is to give the number of build (b) and test (t) failures being reported. I also see that issue http://public.kitware.com/Bug/view.php?id=8675 [^] mentions sending out emails for coverage and dynamic analysis testing. That would give the types of failures:

   b=?: Number of build failures
   t=?: Number of test failures
   c=?: Number of coverage failures
   d=?: Number of dynamic analysis failures

Right now I get several Cdash error messages every night and it would be good to be able to see what these are without having to click and open up the emails.

Again, the most important info to put in the summary line, in order of priority, is:

1) FAILED! (i.e. please don't ignore me)
2) What Trilinos package failed?
3) What failed? (i.e. build? tests?, etc.)

Above I have the (b=?, t=?, ...) before the Trilinos package name even through the Trilinos package identification is more important. However, I think it looks better and makes more sense in the order 'FAILED (what failed): Trilinos/PACKAGE ...'.

Once we get the email summary lines done, the next most high value item would be:

   Filter builds on the main project/subproject page (http://public.kitware.com/Bug/view.php?id=8592 [^])

After that I think the type of information being collected, displayed, and sent out in emails is pretty darn good.

Thanks,

- Ross
(0015614)
Julien Jomier   
2009-03-09 19:51   
Not fully tested, so let me know if the line is not as expected