[Cdash] Report of failed time status

[Julien Jomier]

Hi Olivier,

> 1°) Why not considering CPU times instead of Wall-clock times ?  Is it
> really difficult to implement (maybe more related to CTest than CDash) ?

This is related to CTest. I'll let Dave or Zach (in CC) comment on this.

> 2°) Let's consider we have a 'Test time # max failures before flag =
> 3'.  On the third day with higher test times, are the two first ones
> with test time failures considered in the average ?  I hope not because
> with a coefficient of 0.3 in the average, times before three days ago
> are almost negligible.  I already noticed that test failures not due to
> time are not taken into account into the average.

No the previous failed status are not considered in the average.

> 3°)  From documentation on vtk website:
>
> A test is defined as failing if it verifies the following: if previousSD
> <  thresholdSD then previousSD = thresholdSD.
>
>   if  currentTime>  previousMean+multiplier*previousSD.
>
>
> In my case, with following parameters:
> Test time SD (coefficient): 4.0
> Test time SD threshold : 1
> Test time # max failure before flag : 1
>
> And for a given test (reported on CDash - testcase report - exucution
> time(s) line):
> - mean:31.29
> - std:2.72
> - Execution time : 36.26
>
> =>  previous SD>  threshold =>  threshold not taken into account
> =>  previous mean + multiplier*previous SD = 31.29 + 4*2.72 = 42.17>  36.26
>
> ==>  Test time should be OK but is reported as failed and flagged on main
> CDash page !??
> Maybe reported mean and std  are not the previous ones but current
> ones.  If I go on the previous report, it's reported mean 29.16 -
> std:0.0.  Maybe this previous 0.0 is used.  But clearly on test times
> graph, there is a standard deviation (oscillates between 29 and 41
> during last month).

The average/SD are not recomputed when you change the threshold. From 
the question 2) you see that if before the threshold was not meeting the 
requirement, the average and SD were never computed based on historical 
values. This is probably why you got std:0.0. You should wait a couple 
of days and see if that helps.

> 4°) html link in project configuration in testing tab - in description
> of 'test standard deviation' and 'test standard deviation threshold',
> link to test timing description on the WIKI is wrong; maybe due to the
> fact that I'm using CDash 1.6.2

I added this in the bug tracker. Thanks for the report.

Julien