[Cdash] Does cdash detect changes in a test timing

[Olivier Pierard]

We faced similar problem in the past.  As you explained, test will fail
if and only if execution time is above mean+std for 3 consecutive days
or more.

So, on 21/11 and 22/11, it failed but not reported as #max failures is
set to 3.
On 23/11, it succeeds as 16.20 (exec. time ) < mean + std = 16.97

We already had a discussion on this forum with J. Jomier on this topic. 
To my point of view, mean value should not increase from one day to the
next one if test fails but is not reported (first and second 'failures'
in your case).  BTW, mean seems to be overweighted for recent days and
underweighted  for not so recent ones (TBC).

Olivier

> Huh.
>
> If you look at September 20's test result page:
>
>   http://www.cdash.org/CDash/testDetails.php?test=115301959&build=1548885
>
> You see "Execution Time (s)	2.65 (mean:2.66 std:0.03)" and "Test Timing: Passed"
>
> Then for subsequent days (9/21 through 9/25 ... and similarly beyond):
>
>   "Execution Time (s)	16.25 (mean:6.74 std:5.21)"
>   "Execution Time (s)	16.27 (mean:9.60 std:5.69)"
>   "Execution Time (s)	16.20 (mean:11.58 std:5.39)"
>   "Execution Time (s)	16.26 (mean:12.98 std:4.85)"
>   "Execution Time (s)	16.24 (mean:13.96 std:4.25)"
>
> ....but always with "Test Timing: Passed" even though the timing and
> the std. deviation shot through the roof on 9/21.
>
> I've not looked at the CDash code that measures/detects the test
> timing, but it does seem like there is probably something wrong with
> it.
>
> Sure smells like a CDash bug in here.
>
>
> David C.
>
>
> On Fri, Dec 9, 2011 at 11:32 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>   
>> http://www.cdash.org/CDash/testDetails.php?test=118063952&build=1643508
>>
>> The change occurred around Sept 20 and has increased since then.
>>
>> On Fri, Dec 9, 2011 at 11:28 AM, David Cole <david.cole at kitware.com> wrote:
>>     
>>> There's also a "Test time # max failures before flag:" setting, which
>>> is set to 3. Perhaps it needs to fail timing 3 days in a row before
>>> email is sent?
>>>
>>>
>>> On Fri, Dec 9, 2011 at 11:26 AM, David Cole <david.cole at kitware.com> wrote:
>>>       
>>>> There's an "Enable test timing" check box in the project settings, and
>>>> for "Insight" it is checked at the moment...
>>>>
>>>> What test is it that showed this increase? I would think that it would
>>>> show up as "timing failed" for a 5x increase...
>>>>
>>>>
>>>> On Fri, Dec 9, 2011 at 11:23 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>>>         
>>>>> We recently discovered that a change we made in itk caused a test to
>>>>> increase its time by over a factor of 5. I thought that cdash would
>>>>> report this. Is there a special setting to enable this?
>>>>>
>>>>> Bill
>>>>>
>>>>> --
>>>>> Unpaid intern in BillsBasement at noware dot com
>>>>> _______________________________________________
>>>>> Cdash mailing list
>>>>> Cdash at public.kitware.com
>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>>>>           
>>
>> --
>> Unpaid intern in BillsBasement at noware dot com
>>     
> _______________________________________________
> Cdash mailing list
> Cdash at public.kitware.com
> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>
>
>   


-- 

*Dr. Olivier Pierard*
Senior Research Scientist
Morfeo development team

*Cenaero*
/*Your trustworthy R&T partner*//
//*in modeling and numerical simulation*// /

Tel.: +32 71 910 974 *(NEW)*
Fax: +32 71 910 931 *(NEW)*

*Cenaero headquarters*
Rue des Fre`res Wright, 29
6041 Gosselies
Belgium

Visit our website: www.cenaero.be <http://www.cenaero.be/>

+-+-+- www.cenaero.be/disclaimer <http://www.cenaero.be/disclaimer> +-+-+-

Please consider the environment before printing this e-mail.