[Cdash] Does cdash detect changes in a test timing

[David Cole]

Huh.

If you look at September 20's test result page:

  http://www.cdash.org/CDash/testDetails.php?test=115301959&build=1548885

You see "Execution Time (s)	2.65 (mean:2.66 std:0.03)" and "Test Timing: Passed"

Then for subsequent days (9/21 through 9/25 ... and similarly beyond):

  "Execution Time (s)	16.25 (mean:6.74 std:5.21)"
  "Execution Time (s)	16.27 (mean:9.60 std:5.69)"
  "Execution Time (s)	16.20 (mean:11.58 std:5.39)"
  "Execution Time (s)	16.26 (mean:12.98 std:4.85)"
  "Execution Time (s)	16.24 (mean:13.96 std:4.25)"

....but always with "Test Timing: Passed" even though the timing and
the std. deviation shot through the roof on 9/21.

I've not looked at the CDash code that measures/detects the test
timing, but it does seem like there is probably something wrong with
it.

Sure smells like a CDash bug in here.


David C.


On Fri, Dec 9, 2011 at 11:32 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> http://www.cdash.org/CDash/testDetails.php?test=118063952&build=1643508
>
> The change occurred around Sept 20 and has increased since then.
>
> On Fri, Dec 9, 2011 at 11:28 AM, David Cole <david.cole at kitware.com> wrote:
>> There's also a "Test time # max failures before flag:" setting, which
>> is set to 3. Perhaps it needs to fail timing 3 days in a row before
>> email is sent?
>>
>>
>> On Fri, Dec 9, 2011 at 11:26 AM, David Cole <david.cole at kitware.com> wrote:
>>> There's an "Enable test timing" check box in the project settings, and
>>> for "Insight" it is checked at the moment...
>>>
>>> What test is it that showed this increase? I would think that it would
>>> show up as "timing failed" for a 5x increase...
>>>
>>>
>>> On Fri, Dec 9, 2011 at 11:23 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>>> We recently discovered that a change we made in itk caused a test to
>>>> increase its time by over a factor of 5. I thought that cdash would
>>>> report this. Is there a special setting to enable this?
>>>>
>>>> Bill
>>>>
>>>> --
>>>> Unpaid intern in BillsBasement at noware dot com
>>>> _______________________________________________
>>>> Cdash mailing list
>>>> Cdash at public.kitware.com
>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>
>
>
> --
> Unpaid intern in BillsBasement at noware dot com