MantisBT - CDash
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0007051CDashpublic2008-05-19 12:032008-07-29 13:26
ReporterBrad King 
Assigned ToJulien Jomier 
PrioritynormalSeverityminorReproducibilityN/A
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version1.2 
Target VersionFixed in Version1.2 
Summary0007051: Timing tests have too many false positives
DescriptionThe algorithm to detect time failures needs to be revised. One time sample lying outside the majority of the data is *not* a failure. Single outliers can occur from busy machines or any number of reasons. Only several samples can define a failure.

Statistically, the test would be something like deciding if the last N days are sampled from a distribution with a different mean than the distribution of the preceding M days (with M much bigger than N). Such a formal test would probably be too expensive. Instead we could use some kind of hack to reduce false positives by requiring that the last 3 or 4 days be outliers according to the current test before reporting the failure.
Steps To Reproduce
Additional Information
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2008-05-19 12:03Brad KingNew Issue
2008-05-19 12:06Julien JomierStatusnew => assigned
2008-05-19 12:06Julien JomierAssigned To => Julien Jomier
2008-07-29 13:26Julien JomierStatusassigned => closed
2008-07-29 13:26Julien JomierNote Added: 0012861
2008-07-29 13:26Julien JomierResolutionopen => fixed
2008-07-29 13:26Julien JomierFixed in Version => 1.2

Notes
(0012861)
Julien Jomier   
2008-07-29 13:26   
There is now a "Test time # max failures before flag" variable per project. That means that it doesn't display the timing error if the error occurs less than the max number of testing time failure.

Thanks for the report.