[Cdash] tracing regressions in cdash

[Matthias Brantner]

Hello,

with our open-source XQuery processor Zorba, we are a very happy user  
of CDash. Our projects submits a huge number of tests every day. Among  
these tests is a W3C conformance testsuite with approx. 15k tests.  
Currently, we do not pass all of these tests since we are still  
working on getting version 1.0 out  (i.e. between 100-500 tests are  
still failing depending on the setup). As we are going along and  
trying to fix more and more of these tests, it is a big problem for us  
to trace regressions. For instance, we want to know if a commit makes  
particular tests fail. Although, CDash does report an de-/increase of  
failing/passing tests in the "Fail" and "Pass" columns this is  
unfortunately not good enough for us because CDash doesn't tell us if  
we had a regression. For instance, a commit could break one test but  
make two others pass.

Does anybody else have this problem? Did you already think about  
providing a feature in CDash that allows for such a regression  
analysis? This would be something really helpful for us.

Thanks for doing all this work.

Best regards

Matthias