[Cdash] tracing regressions in cdash
matthias.brantner at 28msec.com
Wed Jun 17 08:17:21 UTC 2009
with our open-source XQuery processor Zorba, we are a very happy user
of CDash. Our projects submits a huge number of tests every day. Among
these tests is a W3C conformance testsuite with approx. 15k tests.
Currently, we do not pass all of these tests since we are still
working on getting version 1.0 out (i.e. between 100-500 tests are
still failing depending on the setup). As we are going along and
trying to fix more and more of these tests, it is a big problem for us
to trace regressions. For instance, we want to know if a commit makes
particular tests fail. Although, CDash does report an de-/increase of
failing/passing tests in the "Fail" and "Pass" columns this is
unfortunately not good enough for us because CDash doesn't tell us if
we had a regression. For instance, a commit could break one test but
make two others pass.
Does anybody else have this problem? Did you already think about
providing a feature in CDash that allows for such a regression
analysis? This would be something really helpful for us.
Thanks for doing all this work.
More information about the CDash