[CMake] test property COST not working in cmake 2.8.3?

[Tyler Roscoe]

On Tue, Nov 30, 2010 at 01:29:37PM -0500, Zach Mullen wrote:
> Hm, yours was a use case we didn't really consider when we were making
> changes to cost behavior.  

Clearly. :)

> The middle ground here would be to respect costs in the non-parallel
> case when they are expressed explicitly

This sounds good.

> but not to cost-order them automatically based on their previous run
> times.

I don't care too much about this behavior, but it seems like a nice
feature. Perhaps if CTest reserved a range for its own COST data (-10 <=
CTest-calculated COST <= 10?) then users could use costs < -10 or > 10
to insure ordering of certain tests?

So what are the next steps? Should I open a bug to track this issue? Is
this an easy change (so easy a monkey like me can do it) or can someone
at Kitware take care of it? I am able and eager to test a CMake nightly
build that fixes this bug/regression/misunderstanding.

Thanks,
tyler

> On Tue, Nov 30, 2010 at 12:43 PM, Tyler Roscoe <tyler at cryptio.net> wrote:
> 
> > On Fri, Nov 26, 2010 at 10:38:44AM -0500, Zach Mullen wrote:
> > > I just realized why this isn't working -- it's actually not a regression.
> >
> > Maybe we have different definitions of "regression". I see a feature
> > that used to do one thing but which now does something else.
> >
> > Here is what the docs say about the COST property:
> >
> >    # COST: Set this to a floating point value. Tests in a test set will be
> >    # run in descending order of cost.
> >
> >    This property describes the cost of a test. You can explicitly set this
> >    value; tests with higher COST values will run first.
> >
> > I don't see anything there about parallel or non-parallel runs. It seems
> > to me that if I set the COST property, I should be able to control the
> > order in which tests run, period. So at the very least, the docs should
> > be updated if you intend to change the behavior.
> >
> > >  In this release we decided that the costs should only be taken into
> > account
> > > in a parallel case (ctest -j N).  Many users have implicit dependencies
> > > based on the order of their add_test calls, so we didn't want to break
> > > backward compatibility for those not using parallel ctest.
> >
> > It looks like ctest -j2 is respecting COST. Currently I have several
> > tests that cannot run at the same time as others (they touch the same
> > resources and/or running two of them at once would crush the machine).
> > If I could get the old COST behavior by running ctest -j1, that might be
> > an acceptable workaround, but it does not appear to work today.
> >
> > > The non-parallel way to specify a test to run last is simply to make it
> > the
> > > last add_test call.
> >
> > My CMake projects are modular (I imagine this is true for many CMake
> > users). Each module is responsible for adding its own unit tests and
> > code quality checks. As I said in my initial email, the code quality
> > checks must run after the unit tests so that accurate code coverage
> > values can be calculated. I can try to insure that my add_unittest()
> > functions all run before my add_code_quality() functions, but that seems
> > brittle and error-prone. It was much nicer when I could just tell
> > add_code_quality() to add all its tests with COST -1000 to guarantee
> > they run after everything else.
> >
> > I can imagine ways to work around this problem, but they all seem rather
> > clunky, especially when COST used to solve the problem so simply and
> > elegantly.
> >
> > I hope we can reach a useful middle ground about the future of the COST
> > property. In its current state, it is of no use to me.
> >
> > Thanks,
> > tyler
> >
> > > On Fri, Nov 26, 2010 at 10:20 AM, Zach Mullen <zach.mullen at kitware.com
> > >wrote:
> > > > On Tue, Nov 23, 2010 at 6:02 PM, David Cole <david.cole at kitware.com
> > >wrote:
> > > >
> > > >> It might be due to this commit:
> > > >>
> > > >>
> > http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=142edf8ad4baccd991a6a8a3e5283d0b575acca2
> > > >> (first released in 2.8.3)
> > > >>
> > > >> Or this one:
> > > >>
> > > >>
> > http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=b4d27dc041c9164d6f3ad39e192f4b7d116ca3b3
> > > >> (first released in 2.8.2)
> > > >>
> > > >> Either way, seems like a bug to me. If you explicitly specify a COST
> > > >> property value, especially a negative one to induce "last run" status,
> > > >> then it should be honored over either historical average measurement
> > > >> or "failed last time, so run it first this time" behavior.
> >