[CMake] test property COST not working in cmake 2.8.3?

[Tyler Roscoe]

I've taken the liberty of adding this bug to the tracker:
http://www.cmake.org/Bug/view.php?id=11561

Can someone give me an idea of how involved this fix is? If it cannot be
fixed in short order, I'll be forced to hack around the change in COST's
behavior in my own scripts.  Fixing the problem at its source would
obviously be preferable for me.

Thanks,
tyler

On Tue, Nov 30, 2010 at 12:14:56PM -0800, Tyler Roscoe wrote:
> On Tue, Nov 30, 2010 at 01:29:37PM -0500, Zach Mullen wrote:
> > Hm, yours was a use case we didn't really consider when we were making
> > changes to cost behavior.  
> 
> Clearly. :)
> 
> > The middle ground here would be to respect costs in the non-parallel
> > case when they are expressed explicitly
> 
> This sounds good.
> 
> > but not to cost-order them automatically based on their previous run
> > times.
> 
> I don't care too much about this behavior, but it seems like a nice
> feature. Perhaps if CTest reserved a range for its own COST data (-10 <=
> CTest-calculated COST <= 10?) then users could use costs < -10 or > 10
> to insure ordering of certain tests?
> 
> So what are the next steps? Should I open a bug to track this issue? Is
> this an easy change (so easy a monkey like me can do it) or can someone
> at Kitware take care of it? I am able and eager to test a CMake nightly
> build that fixes this bug/regression/misunderstanding.
> 
> Thanks,
> tyler
> 
> > On Tue, Nov 30, 2010 at 12:43 PM, Tyler Roscoe <tyler at cryptio.net> wrote:
> > 
> > > On Fri, Nov 26, 2010 at 10:38:44AM -0500, Zach Mullen wrote:
> > > > I just realized why this isn't working -- it's actually not a regression.
> > >
> > > Maybe we have different definitions of "regression". I see a feature
> > > that used to do one thing but which now does something else.
> > >
> > > Here is what the docs say about the COST property:
> > >
> > >    # COST: Set this to a floating point value. Tests in a test set will be
> > >    # run in descending order of cost.
> > >
> > >    This property describes the cost of a test. You can explicitly set this
> > >    value; tests with higher COST values will run first.
> > >
> > > I don't see anything there about parallel or non-parallel runs. It seems
> > > to me that if I set the COST property, I should be able to control the
> > > order in which tests run, period. So at the very least, the docs should
> > > be updated if you intend to change the behavior.
> > >
> > > >  In this release we decided that the costs should only be taken into
> > > account
> > > > in a parallel case (ctest -j N).  Many users have implicit dependencies
> > > > based on the order of their add_test calls, so we didn't want to break
> > > > backward compatibility for those not using parallel ctest.
> > >
> > > It looks like ctest -j2 is respecting COST. Currently I have several
> > > tests that cannot run at the same time as others (they touch the same
> > > resources and/or running two of them at once would crush the machine).
> > > If I could get the old COST behavior by running ctest -j1, that might be
> > > an acceptable workaround, but it does not appear to work today.
> > >
> > > > The non-parallel way to specify a test to run last is simply to make it
> > > the
> > > > last add_test call.
> > >
> > > My CMake projects are modular (I imagine this is true for many CMake
> > > users). Each module is responsible for adding its own unit tests and
> > > code quality checks. As I said in my initial email, the code quality
> > > checks must run after the unit tests so that accurate code coverage
> > > values can be calculated. I can try to insure that my add_unittest()
> > > functions all run before my add_code_quality() functions, but that seems
> > > brittle and error-prone. It was much nicer when I could just tell
> > > add_code_quality() to add all its tests with COST -1000 to guarantee
> > > they run after everything else.
> > >
> > > I can imagine ways to work around this problem, but they all seem rather
> > > clunky, especially when COST used to solve the problem so simply and
> > > elegantly.
> > >
> > > I hope we can reach a useful middle ground about the future of the COST
> > > property. In its current state, it is of no use to me.
> > >
> > > Thanks,
> > > tyler
> > >
> > > > On Fri, Nov 26, 2010 at 10:20 AM, Zach Mullen <zach.mullen at kitware.com
> > > >wrote:
> > > > > On Tue, Nov 23, 2010 at 6:02 PM, David Cole <david.cole at kitware.com
> > > >wrote:
> > > > >
> > > > >> It might be due to this commit:
> > > > >>
> > > > >>
> > > http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=142edf8ad4baccd991a6a8a3e5283d0b575acca2
> > > > >> (first released in 2.8.3)
> > > > >>
> > > > >> Or this one:
> > > > >>
> > > > >>
> > > http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=b4d27dc041c9164d6f3ad39e192f4b7d116ca3b3
> > > > >> (first released in 2.8.2)
> > > > >>
> > > > >> Either way, seems like a bug to me. If you explicitly specify a COST
> > > > >> property value, especially a negative one to induce "last run" status,
> > > > >> then it should be honored over either historical average measurement
> > > > >> or "failed last time, so run it first this time" behavior.
> > >
> _______________________________________________
> Powered by www.kitware.com
> 
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
> 
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
> 
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake