[CMake] 2.8.4 order of tests

David Cole david.cole at kitware.com
Fri Feb 18 10:30:20 EST 2011 

Yup. ctest -N and ctest without -N differ in test ordering.

Congratulations! You've reported the first regression bug in CMake
2.8.4....! :-)

I suspect the change for this bug [
http://public.kitware.com/Bug/view.php?id=11561 ] has introduced this
issue. I'll make a new issue that we can hopefully address fairly
soon.

Thanks,
David


P.S.
=======
The same build tree with a non -N ctest run gives me this:

$ ctest
Test project /Users/davidcole/tmp/b6
      Start 27: enum
 1/52 Test #27: enum .............................   Passed    0.01 sec
      Start 28: set_extent
 2/52 Test #28: set_extent .......................   Passed    3.66 sec
      Start 29: getname
 3/52 Test #29: getname ..........................   Passed    0.06 sec
      Start 30: vfd
 4/52 Test #30: vfd ..............................   Passed    0.03 sec
      Start 31: ntypes
 5/52 Test #31: ntypes ...........................   Passed    0.17 sec
      Start 32: dangle
 6/52 Test #32: dangle ...........................   Passed    0.18 sec
      Start 33: dtransform
 7/52 Test #33: dtransform .......................   Passed    0.02 sec
      Start 34: reserved
 8/52 Test #34: reserved .........................   Passed    0.00 sec
      Start 35: cross_read
 9/52 Test #35: cross_read .......................   Passed    0.02 sec
      Start 36: freespace
10/52 Test #36: freespace ........................   Passed    0.03 sec
      Start 37: mf
11/52 Test #37: mf ...............................   Passed    0.09 sec
...


On Thu, Feb 17, 2011 at 10:41 PM, Allen D Byrne <byrn at hdfgroup.org> wrote:
> David,
>
> I tried ctest -N ---- it lies!
>
> Here is the beginning of the test.log showing the problem (peviously the
> tests were in test number order) :
>
> Start testing: Feb 17 21:13 CST
>
> ----------------------------------------------------------
>
> 435/660 Testing: H5COPY-clear-ext-links
>
> 436/660 Testing: H5COPY-ext_link
>
> 437/660 Testing: H5COPY-ext_link_f
>
> 438/660 Testing: H5COPY-ext_dangle_noobj
>
> 439/660 Testing: H5COPY-DIFF_ext_dangle_noobj
>
> 440/660 Testing: H5COPY-ext_dangle_noobj_f
>
> 441/660 Testing: H5COPY-ext_dangle_nofile
>
> 442/660 Testing: H5COPY-DIFF_ext_dangle_nofile
>
> 443/660 Testing: H5COPY-ext_dangle_nofile_f
>
> 444/660 Testing: H5COPY-ext_link_group
>
> 445/660 Testing: H5COPY-ext_link_group_f
>
> 446/660 Testing: H5COPY-H5LS_h5copy_extlinks_src
>
> 447/660 Testing: H5STAT-h5stat_help1
>
> 448/660 Testing: H5STAT-h5stat_help2
>
> 449/660 Testing: H5STAT-h5stat_filters
>
> 450/660 Testing: H5STAT-h5stat_filters-file
>
> 451/660 Testing: H5STAT-h5stat_filters-F
>
> 452/660 Testing: H5STAT-h5stat_filters-d
>
> 453/660 Testing: H5STAT-h5stat_filters-g
>
> 454/660 Testing: H5STAT-h5stat_filters-dT
>
> 414/660 Testing: H5COPY-DIFF_grp_nested
>
> 415/660 Testing: H5COPY-simple_group
>
> 416/660 Testing: H5COPY-DIFF_simple_group
>
> 417/660 Testing: H5COPY-grp_rename
>
> 418/660 Testing: H5COPY-DIFF_grp_rename
>
> 419/660 Testing: H5COPY-grp_dsets_rename
>
> 420/660 Testing: H5COPY-DIFF_grp_dsets_rename
>
> 421/660 Testing: H5COPY-A_B1_simple
>
> 422/660 Testing: H5COPY-DIFF_A_B1_simple
>
> 423/660 Testing: H5COPY-A_B2_simple2
>
> 424/660 Testing: H5COPY-DIFF_A_B2_simple2
>
> 425/660 Testing: H5COPY-C_D_simple
>
> 426/660 Testing: H5COPY-DIFF_C_D_simple
>
> 427/660 Testing: H5COPY-E_F_grp_dsets
>
> 428/660 Testing: H5COPY-DIFF_E_F_grp_dsets
>
> 429/660 Testing: H5COPY-G_H_grp_nested
>
> 430/660 Testing: H5COPY-DIFF_G_H_grp_nested
>
> 431/660 Testing: H5COPY-H5LS_h5copytst
>
> 432/660 Testing: H5COPY-clear-refs
>
> 433/660 Testing: H5COPY-region_ref
>
> 434/660 Testing: H5COPY-H5LS_h5copy_ref
>
> 476/660 Testing: H5DUMP-thlink-4
>
> 477/660 Testing: H5DUMP-thlink-5
>
> 478/660 Testing: H5DUMP-tcomp-1
>
> 479/660 Testing: H5DUMP-tcomp-2
>
> 480/660 Testing: H5DUMP-tcomp-4
>
> 481/660 Testing: H5DUMP-tnestcomp-1
>
> 482/660 Testing: H5DUMP-tall-1
>
> 483/660 Testing: H5DUMP-tall-2
>
> 484/660 Testing: H5DUMP-tall-3
>
> 485/660 Testing: H5DUMP-tloop-1
>
> 486/660 Testing: H5DUMP-tstr-1
>
> 487/660 Testing: H5DUMP-tstr-2
>
> 488/660 Testing: H5DUMP-tsaf
>
> 489/660 Testing: H5DUMP-tvldtypes1
>
> 490/660 Testing: H5DUMP-tvldtypes2
>
> 491/660 Testing: H5DUMP-tvldtypes3
>
> 492/660 Testing: H5DUMP-tvldtypes4
>
> 493/660 Testing: H5DUMP-tvldtypes5
>
> 494/660 Testing: H5DUMP-tvlstr
>
> 495/660 Testing: H5DUMP-tarray1
>
> 455/660 Testing: H5STAT-h5stat_filters-UD
>
> 456/660 Testing: H5STAT-h5stat_filters-UT
>
> 457/660 Testing: H5STAT-h5stat_tsohm
>
> 458/660 Testing: H5STAT-h5stat_newgrat
>
> 459/660 Testing: H5STAT-h5stat_newgrat-UG
>
> 460/660 Testing: H5STAT-h5stat_newgrat-UA
>
> 461/660 Testing: H5DUMP-tgroup-1
>
> 462/660 Testing: H5DUMP-tgroup-2
>
> 463/660 Testing: H5DUMP-tdset-1
>
> 464/660 Testing: H5DUMP-tdset-2
>
> 465/660 Testing: H5DUMP-tattr-1
>
> 466/660 Testing: H5DUMP-tattr-2
>
> 467/660 Testing: H5DUMP-tattr-3
>
> 468/660 Testing: H5DUMP-tnamed_dtype_attr
>
> 469/660 Testing: H5DUMP-tslink-1
>
> 470/660 Testing: H5DUMP-tudlink-1
>
> 471/660 Testing: H5DUMP-tslink-2
>
> 472/660 Testing: H5DUMP-tudlink-2
>
> 473/660 Testing: H5DUMP-thlink-1
>
> 474/660 Testing: H5DUMP-thlink-2
>
> 475/660 Testing: H5DUMP-thlink-3
>
> 352/660 Testing: H5REPACK-DIFF_chunk_compa
>
> 353/660 Testing: H5REPACK_VERIFY_LAYOUT-chunk_compa
>
> 354/660 Testing: H5REPACK-chunk_conti
>
> 355/660 Testing: H5REPACK-DIFF_chunk_conti
>
> 356/660 Testing: H5REPACK_VERIFY_LAYOUT-chunk_conti
>
> 357/660 Testing: H5REPACK-chunk_18x13
>
> 358/660 Testing: H5REPACK-DIFF_chunk_18x13
>
> 359/660 Testing: H5REPACK_VERIFY_LAYOUT-chunk_18x13
>
> 360/660 Testing: H5REPACK-contig_small_compa
>
> 361/660 Testing: H5REPACK-DIFF_contig_small_compa
>
> 362/660 Testing: H5REPACK_VERIFY_LAYOUT-contig_small_compa
>
> 363/660 Testing: H5REPACK-contig_small_fixed_compa
>
> 364/660 Testing: H5REPACK-DIFF_contig_small_fixed_compa
>
> 365/660 Testing: H5REPACK_VERIFY_LAYOUT-contig_small_fixed_compa
>
> 366/660 Testing: H5REPACK-native_fill
>
> 367/660 Testing: H5REPACK-DIFF_native_fill
>
> 368/660 Testing: H5REPACK-native_attr
>
> 369/660 Testing: H5REPACK-DIFF_native_attr
>
> 370/660 Testing: H5REPACK-layout_long_switches
>
> 371/660 Testing: H5REPACK-DIFF_layout_long_switches
>
> 331/660 Testing: H5REPACK-DIFF_compa
>
> 332/660 Testing: H5REPACK_VERIFY_LAYOUT_ALL-compa
>
> 333/660 Testing: H5REPACK-dset_compa_conti
>
> 334/660 Testing: H5REPACK-DIFF_dset_compa_conti
>
> 335/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_compa_conti
>
> 336/660 Testing: H5REPACK-dset_compa_chunk
>
> 337/660 Testing: H5REPACK-DIFF_dset_compa_chunk
>
> 338/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_compa_chunk
>
> 339/660 Testing: H5REPACK-dset_compa_compa
>
> 340/660 Testing: H5REPACK-DIFF_dset_compa_compa
>
> 341/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_compa_compa
>
> 342/660 Testing: H5REPACK-dset_conti_compa
>
> 343/660 Testing: H5REPACK-DIFF_dset_conti_compa
>
> 344/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_conti_compa
>
> 345/660 Testing: H5REPACK-dset_conti_chunk
>
> 346/660 Testing: H5REPACK-DIFF_dset_conti_chunk
>
> 347/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_conti_chunk
>
> 348/660 Testing: H5REPACK-dset_conti_conti
>
> 349/660 Testing: H5REPACK-DIFF_dset_conti_conti
>
> 350/660 Testing: H5REPACK_VERIFY_LAYOUT-dset_conti_conti
>
> 351/660 Testing: H5REPACK-chunk_compa
>
> 393/660 Testing: H5COPY-chunk
>
> 394/660 Testing: H5COPY-DIFF_chunk
>
> 395/660 Testing: H5COPY-compact
>
> 396/660 Testing: H5COPY-DIFF_compact
>
> 397/660 Testing: H5COPY-compound
>
> 398/660 Testing: H5COPY-DIFF_compound
>
> 399/660 Testing: H5COPY-compressed
>
> 400/660 Testing: H5COPY-DIFF_compressed
>
> 401/660 Testing: H5COPY-named_vl
>
> 402/660 Testing: H5COPY-DIFF_named_vl
>
> 403/660 Testing: H5COPY-nested_vl
>
> 404/660 Testing: H5COPY-DIFF_nested_vl
>
> 405/660 Testing: H5COPY-simple_top
>
> 406/660 Testing: H5COPY-DIFF_simple_top
>
> 407/660 Testing: H5COPY-rename
>
> 408/660 Testing: H5COPY-DIFF_rename
>
> 409/660 Testing: H5COPY-grp_empty
>
> 410/660 Test: H5COPY-DIFF_grp_empty
>
> 411/660 Testing: H5COPY-grp_dsets
>
> 412/660 Testing: H5COPY-DIFF_grp_dsets
>
> 413/660 Testing: H5COPY-grp_nested
>
> 372/660 Testing: H5REPACK_VERIFY_LAYOUT_ALL-layout_long_switches
>
> 373/660 Testing: H5REPACK-layout_short_switches
>
> 374/660 Test: H5REPACK-DIFF_layout_short_switches
>
> 375/660 Testing: H5REPACK_VERIFY_LAYOUT_ALL-layout_short_switches
>
> 376/660 Testing: H5REPACK-global_filters
>
> 377/660 Testing: H5REPACK-DIFF_global_filters
>
> 378/660 Testing: H5REPACK-add_userblock
>
> 379/660 Testing: H5REPACK-DIFF_add_userblock
>
> 380/660 Testing: H5REPACK-add_alignment
>
> 381/660 Testing: H5REPACK-DIFF_add_alignment
>
> 382/660 Testing: H5REPACK-upgrade_layout
>
> 383/660 Testing: H5REPACK-DIFF_upgrade_layout
>
> 384/660 Testing: H5REPACK-gt_mallocsize
>
> 385/660 Testing: H5REPACK-DIFF_gt_mallocsize
>
> 386/660 Testing: H5REPACK-committed_dt
>
> 387/660 Testing: H5REPACK-DIFF_committed_dt
>
> 388/660 Testing: H5REPACK-bug1814
>
> 389/660 Testing: H5REPACK-DIFF_bug1814
>
> 390/660 Testing: H5COPY-clear-objects
>
> 391/660 Testing: H5COPY-simple
>
> 392/660 Testing: H5COPY-DIFF_simple
>
> 600/660 Testing: H5DUMP-XML-thlink.h5
>
> 601/660 Testing: H5DUMP-XML-tloop.h5
>
> 602/660 Testing: H5DUMP-XML-tloop2.h5
>
> 603/660 Testing: H5DUMP-XML-tmany.h5
>
> 604/660 Testing: H5DUMP-XML-tnestedcomp.h5
>
> 605/660 Testing: H5DUMP-XML-tcompound_complex.h5
>
> Allen
>
>> After building HDF5 v 1.8.6 on my Mac, I get this:
>
>>
>
>> $ ctest -N
>
>> Test project /Users/davidcole/tmp/b6
>
>> Test #1: testhdf5
>
>> Test #2: accum
>
>> Test #3: lheap
>
>> Test #4: ohdr
>
>> Test #5: stab
>
>> Test #6: gheap
>
>> Test #7: pool
>
>> Test #8: hyperslab
>
>> Test #9: istore
>
>> Test #10: bittests
>
>> Test #11: dt_arith
>
>> Test #12: dtypes
>
>> Test #13: dsets
>
>> Test #14: cmpd_dset
>
>> Test #15: extend
>
>> Test #16: external
>
>> Test #17: objcopy
>
>> Test #18: links
>
>> Test #19: unlink
>
>> Test #20: big
>
>> Test #21: mtime
>
>> Test #22: fillval
>
>> Test #23: mount
>
>> Test #24: flush1
>
>> Test #25: flush2
>
>> Test #26: app_ref
>
>> Test #27: enum
>
>> Test #28: set_extent
>
>> Test #29: getname
>
>> Test #30: vfd
>
>> Test #31: ntypes
>
>> Test #32: dangle
>
>> Test #33: dtransform
>
>> Test #34: reserved
>
>> Test #35: cross_read
>
>> Test #36: freespace
>
>> Test #37: mf
>
>> Test #38: btree2
>
>> Test #39: fheap
>
>> Test #40: error_test
>
>> Test #41: err_compat
>
>> Test #42: tcheck_version
>
>> Test #43: testmeta
>
>> Test #44: cache
>
>> Test #45: cache_api
>
>> Test #46: ttsafe
>
>> Test #47: h5perf_serial
>
>> Test #48: chunk
>
>> Test #49: iopipe
>
>> Test #50: overhead
>
>> Test #51: perf_meta
>
>> Test #52: zip_perf
>
>>
>
>> Total Tests: 52
>
>>
>
>> I'm not familiar enough with HDF5 to know if this is correct or not.
>
>> Is this the same order you're seeing?
>
>>
>
>> Or can you tell me your expected order? I'll try to analyze the source
>
>> tree and figure out what's going on, but what you expect will tell me
>
>> something, too.
>
>>
>
>>
>
>> Thanks,
>
>> David
>
>>
>
>>
>
>>
>
>> On Thu, Feb 17, 2011 at 5:10 PM, Allen D Byrne <byrn at hdfgroup.org> wrote:
>
>> > Just using standard gcc/gfortran on the fedora distro. It is the 64bit
>
>> > version?
>
>> >
>
>> > You can grab one of our recent src tarballs from
>
>> >
>
>> > "http://www.hdfgroup.org/ftp/HDF5/hdf5-1.8.6/"
>
>> >
>
>> > Allen
>
>> >
>
>> > PS. I will be out of touch for the next 36 hours
>
>> >
>
>> >> On Thu, Feb 17, 2011 at 4:20 PM, Allen D Byrne <byrn at hdfgroup.org>
>> >> wrote:
>
>> >
>
>> >> > This was from a clean build folder. And I just want the test to run
>> >> > in
>
>> >> > the
>
>> >
>
>> >> > order defined in the CMakeLists.txt files. Also never had set any
>> >> > COST
>
>> >
>
>> >> > properties. The surprise was that this never happened before.
>
>> >
>
>> >> >
>
>> >
>
>> >> > Allen
>
>> >
>
>> >> >
>
>> >
>
>> >> >> On Thu, Feb 17, 2011 at 4:08 PM, David Cole <david.cole at kitware.com>
>
>> >
>
>> >> >> wrote:
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > On Thu, Feb 17, 2011 at 3:57 PM, Allen D Byrne <byrn at hdfgroup.org>
>
>> >
>
>> >> >> > wrote:
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> I just installed 2.8.4 and on my fedora 14 linux box the ctest -D
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> Experimental stage decided to run the tests in some unknown to me
>
>> >
>
>> >> >> >> order. The
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> previous versions always ran them in order of definition. Any
>> >> >> >> thing
>
>> >> >> >> I
>
>> >
>
>> >> >> >> can do
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> to get the ordering back?
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >>
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> Allen
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >>
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> _______________________________________________
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> Powered by www.kitware.com
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >>
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> Visit other Kitware open-source projects at
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> http://www.kitware.com/opensource/opensource.html
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >>
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> Please keep messages on-topic and check the CMake FAQ at:
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> http://www.cmake.org/Wiki/CMake_FAQ
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >>
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> Follow this link to subscribe/unsubscribe:
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >> http://www.cmake.org/mailman/listinfo/cmake
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >>
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > Delete the file "Testing/Temporary/CTestCostData.txt" in your
>> >> >> > build
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > tree. It saves run times from run to run and orders them as
>> >> >> > "slowest
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > first" on subsequent runs.
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > A clean build (no CostData file) without any parallel testing, and
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > without any test COST properties defined should give you the top
>> >> >> > to
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > bottom ordering that you're used to.
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > Saving this data from run to run helps us schedule parallel
>> >> >> > testing
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > jobs efficiently on subsequent runs.
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > HTH,
>
>> >
>
>> >> >
>
>> >
>
>> >> >> > David
>
>> >
>
>> >> >
>
>> >
>
>> >> >> >
>
>> >
>
>> >> >
>
>> >
>
>> >> >>
>
>> >
>
>> >> >
>
>> >
>
>> >> >>
>
>> >
>
>> >> >
>
>> >
>
>> >> >> Having said all that: if your goal is to order certain tests with
>
>> >
>
>> >> >
>
>> >
>
>> >> >> respect to each other, see the help for the test property "DEPENDS":
>
>> >
>
>> >> >
>
>> >
>
>> >> >> http://cmake.org/cmake/help/cmake-2-8-docs.html#prop_test:DEPENDS
>
>> >
>
>> >> >
>
>> >
>
>> >> >>
>
>> >
>
>> >>
>
>> >
>
>> >> So there is no "Testing/Temporary/CTestCostData.txt" file at the time
>
>> >
>
>> >> you call "ctest -D Experimental"?
>
>> >
>
>> >>
>
>> >
>
>> >> Can you point me to a source tree that I can build that demonstrates
>
>> >
>
>> >> this behavior?
>
>> >
>
>> >>
>
>> >
>
>> >> If you are not running tests in parallel and you have no "cost data"
>
>> >
>
>> >> (in the form of a cached file or properties set) then the tests should
>
>> >
>
>> >> run in the same top-to-bottom order that they always have.
>
>> >
>
>> >>
>
>> >
>
>> >>
>
>> >
>
>> >> Thx,
>
>> >
>
>> >> David
>
>> >
>
>> >>
>
>>

More information about the CMake mailing list