Notes
(0017515) Bill Hoffman (manager) 2009-09-14 21:24	How would you specify the WHEN part? Can you give an example?

(0017524) Marcel Loose (developer) 2009-09-15 03:57	Well, I don't have any clear thoughts on this. Maybe it should even be SKIPPED_IF, instead of SKIPPED_WHEN. The main issue I wanted to raise is that the outcome of a test program is not always black or white, it could be gray sometimes (e.g., unable to run, due to some external cause: test skipped) Anyway, would something like this work? set_test_properties(outputTest PROPERTIES SKIPPED_IF RETURN_VALUE EQUAL 77) Here the property SKIPPED_IF takes a boolean expression as argument. Of course, you can debate about the exact syntax, but this is the general idea. And to stay consistent with the existing properties, maybe a SKIPPED_IF_REGULAR_EXPRESSION is needed as well.

(0022019) David Cole (manager) 2010-08-31 11:28	We will not be addressing this for the upcoming CMake 2.8.3 release

(0023139) David Cole (manager) 2010-11-10 12:00	ctest already provides for determining if a test passes or fails (black or white, 0 or 1) by the following mechanisms: - return value of the called program: 0 is pass, non-0 is fail - invert the sense of return value: 0 is fail, non-0 is pass - match output against regular expression: match for passing, or match for failing - it also already provides for "not run" tests in the event that the executable to be run cannot be found at test time (for example) In order for this to be a useful feature request, we would need to define the whole scope of what else test results could mean. In addition to pass, fail, and "not run", what other results do you want? How would you implement that while maintaining backwards compatibility? How would the results be indicated on CDash? (as is, or would it require CDash code changes, too?) I think this goes well beyond the scope of what's reasonable to achieve by the 2.8.4 release unless somebody volunteers the time to dedicate to it, and considers all of these issues... Closing for insufficient definition. Feel free to re-open to add more definition.

(0023197) Marcel Loose (developer) 2010-11-12 04:34	I definitely see a use case here. As I already wrote in the original description, there are cases when a test program simply cannot be run successfully. My example being a program that connects to a database and the database being down. The category "skipped" should IMO be a subcategory of "not run". It could simply be reported as "not run" on the Dashboard. If really necessary, an extra subcategory "skipped" could be added as a separate column, but I don't think that's really needed. You already mention that "not run" is used when the executable program cannot be found. You could consider to stretch the definition of "not run" to every situation where "not all prerequisites to run the test are met", instead of just the case that the test program cannot be found.

(0023199) David Cole (manager) 2010-11-12 07:54	How can we define "test prerequisites" such that: - it's easy to define in terms of test properties in the CMakeLists.txt files - ctest can tell if a prerequisite is met or not We recently added the test property 'REQUIRED_FILES': http://cmake.org/cmake/help/cmake-2-8-docs.html#prop_test:REQUIRED_FILES [^] If one of those files does not exist, we treat is as a "not run" sort of test failure and do not even attempt to run the test. Does that help? Or do you still think we need something else in ctest?

(0023200) Rolf Eike Beer (developer) 2010-11-12 08:10	I think giving the test program itself the possibility to say "I can't run because $foo is missing" would be indeed very useful. The missing stuff could basically be anything. For Marcel it's a database (maybe on a different host). We drive tests with that to do some image processing with external pattern generators and whatnot. So missing could be anything like: -there is no external generator connected -the external generator can't do $foo what is needed in this test (there are some versions of them around) -the board has the wrong firmware version None of these things is critical in a way to us so we say "this did not work, we can't release", it's more like "some dude messed up his setup". Finding things as "not run" would be what we like to see there. Messing up the setup could also be "did not install correct version of $dependency so some test executable can not run" which is what we currently have. A property that would define either a regex or a return value to be treated as "not run" for a test would be all we need.

(0023202) Marcel Loose (developer) 2010-11-12 09:52	Earlier, I suggested something like: set_test_properties(outputTest PROPERTIES SKIPPED_IF RETURN_VALUE EQUAL 77) Testing on return values is much easier than using a regex. Furthemore, since the author of the test program and the CMakeLists.txt file will decide on the specific return value to treat as "skipped", there's no such thing as a "magic return value" like the Autotools have. However, if people would also like to be able to specify a regex, it could be something along the lines of: set_test_properties(outputTest PROPERTIES SKIPPED_REGULAR_EXPRESSION <regex>)

(0024863) David Cole (manager) 2011-01-18 11:18	Looking at "older" "feedback" status bugs today... is this still an issue that we want to address moving forward? If so, please remove the "feedback" status and just set it to "assigned"... If not, please resolve it. Thanks.

(0024903) Marcel Loose (developer) 2011-01-19 03:29	Yes, I still think it would be very useful to have this feature implemented. I've assigned it to you, David, since you wrote the last note.

(0024904) Sylwester Arabas (reporter) 2011-01-19 05:42	Hi, it would be perfect to have it in cmake! We have a lot of tests that return 77 in GNU Data Language (gdl), and it would be great to have this supported by cmake as well (GDL can be configured both using autotools and cmake). We use exit code 77 to indicate that a test for a given feature is skipped because GDL was compiled without that feature (e.g. Python bridge or some external file format support), or in case some external resources are needed to proceed with the test (e.g. X-server connection). Regards, Sylwester

(0027139) David Cole (manager) 2011-08-01 11:15	I guess the test property being discussed here is really a map. A map of program return values to test status values. 0 --> passed 77 --> skipped anything not in the map --> failed In addition to cmake and ctest being aware of this map, you would also somehow have to make CDash aware of the same map on a per-project basis. Which completely changes the very top level page of CDash on a per-project basis. You'd have to have "pass, fail, and skip" columns. Or whatever user-defined test statuses there happen to be in a project. Or, you have to map "skip" into a "pass" or "fail" in order to keep the CDash interface to the test results coherent across projects. I don't know, maybe a project-specific layout of the top-level CDash page makes sense, and I'm just biased against it because I don't understand what it would look like. I do not see a short-term way to implement a fix for this issue in a reasonable manner that would still work well with CDash. If somebody can provide me with more insight into a way to do that without having to change CDash's representation of the data in a significant way, please chime in. Patches welcome, because this seems like a *huuuuuuggggeee* can of worms right underneath the surface.

(0027147) Marcel Loose (developer) 2011-08-02 08:20	I don't think CDash has to be aware of this "exit status map". For me, it would be perfectly fine, if a skipped test shows up as "not run" on the dashboard. Actually, I think a tri-state would suffice: pass, fail, indeterminate. Where indeterminate could mean "not run" or "skipped", or what have you. This could be accomplished by adding one test property, e.g., SKIPPED_IF; see my earlier notes. Maybe it's my lack of under-the-hood knowledge of CMake/CTest/CDash, but I really don't see this huge can of worms.

(0029650) David Cole (manager) 2012-06-08 12:56	More discussion on the mailing list starting with this message: http://www.cmake.org/pipermail/cmake/2012-June/050658.html [^]

(0033723) simonh (reporter) 2013-08-16 03:39 edited on: 2013-08-16 03:40	I would also like to request this feature. Sometimes tests have prerequisites which aren't just the executable image and some other files. "Database availability" has already been mentioned. We have tests which require commercially licensed software to be run in order to perform validation; sometimes the licence is already in use elsewhere. In this case, it is not possible to say whether the test passes or not. Again, I support the tri-state solution. If the test runs to completion, it must be PASS, FAIL or NOT_RUN. As we already have PASS_REGULAR_EXPRESSION, FAIL_REGULAR_EXPRESSION, and CDash uses "Not Run", how about NOT_RUN_REGULAR_EXPRESSION and NOT_RUN_RETURN_VALUE test properties?

(0033958) Marcel Loose (developer) 2013-10-01 10:02	Any chance that Rolf's patch (uploaded a year ago) will be accepted?

(0034227) alxanders (reporter) 2013-10-23 11:33	Finer control than pass/fail would be very useful to me for e.g.: - GPU tests if the node happens to not have (the right type of) GPU(s) - tests requiring a remote machine, which is down - tests requiring a network connection to localhost via ssh, while an ssh key pair + authorized_keys is missing. (This also includes any MPI test on localhost.) It is still useful to run on a test node that do not have e.g. GPUs; it may have another environment feature (e.g. OS, compiler) we'd like to test against. Right now with e.g. GPU tests, we cannot see what was really tested: either we get lots of tests failed, or lots of tests passed. In both cases the GPU tests didn't test anything. I can imagine there is some complexity involved adding a new "Skipped" state. But just fold it under "Not Run" as that is what it is. You have already extended it before (with REQUIRED_FILES). We just need to be able to trigger "Not Run" through an exit status (e.g. autotools' 77 or specifiable) from an already started test. Of course, then all Passed or Not Run could be considered fine, while currently cmake always exits with status 8 on Not Run (or Failed). So what about Rolf Eike Beer's patch? Isn't that already close?

(0034393) Burlen (reporter) 2013-11-05 11:05	I'd like to see this feature implemented, I think it will improve tracking OpenGL driver bugs affecting VTK. When a test is run on a driver with known bugs, the test is skipped internally, and shows up as passed in the dashboard. I suspect that it's done this way to avoid polluting the dashboard, and it's probably a good idea to keep it like this. However, I think it's an issue that it's difficult to tell which tests are being skipped on which systems and it would be very useful if these tests showed up in the dashboard as not run rather than passed.

(0034947) Rolf Eike Beer (developer) 2014-01-15 11:50	Fix is in master now: http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=3a0d164bb2fa03966cdd7a17c0ce63b4d54f05b7 [^]

(0036083) Robert Maynard (manager) 2014-06-02 08:38	Closing resolved issues that have not been updated in more than 4 months.

Issue History
Date Modified	Username	Field	Change
2009-02-04 07:42	Marcel Loose	New Issue
2009-02-12 13:06	Bill Hoffman	Status	new => assigned
2009-02-12 13:06	Bill Hoffman	Assigned To	=> Bill Hoffman
2009-09-14 21:24	Bill Hoffman	Note Added: 0017515
2009-09-15 03:57	Marcel Loose	Note Added: 0017524
2010-08-31 11:28	David Cole	Note Added: 0022019
2010-11-10 12:00	David Cole	Note Added: 0023139
2010-11-10 12:00	David Cole	Status	assigned => closed
2010-11-10 12:00	David Cole	Resolution	open => won't fix
2010-11-10 12:00	David Cole	Fixed in Version	=> CMake 2.8.4
2010-11-12 04:34	Marcel Loose	Note Added: 0023197
2010-11-12 04:34	Marcel Loose	Status	closed => feedback
2010-11-12 04:34	Marcel Loose	Resolution	won't fix => reopened
2010-11-12 07:54	David Cole	Note Added: 0023199
2010-11-12 08:10	Rolf Eike Beer	Note Added: 0023200
2010-11-12 09:52	Marcel Loose	Note Added: 0023202
2011-01-18 11:18	David Cole	Note Added: 0024863
2011-01-19 03:29	Marcel Loose	Note Added: 0024903
2011-01-19 03:29	Marcel Loose	Status	feedback => assigned
2011-01-19 05:42	Sylwester Arabas	Note Added: 0024904
2011-08-01 11:15	David Cole	Note Added: 0027139
2011-08-02 08:20	Marcel Loose	Note Added: 0027147
2011-09-05 16:41	David Cole	Fixed in Version	CMake 2.8.4 =>
2011-10-28 12:31	David Cole	Assigned To	Bill Hoffman =>
2011-10-28 12:31	David Cole	Status	assigned => backlog
2012-06-08 12:56	David Cole	Note Added: 0029650
2012-09-21 12:05	Rolf Eike Beer	File Added: 0001-allow-to-mark-a-test-as-Not-Run-with-a-specific-retu.patch
2012-12-31 15:18	David Cole	Relationship added	related to 0013825
2013-08-16 03:39	simonh	Note Added: 0033723
2013-08-16 03:40	simonh	Note Edited: 0033723
2013-10-01 10:02	Marcel Loose	Note Added: 0033958
2013-10-23 11:33	alxanders	Note Added: 0034227
2013-11-05 11:05	Burlen	Note Added: 0034393
2014-01-15 11:50	Rolf Eike Beer	Note Added: 0034947
2014-01-15 11:50	Rolf Eike Beer	Assigned To	=> Rolf Eike Beer
2014-01-15 11:50	Rolf Eike Beer	Status	backlog => resolved
2014-01-15 11:50	Rolf Eike Beer	Resolution	reopened => fixed
2014-01-15 11:50	Rolf Eike Beer	Fixed in Version	=> CMake 3.0
2014-01-15 11:50	Rolf Eike Beer	Target Version	=> CMake 3.0
2014-06-02 08:38	Robert Maynard	Note Added: 0036083
2014-06-02 08:38	Robert Maynard	Status	resolved => closed