[CMake] Coverage information overestimated? Globbing for *.gcno instead of *.gcda?

[Tom Vercauteren]

On Mon, Apr 27, 2009 at 17:01, Bill Hoffman <bill.hoffman at kitware.com> wrote:
> Tom Vercauteren wrote:
>>>>
>>>> I just saw that when ctest is used to generate coverage information,
>>>> it looks for (gc)da files in the build directory - cf.
>>>> cmCTestCoverageHandler::FindGCovFiles in cmCTestCoverageHandler.cxx.
>>>>
>>>> If I understand it correctly, when everything is setup correctly for a
>>>> coverage build:
>>>>  - gcc generates (gc)no files at compile time
>>>>  - (gc)da files are generated at run time - i.e. when the unit tests are
>>>> run
>>>>
>>>> So, it seems to me that for each (gc)da file we are sure that a (gc)no
>>>> file exists but not the other way round.
>>>>
>>>> Since ctest is looking for (gc)da files, it seems to me that it will
>>>> lead to over-estimating the coverage information. Indeed, some code
>>>> can be compiled but never executed by the unit tests.
>>>>
>>> I don't think this is a problem.   Do you have an example that shows the
>>> problem?  If the code is not run, it does not produce any coverage
>>> information in any file for ctest to find???
>>
>> The reason I am looking at this is because I would like to get a
>> global coverage percentage that reflects the coverage of my project as
>> a whole as opposed to only the subset that has some unit tests.
>>
>> So, it would be interesting for me to show on a cdash-dashboard this
>> compiled but untested code (at least not automatically by ctest). One
>> way to do that might be to use gcno files instead of gcda files in
>> ctest.
>>
>> The coverage information that I would get form it is that when a gcno
>> file exists but has no gcda counterpart, there is no coverage at all.
>> Right now, this is silently discarded.
>>
> CTest does that for you.  As you run tests ctest will accumulate the test
> code that is actually run.  I am not sure I understand the coverage
> information you are looking for?  What does compile time coverage mean?

Sorry, if I wasn't clear. I do understand that the information is
accumulated in the .gcda file but this is not my question.

Let me reformulate it by looking at a specific coverage dashboard:
http://www.cdash.org/CDash/viewCoverage.php?buildid=321101

My goal is to get, in the "Coverage Summary" box, a "Total Coverage"
number that is as accurate as possible. This implies that if a given
file (say myfile.cpp) is compiled in my project but its code is never
executed when "make test" is called:
1) I want a line to be added in cdash that says that this file is untested
2) In the "Coverage Summary" box of cdash, the number of "Untested
lines" should account for this untested file
3) In my build directory, myfile.cpp.gcno exists
4) In my build directory, myfile.cpp.gcda does not exists

There are already a few files that are marked as "UNTESTED" in
http://www.cdash.org/CDash/viewCoverage.php?buildid=321101 but my
question is whether ctest is not missing some of these untested files
by looking for the *.gcda files rather than the *.gcno ones.

My guess is that:
1) A file such as myfile.cpp above would currently not appear at all on cdash.
2) If ctest was to report a file as being "UNTESTED" when a .gcno file
is found without its .gcda counterpart, myfile.cpp would appear on
cdash as being "UNTESTED"

It thus seems to me that the number of files reported as "UNTESTED" in
http://www.cdash.org/CDash/viewCoverage.php?buildid=321101 could
(should?) be higher.

But again, I might be missing something, I am not an expert in gcov
nor in cmake... I hope my question makes more sense now.

Tom

P.S.: Even better would be to also report files that are not even
compiled but that is another story I guess... Note that this might
well happen with template code.