[ITK] A suggestion to manage code coverage drops

[Jon Haitz Legarreta]

Hi,
I'd definitely would give it a try.

I'm aware that getting coverage results may delay the merge process, but
depending on the delay I think it is worthwhile.

I had a look at the --diff flag, but it did not provide a clear picture of
how to proceed.

As for the results, an option would be to present the coverage figure along
with a link to a kind of dashboard showing the files (and the lines in
these files; much like the current file-wise diff presentation) whose
coverage has changed with respect to the master/nightly. A separate message
from the dashboard builds sent to the review would be a solution, rather
than holding back the dashboard build results message until both tasks are
completed.

It may take us a while, but I think it adds a value to the toolkit
development process.

We may start narrowing down the space of solutions and hopefully have a
first version of the tool and do some tests to see how it behaves in a few
weeks.

JON HAITZ



On 17 January 2017 at 19:28, Matt McCormick <matt.mccormick at kitware.com>
wrote:

> Hi Jon,
>
> This is a great idea.
>
> We can create coverage metrics with gcov / lcov as in this script:
>
>   https://github.com/InsightSoftwareConsortium/ITK/blob/
> d125b69c44ac0ed7d61df0252a6b3b054c60df83/Utilities/Maintenance/
> computeCodeCoverageLocally.sh
>
> and it looks like lcov has a --diff flag:
>
>   http://ltp.sourceforge.net/coverage/lcov/lcov.1.php
>
> A new script (computeCodeCoverageForPatch.sh?) could be created.
>
> We could run this either with every patch or on demand if it is resource
> intensive. It may take more effort to figure out a way to post the HTML
> output and create a link, but it should be doable.
>
> Thanks,
> Matt
>
> On Tue, Jan 17, 2017 at 11:58 AM, Jon Haitz Legarreta <
> jhlegarreta at vicomtech.org> wrote:
>
>> Hi,
>> I'm wondering whether with the current code coverage tools/CI used, and
>> without excessive work and burden, we would be able to spot merge requests
>> or commits that lower the code coverage.
>>
>> That is, automatically performing a kind of diff between two code
>> coverage reports and identifying the commits (and, thus, the files) that
>> actually make the coverage figure change. Or at least, those that make the
>> figure decrease.
>>
>> If I'm not mistaken some CI tools already provide a coverage report (or
>> at least the figure) for each commit (not 100% sure). I guess that would
>> ideally allow maintainers to take action before a topic or commit makes its
>> way into master.
>>
>> For example, that would allow to automatically ask the developer to raise
>> the coverage, or setting some policy to require at least say, a 95%
>> coverage in new files, or to keep the coverage figure when modifying
>> existing files, although this last may be unfeasible in practice.
>>
>> I am well aware that even if this were possible, it would require some
>> investigation. I am ready to help.
>>
>> Thanks,
>> JON HAITZ
>>
>> --
>>
>>
>> _______________________________________________
>> Community mailing list
>> Community at itk.org
>> http://public.kitware.com/mailman/listinfo/community
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/community/attachments/20170118/9b7c4d52/attachment.html>