[vtk-developers] VTK code review / testing / integration workflow

[David Thompson]

Hi Berk,

> It is time for us to look at our options to replace our current Gerrit instance (review.source.kitware.com). ...
> Here I'll list some requirements that I thought are important in no particular order:
> - Maintain an open and "democratic" review process, a la Gerrit voting support.

+1

> - The workflow needs to be distributed and not require a single (or a few) maintainer to integrate code. A la our Gerrit's review and merge support.

Distributed is great. But it also helps to have stakeholders for particular pieces of code, especially for core functionality, whose buy-off (or at least notification) is required.

> - Support for cdash @ home style testing of code under review before integration.

+1

> - Support for arbitrary server side checks beyond cdash @ home before integration.

As long as arbitrary does not become whimsical. For instance, whitespace rules make it very un-fun to keep third-party libraries up to date in VTK; when a library like netcdf or exodus does not have strict rules about trailing whitespace, merging in a new release from upstream is tedious.

> - Support for "audit trail". There needs to be way of getting to the original discussion and reviews that led to the acceptance of a particular branch.

+1

Also, it should be easy to find the audit trail starting with a git hash.

> I will follow up with a few initial workflow suggestions. Please send your workflow suggestions or requirements.

Suggestions:

- It takes a long time for tests to run because VTK is large. Often, changes to non-core functionality run a lot of unnecessary, unrelated tests. It would be nice to have submitters and/or reviewers force a few tests by name and let the review system choose other tests to run using some static analysis to find tests that have a chance of executing the updated code. Since CDash tracks average test time, it would be easy to have a time budget for choosing tests.

- It would be nice to have tighter integration with the release process and issue tracking so that reports of features added and bugs fixed could be generated. I think what we do of that is mostly by hand now, isn't it?

Suggested Requirements:-)

- Review system comments should be Markdown or rST or something easily presented as HTML which can include links -- at least to the VTK wiki and bug tracker, if nothing else. It would be really nice if pull requests could effectively become parts of a wiki page or blog so that we could curate user-guide-style documentation instead of just doxygen reference docs. Not that every topic would have to become part of a user's guide, but it should be easy to pull topic info into a user's guide.

- An anti-requirement: Less e-mail spam when (1) a topic includes multiple commits and/or (2) automated testing completes. For instance, I can think of one topic I'm a reviewer on that has 40-something commits. There are 16 changesets and reviewers get 1 per commit per changeset-the-commit-is-modified.

	2¢,
	David