[cmake-developers] Keeping unit tests useful and splitting feature support from platform support

[Richard Wackerbarth]

On Oct 16, 2011, at 6:21 PM, Stephen Kelly wrote:

> On 10/16/2011 02:56 PM, Richard Wackerbarth wrote:
>> On Oct 16, 2011, at 6:54 AM, Stephen Kelly wrote:
>>> Having so many old platforms and compilers on the dashboard going red with
>>> failure
>>> So, I think I'm talking about two separate issues. One is whether failures
>>> on platforms I don't care about are my responsibility, and the other is
>>> ensuring that unit tests fail when something breaks.
>> Steve,
>> 
>> This is the "backward compatibility" issue that plagues all projects.

>> I think that it is YOUR responsibility to make any new feature work on all supported platforms.
>> If you cannot support some particular platform, you can petition the CMake community to drop support for certain platforms. But it should not be your unilateral choice to do so.
>> 
>> Richard
> 
> Hi Richard,
> 
> I'm trying to find a solution to an issue I perceive about unit tests not counting for much if we only run them when they are testably already working.

Steve,

In many respects, I agree with your concern about the "unit test" situation. (More on that later)
However, what I was really addressing was the "expected platform support" issue.

With your latest message, it appears that your "reference platforms" are the same as my "supported platforms".
In effect, if some platform is not included in the "reference platform" set, it becomes an unsupported platform.
Deciding to drop support for any platform should not be a choice made by the authors of "features", but rather needs to be a well-thought-out community decision.

With respect to your comparison to Qt, I see a fundamental difference. It is one thing to say "Qt has this existing feature set. If you want to add a new platform, then you need to provide the implementation." On the other hand, if you want to add a feature, then you should have to provide the implementation for all of the already working platforms.

As for the unit test situation, I agree that the present CMake infrastructure addresses only a portion of the development cycle.
Acceptance into "next" implicitly implies that the "feature" is expected to work on ALL supported platforms.

The nightly builds are very effective in assuring that regression issues do not occur.

But, since none of the "feature" developers have access to all of the supported platforms, I think that a scheme for "trial" runs against the supported platforms is needed.

My vision would be to have a system whereby a temporary branch could be submitted for a single dashboard build. But it would not be a part of "next". Only those features that work across all platforms would be committed for the continuing runs on the "next" branch.

In my case, CPU identification does not work on FreeBSD (unless you install Linux emulation). To fix it properly, the code really should be refactored. However, since I have no access to Windows systems, I cannot considering doing the needed refactoring because I don't have adequate testing resources.

Richard

> There may be other ways to make that work, but your response indicates that you strongly think what I proposed is the wrong approach. You might be right, I don't know.
> 
> I came up with this proposal after looking at the Qt approach to platform support -- They define reference platforms, and if others want to make Qt work on WinCE for example, they can do that work and be the experts in it. I don't mean to say that what Qt does will automatically work for CMake (unlikely in fact). I'm trying to explore options.
> 
> All the best,
> 
> Steve.