[Insight-developers] Need to add images for new tests to Data -- how to do that in a Gerrit topic?

Fri Nov 5 21:47:04 EDT 2010

Hi Bill,

Sorry - this is how it was presented to me - the mis-communication
might be on my end (even communication within groups at Kitware can be
challenging), and/or perhaps communication within VTK/CMake teams
needs to be strengthened.

However, I am very confident that Luis/ITK did not initiate the
development of this system...in particular, I know this because I must
admit that I too was not a big fan at first!   Why store hashes of
tests when git or other systems can be used to track revisions, etc
BUT I am now a believer.   Git, Gerrit, sub-projects, modularization,
cdash at home, branchy workflows, and other tools and use cases that ITK
has chosen are causing a file name to be a poor tag for "truth".   If
you instead write a test and want to run it with a particular image
and expect a particular result, hashes on the input and output have
several benefits - including helping maintain "truth".   I won't
(probably cannot) go into all of the cool benefits (e.g., if the test
is highly deterministic, checking the hash of the output MAY be
sufficient for checking the result), but Bill H and others have given
this much thought and developed a highly functional (but undoubtedly
not perfect) implementation.   Also, we don't have to use MIDAS to
store the data or CMake/CTest to generate the hashes, but this system
does need some way of centralized storage of multiple versions of
files tagged by different hashes.   No doubt we could probably extend
gerrit to do this kind of data management, but when looking at the
broader open-source community and realizing the number of projects
using cmake and the fact that ctest and cmake were meant to be used
for testing and test management while gerrit was not, it seems that
the biggest impact on open-source can be had by adding more testing
capabilities to cmake/ctest rather than extending gerrit so that its
use becomes required for testing.

Again - not saying that ITK must use it - but I hope we all can go
into it with an open mind.  Then we can either go with what they got
and help them improve it, or adopt something better, or stick with
what has worked in the past...

I also can't be at the meeting next week either.   I'm sure it will be
a lively discussion.   But, in the end, I'm sure something good will
result.

Now, about simpleITK.... :)

Thanks,
Stephen

On Fri, Nov 5, 2010 at 9:09 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> Stephen,
>
> As a member of the VTK community and the VTK ARB, I have the same
> concerns I mentioned. I do not believe this has been discussed by the
> VTK community. I cannot find any discussion regarding this on the VTK
> wiki.
>
> Bill
>
> On Fri, Nov 5, 2010 at 8:14 PM, Stephen Aylward
> <stephen.aylward at kitware.com> wrote:
>> Hi Bill,
>>
>> I think Luis did a great job at pointing out in a previous email that
>> this is merely a suggestion for ITK, and there have been numerous
>> discussions regarding data hosting and management in repositories
>> intended for source code - particularly when used with gerrit.   There
>> is a problem.  Heck, Luis didn't start this email thread - someone
>> else did when they were trying to determine what to do with their
>> data.
>>
>> My understanding (and it could be wrong on some details, but is
>> correct in the general message) is that the system that will be
>> presented by others (not Luis) was developed by the CMake team at the
>> request of the VTK team, and it is being put into place by those teams
>> for those projects.   As an existing open source solution, we should
>> consider it.    Never did Luis dictate its use.   You know Luis - he
>> isn't like that.
>>
>> But, you bring up a good point.  The presentation should follow the
>> outline you've given - state the problem, discuss how this solution
>> addresses them and falls short (I applaud that they already have
>> hands-on experience with it), listen to suggestions for alternatives,
>> and then pick a way forward as a community.
>>
>> Stephen
>>
>>
>>
>> On Fri, Nov 5, 2010 at 7:45 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>> Luis,
>>>
>>> I am surprised at the approach taken to address this important issue.
>>>
>>> My suggested approach:
>>> ITK needs a better way to manage test data and test baselines. Define
>>> the problem, establish baseline measurements, analyze the root causes,
>>> propose improvements, establish a process to control the solution. All
>>> with community input.
>>>
>>> The current approach is:
>>> ITK needs a better way to manage test data and test baselines. Here is
>>> a solution, please review it.
>>>
>>> The later approach shows that a group has spent time and money for a
>>> solution without community input regarding the problem, requirements
>>> and alternative solutions.
>>>
>>> Bill
>>>
>>> On Fri, Nov 5, 2010 at 2:38 PM, Luis Ibanez <luis.ibanez at kitware.com> wrote:
>>>> Bill,
>>>>
>>>> On Thu, Nov 4, 2010 at 4:04 PM, Bill Lorensen <bill.lorensen at gmail.com>
>>>> wrote:
>>>>>
>>>>> I hope you have some discussion about this in Iowa. I assume it is a
>>>>> proposal and not a mandate.
>>>>
>>>>
>>>> This is an open source project,
>>>>
>>>> We don't do mandates,
>>>> except to keep patented code away     :-)
>>>>
>>>>
>>>>>
>>>>> I hope that Baselines will remain (go back) in the ITK repository (not
>>>>> a submodule). They currently take 17 megs.
>>>>>
>>>>
>>>>
>>>> What really matters is whether the images are in our local
>>>> working tree when they are needed to run the tests, and
>>>> whether they are the right version (of the images) for the
>>>> current version of the source code.
>>>>
>>>>
>>>>>
>>>>> Test data is another issue. If there is a solution it must allow
>>>>> testing without an internet connection.
>>>>>
>>>>
>>>>
>>>> Both the Git submodule and MIDAS options allow running
>>>> testing without a network connection, of course, as long as
>>>>
>>>> a)  We download the data in an initial step, and
>>>> b)  We get the right version of the data.
>>>>
>>>> In practice, in both cases, the work cycle becomes:
>>>>
>>>> 0) Download the data initially
>>>> 1) Run the tests in many instances
>>>> 2) From time to time download
>>>>      new versions of images  that have changed.
>>>>
>>>>
>>>>     Luis
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Kitware offers ITK Training Courses, for more information visit:
>>> http://kitware.com/products/protraining.html
>>>
>>> Please keep messages on-topic and check the ITK FAQ at:
>>> http://www.itk.org/Wiki/ITK_FAQ
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>
>>
>>
>>
>> --
>>
>> ==============================
>> Stephen R. Aylward, Ph.D.
>> Director of Medical Imaging Research
>> Kitware, Inc. - North Carolina Office
>> http://www.kitware.com
>> stephen.aylward (Skype)
>> (919) 969-6990 x300
>>
>

-- 

==============================
Stephen R. Aylward, Ph.D.
Director of Medical Imaging Research
Kitware, Inc. - North Carolina Office
http://www.kitware.com
stephen.aylward (Skype)
(919) 969-6990 x300