[Insight-developers] Need to add images for new tests to Data -- how to do that in a Gerrit topic?

Fri Nov 5 23:16:38 EDT 2010

I don't see how it will work with multiple baselines and/or provide
the ability to specify error thresholds. These are just a couple of
requirements off the top of my head.

I wish there had been input from the ITK Developers.

On Fri, Nov 5, 2010 at 9:47 PM, Stephen Aylward
<stephen.aylward at kitware.com> wrote:
> Hi Bill,
>
> Sorry - this is how it was presented to me - the mis-communication
> might be on my end (even communication within groups at Kitware can be
> challenging), and/or perhaps communication within VTK/CMake teams
> needs to be strengthened.
>
> However, I am very confident that Luis/ITK did not initiate the
> development of this system...in particular, I know this because I must
> admit that I too was not a big fan at first!   Why store hashes of
> tests when git or other systems can be used to track revisions, etc
> BUT I am now a believer.   Git, Gerrit, sub-projects, modularization,
> cdash at home, branchy workflows, and other tools and use cases that ITK
> has chosen are causing a file name to be a poor tag for "truth".   If
> you instead write a test and want to run it with a particular image
> and expect a particular result, hashes on the input and output have
> several benefits - including helping maintain "truth".   I won't
> (probably cannot) go into all of the cool benefits (e.g., if the test
> is highly deterministic, checking the hash of the output MAY be
> sufficient for checking the result), but Bill H and others have given
> this much thought and developed a highly functional (but undoubtedly
> not perfect) implementation.   Also, we don't have to use MIDAS to
> store the data or CMake/CTest to generate the hashes, but this system
> does need some way of centralized storage of multiple versions of
> files tagged by different hashes.   No doubt we could probably extend
> gerrit to do this kind of data management, but when looking at the
> broader open-source community and realizing the number of projects
> using cmake and the fact that ctest and cmake were meant to be used
> for testing and test management while gerrit was not, it seems that
> the biggest impact on open-source can be had by adding more testing
> capabilities to cmake/ctest rather than extending gerrit so that its
> use becomes required for testing.
>
> Again - not saying that ITK must use it - but I hope we all can go
> into it with an open mind.  Then we can either go with what they got
> and help them improve it, or adopt something better, or stick with
> what has worked in the past...
>
> I also can't be at the meeting next week either.   I'm sure it will be
> a lively discussion.   But, in the end, I'm sure something good will
> result.
>
> Now, about simpleITK.... :)
>
> Thanks,
> Stephen
>
>
>
> On Fri, Nov 5, 2010 at 9:09 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>> Stephen,
>>
>> As a member of the VTK community and the VTK ARB, I have the same
>> concerns I mentioned. I do not believe this has been discussed by the
>> VTK community. I cannot find any discussion regarding this on the VTK
>> wiki.
>>
>> Bill
>>
>> On Fri, Nov 5, 2010 at 8:14 PM, Stephen Aylward
>> <stephen.aylward at kitware.com> wrote:
>>> Hi Bill,
>>>
>>> I think Luis did a great job at pointing out in a previous email that
>>> this is merely a suggestion for ITK, and there have been numerous
>>> discussions regarding data hosting and management in repositories
>>> intended for source code - particularly when used with gerrit.   There
>>> is a problem.  Heck, Luis didn't start this email thread - someone
>>> else did when they were trying to determine what to do with their
>>> data.
>>>
>>> My understanding (and it could be wrong on some details, but is
>>> correct in the general message) is that the system that will be
>>> presented by others (not Luis) was developed by the CMake team at the
>>> request of the VTK team, and it is being put into place by those teams
>>> for those projects.   As an existing open source solution, we should
>>> consider it.    Never did Luis dictate its use.   You know Luis - he
>>> isn't like that.
>>>
>>> But, you bring up a good point.  The presentation should follow the
>>> outline you've given - state the problem, discuss how this solution
>>> addresses them and falls short (I applaud that they already have
>>> hands-on experience with it), listen to suggestions for alternatives,
>>> and then pick a way forward as a community.
>>>
>>> Stephen
>>>
>>>
>>>
>>> On Fri, Nov 5, 2010 at 7:45 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>>> Luis,
>>>>
>>>> I am surprised at the approach taken to address this important issue.
>>>>
>>>> My suggested approach:
>>>> ITK needs a better way to manage test data and test baselines. Define
>>>> the problem, establish baseline measurements, analyze the root causes,
>>>> propose improvements, establish a process to control the solution. All
>>>> with community input.
>>>>
>>>> The current approach is:
>>>> ITK needs a better way to manage test data and test baselines. Here is
>>>> a solution, please review it.
>>>>
>>>> The later approach shows that a group has spent time and money for a
>>>> solution without community input regarding the problem, requirements
>>>> and alternative solutions.
>>>>
>>>> Bill
>>>>
>>>> On Fri, Nov 5, 2010 at 2:38 PM, Luis Ibanez <luis.ibanez at kitware.com> wrote:
>>>>> Bill,
>>>>>
>>>>> On Thu, Nov 4, 2010 at 4:04 PM, Bill Lorensen <bill.lorensen at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> I hope you have some discussion about this in Iowa. I assume it is a
>>>>>> proposal and not a mandate.
>>>>>
>>>>>
>>>>> This is an open source project,
>>>>>
>>>>> We don't do mandates,
>>>>> except to keep patented code away     :-)
>>>>>
>>>>>
>>>>>>
>>>>>> I hope that Baselines will remain (go back) in the ITK repository (not
>>>>>> a submodule). They currently take 17 megs.
>>>>>>
>>>>>
>>>>>
>>>>> What really matters is whether the images are in our local
>>>>> working tree when they are needed to run the tests, and
>>>>> whether they are the right version (of the images) for the
>>>>> current version of the source code.
>>>>>
>>>>>
>>>>>>
>>>>>> Test data is another issue. If there is a solution it must allow
>>>>>> testing without an internet connection.
>>>>>>
>>>>>
>>>>>
>>>>> Both the Git submodule and MIDAS options allow running
>>>>> testing without a network connection, of course, as long as
>>>>>
>>>>> a)  We download the data in an initial step, and
>>>>> b)  We get the right version of the data.
>>>>>
>>>>> In practice, in both cases, the work cycle becomes:
>>>>>
>>>>> 0) Download the data initially
>>>>> 1) Run the tests in many instances
>>>>> 2) From time to time download
>>>>>      new versions of images  that have changed.
>>>>>
>>>>>
>>>>>     Luis
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Kitware offers ITK Training Courses, for more information visit:
>>>> http://kitware.com/products/protraining.html
>>>>
>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> ==============================
>>> Stephen R. Aylward, Ph.D.
>>> Director of Medical Imaging Research
>>> Kitware, Inc. - North Carolina Office
>>> http://www.kitware.com
>>> stephen.aylward (Skype)
>>> (919) 969-6990 x300
>>>
>>
>
>
>
> --
>
> ==============================
> Stephen R. Aylward, Ph.D.
> Director of Medical Imaging Research
> Kitware, Inc. - North Carolina Office
> http://www.kitware.com
> stephen.aylward (Skype)
> (919) 969-6990 x300
>