[Insight-developers] Need to add images for new tests to Data -- how to do that in a Gerrit topic?

Stephen Aylward stephen.aylward at kitware.com
Fri Nov 5 23:26:41 EDT 2010 

The meeting is the chance!

You're right - sorry if I was unclear -

>> if the test
>> is highly deterministic, checking the hash of the output MAY be
>> sufficient for checking the result

Instead of "highly deterministic" I should have clarified "has no
error on all platforms"

s

On Fri, Nov 5, 2010 at 11:16 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> I don't see how it will work with multiple baselines and/or provide
> the ability to specify error thresholds. These are just a couple of
> requirements off the top of my head.
>
> I wish there had been input from the ITK Developers.
>
>
> On Fri, Nov 5, 2010 at 9:47 PM, Stephen Aylward
> <stephen.aylward at kitware.com> wrote:
>> Hi Bill,
>>
>> Sorry - this is how it was presented to me - the mis-communication
>> might be on my end (even communication within groups at Kitware can be
>> challenging), and/or perhaps communication within VTK/CMake teams
>> needs to be strengthened.
>>
>> However, I am very confident that Luis/ITK did not initiate the
>> development of this system...in particular, I know this because I must
>> admit that I too was not a big fan at first!   Why store hashes of
>> tests when git or other systems can be used to track revisions, etc
>> BUT I am now a believer.   Git, Gerrit, sub-projects, modularization,
>> cdash at home, branchy workflows, and other tools and use cases that ITK
>> has chosen are causing a file name to be a poor tag for "truth".   If
>> you instead write a test and want to run it with a particular image
>> and expect a particular result, hashes on the input and output have
>> several benefits - including helping maintain "truth".   I won't
>> (probably cannot) go into all of the cool benefits (e.g., if the test
>> is highly deterministic, checking the hash of the output MAY be
>> sufficient for checking the result), but Bill H and others have given
>> this much thought and developed a highly functional (but undoubtedly
>> not perfect) implementation.   Also, we don't have to use MIDAS to
>> store the data or CMake/CTest to generate the hashes, but this system
>> does need some way of centralized storage of multiple versions of
>> files tagged by different hashes.   No doubt we could probably extend
>> gerrit to do this kind of data management, but when looking at the
>> broader open-source community and realizing the number of projects
>> using cmake and the fact that ctest and cmake were meant to be used
>> for testing and test management while gerrit was not, it seems that
>> the biggest impact on open-source can be had by adding more testing
>> capabilities to cmake/ctest rather than extending gerrit so that its
>> use becomes required for testing.
>>
>> Again - not saying that ITK must use it - but I hope we all can go
>> into it with an open mind.  Then we can either go with what they got
>> and help them improve it, or adopt something better, or stick with
>> what has worked in the past...
>>
>> I also can't be at the meeting next week either.   I'm sure it will be
>> a lively discussion.   But, in the end, I'm sure something good will
>> result.
>>
>> Now, about simpleITK.... :)
>>
>> Thanks,
>> Stephen
>>
>>
>>
>> On Fri, Nov 5, 2010 at 9:09 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>> Stephen,
>>>
>>> As a member of the VTK community and the VTK ARB, I have the same
>>> concerns I mentioned. I do not believe this has been discussed by the
>>> VTK community. I cannot find any discussion regarding this on the VTK
>>> wiki.
>>>
>>> Bill
>>>
>>> On Fri, Nov 5, 2010 at 8:14 PM, Stephen Aylward
>>> <stephen.aylward at kitware.com> wrote:
>>>> Hi Bill,
>>>>
>>>> I think Luis did a great job at pointing out in a previous email that
>>>> this is merely a suggestion for ITK, and there have been numerous
>>>> discussions regarding data hosting and management in repositories
>>>> intended for source code - particularly when used with gerrit.   There
>>>> is a problem.  Heck, Luis didn't start this email thread - someone
>>>> else did when they were trying to determine what to do with their
>>>> data.
>>>>
>>>> My understanding (and it could be wrong on some details, but is
>>>> correct in the general message) is that the system that will be
>>>> presented by others (not Luis) was developed by the CMake team at the
>>>> request of the VTK team, and it is being put into place by those teams
>>>> for those projects.   As an existing open source solution, we should
>>>> consider it.    Never did Luis dictate its use.   You know Luis - he
>>>> isn't like that.
>>>>
>>>> But, you bring up a good point.  The presentation should follow the
>>>> outline you've given - state the problem, discuss how this solution
>>>> addresses them and falls short (I applaud that they already have
>>>> hands-on experience with it), listen to suggestions for alternatives,
>>>> and then pick a way forward as a community.
>>>>
>>>> Stephen
>>>>
>>>>
>>>>
>>>> On Fri, Nov 5, 2010 at 7:45 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>>>> Luis,
>>>>>
>>>>> I am surprised at the approach taken to address this important issue.
>>>>>
>>>>> My suggested approach:
>>>>> ITK needs a better way to manage test data and test baselines. Define
>>>>> the problem, establish baseline measurements, analyze the root causes,
>>>>> propose improvements, establish a process to control the solution. All
>>>>> with community input.
>>>>>
>>>>> The current approach is:
>>>>> ITK needs a better way to manage test data and test baselines. Here is
>>>>> a solution, please review it.
>>>>>
>>>>> The later approach shows that a group has spent time and money for a
>>>>> solution without community input regarding the problem, requirements
>>>>> and alternative solutions.
>>>>>
>>>>> Bill
>>>>>
>>>>> On Fri, Nov 5, 2010 at 2:38 PM, Luis Ibanez <luis.ibanez at kitware.com> wrote:
>>>>>> Bill,
>>>>>>
>>>>>> On Thu, Nov 4, 2010 at 4:04 PM, Bill Lorensen <bill.lorensen at gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> I hope you have some discussion about this in Iowa. I assume it is a
>>>>>>> proposal and not a mandate.
>>>>>>
>>>>>>
>>>>>> This is an open source project,
>>>>>>
>>>>>> We don't do mandates,
>>>>>> except to keep patented code away     :-)
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I hope that Baselines will remain (go back) in the ITK repository (not
>>>>>>> a submodule). They currently take 17 megs.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> What really matters is whether the images are in our local
>>>>>> working tree when they are needed to run the tests, and
>>>>>> whether they are the right version (of the images) for the
>>>>>> current version of the source code.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Test data is another issue. If there is a solution it must allow
>>>>>>> testing without an internet connection.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Both the Git submodule and MIDAS options allow running
>>>>>> testing without a network connection, of course, as long as
>>>>>>
>>>>>> a)  We download the data in an initial step, and
>>>>>> b)  We get the right version of the data.
>>>>>>
>>>>>> In practice, in both cases, the work cycle becomes:
>>>>>>
>>>>>> 0) Download the data initially
>>>>>> 1) Run the tests in many instances
>>>>>> 2) From time to time download
>>>>>>      new versions of images  that have changed.
>>>>>>
>>>>>>
>>>>>>     Luis
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Powered by www.kitware.com
>>>>>
>>>>> Visit other Kitware open-source projects at
>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>
>>>>> Kitware offers ITK Training Courses, for more information visit:
>>>>> http://kitware.com/products/protraining.html
>>>>>
>>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>>>
>>>>> Follow this link to subscribe/unsubscribe:
>>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> ==============================
>>>> Stephen R. Aylward, Ph.D.
>>>> Director of Medical Imaging Research
>>>> Kitware, Inc. - North Carolina Office
>>>> http://www.kitware.com
>>>> stephen.aylward (Skype)
>>>> (919) 969-6990 x300
>>>>
>>>
>>
>>
>>
>> --
>>
>> ==============================
>> Stephen R. Aylward, Ph.D.
>> Director of Medical Imaging Research
>> Kitware, Inc. - North Carolina Office
>> http://www.kitware.com
>> stephen.aylward (Skype)
>> (919) 969-6990 x300
>>
>



-- 

==============================
Stephen R. Aylward, Ph.D.
Director of Medical Imaging Research
Kitware, Inc. - North Carolina Office
http://www.kitware.com
stephen.aylward (Skype)
(919) 969-6990 x300

More information about the Insight-developers mailing list