[Insight-developers] Need to add images for new tests to Data -- how to do that in a Gerrit topic?

Fri Nov 5 23:39:15 EDT 2010

Sorry, I won't be there.

On Fri, Nov 5, 2010 at 11:26 PM, Stephen Aylward
<stephen.aylward at kitware.com> wrote:
> The meeting is the chance!
>
> You're right - sorry if I was unclear -
>
>>> if the test
>>> is highly deterministic, checking the hash of the output MAY be
>>> sufficient for checking the result
>
> Instead of "highly deterministic" I should have clarified "has no
> error on all platforms"
>
> s
>
> On Fri, Nov 5, 2010 at 11:16 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>> I don't see how it will work with multiple baselines and/or provide
>> the ability to specify error thresholds. These are just a couple of
>> requirements off the top of my head.
>>
>> I wish there had been input from the ITK Developers.
>>
>>
>> On Fri, Nov 5, 2010 at 9:47 PM, Stephen Aylward
>> <stephen.aylward at kitware.com> wrote:
>>> Hi Bill,
>>>
>>> Sorry - this is how it was presented to me - the mis-communication
>>> might be on my end (even communication within groups at Kitware can be
>>> challenging), and/or perhaps communication within VTK/CMake teams
>>> needs to be strengthened.
>>>
>>> However, I am very confident that Luis/ITK did not initiate the
>>> development of this system...in particular, I know this because I must
>>> admit that I too was not a big fan at first!   Why store hashes of
>>> tests when git or other systems can be used to track revisions, etc
>>> BUT I am now a believer.   Git, Gerrit, sub-projects, modularization,
>>> cdash at home, branchy workflows, and other tools and use cases that ITK
>>> has chosen are causing a file name to be a poor tag for "truth".   If
>>> you instead write a test and want to run it with a particular image
>>> and expect a particular result, hashes on the input and output have
>>> several benefits - including helping maintain "truth".   I won't
>>> (probably cannot) go into all of the cool benefits (e.g., if the test
>>> is highly deterministic, checking the hash of the output MAY be
>>> sufficient for checking the result), but Bill H and others have given
>>> this much thought and developed a highly functional (but undoubtedly
>>> not perfect) implementation.   Also, we don't have to use MIDAS to
>>> store the data or CMake/CTest to generate the hashes, but this system
>>> does need some way of centralized storage of multiple versions of
>>> files tagged by different hashes.   No doubt we could probably extend
>>> gerrit to do this kind of data management, but when looking at the
>>> broader open-source community and realizing the number of projects
>>> using cmake and the fact that ctest and cmake were meant to be used
>>> for testing and test management while gerrit was not, it seems that
>>> the biggest impact on open-source can be had by adding more testing
>>> capabilities to cmake/ctest rather than extending gerrit so that its
>>> use becomes required for testing.
>>>
>>> Again - not saying that ITK must use it - but I hope we all can go
>>> into it with an open mind.  Then we can either go with what they got
>>> and help them improve it, or adopt something better, or stick with
>>> what has worked in the past...
>>>
>>> I also can't be at the meeting next week either.   I'm sure it will be
>>> a lively discussion.   But, in the end, I'm sure something good will
>>> result.
>>>
>>> Now, about simpleITK.... :)
>>>
>>> Thanks,
>>> Stephen
>>>
>>>
>>>
>>> On Fri, Nov 5, 2010 at 9:09 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>>> Stephen,
>>>>
>>>> As a member of the VTK community and the VTK ARB, I have the same
>>>> concerns I mentioned. I do not believe this has been discussed by the
>>>> VTK community. I cannot find any discussion regarding this on the VTK
>>>> wiki.
>>>>
>>>> Bill
>>>>
>>>> On Fri, Nov 5, 2010 at 8:14 PM, Stephen Aylward
>>>> <stephen.aylward at kitware.com> wrote:
>>>>> Hi Bill,
>>>>>
>>>>> I think Luis did a great job at pointing out in a previous email that
>>>>> this is merely a suggestion for ITK, and there have been numerous
>>>>> discussions regarding data hosting and management in repositories
>>>>> intended for source code - particularly when used with gerrit.   There
>>>>> is a problem.  Heck, Luis didn't start this email thread - someone
>>>>> else did when they were trying to determine what to do with their
>>>>> data.
>>>>>
>>>>> My understanding (and it could be wrong on some details, but is
>>>>> correct in the general message) is that the system that will be
>>>>> presented by others (not Luis) was developed by the CMake team at the
>>>>> request of the VTK team, and it is being put into place by those teams
>>>>> for those projects.   As an existing open source solution, we should
>>>>> consider it.    Never did Luis dictate its use.   You know Luis - he
>>>>> isn't like that.
>>>>>
>>>>> But, you bring up a good point.  The presentation should follow the
>>>>> outline you've given - state the problem, discuss how this solution
>>>>> addresses them and falls short (I applaud that they already have
>>>>> hands-on experience with it), listen to suggestions for alternatives,
>>>>> and then pick a way forward as a community.
>>>>>
>>>>> Stephen
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Nov 5, 2010 at 7:45 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>>>>> Luis,
>>>>>>
>>>>>> I am surprised at the approach taken to address this important issue.
>>>>>>
>>>>>> My suggested approach:
>>>>>> ITK needs a better way to manage test data and test baselines. Define
>>>>>> the problem, establish baseline measurements, analyze the root causes,
>>>>>> propose improvements, establish a process to control the solution. All
>>>>>> with community input.
>>>>>>
>>>>>> The current approach is:
>>>>>> ITK needs a better way to manage test data and test baselines. Here is
>>>>>> a solution, please review it.
>>>>>>
>>>>>> The later approach shows that a group has spent time and money for a
>>>>>> solution without community input regarding the problem, requirements
>>>>>> and alternative solutions.
>>>>>>
>>>>>> Bill
>>>>>>
>>>>>> On Fri, Nov 5, 2010 at 2:38 PM, Luis Ibanez <luis.ibanez at kitware.com> wrote:
>>>>>>> Bill,
>>>>>>>
>>>>>>> On Thu, Nov 4, 2010 at 4:04 PM, Bill Lorensen <bill.lorensen at gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I hope you have some discussion about this in Iowa. I assume it is a
>>>>>>>> proposal and not a mandate.
>>>>>>>
>>>>>>>
>>>>>>> This is an open source project,
>>>>>>>
>>>>>>> We don't do mandates,
>>>>>>> except to keep patented code away     :-)
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I hope that Baselines will remain (go back) in the ITK repository (not
>>>>>>>> a submodule). They currently take 17 megs.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> What really matters is whether the images are in our local
>>>>>>> working tree when they are needed to run the tests, and
>>>>>>> whether they are the right version (of the images) for the
>>>>>>> current version of the source code.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Test data is another issue. If there is a solution it must allow
>>>>>>>> testing without an internet connection.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Both the Git submodule and MIDAS options allow running
>>>>>>> testing without a network connection, of course, as long as
>>>>>>>
>>>>>>> a)  We download the data in an initial step, and
>>>>>>> b)  We get the right version of the data.
>>>>>>>
>>>>>>> In practice, in both cases, the work cycle becomes:
>>>>>>>
>>>>>>> 0) Download the data initially
>>>>>>> 1) Run the tests in many instances
>>>>>>> 2) From time to time download
>>>>>>>      new versions of images  that have changed.
>>>>>>>
>>>>>>>
>>>>>>>     Luis
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Powered by www.kitware.com
>>>>>>
>>>>>> Visit other Kitware open-source projects at
>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>>
>>>>>> Kitware offers ITK Training Courses, for more information visit:
>>>>>> http://kitware.com/products/protraining.html
>>>>>>
>>>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>>>>
>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> ==============================
>>>>> Stephen R. Aylward, Ph.D.
>>>>> Director of Medical Imaging Research
>>>>> Kitware, Inc. - North Carolina Office
>>>>> http://www.kitware.com
>>>>> stephen.aylward (Skype)
>>>>> (919) 969-6990 x300
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> ==============================
>>> Stephen R. Aylward, Ph.D.
>>> Director of Medical Imaging Research
>>> Kitware, Inc. - North Carolina Office
>>> http://www.kitware.com
>>> stephen.aylward (Skype)
>>> (919) 969-6990 x300
>>>
>>
>
>
>
> --
>
> ==============================
> Stephen R. Aylward, Ph.D.
> Director of Medical Imaging Research
> Kitware, Inc. - North Carolina Office
> http://www.kitware.com
> stephen.aylward (Skype)
> (919) 969-6990 x300
>