[Insight-developers] benchmarking biomedical algorithms

[Luis Ibanez]

Hi Dave

This is great.

Thanks for pursuing this initiative.

I'm not aware of a benchmark suit of this type. It will be great
to create one , particularly if it is made out of publicly available
block (code, data, parameters, scripts...).

A benchmark tool must be fully available,
for it to be taken seriously.

Would you like to create a Wiki page for this initiative ?

In this way it will be easier to gather feedback from the community
and to keep track of the comments and suggestions.

In the meantime I'll suggest to include:

1) Deformable registration
    - Demons
    - BSplines
2) Connected components filters
3) Region growing filters
4) Level Sets


    Luis


-----------------------------------------------
Holmes, David R. III, Ph.D. [RO RM1-24] wrote:
> Colleagues-
> 
>  
> 
> I apologize if this note is misplaced.  Please feel free to disregard 
> completely; however, I have been asked by a hardware manufacturer to 
> help determine what a “benchmark suite” of biomedical imaging algorithms 
> would look like.  I, personally, believe that this developer list is 
> viewed by the top developer, and I would appreciate your input.
> 
>  
> 
> My eventual goal for this is to spec out a list of algorithms to be used 
> in test suite.  The algorithms would have to include specific reference 
> which describe them and representative test data (image data and final 
> result).  This is somewhat similar to the dashboard testing, but the 
> data must be real, representative data.  It would be ideal if the 
> algorithms include code or pseudo-code.  I am hoping to draw the data 
> from the NLM-Mayo data collection as it is freely available.  For some 
> algorithms, I am hoping to draw directly from ITK; although in some 
> cases the generic programming/templated code may not be well-suited for 
> this type of benchmarking.
> 
>  
> 
> I would hope that this well-defined collection would be used, at a 
> minimum, to benchmark new CPU/GPU technologies; however, this would also 
> serve as a nice benchmark to test out different implementations of 
> algorithms.  I know that this is something that the ITK community has 
> repeatedly talked about, but I don’t yet know of a test suite.
> 
>  
> 
> I plan to span a wide variety of algorithms, and I would initially like 
> to avoid very complex algorithms (or stochastic algorithms) as they may 
> encumber the purpose of the suite (e.g. benchmarking). Rather, I am 
> interested in basic building blocks which would be incorporated into 
> more complex and valuable algorithms.   I’m after algorithms which 
> require different numerical precision, different regions of support, 
> different dimensionality, etc.  For example, I would by default include:
> 
>  
> 
> -Thresholding
> 
> -Erode or Dilate Morphologic operation
> 
> -Volume transformation
> 
> -simple volumetric projection – MIP or summed-voxel projection
> 
> -linear or non-linear filtering – average, rank, gradient filter.
> 
> -histogram calculation
> 
>  
> 
> Anyway, I would appreciate any input that you might have.  I guess I 
> should really first ask if someone has already developed such a suite.  
> Again, I apologize if this is not the right forum.
> 
>  
> 
> Thanks
> 
>  
> 
> david
> 
>  
> 
> ============================================
> 
> David R Holmes III
> 
> Department of Physiology and Biomedical Engineering
> 
> Mayo Clinic College of Medicine
> 
>  
> 
> Ph:  507-266-4250       Fax:  507-284-1632
> 
> holmes.david3 at mayo.edu <mailto:holmes.david3 at mayo.edu>
> 
> ============================================
> 
>  
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Insight-developers mailing list
> Insight-developers at itk.org
> http://www.itk.org/mailman/listinfo/insight-developers