[Insight-users] Re: Tumor Measurements..
Luis Ibanez
luis.ibanez at kitware.com
Sun Nov 7 10:37:46 EST 2004
Hi Lino,
The main two aspects that you may want to evaluate of a
segmentation method are:
1) Is it "at least" as repeatable as a human rater ?
2) Are its differences with a human rater smaller or
equal to the differences between two different raters ?
For that purpose you *must keep separate* the statistics of
intra-rater variability and inter-rater variability
Intra-rater variability can be estimated by having the same
human rater segment the same structure several times, ideally
with several days of interval.
Inter-rater variability will be estimated by comparing the means
from different raters.
A computer assisted segmentation method will be "equivalent" to
a human operator when the repeatability of the computer method
will be similar to the one of the human operators. AND when
the differences between the computer method and any human
rater will be similar to the differences between two human raters.
For a paper discussing the issues of segmentation validation, you
may want to look at:
"VALMET: A New Validation Tool for Assessing and improving
3D Object Segmentation"
G. Gerig, M. Jomier, M. Chackos,
MICCAI 2001, Springer Lecture Notes in Computer Science, p. 516.
http://www.springerlink.com/app/home/contribution.asp?wasp=bn42d3kp5g1ytjbb274p&referrer=parent&backto=issue,59,250;journal,1012,1768;linkingpublicationresults,1:105633,1
Don't forget to also look at the paper suggested by Simon Warfield
http://www.itk.org/pipermail/insight-users/2004-November/010913.html
http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=pubmed&dopt=Abstract&list_uids=11208850
Regards,
Luis
--------------------------
Lino Ramirez wrote:
>Hi Luis,
>
>Thank you very much for your comprehensive explanation.
>
>I am designing the experiments to validate a segmentation approach I am
>developing. The final goal of this segmentation is to provide clinicians
>with a set of measurements they use in their daily practice.
>
>Now, I have some questions about comparing the results obtained by the
>expert(s) and the computer method:
>
>1.- Should I pool together the results obtained by the same expert at
>different time intervals with the results obtained by different experts
>(to get let's say at least 30 results) and then compare those results with
>the computer generated results using for example paired t-tests (or other
>type of statistical test)?
>
>2.- Should I keep the inter-observer measurements separated from the
>intra-observer measurements and do the individual comparisons of these
>results with the computer-generated results?
>
>3.- Should I repeat the comparisons for each parameter I can set up in the
>computer segmentation method?
>
>4.- Do you know of any paper in which the authors present a comprehensive
>comparison of expert-made measurements with computer-generated
>measurements? I want to produce a tool that could be used in clinical
>practice and I think that the best way of convincing clinicians of the
>validity of the tool is providing a sound statistical analysis.
>
>Thank you very much for all your help.
>
>Have a nice day
>
>Lino
>
>
>
>
>>Hi Lino,
>>
>>When measuring a value there is always variability
>>in the outcome of the measurement.
>>
>>This is due to the fact that the measurement process
>>itself is regulated by parameters. Some of those
>>parameters are under control and can be regulated,
>>some other parameters are not under control, and
>>may even be unknown.
>>
>>
>>Let's consider both cases:
>>
>>1) Parameters not under control.
>>
>> When dealing with parameters of the measurement
>> process that are not under control, the only way to
>> estimate values is to take the measure multiple times
>> and attempt to estimate the statistical distribution
>> of the outcome
>>
>> In most cases, Gaussian distributions are assumed,
>> usually for the lack of any other better model.
>>
>> When you follow this method, the set of values
>> resulting from your measurements will have a Mean
>> and a Standard Deviation. In a Gaussian distribution
>> you can claim that if you randomly sample the
>> distribution, X% of the samples will be at a distance
>> of K*(Standard Deviations) from the Mean. The relation
>> ship between X and K is given by the erf() function.
>> (the integral of the Gaussian).
>>
>> X% is called the confidence and K*STD define your interval.
>> A typical common confidence value to use is 95% which
>> corresponds to +-3*STD. Meaning that if you integrate the
>> area of the Gaussian from (Mean - 3*STD) to (Mean+3*STD)
>> you get 95% of the total area under the full Gaussian.
>>
>>
>> This is the typical process to follow when segmentation
>> is done under human supervision, since it allows to account
>> for the variability of human judgment, both in the same
>> individual over time, and between multiple different
>> individuals.
>>
>>
>>
>>2) Parameters under control.
>>
>> For parameters that are under control, the uncertainty of
>> the final measured value is related to the sensitivity of
>> the measurement as response to perturbations of the parameters.
>>
>> For a trivial example:
>>
>> If you use a thresholding method in order to segment a tumor
>> that has been made visible by Gadolinium enhancement in an
>> MRI image, the value of the threshold is a critical parameter.
>> Small changes in the value of the threshold will produce
>> variations on the segmentation result and therefore will affect
>> the volume estimation of the tumor.
>>
>> From the point of view of Measurement Theory, the preferred
>> value for the threshold should be the one that at which perturbations
>> result in minimal variations of the value to be measured (in this
>> case the tumor volume).
>>
>> In order to bound the range of the thresholding value, you must
>> establish what are the limits in which the resulting segmentation
>> is still plausible. For example, let's say that you find that
>> plausible segmentations are produced when the thresholding value
>> is between 150 and 180. That means that at threshold values less
>> than 150 you observe that the segmentation is missing regions that
>> are clearly part of the tumor, or that it is including regions of
>> healthy tissue.
>>
>> At this point you can generate a population of threshold values
>> that follow a uniform statistical distribution inside the plausible
>> interval. For each one of those thresholding values the segmentation
>> will result in a specific tumor volume. If you collect those volumes
>> and compute their Mean and Standard deviation, then you will be
>> able to generate an interval around the mean that correspond to the
>> confidence level that you selected.
>>
>>
>>What is challenging in most segmentation methods is that there are
>>many input parameters that may affect the outcome of the measurement.
>>Therefore, plausible ranges must be established for each parameters
>>and using their collective ranges, a population of segmentation can
>>be generated.
>>
>>
>>There are a number of tools that can help you to explore the parameter
>>space and estimate the statistical distribution of the measured value.
>>
>>
>>
>> Please let us know if you have further questions.
>>
>>
>> Thanks
>>
>>
>> Luis
>>
>>
>>--------------------
>>Lino Ramirez wrote:
>>
>>
>>
>>>Hi Luis,
>>>
>>>Since you pointed out that
>>>
>>>----------------------------------------
>>>Do not forget also that when you report measurement
>>>they must be accompanied by the uncertainty in the
>>>measure and the confidence of this interval.
>>>
>>>For example,
>>>the tumor volume should be reported as:
>>>
>>> 30 mm^3 +/- 5 mm^3 with 95% confidence.
>>>---------------------------
>>>
>>>Could you please provide some references on how to compute the
>>>uncertainty
>>>and confidence interval of the measurements obtained after using a
>>>segmentation method?
>>>
>>>Thank you,
>>>
>>>Lino
>>>
>>>
>>>_______________________________________________
>>>Insight-users mailing list
>>>Insight-users at itk.org
>>>http://www.itk.org/mailman/listinfo/insight-users
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>
>
>
>_______________________________________________
>Insight-users mailing list
>Insight-users at itk.org
>http://www.itk.org/mailman/listinfo/insight-users
>
>
>
More information about the Insight-users
mailing list