[Insight-developers] Validation Directory
Jisung Kim
bahrahm@yahoo.com
Tue, 17 Sep 2002 15:32:16 -0700 (PDT)
HI Sayan.
I think we don't have much time left before final
release. I think the most important thing in
coordinating validation studies is that making things
comparable to be easily comparable. What I mean is
that studies that use similar or exactly same data for
similar purpose should be easily comparable between
different groups' studies.
For example, your IBSR classification study includes
GaussianClassifier, KmeansClassifier,
MRFGaussianClassifier, and MRFKmeansClassifier. My
validation study will include k-d tree based Kmeans,
expectation-maximization mixture modelling, and
goodness-of-fit mixture modelling. I plan to use
BrainWeb data for multi-channel experiments and IBSR
for single-channel experiments. Users might want to
compare the results from your KmeansClassifier and my
k-d tree based Kmeans clustering or the results from
you MRFKmeansClassifier and my EM stuff.
I think at least we should agree on two things for
this. First, since you and I will use the IBSR data,
by creating a common data discription that explains
how to get the data and includes proper credentials,
we can prevent duplicate and confusing description
about the same data. I think it is even better if we
can agree on even the image file format. I prefer
meta-image format, because of its simplicity and easy
to read through text editors (at least, you can get
basic information about the image from it). Second, it
will be nice if we have a consistent formats for the
experiments' output that allows users to read them on
their favorite data analysis tools to summarize and
plot them to get more comparable and intuitive
comparisons.
Here are my some suggestions:
1) directory structure
* Validation/Data
has subdirectories for each datasets. For an
example, Validation/Data/IBSR and
Validation/Data/BrainWeb. the IBSR directory may
includes a credentials or copyright statement of IBSR
dataset, how-to-get-it documents, and meta image
headers (only headers).
* Validataion/Utilities or Validation/Common
I found that plotting data before doing any real
processing is quite important to understand the data
and make proper plans for data analysis. So I created
a utility that samples data point from the dataset and
create a table in a file that I can use for plotting
in a statistical package. I also have a preprocessor
that maskes out some tissue classes from the data
using class mask image and then normalize images using
images' means and standard deviations. I believe
normalizing process is quite common for multivariate
data analysis. I think also some basic UI stuff can be
placed here too.
* Validation/"Your Own Studies"
In this directory, we put our own validation stuff
specific to each study. I like your idea, having at
least three common subdirectories that you already
have with your validation stuff, "Code", "Inputs", and
"Results".
2) Experiments Output format
* table with headers, each class statistics is a
record ( a row) in the table.
For example, a clustering algorithm produces 3
different Gaussian classes with each class has a mean
and a standard deviation as its parameters. And in
addition to that it has two common field for an case,
"number of iterations" and "elapsed time". And I ran
it with two different sets of initial parameters for
classes (say, two cases in the experiment). Then the
output file would look like:
"case" "class" "mean" "standard deviation"
"iterations" "elapsed time"
1 1 200 40 2000 20.35
1 2 300 20 2000 20.35
1 3 100 50 2000 20.35
2 1 202 38 2020 21.23
2 2 298 19 2020 21.23
2 3 98 47 2020 21.23
I also want to name such output file or other table
like files ( such as intial parameters files) have the
same file extension such as .dat for each search. :)
What do you think, Sayan and other people working on
validation?
--- Sayan Pathak <spathak@insightful.com> wrote:
> Hi Jisung,
> I wanted to checkin my code in the validation
> directory too. In the last TCON, this issue was
> discussed. I had asked about the rules if any. The
> answer was that we have to first agree what goes in
> there. So for the time being I checked the code in
> example. Thanks for bringing this up again and may
> be we will have a concensus on the shape of the
> validation directory soon.
>
> Sayan
>
> > -----Original Message-----
> > From: Jisung Kim [mailto:bahrahm@yahoo.com]
> > Sent: Tuesday, September 17, 2002 9:18 AM
> > To: insight-dev-list
> > Subject: [Insight-developers] Validation Directory
> >
> >
> > Hi.
> >
> > I want to check in some codes and documents
> related
> > with my validation work. I think we decided to use
> > "Insight source directory"/Validation directory
> for
> > this purpose.
> > Sayan alread put some of his stuff under the
> Examples
> > directory. I want to use the Validation directory
> from
> > the beginning. To do that, I think I have to
> create a
> > build option (e.g. BUILD_VALIDATIONS)for
> validation
> > stuff just like the "BUILD_EXAMPLES" option in the
> > CMakeLists.txt file in the root source directory,
> and
> > create a subdirectory and a CMakelists.txt file
> for my
> > validation apps under the Validation directory.
> >
> > By doing that, will the dashboard will compile
> > validation stuff and report errors and warnings?
> Whom
> > should I ask for help if I want to validation
> stuff
> > build for the dashboard?
> >
> > Another question is related with the proposed one
> big
> > binary directory. Do those validation binaries
> have to
> > be in the big binary directory too? I guess the
> answer
> > is yes. If so, could anybody give me an example
> of, or
> > some information on CMakeLists.txt that builds
> apps
> > using and places binaries in a specific
> directory?
> >
> > Do I have to wait until existing binaries have
> moved
> > to the big binary directory before doing this?
> >
> > Thanks,
> >
> >
> >
> > =====
> > Jisung Kim
> > bahrahm@yahoo.com
> > 106 Mason Farm Rd.
> > 129 Radiology Research Lab., CB# 7515
> > Univ. of North Carolina at Chapel Hill
> > Chapel Hill, NC 27599-7515
> >
> > __________________________________________________
> > Do you Yahoo!?
> > Yahoo! News - Today's headlines
> > http://news.yahoo.com
> > _______________________________________________
> > Insight-developers mailing list
> > Insight-developers@public.kitware.com
> >
>
http://public.kitware.com/mailman/listinfo/insight-developers
> >
> >
> _______________________________________________
> Insight-developers mailing list
> Insight-developers@public.kitware.com
>
http://public.kitware.com/mailman/listinfo/insight-developers
=====
Jisung Kim
bahrahm@yahoo.com
106 Mason Farm Rd.
129 Radiology Research Lab., CB# 7515
Univ. of North Carolina at Chapel Hill
Chapel Hill, NC 27599-7515
__________________________________________________
Do You Yahoo!?
Yahoo! Health - Feel better, live better
http://health.yahoo.com