[Insight-developers] Validation Directory
Sayan Pathak
spathak@insightful.com
Wed, 18 Sep 2002 10:35:34 -0700
Hi Josh,
I agree with your opinion. That is the view I have too. Given the time =
frame that is the best we can achieve.
Sayan
> -----Original Message-----
> From: Joshua Cates [mailto:cates@sci.utah.edu]
> Sent: Wednesday, September 18, 2002 9:56 AM
> To: Jisung Kim
> Cc: Sayan Pathak; insight-dev-list
> Subject: RE: [Insight-developers] Validation Directory
>=20
>=20
> Hi Jisung, Sayan, Lydia,
>=20
> I think the simplest course of action is just to follow the=20
> example that
> Lydia has set with her validation work. Relevant code goes in the
> Validation directory, the write up and results (i.e. tables, figures,
> stats, etc) go in InsightDocuments, and any large input or=20
> output data
> files go in the Insight data ftp repository. =20
>=20
> Organize your code and results under the appropriate=20
> directory in whatever
> way makes sense for your particular study.
>=20
> Whether or not we want Validation code on the dashboard right=20
> now is an=20
> open question, but my vote is to leave it off until things have=20
> stabilized.
>=20
> Josh.
>=20
> ______________________________
> Josh Cates =09
> School of Computer Science=09
> University of Utah
> Email: cates@sci.utah.edu
> Phone: (801) 587-7697
> URL: www.cs.utk.edu/~cates
>=20
>=20
> On Tue, 17 Sep 2002, Jisung Kim wrote:
>=20
> > HI Sayan.
> >=20
> > I think we don't have much time left before final
> > release. I think the most important thing in
> > coordinating validation studies is that making things
> > comparable to be easily comparable. What I mean is
> > that studies that use similar or exactly same data for
> > similar purpose should be easily comparable between
> > different groups' studies.=20
> >=20
> > For example, your IBSR classification study includes
> > GaussianClassifier, KmeansClassifier,
> > MRFGaussianClassifier, and MRFKmeansClassifier. My
> > validation study will include k-d tree based Kmeans,
> > expectation-maximization mixture modelling, and
> > goodness-of-fit mixture modelling. I plan to use
> > BrainWeb data for multi-channel experiments and IBSR
> > for single-channel experiments. Users might want to
> > compare the results from your KmeansClassifier and my
> > k-d tree based Kmeans clustering or the results from
> > you MRFKmeansClassifier and my EM stuff.=20
> >=20
> > I think at least we should agree on two things for
> > this. First, since you and I will use the IBSR data,
> > by creating a common data discription that explains
> > how to get the data and includes proper credentials,
> > we can prevent duplicate and confusing description
> > about the same data. I think it is even better if we
> > can agree on even the image file format. I prefer
> > meta-image format, because of its simplicity and easy
> > to read through text editors (at least, you can get
> > basic information about the image from it). Second, it
> > will be nice if we have a consistent formats for the
> > experiments' output that allows users to read them on
> > their favorite data analysis tools to summarize and
> > plot them to get more comparable and intuitive
> > comparisons.
> >=20
> > Here are my some suggestions:
> >=20
> > 1) directory structure
> > * Validation/Data
> > has subdirectories for each datasets. For an
> > example, Validation/Data/IBSR and
> > Validation/Data/BrainWeb. the IBSR directory may
> > includes a credentials or copyright statement of IBSR
> > dataset, how-to-get-it documents, and meta image
> > headers (only headers).
> >=20
> > * Validataion/Utilities or Validation/Common
> > I found that plotting data before doing any real
> > processing is quite important to understand the data
> > and make proper plans for data analysis. So I created
> > a utility that samples data point from the dataset and
> > create a table in a file that I can use for plotting
> > in a statistical package. I also have a preprocessor
> > that maskes out some tissue classes from the data
> > using class mask image and then normalize images using
> > images' means and standard deviations. I believe
> > normalizing process is quite common for multivariate
> > data analysis. I think also some basic UI stuff can be
> > placed here too.
> >=20
> > * Validation/"Your Own Studies"
> > In this directory, we put our own validation stuff
> > specific to each study. I like your idea, having at
> > least three common subdirectories that you already
> > have with your validation stuff, "Code", "Inputs", and
> > "Results".
> >=20
> > 2) Experiments Output format
> > * table with headers, each class statistics is a
> > record ( a row) in the table.
> >=20
> > For example, a clustering algorithm produces 3
> > different Gaussian classes with each class has a mean
> > and a standard deviation as its parameters. And in
> > addition to that it has two common field for an case,
> > "number of iterations" and "elapsed time". And I ran
> > it with two different sets of initial parameters for
> > classes (say, two cases in the experiment). Then the
> > output file would look like:
> >=20
> > "case" "class" "mean" "standard deviation"
> > "iterations" "elapsed time"
> > 1 1 200 40 2000 20.35
> > 1 2 300 20 2000 20.35
> > 1 3 100 50 2000 20.35
> > 2 1 202 38 2020 21.23
> > 2 2 298 19 2020 21.23
> > 2 3 98 47 2020 21.23
> >=20
> > I also want to name such output file or other table
> > like files ( such as intial parameters files) have the
> > same file extension such as .dat for each search. :)
> >=20
> >=20
> > What do you think, Sayan and other people working on
> > validation?
> >=20
> >=20
> > =20
> >=20
> >=20
> >=20
> > --- Sayan Pathak <spathak@insightful.com> wrote:
> > > Hi Jisung,
> > > I wanted to checkin my code in the validation
> > > directory too. In the last TCON, this issue was
> > > discussed. I had asked about the rules if any. The
> > > answer was that we have to first agree what goes in
> > > there. So for the time being I checked the code in
> > > example. Thanks for bringing this up again and may
> > > be we will have a concensus on the shape of the
> > > validation directory soon.
> > >=20
> > > Sayan
> > >=20
> > > > -----Original Message-----
> > > > From: Jisung Kim [mailto:bahrahm@yahoo.com]
> > > > Sent: Tuesday, September 17, 2002 9:18 AM
> > > > To: insight-dev-list
> > > > Subject: [Insight-developers] Validation Directory
> > > >=20
> > > >=20
> > > > Hi.
> > > >=20
> > > > I want to check in some codes and documents
> > > related
> > > > with my validation work. I think we decided to use
> > > > "Insight source directory"/Validation directory
> > > for
> > > > this purpose.=20
> > > > Sayan alread put some of his stuff under the
> > > Examples
> > > > directory. I want to use the Validation directory
> > > from
> > > > the beginning. To do that, I think I have to
> > > create a
> > > > build option (e.g. BUILD_VALIDATIONS)for
> > > validation
> > > > stuff just like the "BUILD_EXAMPLES" option in the
> > > > CMakeLists.txt file in the root source directory,
> > > and
> > > > create a subdirectory and a CMakelists.txt file
> > > for my
> > > > validation apps under the Validation directory.=20
> > > >=20
> > > > By doing that, will the dashboard will compile
> > > > validation stuff and report errors and warnings?
> > > Whom
> > > > should I ask for help if I want to validation
> > > stuff
> > > > build for the dashboard?
> > > >=20
> > > > Another question is related with the proposed one
> > > big
> > > > binary directory. Do those validation binaries
> > > have to
> > > > be in the big binary directory too? I guess the
> > > answer
> > > > is yes. If so, could anybody give me an example
> > > of, or
> > > > some information on CMakeLists.txt that builds
> > > apps
> > > > using and places binaries in a specific
> > > directory?
> > > >=20
> > > > Do I have to wait until existing binaries have
> > > moved
> > > > to the big binary directory before doing this?
> > > >=20
> > > > Thanks,
> > > >=20
> > > >=20
> > > >=20
> > > > =3D=3D=3D=3D=3D
> > > > Jisung Kim
> > > > bahrahm@yahoo.com
> > > > 106 Mason Farm Rd.
> > > > 129 Radiology Research Lab., CB# 7515
> > > > Univ. of North Carolina at Chapel Hill
> > > > Chapel Hill, NC 27599-7515
> > > >=20
> > > > __________________________________________________
> > > > Do you Yahoo!?
> > > > Yahoo! News - Today's headlines
> > > > http://news.yahoo.com
> > > > _______________________________________________
> > > > Insight-developers mailing list
> > > > Insight-developers@public.kitware.com
> > > >
> > >
> > http://public.kitware.com/mailman/listinfo/insight-developers
> > > >=20
> > > >=20
> > > _______________________________________________
> > > Insight-developers mailing list
> > > Insight-developers@public.kitware.com
> > >
> > http://public.kitware.com/mailman/listinfo/insight-developers
> >=20
> >=20
> > =3D=3D=3D=3D=3D
> > Jisung Kim
> > bahrahm@yahoo.com
> > 106 Mason Farm Rd.
> > 129 Radiology Research Lab., CB# 7515
> > Univ. of North Carolina at Chapel Hill
> > Chapel Hill, NC 27599-7515
> >=20
> > __________________________________________________
> > Do You Yahoo!?
> > Yahoo! Health - Feel better, live better
> > http://health.yahoo.com
> > _______________________________________________
> > Insight-developers mailing list
> > Insight-developers@public.kitware.com
> > http://public.kitware.com/mailman/listinfo/insight-developers
> >=20
>=20
>=20
>=20