[Insight-users] about the Information Theory section in the ITK Guide

Tue Feb 13 14:57:29 EST 2007

OK, thank you Wagner!
Well, my suggestion is in fact a correction.
In fact, it is a rather theoretical correction that I am sure it is not
central to the purpose of this list,
so I think anyone who is not the guy who wrote the part of the ITK Guide of
which I am about to talk
can safely ignore the rest of the message.

Please take this as a humble attempt to contribute.

On to the subject: I am a teacher and researcher working in both image
processing and information theory. While zapping through the guide I found a
small section on Information Theory operators (page 663).
The referred section is wrong in many ways, and I humbly think that being
this package so popular among researchers, it is not good that it remains
that way as it can bias the research direction of many people who happen to
read it.
This is what the section says:
--------------
Computing Image Entropy

The concept of Entropy has been introduced into image processing as a crude
mapping from
its application in Communications. The notions of Information Theory can be
deceiving and
misleading when applied to images because their language from Communication
Theory does
not necessarily maps to what people in the Imaging Community use.

For example, it is commonly said that
"The Entropy of an image is a measure of the amount of information contained
in an image".
This statement is fundamentally incorrect.

The way the notion of Entropy is commonly measured in images is by ﬁrst
assuming that the
spatial location of a pixel in an image is irrelevant! That is, we simply
take the statistical
distribution of the pixel values as it can be evaluated in a histogram and
from that histogram we
estimate the frequency of the value associated to each bin. In other words,
we simply assume
that the image is a set of pixels that are passing through a channel, just
as things are commonly
considered for communication purposes.
--------------

The first confusion is to say that a "zero order empirical entropy" is "the
definition of entropy".
To be more specific: in Info Theory, Entropy is defined in terms of a
*probabilistic model* for the data at hand. To consider pixels regardless of
their position and relation to other pixels within the image is the simplest
possible model and is widely known and accepted in both Info Theory and
Image Processing fields that is highly innefective for almost any kind of
image class. (*)
"zero order" refers to the fact that the influence of "zero other pixels" is
taken into account to *model* the probability of a given other pixel.
*empirical* entropy refers to the fact that the entropy calculated is based
on a probabilistic model fitted to  empirical data, not a *real*
probability, which is generally not available when modeling any kind of
signal.

It is also misleading to say that the ideas were "mapped" from
"communications" to "image processing". The current theoretical framework of
Info. Theory has long departed from its communication origins, and it is
formulated as a general problem over any kind of data. The concept of
"channel" is often used to model certain distortion processes that can
modify images.

It is also not true that the applications of information theory to image
processing are in their early infancy. There is a lot of work to do, but
algorithms such as the ones in the JPEG2000 or JPEG-LS image compression
formats make heavy and effective use of advanced information theory
concepts.

There are also many effective applications of modern IT in segmentation (see
http://citeseer.ist.psu.edu/leclerc94constructing.html for an example),
texture simulation, restoration and denoising (which is what I have been
doing as a researcher in the last two years) and many more areas.

(*) Later in the next page there is a discussion of how the real concepts of
IT should be used in image processing.  The suggested directions, which are
right, are being used since the 80's.

Thank you,
Ignacio.

2007/2/13, Wagner Sales <wsalles2003 at yahoo.com.br>:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Ignacio,
>
> First of all, take my answer just a suggestion ( I'm not a ITK
> developer and not from Kitware ), but like experienced user, just try
> to put the question here and may be someone can help you.
>
> Regards,
>
> Wagner Sales
>
>
> Ignacio Ramirez Paulino wrote:
> > Hi, I am new to this list. Hi have a very specific suggestion
> > regarding the ITK documentation. Should I post it here? I don't
> > want to subscribe to the developers list just for this.
> >
> > -- --------------------------------------------------------------
> > Ignacio Francisco Ramírez Paulino Magister en Ingeniería Eléctrica
> > --------------------------------------------------------------
> > Instituto de Ingeniería Eléctrica Facultad de Ingeniería -
> > Universidad de la República Tel: 598 2 7110974 x 117
> > MSN:ignacio_francisco_ramirez_paulino at hotmail.com
> > <mailto:MSN:ignacio_francisco_ramirez_paulino at hotmail.com> jabber:
> > ignacio_ramirez at jabber.org <mailto:ignacio_ramirez at jabber.org>
> > Skype: ignacio_ramirez_paulino
> > --------------------------------------------------------------
> >
> > ----------------------------------------------------------------------
> >
> >
> > _______________________________________________ Insight-users
> > mailing list Insight-users at itk.org
> > http://www.itk.org/mailman/listinfo/insight-users
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (GNU/Linux)
> Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org
>
> iD8DBQFF0gDzG0tpd3N7UOYRAhNPAJ9HdxRVGQjIvIRcApRhdLtferxJhwCfUTst
> 5CNdVluDr/uB2IEgbdDTIU8=
> =ehOw
> -----END PGP SIGNATURE-----
>
>
>
> _______________________________________________________
> Yahoo! Mail - Sempre a melhor opção para você!
> Experimente já e veja as novidades.
> http://br.yahoo.com/mailbeta/tudonovo/
>
>
> _______________________________________________
> Insight-users mailing list
> Insight-users at itk.org
> http://www.itk.org/mailman/listinfo/insight-users
>

-- 
--------------------------------------------------------------
Ignacio Francisco Ramírez Paulino
Magister en Ingeniería Eléctrica
--------------------------------------------------------------
Instituto de Ingeniería Eléctrica
Facultad de Ingeniería - Universidad de la República
Tel: 598 2 7110974 x 117
MSN:ignacio_francisco_ramirez_paulino at hotmail.com
jabber: ignacio_ramirez at jabber.org
Skype: ignacio_ramirez_paulino
--------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://public.kitware.com/pipermail/insight-users/attachments/20070213/541eb430/attachment.html