[Insight-developers] IJ Volunteers : THE MATRIX : RED PILL
Luis Ibanez
luis.ibanez at kitware.com
Sun Sep 17 18:26:10 EDT 2006
Hi Zach,
I have to admit being trapped in the chicken-egg fallacy of my own
argument.
Your are right, there is indeed a group of developers who steers
the direction of ITK.
These developers, however, are aware that it is not up to them to
setup the direction of the toolkit. That such decisions must be
driven by the needs and interests of the community at large.
The challenge that this group of developers faced two years ago was:
How to transfer that steering power to the community?
...with the goal of bringing ITK to become a self-sustained system.
Setting up the IJ system was an attempt to bootstrap that process of
power transfer to happen. Here is how the wild dream was supposed to
unfold:
1) Create an electronic Online Open resource
2) Give everybody access to Publish and to Review contributions
3) Empower contributors with a community resource where they
can post their source code, data, parameters, experiences.
4) Let the community decide what contributions were good enough
to go permanently into the toolkit, via public reproducible
reviews of that material.
5) Grow the group of developers to include contributors of
source code.
In that scenario, the steering committee was supposed to do less and
less, and let the community do more and more. The older developers
were supposed to just make sure that the wheels of the system were
turning, and providing support for new developers that were not
familiar with all the details of the software.
What we have learned, however, is that Freedom is an privilege that
not everybody wants to exercise. That most people are still more
comfortable with letting others make decisions for them. That they
are still used to higher authorities doing the thinking for them.
The avalanche of user-provided reviews
that we were expecting...
have not happened.
Why ?
...It is hard to tell at this point...
Maybe because we didn't provided enough education and motivation
to the users to convince them of the advantages of exercising
their right to make decision on the future of the toolkit.
Maybe because the real meaning of peer-review is still contaminated
with the decadence of a structure that today is only used for
deciding if an academic should be hired, funded, promoted or fired.
At at the end, nobody really cares about the content of what is
being published. whether it works or not, whether it is true or not,
as long as the so necessary credits are admonished to the authors by
the review process.
Maybe because the concept of "Publishing" never really conveyed
the idea that the purpose is to share results with the community.
And most users are still addicted to submitting to reputation-based
journals in order to get the academic credits their need for their
annual reports.
Maybe it was a mistake to attempt to solve two different problem:
a) The need for an honest peer-review publication system
b) The need for adopting contributions to the toolkit
With a single answer:
and online Open Access, Open Review publication system
The fact that we are going through all this discussion among
the group that was supposed to have a clear idea of how the
system was intended to work is an unsettling evidence that
this probably wasn't such a good idea after all.
On the bright side, we still have the comforting option that we
can go back to reading IEEE-TMI the papers that were submitted two
years ago, reviewed anonymously without any attempt for verifying
reproducibility, and we can go back to writing code that reimplements
algorithms by reverse engineering a verbal description scattered
through papers. Enjoy the detective work of fishing for those untold
values of parameters that make the difference between success and
failure when running an algorithm.
The fact that Biologist were ready to embrace Open Access and
online publishing doesn't mean that computer scientists were
ready for living up to the same standards.
Luis
========================
Zachary Pincus wrote:
> I don't think anyone is suggesting that there is a small circle of
> "all-powerful developers". However, there is a circle of developers,
> who, in an admirably open fashion, minister to ITK. This circle is
> overseen by the ITK Oversight Committee ( http://www.itk.org/Wiki/
> ITK_Oversight_Committee ).
>
> It was some subset of these developers and the oversight committee who
> decided upon (again in an open fashion) the precise algorithm for
> transferring work from the IJ to ITK, and the exact parameter settings
> for that algorithm. (Of which there are many! What's the threshold of
> review-goodness for accepting a work? How many reviews are needed? Are
> works from old "editions" of the IJ re-visited?)
>
> Thus, when Gaëtan suggested a change to some of the algorithm settings,
> it is reasonable and natural that that suggestion be sent to the
> itk-dev list, where, by and large, the people who decided upon the
> algorithm and settings (in an open and democratic fashion) and may
> continue to refine them, can be reached. Before today's emails, I would
> have presumed that the open nature of ITK would encourage suggestions
> of this sort.
>
> Again, to review:
> There is a problem in getting work from the IJ to ITK via the algorithm
> sketched below. This can be addressed by:
> (a) addressing at the algorithm. In which case, addressing the people
> involved in setting and changing that algorithm is appropriate.
> (b) addressing the IJ. In which case, "complaining" to the itk-dev list
> is not appropriate, but volunteering to do reviews, to be an associate
> editor, and many other such things *is* appropriate.
>
> Gaëtan and I think that perhaps (a) *and* (b) need to be addressed. And
> I find it wholly inappropriate to castigate him, repeatedly and
> publicly, about failing to do the right things, regarding (b), when in
> fact his concern was regarding (a) to begin with! (And not to mention
> that few people have been doing more, (b)-wise, than he.)
>
> Please do not conflate issues (a) and (b). I'll chalk this
> misunderstanding up to the nature of email and language barriers.
>
> Let me make some suggestions, regarding these separate issues:
>
> Regarding (a):
> - Perhaps three reviews is too many to require? A single review from a
> highly-trusted source might be enough in some cases.
> - Perhaps in cases when repeatedly-requested or otherwise desired
> functionality for ITK is in the IJ, reviews could be directly solicited.
> - Perhaps during the quarterly review sessions, submissions that appear
> to provide desired functionality but which lack reviews could be noted
> and reviews requested. (Similar to above suggestion.)
> - I don't know if submissions from "old editions" of the IJ are ever
> re-visited. Perhaps they should be.
>
> Regarding (b):
> - I'm not sure what the point of the different "editions" of the
> journal is. Why should time-of-submission matter at all? It seems to
> serve to bury older work.
>
> Regarding (a) and (b) together:
> - Perhaps there isn't sufficient motivation for people like me to write
> reviews? Maybe no submissions should be accepted in the ITK unless the
> submitter has made three reviews of his or her own? This might not
> provide the best kind of motivation (!), but it seems that somehow
> making good reviews needs to be better-incentivized.
> - I'd like to see more of the submissions put into the ITK distribution
> somewhere under the "review" directory. Maybe if (1) the submitter
> signs over the copyright, and (2) the buildbot can build it, the
> submission could be deposited there.
>
> You will note that most of these suggestions have to do with procedure
> for ITK, or for the interaction of ITK and the IJ. Issues which might
> best be discussed and resolved by the denizens of the itk- dev list, and
> especially those on the oversight committee and the cast of characters
> who makes up the regular conference-call attendees.
>
>
> Zach
>
>
>
>
> On Sep 16, 2006, at 5:39 PM, Luis Ibanez wrote:
>
>>
>>
>> Hi Gaëtan, Zach
>>
>>
>> It seems that the fundamental misperception still remains
>> in this discussion:
>>
>>
>>
>> The notion that there is a small circle of
>> all-powerful "developers" who decides the
>> destiny of the toolkit.
>>
>>
>>
>> That shows that the process for adding new code into ITK
>> has not been communicated properly to the community.
>>
>>
>>
>>
>> The
>>
>> "...people in ITK developers who are choosing
>> how to get code contributions in the toolkit..."
>>
>>
>>
>> is simply the following *Algorithm*:
>>
>>
>> 1) Sort the contributions on the Insight Journal
>> based on their ratings as provided by reviews.
>> Reviews that are public and that can be posted
>> by anybody.
>>
>>
>> 2) Request copyright transfers of the source code
>> to the Insight Software Consortium.
>>
>>
>>
>>
>> This means that the "readers/reviewers/raters" of papers
>> in the Insight Journal are *the ones who decide* what
>> gets included in the toolkit.
>>
>>
>>
>> There is nothing in this process that is closed or that
>> excludes anybody.
>>
>>
>> There are no different grades of "developers".
>> There is no "root" user in ITK.
>>
>>
>>
>>
>> The fact that even though you have write access to the
>> CVS repository, you will:
>>
>>
>> "...not take the freedom to put in the
>> toolkit all the nice work we can
>> find in the IJ, even if cvs would
>> let me do that..."
>>
>>
>> Simply means that you are as responsible as all other ITK
>> developers. We all share that same feeling and that same
>> responsibility. And curiously enough, is this careful
>> consideration and respect for the toolkit, one of the
>> elements that makes so slow the process of integrating
>> new code into the toolkit.
>>
>> In the same way that you will not dare to include code out of
>> your own initiative, anyone of us older developers, will not
>> dare either to make arbitrary unilateral decisions on what to
>> include in the toolkit.
>>
>>
>>
>> It is not the fact that some "privileged group" of "developers"
>> are the only ones that make arbitrary decisions on what to include
>> in the toolkit. Instead, it is the fact that in order to include
>> a piece of code into a toolkit that is used by a more than 2,000
>> people in more than 40 different countries, and that should be
>> maintained working for the next ten or so years, we want to get
>> *at least* the opinions of *three* people by rating a contribution
>> submitted to the Insight Journal.
>>
>>
>> It is simply an *Open Process*
>> that requires a *Participatory Community*.
>>
>>
>>
>> *EVERYTHING* about ITK is Open. Every document, from every
>> meeting since 1999 is available on the Web. The agendas and
>> minutes from every weekly conference are posted on the Wiki.
>> The plans for releases, the bugs, the strengths and weaknesses
>> of the toolkit. Everything is openly and publicly discussed.
>>
>>
>> I would be surprised to find an Open Source project that has
>> a more public and open process than ITK.
>>
>>
>>
>>
>>
>>
>> That the process of moving source code
>> from the Insight Journal to the Toolkit
>> is slow....?
>>
>>
>> YES,
>>
>>
>> You are right. It is painfully slow.
>>
>>
>>
>> It was not supposed to be slow.
>> It was not supposed to be painful.
>>
>>
>>
>> Why is it slow?
>> and how it could be improved ?
>>
>>
>>
>> That's the *positive* discussion
>> that we should be holding.
>>
>>
>>
>>
>> Luis
>>
>>
>>
>>
>> ======================
>> Gaëtan Lehmann wrote:
>>
>>> Hi Luis,
>>> You're wrong, I'm not more a developer than before, and the write
>>> access to ITK cvs is just shortening the time needed to fix some
>>> bugs, but it doesn't make any difference regarding the code
>>> contribution process. I will not take the freedom to put in the
>>> toolkit all the nice work we can find in the IJ, even if cvs would
>>> let me do that. That's something only you can do. "The term
>>> developers" was surely badly chosen - please replace it with
>>> something better, like "people in ITK developers who are choosing
>>> how to get code contibutions in the toolkit".
>>> For me, open access and peer review are an evidence: they are the
>>> future of science publication. In the mean time, ITK is loosing a
>>> part of its active community.
>>> The red pill can be difficult to take if too few people are taking
>>> it at the same time
>>> Gaetan
>>> Le Sat, 16 Sep 2006 21:06:53 +0200, Luis Ibanez
>>> <luis.ibanez at kitware.com> a écrit:
>>>
>>>>
>>>>
>>>> Hi Gaetan,
>>>>
>>>>
>>>> I would suggest that you approach this situation with a more positive
>>>> attitude.
>>>>
>>>>
>>>> You are referring to the IJ the same way a tax-payer will complain
>>>> about the government.
>>>>
>>>>
>>>>
>>>> Complaining and whining is not an effective strategy when you are
>>>> trying
>>>> to change the status quo of a community that has been mushy- minded by
>>>> decades of "Publish or Perish" propaganda.
>>>>
>>>>
>>>>
>>>> The Insight Journal *is not* an Institution,
>>>> The Insight Journal *is* a Community Resource.
>>>>
>>>>
>>>>
>>>> There is no such a thing as "The Owners" of the Insight Journal.
>>>>
>>>>
>>>>
>>>> If you want to define such concept, probably the closest thing to the
>>>> "Owners" of the Journal are the 1,200 subscribers to the ITK users
>>>> list.
>>>>
>>>>
>>>>
>>>> This is a Journal intended to be supported and managed by the
>>>> community.
>>>> This is why it is poised to be a real *peer-review* Journal.
>>>>
>>>>
>>>>
>>>> The concept of "Freedom" is too new for our community, because the
>>>> traditional peer-review process treats authors and readers as children
>>>> who are supervised by the "adult" reviewers. The "adult" reviewers are
>>>> the ones who decide what is good for the children to read. Such roles,
>>>> atrophied critical thinking in the readers and lead them to passively
>>>> accept any published paper as "absolute truth", because it has passed
>>>> through the supposedly "holy" process of review by the "adults".
>>>>
>>>>
>>>>
>>>> Members of our community read technical journals with the same level
>>>> of critical thinking that a watcher of TV-reality-shows exercise while
>>>> sitting empty-minded for hours in front of the TV-set. The concept
>>>> that
>>>> they are actually entitled to question the content of papers is too
>>>> alien for them. The notion that a published paper may contain
>>>> mistakes,
>>>> is a sacrilege for them. The notion that the claims made in a paper
>>>> are supposed to be verified, is a blasphemy to them.
>>>>
>>>>
>>>>
>>>> After all,
>>>>
>>>> "Who are readers to question the judgment of the holy reviewers" ?
>>>>
>>>>
>>>>
>>>> As a consequence our community has developed the laziness of the kid
>>>> that doesn't do his homework until the parents tell them to do so.
>>>>
>>>>
>>>> The concept that *everybody* is entitled to write a review, is too new
>>>> for our community. Most readers and authors assume that "reviewing" is
>>>> an activity exclusively reserved for some "supernatural" beings who
>>>> are
>>>> the "Chosen Ones", the "Holy Reviewers".
>>>>
>>>>
>>>> Authors are still trained to submit papers to Journals and "Pray" for
>>>> the papers to be accepted by the "Chosen Ones".
>>>>
>>>>
>>>> It takes a lot of education, motivation and *repetition* to
>>>> rehabilitate
>>>> readers and authors from the damage that the "Publish or Perish"
>>>> disease
>>>> has inflicted on their minds.
>>>>
>>>>
>>>>
>>>> In the meantime,
>>>> You are discharging your frustration on the wrong crowd.
>>>>
>>>>
>>>> Instead of complaining about the management of the Journal and its
>>>> supposed "Owners", you should brainstorm on ways of encouraging all
>>>> under-graduate and graduate students to act as reviewers. They should
>>>> learn that they *ARE* qualified to be reviewers. They should learn
>>>> that
>>>> they *are* the "PEERS" that the term "Peer-Review" refers to:
>>>>
>>>>
>>>> It is not "Supervisor-Review"
>>>> It is not "Nobel-Prize-Review"
>>>> It is not "Chosen-One-Review"
>>>> It is not "Friend-of-The-Journal-Editor-Review"
>>>> It is not "Owner-of-The-Journal-Review"
>>>> It is not "Anonymous-Competitor-Review"
>>>>
>>>>
>>>> It is "PEER-Review"
>>>>
>>>>
>>>>
>>>> Definition of "PEER":
>>>>
>>>> One that is of equal standing with another :
>>>> EQUAL; one belonging to the same societal group
>>>> especially based on *age*, *grade*, or *status*
>>>>
>>>>
>>>>
>>>> The "PEERS" of Graduate Students are *OTHER GRADUATE STUDENTS*.
>>>>
>>>>
>>>>
>>>> *EVERY* person that uses ITK *IS* qualified to act as a reviewer
>>>> of the Insight Journal.
>>>>
>>>>
>>>>
>>>> There is no need for being "Friend" of anybody.
>>>> There is no need for being consecrated through some "Holy Process".
>>>> There is no need for belonging to some influence group.
>>>> There is no need for being member of any social circle.
>>>> There should not be a need to be pressured by an associate editor.
>>>>
>>>>
>>>> That is the message that we must pass across the community.
>>>>
>>>> Having 1,200 users in the mailing list, there is no shortage of
>>>> brains for reviewing less than 100 papers.
>>>>
>>>>
>>>>
>>>> Do not make the mistake of thinking
>>>> that the Insight Journal is simply the
>>>> gate for bringing source code into ITK.
>>>>
>>>>
>>>>
>>>> The Insight Journal is the RED PILL that may help our community
>>>> to wake up from THE MATRIX of the "Publish or Perish" stratagem.
>>>>
>>>>
>>>> As a community resource, the *Exercise of participating* in the review
>>>> process is *MORE* important than the act of bringing code into ITK.
>>>>
>>>>
>>>> It is a simple matter of teaching readers to stand on their own feet,
>>>> to judge with their own minds, and to take responsibility for their
>>>> own future.
>>>>
>>>>
>>>>
>>>> "Publish or Perish" was invented by budget managers and administrators
>>>> who didn't wanted, or were incapable of understanding the science they
>>>> were managing. It was an easy trick that made possible for them to
>>>> "count numbers of papers" in an annual report instead of having to
>>>> "read those papers" and try to understand their implications.
>>>>
>>>>
>>>>
>>>> We as a community, are all guilty of nourishing this primitive and
>>>> decadent practice. We still assume that "number of papers" equals
>>>> "productivity", regardless of what the paper content is.
>>>>
>>>>
>>>>
>>>> As dogs in training, the "Publish or Perish" stratagem, educated our
>>>> community members to receive cookies (productivity credits) when they
>>>> bark (publish), no matter what the barking was about. In the "Publish
>>>> or Perish" MATRIX, there are no cookies for the ones who listen to the
>>>> barking (read the publications). There are no cookies for the ones who
>>>> attempt to repeat the barking of others (reproduce publications). Only
>>>> "original barking" is supposed to be worth of cookies.
>>>>
>>>>
>>>>
>>>> The Insight Journal is like a public library. It is there, open for
>>>> everybody. However, we can not force people to go to this library,
>>>> we can not force people to contribute reviews.
>>>>
>>>>
>>>> The challenge we face at this point is to untrain the damage that
>>>> years
>>>> of traditional publishing system has made. Every time that we hear a
>>>> senior researcher telling a junior researcher "You should publish" we
>>>> must jump in alert and examine "WHY" and "HOW" publishing should be
>>>> done.
>>>>
>>>>
>>>>
>>>>
>>>> You are now *an ITK developer*, at this point, you have access to all
>>>> the same resources and information that any other "ITK developer" has.
>>>> You have write access to the CVS repository, access to the mailing
>>>> lists, access to the Wiki, access to the bug tracker. As you can see
>>>> there is no magic aura that suddenly appears, no sparkly dust falling
>>>> down while you walk. It is just one more entry among many other things
>>>> in your to-do list.
>>>>
>>>>
>>>> You can complain about other "ITK developers" not doing enough, but
>>>> that's not going to make the world move any faster. The place where
>>>> you could invest your energy in a positive way is by going to all
>>>> junior researchers around you and letting them know that "Publish or
>>>> Perish" is a lie, and that if they don't take responsibility for the
>>>> quality of work shared in their community, nobody else will. At least
>>>> nobody without an agenda of their own...
>>>>
>>>>
>>>>
>>>>
>>>> Luis
>>>>
>>>>
>>>>
>>>> =======================
>>>> Gaëtan Lehmann wrote:
>>>>
>>>>> Stephen,
>>>>> Have I said or done something wrong ?
>>>>> I'm writting a review for each new paper I'm able to review -
>>>>> perhaps I'm late of 1 review or 2 - but I'll not submit 3
>>>>> reviews per paper, and will not submit reviews for my own ones.
>>>>> I'm asking about reviewer associate because their role is to
>>>>> assign some reviewers to a paper, and most of the papers in the
>>>>> IJ are still lacking of reviews - Note that I'm only talking
>>>>> about the contributions to ITK, not about the articles for the
>>>>> MICCAI. I would be pleased to incite potential reviewers to
>>>>> write more reviews, but again, I have seen nothing about the
>>>>> editor associate since your last mail.
>>>>> Alsso, it seem that the ITK developers are not getting that
>>>>> letting a contribution for 6 or 9 months in the insight journal
>>>>> without much activity is like saying
>>>>> "we don't care about your work and the time you spent to write
>>>>> your article".
>>>>> If there can't be more reviewers for the contributions to the
>>>>> insight journal, do not require 3 reviews to integrate it in the
>>>>> toolkit, are stop asking people to post their code to the IJ. If
>>>>> you (developers) really don't care, just keep things like that -
>>>>> that's fine.
>>>>> Some days like this one, I'm really not sure why I'm still using
>>>>> the IJ, and warning their owners about what is wrong - it also
>>>>> look like a huge waste of time
>>>>> Gaetan
>>>>> Le Sat, 16 Sep 2006 01:52:17 +0200, Stephen R. Aylward
>>>>> <Stephen.Aylward at kitware.com> a écrit:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Thanks for volunteering to provide more reviews! There are
>>>>>> instructions on the wiki on how to write reviews and contribute
>>>>>> to the IJ.
>>>>>> As each new paper arrives - I am assigning it to reviewers, but
>>>>>> we always welcome additional reviews.
>>>>>>
>>>>>> Thanks,
>>>>>> Stephen
>>>>>>
>>>>>> ___
>>>>>> Sent with SnapperMail
>>>>>> www.snappermail.com
>>>>>>
>>>>>> ..... Original Message .......
>>>>>> On Fri, 15 Sep 2006 22:34:56 +0200 Gaëtan Lehmann
>>>>>> <gaetan.lehmann at jouy.inra.fr> wrote:
>>>>>>
>>>>>>> Le Mon, 21 Aug 2006 18:19:37 +0200, Stephen R. Aylward
>>>>>>> <Stephen.Aylward at Kitware.com> a écrit:
>>>>>>>
>>>>>>>> Thanks for all who have already volunteered to be Associate
>>>>>>>> Editors for
>>>>>>>> the Insight Journal!
>>>>>>>> http://insightsoftwareconsortium.org/wiki/index.php/IJ-Whos-Who
>>>>>>>>
>>>>>>>> We are still looking for a few more volunteers. You've
>>>>>>>> contributed
>>>>>>>> code to open-source medical image analysis...why not help to
>>>>>>>> spread the
>>>>>>>> news of your contribution and the contributions of others...
>>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> What is the status of this initiative ?
>>>>>>> I've seen nothing about it since this mail, and there is lots of
>>>>>>> contributions in the IJ which again will not get their 3
>>>>>>> reviews before
>>>>>>> the sept 20 (http://www.itk.org/Wiki/ITK_Release_Schedule). If
>>>>>>> things stay
>>>>>>> like that, some of them will be more than one year old without
>>>>>>> getting
>>>>>>> their 3 reviews.
>>>>>>> If things stay like that, lots of contributors will be discouraged
>>>>>>>
>>>>>>> Believe me, I'm a contributor
>>>>>>>
>>>>>>> And what a waste of time.
>>>>>>> If things have been done earlier, some classes can have been
>>>>>>> integrated in
>>>>>>> the toolkit 2 releases ago
>>>>>>>
>>>>>>> Gaetan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- Gaëtan Lehmann
>>>>>>> Biologie du Développement et de la Reproduction
>>>>>>> INRA de Jouy-en-Josas (France)
>>>>>>> tel: +33 1 34 65 29 66 fax: 01 34 65 29 09
>>>>>>> http://voxel.jouy.inra.fr
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Insight-developers mailing list
>> Insight-developers at itk.org
>> http://www.itk.org/mailman/listinfo/insight-developers
>
>
> _______________________________________________
> Insight-developers mailing list
> Insight-developers at itk.org
> http://www.itk.org/mailman/listinfo/insight-developers
>
>
More information about the Insight-developers
mailing list