[Insight-users] NIH Weighs Big Changes in Peer Review

[Luis Ibanez]

News of the Week
RESEARCH POLICY:
NIH Weighs Big Changes in Peer Review
Jennifer Couzin

Science 14 December 2007:
Vol. 318. no. 5857, pp. 1708 - 1709
DOI: 10.1126/science.318.5857.1708b


Peer review, a cornerstone of biomedical science, appears headed for an
overhaul, to judge by a sweeping examination unveiled at the National
Institutes of Health (NIH) last week. Since July, scientists have
flooded two working groups established by NIH Director Elias Zerhouni
with several thousand comments and ideas. This outpouring indicates that
the community is frustrated by the system's administrative burden and
deeply concerned about the fate of talented new investigators. Zerhouni
has promised quick action.

At a meeting of the advisory committee to the NIH director on NIH's
Bethesda, Maryland, campus last week, leaders of this review highlighted
the recommendations they may deliver to Zerhouni in February. No final
decisions have been made, however, and the committees are weighing
everything, including shortening grant applications to seven pages from
the current 25 pages and recommending an "editorial board" model that
would refer some grant proposals to outside experts.

Molecular biologist Keith Yamamoto of the University of California, San
Francisco, who also serves as co-chair of the external working group
that solicited comments from outside NIH, suggested ways to ease a
reviewing backlog. (Lawrence Tabak, director of the National Institute
of Dental and Craniofacial Research, co-chaired the internal group.)
Currently, Yamamoto noted, most applicants are permitted to resubmit
their proposal twice if it's rejected the first time around, which
happens most of the time. But the appeals, Zerhouni said at the meeting,
have created a "traffic jam" and a system that "penalizes the new
entrant to a very extreme degree."

Yamamoto thinks reviewers ought to assess applications first for their
scientific impact and, in cases that seem hopeless, communicate that
unequivocally to the applicant without allowing resubmissions. "Right
now, if an application is triaged"--left unscored--"many times it's
unclear what the reason is," said Yamamoto in a conversation after the
meeting. "Here, the goal is to say, 'Let's stop all that.' "

Streamlining applications--perhaps by vastly reducing the amount of
preliminary data that's included--is also a possibility, as is
eliminating the current scoring system and having reviewers rank only
the top 10 grant proposals that they consider. Some study sections, the
working groups believe, have too many members, having ballooned from the
usual 15 or 20 members to as many as 80, to accommodate the increasingly
specialized science being proposed. Sending applications containing
certain technical details to outside experts, who would consider those
elements alone and report back to the study section, is one way to slim
study sections down. Shorter grant proposals, meanwhile, could allow
each one to be evaluated by four people instead of the usual two.

Zerhouni and his advisory committee seemed enthusiastic, but several
members wondered if the proposed changes went far enough. "The biggest
[issue] on the minds of the people I talk to is getting the best people
to serve on study sections," said advisory committee member Thomas
Kelly, director of the Sloan-Kettering Institute in New York City. And,
he added, "I'm skeptical" whether the incentives proposed will be
sufficient to coax these people to serve. Bioengineer Annelise Barron of
Stanford University in Palo Alto, California, worried that slimming down
the applications might mean those from less prestigious universities
would not fare as well, because with shorter applications, name
recognition could carry more weight. The NIH system cannot allow such a
bias or must find a way to manage it, agreed David Botstein of Princeton
University, who's working with Yamamoto on reviewing peer review. Some
wondered whether blinding the names and affiliations of grantees would
be possible.

No matter what the working groups decide, it's critical that NIH retains
the scientists it's helped train and gives investigators "a sense of
commitment that's real," says pharmacologist and cardiologist Garret
FitzGerald of the University of Pennsylvania, an NIH adviser not
directly involved in this review. "Otherwise," as the average age of
first-time grantees continues to rise, says FitzGerald, "what rational
person would choose to go into a career where you begin to be
independent when you're 45?"