Researchers from two of the nation’s most eminent universities, Harvard and Stanford, recently unveiled studies on different, but equally controversial, education issues. One was on school accountability policy, the other on teacher certification, but critics’ responses were the same.
They complained that the studies were “not quite ready for prime time” because they had not undergone the independent scrutiny that usually comes when studies are published in academic journals.
It’s a refrain that has become common lately as research on hot-button issues moves directly from the author’s computer printer to a press release. The criticism has some scholars wondering whether education researchers need to do a better job of policing themselves.
“I think there have been abuses on all sides,” said James. G. Cibulka, the dean of the college of education at the University of Kentucky, in Lexington. “As a profession, we really need to have some agreed-upon criteria for vetting studies before they make their way to the public.”
Whether the buzz about the need for giving studies some level of professional review before they hit the public arena is new, or even limited to education research, is an open question.
Studies generally undergo an evaluation process before they are accepted for publication in an academic journal, presented at a conference, or released by a think tank.
Peer review for articles published in a professional journal typically involves three or more reviewers who critique an author’s work in writing. The reviewers are “blind,” meaning that they are chosen by the journal editor and that the author does not know their identities.
Peer review for presentations at a professional conference is a less-rigorous procedure. Reviewers often accept or reject a proposed presentation on the basis of an abstract or a three- or four-page paper.
Peer review at a nonpartisan think tank, such as the New York City-based MDRC or the Washington-based American Institutes for Research, takes place mostly in-house as think tank researchers review one another’s work. In the later stages of its three-tiered review process, MDRC also enlists board members, outside experts in the topic under study, and participants from the study sites. For the Manhattan Institute, a think tank widely seen to support conservative perspectives, researcher Jay P. Greene said he typically circulates his papers to several outside colleagues before releasing them to the public.
“One of the traditions of the education research enterprise has been the high frequency of advocacy research,” said Grover J. “Russ” Whitehurst, the director of the U.S. Department of Education’s Institute of Education Sciences. “I would be skeptical that there’s been an uptick on this.”
Others disagree. They say the debate over the need for tighter peer-review mechanisms sharpened last summer after the American Federation of Teachers produced an unreviewed study suggesting that some students in public elementary schools outperformed their counterparts in the same grades in charter schools on national tests of reading and mathematics. (“AFT Charter School Study Sparks Heated National Debate,” Sept. 1, 2004.)
When the AFT study’s findings became the subject of the lead story in The New York Times on Aug. 17, supporters of charter schools mounted a counterattack that included editorials, letters to the editor, and a full-page advertisement in national newspapers. The ad, signed by 31 academics, mostly criticized the study on methodological grounds, but it also called for closer scrutiny of research before findings percolate into the mainstream media. It said that “such studies need to be vetted by independent scholars, as is commonly done in coverage of research on the biological and physical sciences.”
Since then, what has irked some other researchers is that three of the signers went on to do just what the teachers’ union had done: They took research findings to the press before the studies had undergone a formal peer-review process. That drew charges of hypocrisy from some of their research colleagues.
Jay P. Greene, one of the signers who was criticized, said he saw the ad as more of an admonition to journalists than to researchers.
“The ad never said all studies have to be peer-reviewed before people talk about them in public,” said Mr. Greene, a senior research fellow who works at the Davie, Fla., branch of the New York City-based Manhattan Institute. “I don’t think that’s a reasonable standard.”
Another signer, David N. Figlio, an economist at the University of Florida, in Gainesville, said he viewed the ad similarly. He was not one of the researchers who later publicized unreviewed work.
Mr. Figlio said he signed the original ad to protest the prominent play the Times had given to research that had never been professionally vetted. When the same ad appeared in Education Week, he said, he removed his name because he thought its coverage of the study had been more balanced.
“I feel strongly no unrefereed study should get front-page, above-the-fold treatment,” he said.
Setting Clear Standards
But Mr. Cibulka of the University of Kentucky, who also signed the ad, says the responsibility belongs to researchers from the start. “The problem is not with journalists,” he said. “The problem is with the profession for having failed to promulgate some clear standards.”
Mr. Cibulka said a trend toward making premature research findings public began as education became more prominent in federal and state policymaking, and as think tanks sprang up to try to influence those policymaking decisions.
David C. Berliner, an education professor at Arizona State University, in Tempe, added that “everybody has to have news all the time, and you can get stuff out there if you can tap into that.”
Mr. Greene said that one reason he sends his reports directly to the news media is that it’s quicker than publishing in traditional journals, which can take up to two years.
“In that time, people who are not researchers are commenting all the time on the merits of certain policies, and they’re doing so in the absence of evidence,” he said. “If we say that researchers can’t comment with evidence until that research has been peer-reviewed, ... we’ll actually have worse public-policy decisionmaking.”
Such practices, in comparison, would be taboo in biomedical science, according to Joann Rodgers, the director of media relations for Johns Hopkins medical school. She said her office won’t issue a press release on a researcher’s study unless it’s been accepted for publication in a peer-reviewed journal. She said medical school researchers respect those guidelines because the penalty for violating them is heavy. Journals such as Nature, The New England Journal of Medicine, and Science won’t accept a study if its findings have appeared elsewhere first, she said.
Peer-Review Critique
Even so, many researchers say peer review is never a guarantee of quality in any field.
“Not only do some lower-quality studies get published when the peer-review process is not rigorous enough, some important and high-quality studies may not get published because there is not a suitable outlet for their topic, methodology, or length,” Linda Darling-Hammond, the Stanford University education professor whose teacher-certification study was criticized last month, said in an e-mail interview.
“So any discussion of barriers to making research results public raises the possibility of staunching the free flow of ideas that is necessary for knowledge to move forward,” she added in her e-mail.
Ms. Darling-Hammond issued a press release on her study after circulating it among colleagues and then presenting it in Montreal last month at the annual meeting of the American Educational Research Association. (“Study Sees Positive Effects of Teacher Certification,” April 27, 2005.)
The Harvard University researchers who were criticized last month, Paul E. Peterson and Martin R. West, followed the same procedure when they presented their school accountability study at a British economics conference.
Felice J. Levine, the executive director of the 22,000-member, Washington-based AERA, said it’s important for researchers to be able to present their findings at public professional conferences because that’s one way to gather feedback from colleagues on how to improve the work.
Conference presentations do undergo some review in order to land a spot on the AERA agenda, she pointed out. But it’s not quite the same as undergoing a journal review, which typically involves a careful reading by at least three “blind” reviewers—colleagues, in other words, who have been chosen by the editor and whose identity is unknown to the author.
Perhaps what the profession needs, some education researchers said last week, is a review mechanism in the middle. One model, they said, might be the National Bureau of Education Research, which posts “working papers” on its Web site for colleagues’ reaction.
Mr. Cibulka suggested creating a national panel that could set standards for when to publicize research.
In retrospect, said Mr. Figlio of the University of Florida, “I think the charter school dust-up may have done us all a big favor in terms of thinking about how do we balance readiness and timeliness.”
