After five years of providing critical reviews of education-related reports by nonacademic think tanks, education professors Alex Molnar and Kevin G. Welner hope to expand their own reach with a new, broader research center.
The new National Education Policy Center, based at Mr. Welner’s academic home, the University of Colorado at Boulder, will consolidate his Education and the Public Interest Center and Mr. Molnar’s Education Policy Research Unit, previously at Arizona State University. It will review existing research, conduct new research, and, for the first time for both groups, make policy recommendations.
It will also house the Education Review, NEPC senior researcher Gene V. Glass’ education book-review journal; a new international journal, Education Research Global Observatory; and a new Center on Diversity, Equity, and Learning, for which the NEPC has won a $1 million grant from the Ford Foundation, of New York City, to study racial-equity issues in education.
Read a related story about the National Education Policy Center’s first major effort, “Book Critiques Research Base for Obama’s ESEA Blueprint.”
“Five years from now, I hope and expect that policymakers will look to us before they make policy decisions based on the evidence that’s been put in their hands by interested parties,” said Mr. Molnar, the publications director for the NEPC. “We want to move the discussion in education policy toward valuing high-quality research and incorporating the research into policy formation.”
Mr. Molnar and Mr. Welner, the new center’s director, already are well known in the academic community for the centers they’ve been running, which study the effects of private and for-profit influences on education and of policy interventions, such as charters and other forms of school choice.
Their highest-profile venture, though, is the Think Tank Review Project, launched in 2006 with the aim of countering what its founders saw as “a hodgepodge of private ‘think tanks’ at both the state and national levels [that] wield significant and very often undeserved influence in policy discussions by cranking out an array of well-funded and slickly produced—yet ideologically driven—research.” Possibly the only effort of its kind, the project enlists scholars to write quick-turnaround critiques of attention-grabbing studies published by prominent think tanks.
The launch of the new center, coinciding with its publication of a book collecting 21 of the review project’s 59 critiques of think tank research, renews the debate over the quality of education research in both the think tank and academic sectors—as well as questions about the objectivity of the review project itself.
“I think it’s a major contribution to the field because these reports come out and the media just report the findings as though it was truthful, so we need someone to go back and review the material,” said Gerald E. Sroufe, the director of government relations for the American Educational Research Association. “They have devoted such passion to reviewing the conservative [reports], and it would be great if they could bring equal passion to the array of reports across the board.”
‘We Aren’t Referees’
Some critics argue that Mr. Molnar, Mr. Welnar, and the 100 fellows affiliated with the project disproportionately critique conservative and “free market” institutions’ research, and are less upfront than advocacy organizations about their funding.
“It’s a free country; it’s fine for them to look at research produced by think tanks that hold other views and try to critique them,” said Frederick M. Hess, the director of education policy studies for the right-leaning American Enterprise Institute, in Washington, and the author of a blog for Education Week‘s website. “It’s only problematic when they try to pass themselves off as objective, even-handed arbiters of good research.”
Mr. Welner and Mr. Molnar say, however, they have focused mostly on conservative think tanks’ research because it has been the most prolific and influential in education policy. The project has also reviewed work from moderate organizations such as the Brookings Institution and the Center on Education Policy, both based in Washington.
“To criticize us for overwhelmingly reviewing reports from free-market think tanks, it’s accurate, but I don’t see it as any sort of substantive critique of what we’re doing,” Mr. Welner said.
“I’m baffled in an annoyed sort of way at the charge [of bias],” he said. “There’s this idea that we aren’t good referees, and I would agree, we aren’t referees.”
“We very much believe,” he added, “the outcome of the game should be influenced by at least two things: universal opportunities for all students and policies being guided by high-quality research. We don’t claim to be referees for that reason.”
Most experts agree that studies produced by nonacademic brain trusts, regardless of their political, policy, or philosophical orientation, are often agenda-driven.
In the book Mr. Molnar and Mr. Welner edited, Think Tank Research Quality: Lessons for Policy Makers, the Media, and the Public, they argue that think tank studies should undergo the same blind-expert peer review used by academic researchers, which would reduce the common flaws their reviewers noted. Those include: failing to identify and correct selection bias, confusing correlations with causation, conducting selective literature reviews, and overstating conclusions based on the data.
For example, in one chapter, the editors note that reports on privatization in schools included helpful information for administrators on understanding outside contractors, but did not provide sufficient data on how those services compare with in-district services, which the editors said led to later misunderstandings by policymakers and the public.
Washington think tankers, from Mr. Hess of the AEI to Jack Jennings, the founder of the CEP, and Kevin Carey, the policy director for the center-left think tank Education Sector, said the Think Tank Review Project’s analysis has been a mix of “valid observations” and “conclusions flawed to the point of being nonsensical.”
All three argue that blind reviews aren’t necessarily the best solution. Regardless of whether research undergoes blind, academic peer review or open review by contracted experts, research quality varies among studies produced by academic institutions and think tanks alike, they say.
“A lot of these issues are very much subject to reasonable debate and are not simply a question of whether or not established rules and norms of research have and have not been followed,” Mr. Carey added.
Expecting Scrutiny
Mr. Jennings, whose organization issued a 2007 study on student achievement that is critiqued in the book, said that while he “understand[s]” what the Think Tank Review project is doing and thinks it has “the right idea,” he disagreed with the review of the study and with the NEPC’s allegation that think tanks’ studies undergo less stringent review than academic papers.
“When anybody in the think tank community issues a report, they know it’s going to be scrutinized,” and take pains to find expert reviewers, said Mr. Jennings, a former education aide to congressional Democrats. The flaws cited by think tank critics are typical of much education research, he added. “You’d think it ought to be the heyday of education research, but it isn’t,” Mr. Jennings said, “because there isn’t enough of it, and it isn’t of good enough quality to be translated into the field of practice.”
Neal McCluskey, the associate director of the libertarian Center for Educational Freedom at the Cato Institute, in Washington, said the book’s review of his own report on the future of the No Child Left Behind Act accused him of “unsupported advocacy of school choice policies” without acknowledging supporting studies in the report and footnotes.
“Everybody benefits from critical analysis of what they are doing,” Mr. McCluskey said. “To the extent [a Think Tank Review Project report] provides substantive critiques, it’s valuable, but the problem is when they twist the research and demonize think tanks.”
Argument Resonates
The NEPC’s argument that think tank studies can promote lower-quality research to policymakers resonated with Diane Ravitch, a research professor of education at New York University and a senior fellow at the Brookings Institution, a Washington think tank with a centrist reputation.
“What public-policy makers are doing now is just cherry-picking,” said Ms. Ravitch, a former federal education research official under President George H.W. Bush who often writes critically about education studies in a blog she co-authors for edweek.org. “They know what they want to do, and they look for the research that supports it. If they can’t find research, then they go to think tank reports.”
One reason that the NEPC’s incorporated centers draw criticism from other think tankers is that they receive funding from the Great Lakes Center for Education Research and Practice, an East Lansing, Mich.-based group founded by the Michigan Education Association, an affiliate of the National Education Association. The funding relationship is seen as ironic because Think Tank Review Project reviewers often highlight ties between prominent think tanks and free-market-oriented groups such as the Foundation for Educational Choice, formerly the Milton and Rose D. Friedman Foundation.
Mr. Welner laughed at the idea that unions might direct center research. “You can imagine the reaction we’d get if they tried,” he said. “Our careers don’t depend on this project; if anything, our careers have probably been held back by this project, so we wouldn’t hesitate to leave the project behind if we didn’t have that freedom.”
Ms. Ravitch agreed that there will always be a tension between the reports issued by think tanks and academia.
“The problem is think tanks tend to have a point of view; that’s not research, it’s a think tank report, and there’s a distinction,” she said.
Mr. Molnar is undeterred by criticism and said the center has already applied its review process to an even bigger project: the Obama administration’s education reform proposals. Mr. Molnar and Mr. Welner will hold a panel on recommendations from the think tank project at the Economic Policy Institute on Sept. 29. (“Scholars Dissect Research Behind ESEA Blueprint,” this issue.)
“We intend to continue producing high-quality research and help policymakers understand when research is not high-quality.”
