Includes updates and/or revisions.
The authors of a recent national study that found students in regular public schools outperforming their charter school peers are rebutting criticism that their research suffered from a “serious mathematical mistake” that negatively skewed the results.
In a memorandum posted online last week, researchers from the Center for Research on Education Outcomes, or CREDO, at Stanford University responded to criticism made last month by their Stanford colleague Caroline M. Hoxby. Ms. Hoxby, an economics professor, issued a memo critiquing the CREDO study in tandem with results from her own study of charter schools in New York City.
That study showed that charter schools in the city were having the opposite effect on their students’ achievement as the CREDO researchers found. Compared with their public school counterparts, Ms. Hoxby found, students in charter schools made more progress in closing achievement gaps with better-off peers in the suburbs. (“N.Y.C. Charters Found to Close Gaps,” Sept. 22, 2009.)
“She claims that we modeled charter schools in a particular way and that was inaccurate,” Margaret E. Raymond, CREDO’s director, said of Ms. Hoxby’s critique of the statistical analysis. “And her proof was based on her inaccurate specification.”
Ms. Raymond said Ms. Hoxby’s critique also failed to consider the many factors that the CREDO study took into account in its analysis.
Stakes High
“Imagine that we’re going to compare height and shoe size, and that we’re going to do it for all the people in the office,” Ms. Raymond said. “Those two things might actually correlate in a particular way because they’re only two factors. But if we’re going to predict shoe size based on many more factors, like age or parents’ height or parents’ shoe sizes, that’s a much more precise way of estimating.”
With charter schools a high-profile policy issue, researchers who came to opposite conclusions about their effect on student achievement are debating their analyses.
“She claims that we modeled charter schools in a particular way, and that was inaccurate. And her proof was based on her inaccurate speciication.”
Margaret E. Raymond
Director, Center for Research on Education Outcomes
Stanford University
“They assert that the proof proceeds in a way that it does not. Since their main rebuttal is based on misreading a simple proof, their rebuttal is wrong.”
Caroline M. Hoxby
Professor of Economics
Stanford University
The CREDO researchers also argue in their memorandum that Ms. Hoxby’s estimates of the magnitude of the bias in their results was faulty.
The researchers’ arguments failed, however, to persuade Ms. Hoxby. In an e-mail response to Education Week, she said CREDO’s analysis contained “logically incorrect assertions.”
“They assert that the proof proceeds in a way that it does not,” Ms. Hoxby wrote. “Since their main rebuttal is based on misreading a simple proof, their rebuttal is wrong.”
If the debate is taking on a bitter tone, it’s because the stakes are high. U.S. Secretary of Education Arne Duncan has been pressuring states to remove caps on charter schools in order to qualify for some of the $100 billion in economic-stimulus funds targeted to education. But opposition to the schools, which are public schools that operate largely free of the usual regulations, continues to come from teachers’ union officials and other education leaders.
“I don’t think the field of education research or policymakers are well served by scholars going back and forth with dueling memos, without peer review and without ample time to think it through,” said Sean F. Reardon, a colleague of both scholars at Stanford. “But I don’t think either side got it right,” he added.
An associate professor of education and sociology, Mr. Reardon is reviewing the New York City study. But he said he’s also not yet convinced that the CREDO study “does enough to ensure that it’s entirely free of bias.”
“Like any nonexperimental study,” he added, “it relies on some strong assumptions that are hard to justify.”
Part of the problem, he said, is that neither study provides enough statistical detail about the data and the methods used to help reviewers settle the debate.
Also, neither study has been published yet in a peer-reviewed journal. However, an earlier version of Ms. Hoxby’s report is posted as a working paper on the Web site of the National Bureau of Economic Research, which provides a forum for economists seeking feedback from the field on their work. Ms. Hoxby said her paper is also under consideration for publication by two journals.
The CREDO study was reviewed by four outside experts before its release this past June, according to a publicist. (“Study Casts Doubt on Charter School Results,” June 15, 2009)
The memo last week from CREDO suggests that it’s not surprising that the two studies should yield different results, given that a central finding of the CREDO study, which encompasses 15 states and the District of Columbia, is that charter school performance varies considerably from school to school.
“Some communities and states have gotten the policy right, and are able to demonstrate positive charter school results,” the CREDO researchers write. “The recent results for New York City indicate it, too, has a focused and effective charter policy.
“Our hope is that we can get beyond the distractions of this technical discussion,” the rebuttal adds, “and return to the larger policy issues.”
