Federal

Scholars Spar Over Research Methods Used to Evaluate Charters

By Debra Viadero — October 08, 2009 4 min read
  • Save to favorites
  • Print

Includes updates and/or revisions.

The authors of a recent national study that found students in regular public schools outperforming their charter school peers are rebutting criticism that their research suffered from a “serious mathematical mistake” that negatively skewed the results.

In a memorandum posted online last week, researchers from the Center for Research on Education Outcomes, or CREDO, at Stanford University responded to criticism made last month by their Stanford colleague Caroline M. Hoxby. Ms. Hoxby, an economics professor, issued a memo critiquing the CREDO study in tandem with results from her own study of charter schools in New York City.

That study showed that charter schools in the city were having the opposite effect on their students’ achievement as the CREDO researchers found. Compared with their public school counterparts, Ms. Hoxby found, students in charter schools made more progress in closing achievement gaps with better-off peers in the suburbs. (“N.Y.C. Charters Found to Close Gaps,” Sept. 22, 2009.)

“She claims that we modeled charter schools in a particular way and that was inaccurate,” Margaret E. Raymond, CREDO’s director, said of Ms. Hoxby’s critique of the statistical analysis. “And her proof was based on her inaccurate specification.”

Ms. Raymond said Ms. Hoxby’s critique also failed to consider the many factors that the CREDO study took into account in its analysis.

Stakes High

“Imagine that we’re going to compare height and shoe size, and that we’re going to do it for all the people in the office,” Ms. Raymond said. “Those two things might actually correlate in a particular way because they’re only two factors. But if we’re going to predict shoe size based on many more factors, like age or parents’ height or parents’ shoe sizes, that’s a much more precise way of estimating.”

Criticism and Rebuttal

With charter schools a high-profile policy issue, researchers who came to opposite conclusions about their effect on student achievement are debating their analyses.

“She claims that we modeled charter schools in a particular way, and that was inaccurate. And her proof was based on her inaccurate speciication.”

Margaret E. Raymond
Director, Center for Research on Education Outcomes
Stanford University

“They assert that the proof proceeds in a way that it does not. Since their main rebuttal is based on misreading a simple proof, their rebuttal is wrong.”

Caroline M. Hoxby
Professor of Economics
Stanford University

The CREDO researchers also argue in their memorandum that Ms. Hoxby’s estimates of the magnitude of the bias in their results was faulty.

The researchers’ arguments failed, however, to persuade Ms. Hoxby. In an e-mail response to Education Week, she said CREDO’s analysis contained “logically incorrect assertions.”

“They assert that the proof proceeds in a way that it does not,” Ms. Hoxby wrote. “Since their main rebuttal is based on misreading a simple proof, their rebuttal is wrong.”

If the debate is taking on a bitter tone, it’s because the stakes are high. U.S. Secretary of Education Arne Duncan has been pressuring states to remove caps on charter schools in order to qualify for some of the $100 billion in economic-stimulus funds targeted to education. But opposition to the schools, which are public schools that operate largely free of the usual regulations, continues to come from teachers’ union officials and other education leaders.

“I don’t think the field of education research or policymakers are well served by scholars going back and forth with dueling memos, without peer review and without ample time to think it through,” said Sean F. Reardon, a colleague of both scholars at Stanford. “But I don’t think either side got it right,” he added.

An associate professor of education and sociology, Mr. Reardon is reviewing the New York City study. But he said he’s also not yet convinced that the CREDO study “does enough to ensure that it’s entirely free of bias.”

“Like any nonexperimental study,” he added, “it relies on some strong assumptions that are hard to justify.”

Part of the problem, he said, is that neither study provides enough statistical detail about the data and the methods used to help reviewers settle the debate.

Also, neither study has been published yet in a peer-reviewed journal. However, an earlier version of Ms. Hoxby’s report is posted as a working paper on the Web site of the National Bureau of Economic Research, which provides a forum for economists seeking feedback from the field on their work. Ms. Hoxby said her paper is also under consideration for publication by two journals.

The CREDO study was reviewed by four outside experts before its release this past June, according to a publicist. (“Study Casts Doubt on Charter School Results,” June 15, 2009)

The memo last week from CREDO suggests that it’s not surprising that the two studies should yield different results, given that a central finding of the CREDO study, which encompasses 15 states and the District of Columbia, is that charter school performance varies considerably from school to school.

“Some communities and states have gotten the policy right, and are able to demonstrate positive charter school results,” the CREDO researchers write. “The recent results for New York City indicate it, too, has a focused and effective charter policy.

“Our hope is that we can get beyond the distractions of this technical discussion,” the rebuttal adds, “and return to the larger policy issues.”

A version of this article appeared in the October 14, 2009 edition of Education Week as Charter School Researchers Spar Over Analyses

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Webinar
Reflections on Evidence-Based Grading Practices: What We Learned for Next Year
Get real insights on evidence-based grading from K-12 leaders.
Content provided by Otus
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Creating Resilient Schools with a Trauma-Responsive MTSS
Join us to learn how school leaders are building a trauma-responsive MTSS to support students & improve school outcomes.
School & District Management Live Online Discussion A Seat at the Table: We Can’t Engage Students If They Aren’t Here: Strategies to Address the Absenteeism Conundrum
Absenteeism rates are growing fast. Join Peter DeWitt and experts to learn how to re-engage students & families.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Opinion The U.S. Dept. of Ed. Has Been Cut in Half. We Have Thoughts
Absent clear explanation and deft management, the push to downsize the department invites confusion and risks political blowback.
7 min read
The United States Capitol building as a bookcase filled with red, white, and blue policy books in a Washington DC landscape.
Luca D'Urbino for Education Week
Federal Linda McMahon Abruptly Tells States Their Time to Spend COVID Relief Has Passed
Secretary Linda McMahon said the Education Department would no longer honor the extensions it had granted states.
3 min read
Education Secretary Linda McMahon arrives before President Donald Trump attends a reception for Women's History Month in the East Room of the White House, Wednesday, March 26, 2025, in Washington.
Education Secretary Linda McMahon arrives before President Donald Trump attends a reception for Women's History Month in the East Room of the White House, Wednesday, March 26, 2025, in Washington. In a letter Friday, McMahon told state leaders on March 28 that their time to spend remaining COVID relief funds would end that same day.
Jacquelyn Martin/AP
Federal McMahon Says Schools With 'Gender Plans' Could Be Violating Federal Privacy Law
The U.S. Department of Education opened investigations under FERPA into two states, alleging violations of parents' rights.
5 min read
Secretary of Education Linda McMahon speaks to reporters at the White House in Washington, Thursday, March 20, 2025.
Secretary of Education Linda McMahon speaks to reporters at the White House in Washington, Thursday, March 20, 2025. McMahon said that the U.S. Department of Education would make a "revitalized effort" to pursue federal student privacy law violations for parents' rights, asserting that school "gender plans" that aren't available to parents violate the federal law.
Ben Curtis/AP
Federal Dramatic Cuts to Ed. Data Programs Will Have Far-Reaching Consequences, Researchers Warn
Education research organizations asked Congress to intervene in cuts to ed. data, research staff.
6 min read
Image of performance data analysis.
NicoElNino/iStock/Getty