More Than One Way To Read Voucher Study

The research war over private school vouchers rages on.

In August, researchers led by Paul E. Peterson, a professor at Harvard University, reported results from a three-city voucher experiment indicating that black students using vouchers to attend private schools had averaged 6 percentile points higher on a standardized test after two years than a control group that had applied for the vouchers but remained in public schools.

The study involved recipients of privately financed vouchers in New York City, Washington, and Dayton, Ohio. (“Privately Financed Vouchers Help Black Students, Two Studies Find,” Sept. 6, 2000.)

But on Sept. 15, a research organization involved in the New York City portion of the study issued a more detailed report of the results, as well as a caution that “voucher claims of success are premature” in that city.

Mathematica Policy Research Inc., based in Princeton, N.J., expressed concern that the New York City results were being overstated because only black 6th graders had shown statistically significant test gains after two years, while African-American students in three other grades tested showed no gains.

“My caution is, when you put a lot of weight on the average, you want to be comfortable with it,” said David Myers, the project director for Mathematicas research on the New York program. “In my view, its premature to conclude that, in New York, vouchers have an impact on the test scores of African-American students.”

The Mathematica release generated a round of stories in the news media that raised questions about the three-city study. And it provided new ammunition for critics of Mr. Peterson, who have long contended that his research is biased in favor of vouchers, a charge the researcher denies.

For example, People for the American Way, a Washington advocacy group that strongly opposes vouchers, said in a release last week that Mr. Peterson had been “rebuked” by his own research partner.

Small Disagreement

But upon close examination, there appears to be more complexity to Mathematica’s concerns about the study.

For one thing, the more detailed report on New York City bears not only Mr. Myers name as an author, but also that of Mr. Peterson.

The Harvard professor said last week that he had no disagreement with the Mathematica report on New York City. He disagrees with Mr. Myers only about the validity of relying on an average that is based on a particularly strong effect in one grade.

“The range of disagreement is really very small,” Mr. Peterson said.

He also defended the practice of averaging the results from the three cities. African- American voucher recipients scored an average of 4.3 percentage points better than their public school peers on a standardized test in New York City, 6.5 points better in Dayton, and 9 points better in Washington, for an overall average difference of 6.3 percentage points.

“This looks reasonably robust when you find similar results in three locations,” Mr. Peterson said. “The aggregated effect is more likely to be a stable finding than when you disaggregate.”

Mr. Myers said he agreed that looking across the three cities for similar results was valid.

“But when you see [some gains] so concentrated, it makes you want to pause,” he said. “I want to understand this better before I put much policy weight on it.”

Jennifer Hochschild, a professor of politics and government at Princeton University who critiqued Mr. Peterson’s three-city report for a recent political science meeting, said she had some concerns about the rush to draw policy conclusions from the results.

“I don’t think all the evidence was laid in this paper in a way that was convincing,” said Ms. Hochschild, who has since joined the government department in Harvard’s college of arts and sciences. “What is interesting is how much weight people are placing on a relatively small program. It’s the politics of vouchers, I guess.”