Includes updates and/or revisions.
Nearly four years after a front-page story in The New York Times sparked a fierce debate by suggesting that charter school students nationally were lagging academically behind their peers in regular public schools, the national testing program that informed the controversy has generated far more data for researchers and advocates to scrutinize.
Yet the more recent findings from the National Assessment of Educational Progress have garnered much less attention and analysis than the 2003 results.
The picture that emerges from the growing data set appears mixed for charter schools. While many analysts urge caution in using NAEP to judge the 4,300-school charter sector, the latest data do not bolster the early hopes of charter advocates that the sector as a whole would significantly outperform regular public schools.
The overall scores of charter students tested in 2007 in the nationally representative assessment program were lower than for students in regular public schools in 4th grade reading and mathematics, and in 8th grade math, all by statistically significant margins.
In 4th grade reading, charter students had an average score of 214, compared with 220 for regular public schools, on a 500-point scale. Looked at another way, 59 percent of charter students scored at or above the “basic” reading level on the NAEP test, compared with 66 percent in other public schools.
But in 8th grade reading, charter students appeared to essentially close a gap from 2005, with charter and regular public school students scoring about the same in 2007.
Digging deeper in the data reveals a more complex story, though limited sample sizes for charter schools make many score differences hard to interpret with confidence. For low-income black students—a key population served by many urban charters—the 2007 performance gaps between charter and noncharter schools generally appeared smaller than those between the two sectors’ populations as a whole, and none was large enough to be deemed statistically significant.
Hispanic charter students, meanwhile, appeared to do about as well as or better than their peers in regular public schools across grades and subjects. But here, too, limited sample sizes make the differences too small to state with confidence.
Piece of the Puzzle
Researchers emphasize that because of NAEP’s design, the program has serious limitations in assessing charter schools, or comparing them with other public schools. Some experts argue that the tests are altogether ill-suited for the purpose.
The congressionally mandated NAEP—known as “the nation’s report card”—provides a snapshot of performance at a single point in time, testing different students each cycle. It also does not take into account prior achievement.
Even with those and other caveats, some researchers—though not all—suggest the results are a useful part of the growing research base on charter achievement.
“I personally don’t believe we’re going to have a single, definitive study to tell us whether charters are working,” said Gary J. Miron, a principal research associate at the evaluation center at Western Michigan University, who studies charter schools. “The NAEP is one more piece of the puzzle. … And now we have multiple snapshots.”
The debate over charter schools and NAEP first came to public prominence following an August 2004 story in The New York Times. The article focused on an analysis conducted by the American Federation of Teachers, which had called for a national moratorium on new charter schools in 2002.
The AFT study used reading and math scores in 2003, the first time charter students’ NAEP scores were reported separately. It concluded that charter students trailed those in regular public schools as a whole, and also when results were broken down by such subgroups as black students or students eligible for free or reduced-price lunches.
The report and news story spurred an outcry from charter advocates and some researchers, who suggested its methodology was flawed and the results misleading. The 2003 NAEP data were the subject of multiple re-evaluations, including a federal analysis issued in 2006 finding that scores in charter schools, taking into account a range of background characteristics of students and schools, trailed those in regular public schools that year in reading and math. (“Reanalysis of NAEP Scores Finds Charter Schools Lagging,” Aug. 30, 2006.)
Adding to the Debate?
Charter schools, which are publicly funded but largely independent, have been the focus of a growing body of studies. Many experts say the overall findings so far are inconclusive, with a blend of positive, neutral, and negative outcomes for charter schools compared with other public schools.
Todd M. Ziebarth, a senior policy analyst at the National Alliance for Public Charter Schools, a Washington research and advocacy group, said his group is keenly aware of NAEP’s limitations, but keeps an eye on the results. “They’re important to look at and learn from, but you learn so little compared to [other] assessments,” he said.
Chester E. Finn Jr., the president of the Thomas B. Fordham Foundation, a Washington think tank that is the authorizer for a set of Ohio charter schools, was among the charter advocates who urged the separate reporting of results for charter students on the 2003 NAEP tests. He said he believed it would be good for charter schools and policymakers.
“I hoped they also would be good for traditional public schools, and might show them up,” he said.
Mr. Finn said he now believes the most important distinctions are between different kinds of charters, such as those that operate in networks, or from state to state where different governing laws apply.
“I’m not very interested in the average performance of charters,” he said. “The word ‘charter’ signals so little about them, and the diversity within that universe is at least as great as the diversity outside it.”
The Charter Sample
The NAEP sample of charter students is designed to be nationally representative, but is dwarfed by the number of students tested in regular public schools.
Recent samples have ranged from 2,300 to 3,300 charter students for each grade level in reading, and from 150 to more than 200 schools. By contrast, in 2007, the national public school sample in 4th grade reading included 183,400 students from 7,310 schools, with somewhat lower 8th grade figures.
The population that charter schools typically serve is substantially different from that served by public schools as a whole, complicating comparisons across school types. In 2007, the NAEP charter sample had more than twice as many black students on a percentage basis and far more students living in cities—groups that generally score below national averages—than did the pool of NAEP test-takers overall.
And the charter population itself is changing, the test samples suggest. For example, 53 percent of charter students tested in 4th grade math in 2007 were eligible for free or reduced-price lunches, up from 42 percent in 2003.
With three rounds of reading and math scores for 4th grade charter students, the NAEP results have not shown clear trends. In reading, charter students’ scores apparently rose in 2005 but dropped slightly in 2007; the changes were not large enough, however, to be statistically significant. With 4th graders in regular public schools posting reading gains in 2007, the charter students slipped behind them.
In 4th grade math, charter students have shown what appear to be steady gains from 2003 to 2007—again, not statistically significant—but made little headway in closing the gap with students in regular public schools, who saw a similar growth trend.
Looking at students by race and income status, as measured by eligibility for free or reduced-price lunches, seems to alter the picture.
In 4th grade math, charter students overall were 5 points behind noncharter students in both 2005 and 2007 on the 500-point scale, statistically significant differences.
But for low-income black students, the difference between those in charter schools and those in regular public schools was 1 point in 2005 and 2 points in 2007, neither of which was statistically significant. Measured another way, 56 percent of low-income black students from charter schools scored at the basic level or above in 2007, compared with 59 percent of such students in regular public schools.
“Certainly, the raw data suggest charter students are behind,” said Sarah Theule Lubienski, an associate professor of education at the University of Illinois at Urbana-Champaign, after reviewing the recent results for 4th grade students in math and reading. But she said that when controlling for race and low-income status, it appears that “neither one is ahead.”
In a press release last September, the national charter alliance highlighted the gains of 8th grade charter students in 2007, though those apparent increases were not deemed statistically significant. The average scale score rose from 255 in 2005 to 260 in 2007 in reading, and from 268 to 273 in math.
Growth Rates Differ?
The alliance concluded that charter students’ achievement “increased at a notably faster rate” than that of their peers in regular public schools. It also noted stronger 8th grade gains in certain categories, such as the reading scores for African-American and low-income students.
The charter alliance also noted the performance of Hispanic charter students. In 8th grade math, their average score was 9 points higher than for Hispanics in regular public schools in 2005, and 8 points higher in 2007. Because of the limited charter sample size, though, the differences were not deemed statistically significant.
For its part, the AFT has been quiet about the recent NAEP data. Asked about the results, union spokesman Dan Murphy said: “Basically, what the NAEP data suggests is what we already know. There are some excellent charter schools and some charter schools that fall short. … And the same goes for regular public schools.”
