A suburban school district just outside St. Louis has become the latest in what appears to be a small pool of school systems to explore how their students stack up against international peers.
The 2,500-student Clayton district came out “first in the world in science and reading; second in the world in math,” in its words, based on the scores of its 15-year-olds on the Program for International Student Assessment, or PISA.
But experts caution that the Missouri district’s claim isn’t quite that simple, as it’s based on comparing the results for one small, relatively affluent district with the averages posted by other nations.
Nonetheless, some observers give points to Clayton and other districts that put themselves in that position.
“The fact that this district has gone to benchmark themselves against PISA scores, I think, is a very positive reaction because it says they’re looking up and seeing that they’re competing in the world,” said Eric A. Hanushek, an economist at Stanford University who has studied international achievement.
“But I don’t think we can generalize very much from these results,” he said. “If we take a high-performing, high-income district in St. Louis and say it beats the average performance of kids in a bunch of other countries, we don’t know quite what to make of that. ... It would be really striking if [a district like Clayton] didn’t.”
District Superintendent Mary Herrmann said she found the outcome to be “very affirming” of her school system’s work, especially given PISA’s emphasis on applying learning in real-world contexts, even as she cautioned that the results reflect only a “snapshot” of achievement.
“When you set out to ask the question, ‘Are our students globally competitive?’ and get results that are obviously very favorable, that’s ... good information,” she said.
‘Clayton Versus Clayton’
The district’s performance in reading, math, and science was far above the average for U.S. participants on PISA in 2009, as the United States as a whole trailed many leading nations.
The average reading score for Clayton 15-year-olds was 572 on the PISA scale, which goes from 1 to 1,000—above the 556 for Shanghai, China, 539 for South Korea, and 536 for Finland, the three top performers. The U.S. average was 500.
In science, Clayton’s average was 581, compared with 575 for Shanghai, Finland’s 554, and Hong Kong’s 549. The U.S. average was 502.
In math, Clayton’s score of 582 was below Shanghai’s 600, the highest. Singapore received a 562, Hong Kong 555, and South Korea 546. The U.S. average was 487.
“The numbers look pretty impressive, and that’s good because if the numbers didn’t look impressive, then we would be worried for a district like Clayton,” said Andrew Chen, a former research scientist at the Massachusetts Institute of Technology who is the president of EduTron Corp., a Winchester, Mass.-based company that provides math and science education products and services. “It would be nice if we could really have data on our Clayton versus their ‘Clayton.’ ”
Chris Tennill, a district spokesman, said Clayton serves a “reasonably affluent community,” but it is “fairly demographically diverse.”
About 16 percent of students are eligible for free or reduced-price lunches, he said. While the majority of students are white, 22 percent are African-American, 10 percent are Asian, and 3 percent Hispanic.
Small Sample
In all, 60 nations participated in PISA in 2009, as well as several large, non-national education systems, such as Shanghai. Experts say Shanghai is seen as a leader in school performance in China.
Clayton was not part of the U.S. pool, but instead took the exam as part of a study led by ACT Inc., the Iowa City, Iowa-based testing company. That study, not yet released, aims to evaluate the college-readiness benchmarks for ACT’s 10th grade PLAN assessment with respect to international performance, according to the district. ACT officials declined to discuss the report or Clayton’s results.
Mr. Tennill said that of the districts in the ACT study, his was the only one in which all sophomores—not just a sample—took the assessment, a special request made by the district.
Of the Clayton students to take the exam—all at the district’s one high school—176 fit the age criteria to be comparable on PISA.
Because of the relatively small number of students tested, experts say, there is likely a higher variability in the reliability of the scores, though even if the scores were slightly lower, the district’s results would remain at or near the top.
No information was made available on the Clayton district’s “confidence interval”—a reliability measure that indicates the range within which the score for those tested is likely to fall 95 percent of the time.
“Their confidence interval almost assuredly is high, probably more than plus or minus 10 points,” said Daniel McGrath, a program director at the National Center for Education Statistics, which coordinates U.S. participation in PISA. That matters, he said, because no single student takes all the test items, but instead takes just a portion of them.
This is not the first time a district has compared itself against nations on PISA. He noted that several did so in 2006, including Scarsdale and Herricks in New York state.
A 2008 report from the Scarsdale district, long seen as high-performing, proclaimed that “if the Republic of Scarsdale really were a country, we’d be the highest-scoring nation in the world.”
Mr. McGrath said the NCES no longer makes available to smaller districts the option of oversampling so they can obtain PISA scores, because of what he called the imprecise results obtained in 2006, given the small number of students tested.
Some districts have been able to rate their performance against that of nations on the Trends in International Mathematics and Science Study, or TIMSS. For example, a batch of school systems and district consortia participated in a 1999 benchmarking study that enabled such comparisons.
Mr. Chen from EduTron, who recently visited the Clayton district to make a presentation to parents about the PISA results and other topics, said he respects the district’s eagerness to compare its students with those around the world.
“It takes some guts to want to do this,” he said. “We all wonder: What would our school look like on that chart?”
