Federal

Study Puts Results of International Tests on Common Metric

By Sean Cavanagh — June 16, 2009 5 min read
  • Save to favorites
  • Print

International tests known by odd acronyms like PISA and TIMSS have become fixed in the American educational and political vernacular. Newspaper editorial writers and elected officials at all levels hash over U.S. students’ scores on those nation-by-nation exams with a zeal and frequency they once reserved for the release of state and local test results.

Now an American researcher has attempted to make those comparisons more meaningful to the public, by comparing the performance of students in U.S. states and cities against that of their foreign peers using a well-understood metric: letter grades.

A new study, “The Second Derivative: International Benchmarks in Mathematics for U.S. States and School Districts,” shows students in almost all U.S. states and cities performing at a C+ or C level. That is below the B grade that the study’s author contends should be the international target for American students, and which is achieved by the highest-performing nations.

In addition, the study reveals that the gap between a B and a C grade is a “quantum leap,” according to the author of the report, Gary W. Phillips, a vice president and a chief scientist at the American Institutes for Research, in Washington. The gap that separates the United States from top-performing jurisdictions like Hong Kong is about the size of the gulf between the highest-achieving U.S. state in the study, Massachusetts, and the lowest, Mississippi.

“We’re below the international benchmark, and we’re way below it,” Mr. Phillips said in an interview. The goal of the study, he said, is to compare U.S. and international performance in a clear fashion, which “resonates with readers and does it in a way that’s scientifically justifiable.”

Letter Grades Assigned

The study allows grade comparisons across cities, states, and nations. Florida and Washington state, for instance, earn C+ grades in 4th grade math, which the study shows as comparable to the performance of England, Germany, and Denmark, among others.

In two previous studies, Mr. Phillips compared the performance of students in U.S. states and cities, respectively, on an international scale. He did so by statistically linking the results of a prominent global exam, the Trends in International Mathematics and Science Study, or TIMSS, with the results of a U.S. test, the National Assessment of Educational Progress, or NAEP.

City Grades: A Global Perspective

A study of international achievement grades nations, U.S. states, and select cities on a common scale, using American jurisdictions’ NAEP scores to estimate their performance on an international test, TIMSS. The OECD figure represents the average for developed nations.

BRIC ARCHIVE

SOURCE: American Institutes for Research

In his latest effort, Mr. Phillips uses the cut scores on the TIMSS test to fashion a new statistical comparison with NAEP, and then equates scores with letter grades ranging from A to “BD,” or below D. Linking the two tests is possible, Mr. Phillips says, because TIMSS and NAEP are administered to equivalent, representative samples of students, in the same grades.

He selects a B grade as the U.S. target because his methodology shows that students who reached the “proficient” level on NAEP are statistically comparable to those reaching the “high” international level on the TIMSS.

American students’ performance, as judged by this metric, is mixed.

In 4th grade math, roughly half of U.S. states perform above the C+ mean for developed nations, according to the report. Just five states—Massachusetts, Minnesota, New Jersey, New Hampshire, and Kansas—reach the international target grade of B, and one state, Vermont, scores a B-minus in 4th grade.

In 8th grade, the U.S. states’ results were even less impressive, with only Massachusetts scoring a B-minus. Those scores fit a pattern: On the whole, American students’ math performance was weaker at 8th grade than at 4th grade, a disparity that has emerged in other international studies. (“U.S. Students’ Scores Drop By 8th Grade,” Dec. 13, 2000.)

Of 11 large U.S. cities studied, the majority scored a C grade at the 4th and 8th grade level, and none reached a B grade. Just two—Charlotte, N.C., and Austin, Texas—notched scores above the estimated TIMSS score, in both 4th and 8th grade. The District of Columbia schools had the lowest scores for both 4th and 8th grade.

No jurisdiction in the study—not even top-performing nations like Japan—earned an A grade. That result caused one expert on testing, Sharif Shakrani of Michigan State University, to question whether Mr. Phillips’ methods had set too high a bar, which might in turn make the U.S. results look worse than they actually are. Mr. Shakrani, who directs his university’s Education Policy Center, has helped develop math content for NAEP.

“Nobody got an ‘A,’ so is this [goal] a pie in the sky?” Mr. Shakrani said in an interview. “A very, very, good student should be able to reach that level,” he argued, as should a top-tier nation.

U.S. States as Models?

Mr. Shakrani attributed the drop-off in U.S. performance between 4th and 8th grade to American middle schools’ tendency to track students into middle grades math classes—algebra, pre-algebra, and very basic math—of strikingly different quality. Higher-performing nations have more uniform expectations for what math content is taught and tested, he said.

Comparing States on an International Scale

Most U.S. states meet or exceed the average of developed nations in 4th grade math.

BRIC ARCHIVE

SOURCE: American Institutes for Research
* U.S. Department of Defense schools

Hal Salzman, a professor of public policy at Rutgers University, said in an e-mail that the study reinforces an “unproved assumption” that international test scores are connected to a country’s ability to compete economically. Mr. Salzman has argued that U.S. policymakers misinterpret international exam results and exaggerate worries about American performance.

“Using these scores and international comparisons too often substitutes for substantive discussion of what is needed for real education improvement,” Mr. Salzman added.

More variation in test scores occurs within countries, among their own student populations, than from one nation to the next, Mr. Salzman said. American policymakers would be better off focusing on closing gaps between high- and low-perfoming states and districts, than on nation-by-nation comparisons, he argued.

In an e-mail, Mr. Phillips said that, on his grading scale, it is possible, albeit very difficult, for jurisdictions to obtain an A. He chose a B grade as the international target because it was “high, but achievable.”

Mr. Phillips responded to Mr. Salzman’s point by saying that it is important for U.S. officials to be aware of the high standards set by foreign countries.

But he agreed with the Rutgers professor that American policymakers should probe the successes of states like Massachusetts. Those states’ high grades, he wrote in the study, show “it is possible in the United States for students to learn mathematics at a level that is competitive with the best in the world.”

A version of this article appeared in the June 17, 2009 edition of Education Week as Study Puts Results Of International Tests On Common Metric

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Assessment Webinar
Reflections on Evidence-Based Grading Practices: What We Learned for Next Year
Get real insights on evidence-based grading from K-12 leaders.
Content provided by Otus
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Student Well-Being Webinar
Creating Resilient Schools with a Trauma-Responsive MTSS
Join us to learn how school leaders are building a trauma-responsive MTSS to support students & improve school outcomes.
School & District Management Live Online Discussion A Seat at the Table: We Can’t Engage Students If They Aren’t Here: Strategies to Address the Absenteeism Conundrum
Absenteeism rates are growing fast. Join Peter DeWitt and experts to learn how to re-engage students & families.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Trump Admin. Tells Schools: No Federal Funds If You're Using DEI
A letter sent out Thursday is another Trump administration to curb diversity, equity, and inclusion in schools—and use funding as leverage.
6 min read
Vector illustration of a large hand holding a contract and a smaller man with a large pen signing the contract while a woman in the background is clutching a gold coin and watching as he signs.
DigitalVision Vectors/Getty
Federal Opinion The U.S. Dept. of Ed. Has Been Cut in Half. We Have Thoughts
Absent clear explanation and deft management, the push to downsize the department invites confusion and risks political blowback.
7 min read
The United States Capitol building as a bookcase filled with red, white, and blue policy books in a Washington DC landscape.
Luca D'Urbino for Education Week
Federal Linda McMahon Abruptly Tells States Their Time to Spend COVID Relief Has Passed
Secretary Linda McMahon said the Education Department would no longer honor the extensions it had granted states.
3 min read
Education Secretary Linda McMahon arrives before President Donald Trump attends a reception for Women's History Month in the East Room of the White House, Wednesday, March 26, 2025, in Washington.
Education Secretary Linda McMahon arrives before President Donald Trump attends a reception for Women's History Month in the East Room of the White House, Wednesday, March 26, 2025, in Washington. In a letter Friday, McMahon told state leaders on March 28 that their time to spend remaining COVID relief funds would end that same day.
Jacquelyn Martin/AP
Federal McMahon Says Schools With 'Gender Plans' Could Be Violating Federal Privacy Law
The U.S. Department of Education opened investigations under FERPA into two states, alleging violations of parents' rights.
5 min read
Secretary of Education Linda McMahon speaks to reporters at the White House in Washington, Thursday, March 20, 2025.
Secretary of Education Linda McMahon speaks to reporters at the White House in Washington, Thursday, March 20, 2025. McMahon said that the U.S. Department of Education would make a "revitalized effort" to pursue federal student privacy law violations for parents' rights, asserting that school "gender plans" that aren't available to parents violate the federal law.
Ben Curtis/AP