Federal

Study Puts Results of International Tests on Common Metric

By Sean Cavanagh — June 16, 2009 5 min read
  • Save to favorites
  • Print

International tests known by odd acronyms like PISA and TIMSS have become fixed in the American educational and political vernacular. Newspaper editorial writers and elected officials at all levels hash over U.S. students’ scores on those nation-by-nation exams with a zeal and frequency they once reserved for the release of state and local test results.

Now an American researcher has attempted to make those comparisons more meaningful to the public, by comparing the performance of students in U.S. states and cities against that of their foreign peers using a well-understood metric: letter grades.

A new study, “The Second Derivative: International Benchmarks in Mathematics for U.S. States and School Districts,” shows students in almost all U.S. states and cities performing at a C+ or C level. That is below the B grade that the study’s author contends should be the international target for American students, and which is achieved by the highest-performing nations.

In addition, the study reveals that the gap between a B and a C grade is a “quantum leap,” according to the author of the report, Gary W. Phillips, a vice president and a chief scientist at the American Institutes for Research, in Washington. The gap that separates the United States from top-performing jurisdictions like Hong Kong is about the size of the gulf between the highest-achieving U.S. state in the study, Massachusetts, and the lowest, Mississippi.

“We’re below the international benchmark, and we’re way below it,” Mr. Phillips said in an interview. The goal of the study, he said, is to compare U.S. and international performance in a clear fashion, which “resonates with readers and does it in a way that’s scientifically justifiable.”

Letter Grades Assigned

The study allows grade comparisons across cities, states, and nations. Florida and Washington state, for instance, earn C+ grades in 4th grade math, which the study shows as comparable to the performance of England, Germany, and Denmark, among others.

In two previous studies, Mr. Phillips compared the performance of students in U.S. states and cities, respectively, on an international scale. He did so by statistically linking the results of a prominent global exam, the Trends in International Mathematics and Science Study, or TIMSS, with the results of a U.S. test, the National Assessment of Educational Progress, or NAEP.

City Grades: A Global Perspective

A study of international achievement grades nations, U.S. states, and select cities on a common scale, using American jurisdictions’ NAEP scores to estimate their performance on an international test, TIMSS. The OECD figure represents the average for developed nations.

BRIC ARCHIVE

SOURCE: American Institutes for Research

In his latest effort, Mr. Phillips uses the cut scores on the TIMSS test to fashion a new statistical comparison with NAEP, and then equates scores with letter grades ranging from A to “BD,” or below D. Linking the two tests is possible, Mr. Phillips says, because TIMSS and NAEP are administered to equivalent, representative samples of students, in the same grades.

He selects a B grade as the U.S. target because his methodology shows that students who reached the “proficient” level on NAEP are statistically comparable to those reaching the “high” international level on the TIMSS.

American students’ performance, as judged by this metric, is mixed.

In 4th grade math, roughly half of U.S. states perform above the C+ mean for developed nations, according to the report. Just five states—Massachusetts, Minnesota, New Jersey, New Hampshire, and Kansas—reach the international target grade of B, and one state, Vermont, scores a B-minus in 4th grade.

In 8th grade, the U.S. states’ results were even less impressive, with only Massachusetts scoring a B-minus. Those scores fit a pattern: On the whole, American students’ math performance was weaker at 8th grade than at 4th grade, a disparity that has emerged in other international studies. (“U.S. Students’ Scores Drop By 8th Grade,” Dec. 13, 2000.)

Of 11 large U.S. cities studied, the majority scored a C grade at the 4th and 8th grade level, and none reached a B grade. Just two—Charlotte, N.C., and Austin, Texas—notched scores above the estimated TIMSS score, in both 4th and 8th grade. The District of Columbia schools had the lowest scores for both 4th and 8th grade.

No jurisdiction in the study—not even top-performing nations like Japan—earned an A grade. That result caused one expert on testing, Sharif Shakrani of Michigan State University, to question whether Mr. Phillips’ methods had set too high a bar, which might in turn make the U.S. results look worse than they actually are. Mr. Shakrani, who directs his university’s Education Policy Center, has helped develop math content for NAEP.

“Nobody got an ‘A,’ so is this [goal] a pie in the sky?” Mr. Shakrani said in an interview. “A very, very, good student should be able to reach that level,” he argued, as should a top-tier nation.

U.S. States as Models?

Mr. Shakrani attributed the drop-off in U.S. performance between 4th and 8th grade to American middle schools’ tendency to track students into middle grades math classes—algebra, pre-algebra, and very basic math—of strikingly different quality. Higher-performing nations have more uniform expectations for what math content is taught and tested, he said.

Comparing States on an International Scale

Most U.S. states meet or exceed the average of developed nations in 4th grade math.

BRIC ARCHIVE

SOURCE: American Institutes for Research
* U.S. Department of Defense schools

Hal Salzman, a professor of public policy at Rutgers University, said in an e-mail that the study reinforces an “unproved assumption” that international test scores are connected to a country’s ability to compete economically. Mr. Salzman has argued that U.S. policymakers misinterpret international exam results and exaggerate worries about American performance.

“Using these scores and international comparisons too often substitutes for substantive discussion of what is needed for real education improvement,” Mr. Salzman added.

More variation in test scores occurs within countries, among their own student populations, than from one nation to the next, Mr. Salzman said. American policymakers would be better off focusing on closing gaps between high- and low-perfoming states and districts, than on nation-by-nation comparisons, he argued.

In an e-mail, Mr. Phillips said that, on his grading scale, it is possible, albeit very difficult, for jurisdictions to obtain an A. He chose a B grade as the international target because it was “high, but achievable.”

Mr. Phillips responded to Mr. Salzman’s point by saying that it is important for U.S. officials to be aware of the high standards set by foreign countries.

But he agreed with the Rutgers professor that American policymakers should probe the successes of states like Massachusetts. Those states’ high grades, he wrote in the study, show “it is possible in the United States for students to learn mathematics at a level that is competitive with the best in the world.”

A version of this article appeared in the June 17, 2009 edition of Education Week as Study Puts Results Of International Tests On Common Metric

Events

School & District Management Webinar Crafting Outcomes-Based Contracts That Work for Everyone
Discover the power of outcomes-based contracts and how they can drive student achievement.
School & District Management Webinar EdMarketer Quick Hit: What’s Trending among K-12 Leaders?
What issues are keeping K-12 leaders up at night? Join us for EdMarketer Quick Hit: What’s Trending among K-12 Leaders?
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Artificial Intelligence Webinar
Teaching Students to Use Artificial Intelligence Ethically
Ready to embrace AI in your classroom? Join our master class to learn how to use AI as a tool for learning, not a replacement.
Content provided by Solution Tree

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal How Trump Could Roll Back Access to Free School Lunches
Project 2025 and a GOP budget proposal call for axing a federal rule that allows public schools to serve free meals to all students.
5 min read
Cafeteria workers serve student lunches at Firebaugh High School in Lynwood, Calif. on Wednesday, April 3, 2024. Demand for school lunches has increased after California guaranteed free meals to all students regardless of their family's income.
Cafeteria workers serve lunches at Firebaugh High School in Lynwood, Calif., on Wednesday, April 3, 2024. Demand for school lunches has increased after California guaranteed free meals to all students regardless of their family's income. A federal school lunch provision that makes it easier for public schools to provide universal free meals may be a target for elimination in President-elect Donald Trump's upcoming term if some conservative activists and lawmakers get their way.
Richard Vogel/AP
Federal A Bill to Kill the Education Department Is Already Filed. Here's What It Says
The bill represents another attempt at a long-term Republican goal.
6 min read
People walk outside the U.S Capitol building in Washington, June 9, 2022.
People walk outside the U.S Capitol building in Washington, June 9, 2022. Legislation has been introduced in the Senate to abolish the Department of Education.
Patrick Semansky/AP
Federal Video Linda McMahon: 5 Things to Know About Trump's Choice for Education Secretary
President-elect Donald Trump plans to nominate former pro-wrestling CEO Linda McMahon to lead the education department.
1 min read
Federal The K-12 World Reacts to Linda McMahon, Trump's Choice for Education Secretary
Some question her lack of experience in education, while supporters say her business background is a major asset.
7 min read
Linda McMahon, former Administrator of Small Business Administration, speaks during the Republican National Convention on July 18, 2024, in Milwaukee.
Linda McMahon speaks during the Republican National Convention on July 18, 2024, in Milwaukee. McMahon has been selected by President-elect Trump to serve as as the next secretary of education.
J. Scott Applewhite/AP