Federal

Study Puts Results of International Tests on Common Metric

By Sean Cavanagh — June 16, 2009 5 min read
  • Save to favorites
  • Print

International tests known by odd acronyms like PISA and TIMSS have become fixed in the American educational and political vernacular. Newspaper editorial writers and elected officials at all levels hash over U.S. students’ scores on those nation-by-nation exams with a zeal and frequency they once reserved for the release of state and local test results.

Now an American researcher has attempted to make those comparisons more meaningful to the public, by comparing the performance of students in U.S. states and cities against that of their foreign peers using a well-understood metric: letter grades.

A new study, “The Second Derivative: International Benchmarks in Mathematics for U.S. States and School Districts,” shows students in almost all U.S. states and cities performing at a C+ or C level. That is below the B grade that the study’s author contends should be the international target for American students, and which is achieved by the highest-performing nations.

In addition, the study reveals that the gap between a B and a C grade is a “quantum leap,” according to the author of the report, Gary W. Phillips, a vice president and a chief scientist at the American Institutes for Research, in Washington. The gap that separates the United States from top-performing jurisdictions like Hong Kong is about the size of the gulf between the highest-achieving U.S. state in the study, Massachusetts, and the lowest, Mississippi.

“We’re below the international benchmark, and we’re way below it,” Mr. Phillips said in an interview. The goal of the study, he said, is to compare U.S. and international performance in a clear fashion, which “resonates with readers and does it in a way that’s scientifically justifiable.”

Letter Grades Assigned

The study allows grade comparisons across cities, states, and nations. Florida and Washington state, for instance, earn C+ grades in 4th grade math, which the study shows as comparable to the performance of England, Germany, and Denmark, among others.

In two previous studies, Mr. Phillips compared the performance of students in U.S. states and cities, respectively, on an international scale. He did so by statistically linking the results of a prominent global exam, the Trends in International Mathematics and Science Study, or TIMSS, with the results of a U.S. test, the National Assessment of Educational Progress, or NAEP.

City Grades: A Global Perspective

A study of international achievement grades nations, U.S. states, and select cities on a common scale, using American jurisdictions’ NAEP scores to estimate their performance on an international test, TIMSS. The OECD figure represents the average for developed nations.

BRIC ARCHIVE

SOURCE: American Institutes for Research

In his latest effort, Mr. Phillips uses the cut scores on the TIMSS test to fashion a new statistical comparison with NAEP, and then equates scores with letter grades ranging from A to “BD,” or below D. Linking the two tests is possible, Mr. Phillips says, because TIMSS and NAEP are administered to equivalent, representative samples of students, in the same grades.

He selects a B grade as the U.S. target because his methodology shows that students who reached the “proficient” level on NAEP are statistically comparable to those reaching the “high” international level on the TIMSS.

American students’ performance, as judged by this metric, is mixed.

In 4th grade math, roughly half of U.S. states perform above the C+ mean for developed nations, according to the report. Just five states—Massachusetts, Minnesota, New Jersey, New Hampshire, and Kansas—reach the international target grade of B, and one state, Vermont, scores a B-minus in 4th grade.

In 8th grade, the U.S. states’ results were even less impressive, with only Massachusetts scoring a B-minus. Those scores fit a pattern: On the whole, American students’ math performance was weaker at 8th grade than at 4th grade, a disparity that has emerged in other international studies. (“U.S. Students’ Scores Drop By 8th Grade,” Dec. 13, 2000.)

Of 11 large U.S. cities studied, the majority scored a C grade at the 4th and 8th grade level, and none reached a B grade. Just two—Charlotte, N.C., and Austin, Texas—notched scores above the estimated TIMSS score, in both 4th and 8th grade. The District of Columbia schools had the lowest scores for both 4th and 8th grade.

No jurisdiction in the study—not even top-performing nations like Japan—earned an A grade. That result caused one expert on testing, Sharif Shakrani of Michigan State University, to question whether Mr. Phillips’ methods had set too high a bar, which might in turn make the U.S. results look worse than they actually are. Mr. Shakrani, who directs his university’s Education Policy Center, has helped develop math content for NAEP.

“Nobody got an ‘A,’ so is this [goal] a pie in the sky?” Mr. Shakrani said in an interview. “A very, very, good student should be able to reach that level,” he argued, as should a top-tier nation.

U.S. States as Models?

Mr. Shakrani attributed the drop-off in U.S. performance between 4th and 8th grade to American middle schools’ tendency to track students into middle grades math classes—algebra, pre-algebra, and very basic math—of strikingly different quality. Higher-performing nations have more uniform expectations for what math content is taught and tested, he said.

Comparing States on an International Scale

Most U.S. states meet or exceed the average of developed nations in 4th grade math.

BRIC ARCHIVE

SOURCE: American Institutes for Research
* U.S. Department of Defense schools

Hal Salzman, a professor of public policy at Rutgers University, said in an e-mail that the study reinforces an “unproved assumption” that international test scores are connected to a country’s ability to compete economically. Mr. Salzman has argued that U.S. policymakers misinterpret international exam results and exaggerate worries about American performance.

“Using these scores and international comparisons too often substitutes for substantive discussion of what is needed for real education improvement,” Mr. Salzman added.

More variation in test scores occurs within countries, among their own student populations, than from one nation to the next, Mr. Salzman said. American policymakers would be better off focusing on closing gaps between high- and low-perfoming states and districts, than on nation-by-nation comparisons, he argued.

In an e-mail, Mr. Phillips said that, on his grading scale, it is possible, albeit very difficult, for jurisdictions to obtain an A. He chose a B grade as the international target because it was “high, but achievable.”

Mr. Phillips responded to Mr. Salzman’s point by saying that it is important for U.S. officials to be aware of the high standards set by foreign countries.

But he agreed with the Rutgers professor that American policymakers should probe the successes of states like Massachusetts. Those states’ high grades, he wrote in the study, show “it is possible in the United States for students to learn mathematics at a level that is competitive with the best in the world.”

A version of this article appeared in the June 17, 2009 edition of Education Week as Study Puts Results Of International Tests On Common Metric

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Reading & Literacy Webinar
Literacy Success: How Districts Are Closing Reading Gaps Fast
67% of 4th graders read below grade level. Learn how high-dosage virtual tutoring is closing the reading gap in schools across the country.
Content provided by Ignite Reading
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Artificial Intelligence Webinar
AI and Educational Leadership: Driving Innovation and Equity
Discover how to leverage AI to transform teaching, leadership, and administration. Network with experts and learn practical strategies.
This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
School Climate & Safety Webinar
Investing in Success: Leading a Culture of Safety and Support
Content provided by Boys Town

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

Federal Trump's K-12 Record in His First Term Offers a Blueprint for What Could Be Next
In his first term, Trump sought to significantly expand school choice, slash K-12 spending, and tear down the U.S. Department of Education.
11 min read
Education Secretary Betsy DeVos listens at left as President Donald Trump speaks during a round table discussion at Saint Andrew Catholic School on March 3, 2017, in Orlando, Fla.
Education Secretary Betsy DeVos listens at left as President Donald Trump speaks during a round table discussion at Saint Andrew Catholic School on March 3, 2017, in Orlando, Fla. The education policies Trump pursued in his first term offer clues for what a second Trump term would look like for K-12 schools.
Alex Brandon/AP
Federal From Our Research Center How Educators Say They'll Vote in the 2024 Election
Educators' feelings on Vice President Kamala Harris and former President Donald Trump vary by age and the communities where they work.
4 min read
Jacob Lewis, 3, waits at a privacy booth as his grandfather, Robert Schroyer, fills out his ballot while voting at Sabillasville Elementary School, Nov. 8, 2022, in Sabillasville, Md.
Jacob Lewis, 3, waits at a privacy booth as his grandfather, Robert Schroyer, fills out his ballot while voting at Sabillasville Elementary School, Nov. 8, 2022, in Sabillasville, Md.
Julio Cortez/AP
Federal Q&A Oklahoma State Chief Ryan Walters: 'Trump's Won the Argument on Education'
The state schools chief's name comes up as Republicans discuss who could become education secretary in a second Trump administration.
8 min read
Ryan Walters, then-Republican candidate for Oklahoma State Superintendent, speaks at a rally, Nov. 1, 2022, in Oklahoma City.
Ryan Walters speaks at a rally on Nov. 1, 2022, in Oklahoma City as a candidate for state superintendent of public instruction. He won the race and has built a national profile for governing in the MAGA mold.
Sue Ogrocki/AP
Federal Why Trump and Harris Have Barely Talked About Schools This Election
Kamala Harris and Donald Trump haven't outlined many plans for K-12 schools, reflecting what's been the norm in recent contests for the White House.
6 min read
Republican presidential nominee former President Donald Trump and Democratic presidential nominee Vice President Kamala Harris participate during an ABC News presidential debate at the National Constitution Center, Tuesday, Sept.10, 2024, in Philadelphia.
Republican presidential nominee former President Donald Trump and Democratic presidential nominee Vice President Kamala Harris participate in an ABC News presidential debate at the National Constitution Center on Sept.10, 2024, in Philadelphia.
Alex Brandon/AP