In March 1996, the nation’s governors met in Palisades, N.Y., and called for an “external, independent, nongovernmental effort” to measure and report on each state’s annual progress in raising student achievement and improving the public schools.
Education Week responded by launching the first edition of Quality Counts in January 1997, with the goal of producing an annual report card on public education in the 50 states and the District of Columbia.
At the time, standards-based education was in its infancy, spawned largely by frustration with a fragmented public education system whose governance was spread across more than 14,000 school districts, 50 states, and the federal government, and by low levels of student achievement, especially among poor and minority students.
The concept of standards-based education, as described by researchers, was to invite policy coherence where none existed. States would set clear and challenging content standards for what students should know and be able to do, ensure teachers were prepared to teach to those standards, provide the resources necessary to achieve the goals, and then develop tests and accountability systems closely aligned with the standards to measure progress and to prod and push schools to reach higher.
The coverage and research in Quality Counts largely mirrored this standards-based framework, including its assumption that the primary and constitutional authority for public education resided in the states.
A decade later, it’s fair to ask: Has student achievement improved? To what extent have states put in place the pieces of standards-based education? And is there any evidence that the two are related?
To answer those questions, this 10th edition of Quality Counts first examines scores on the National Assessment of Educational Progress, often known as “the nation’s report card,” from 1992, near the beginning of the standards movement, through 2005, both nationally and state by state. In addition to analyzing changes in average student achievement on NAEP, Quality Counts looks at progress for those students all too often left behind: poor, black, and Hispanic young people. And it asks whether some states have moved further or faster than others. The Educational Testing Service, of Princeton, N.J., conducted the NAEP analyses for Quality Counts.
Second, the report examines the progress states have made on some of the core indicators that Quality Counts has tracked over the past decade in the areas of standards, assessments, accountability, and efforts to improve teacher quality. (“A Decade of Policy Indicators”, this report).
Read related story, A Decade of Policy Indicators
Finally, the Editorial Projects in Education Research Center conducted a series of analyses for this report to examine the relationship between states’ increasing embrace of standards-based education and gains in student achievement. Profiles of the experiences in a handful of states expand on those analyses.
Such exploratory analyses have their limits.
First, gains on NAEP may fall short of measuring changes in student achievement at the state level. That’s in part because NAEP tests are given only to a sample of students in each state, do not carry any stakes for individual students or schools, and may not completely match state content standards and priorities.
Second, as researcher Henry I. Braun and his colleagues from the ETS point out, state responses to the call for improving education have been far from uniform. States vary in the specific policies they employ, how consistently and coherently they carry them out, when they put them into effect, and the degree to which those policies are actually picked up and put into action by districts and schools. Any longitudinal analysis also must be approached with care because of the many incremental changes to Quality Counts’ indicators over time, although the analysis tries to account for that evolution by focusing on a subset of indicators that have remained consistent over the 10-year period. Finally, from a statistical perspective, the analysis relies on a small sample, the 50 states. That fact provides another reason for caution in interpreting the results.
Despite those caveats, the time and energy put into standards-based education over the past decade suggest it’s appropriate to examine the effects of state policies as a way to help inform state practices in the future.
The conclusions are at once heartening and sobering. They’re heartening because when looked at over this span from 1992 to 2005, student achievement has gotten better, particularly in mathematics and particularly for those students who started furthest behind. Meanwhile, an increasing number of states have embraced a standards-based-education framework, with some of the earliest and most ardent adopters of standards-based accountability systems making some of the most progress in raising achievement, as highlighted by the case studies on Delaware, Massachusetts, New York, North Carolina, and Texas in this report.
A Mixed Picture
Standards-based education, explains Charlotte Ciancio, the superintendent of the 6,000-student Mapleton, Colo., public schools, has provided “the leverage that we’ve been able to use to make the changes that have been necessary in Mapleton. All of our reform work has been based on a standards-based design.”
At the same time, it would be hard to ignore the fact that progress has not come nearly far or fast enough. That’s particularly true in reading, where average scores nationally have barely budged since 1992. It’s also true that, despite the solid gains of poor, African-American, and Hispanic students during this period, the achievement gaps between those students and their more affluent and white peers remain disturbingly deep—at least 20 points in both grades and subjects, or the equivalent of two grade levels or more.
After widening a bit during the mid-1990s, those gaps have begun to close again. But in many cases, the gaps now mirror what they were in the early ’90s, and progress in closing them has been less dramatic since 2003.
An EPE Research Center analysis found a positive relationship between strengthening implementation of standards-based education over the past decade and gains in student achievement on NAEP math tests, but a slight negative relationship for reading. A closer examination found a negative relationship between state efforts to improve teacher quality, as measured by Quality Counts’ indicators, and gains in student achievement. After eliminating the teacher-quality indicators from the analyses, the relationship between state implementation of standards, assessments, and accountability systems and gains on NAEP became positive in both reading and math. The implication is that some elements of standards-based reform are more effective (or more effectively implemented) than others.
The good news is that, when examined from 1992 to 2005, the nation has made gains in student learning, particularly in math.
Student Achievement
Nationally, NAEP scores on 4th grade math have increased by 18.5 points on a 500-point scale, or nearly two grade levels, since 1992. Even more heartening are the gains for black and Hispanic 4th graders: 27.7 points and 24.2 points, respectively, during that same period.
One way to think about those math gains is that if the scores for white students had not also improved, the advances would have been enough to shrink the black-white achievement gap that existed in 1992 by 80 percent, and the gap between Hispanic 4th graders and their non-Hispanic white peers by 94 percent, virtually closing the gap between those groups. The scores for low-income students, which NAEP began reporting in 1996, basically mirror the average national increase for the 4th grade.
Since 1992, average student achievement on the National Assessment of Educational Progress has gotten better in mathematics, but results in reading are discouraging.
Note: Trends between 1992 and 2005 reflect statistically significant increases for math in grades 4 and 8 and reading in grade 8. Data from 1996 to 2005 reflect the use of accommodations for students with disabilities and English-language learners. Accommodations were not permitted in 1992 and 1994.
SOURCE: Editorial Projects in Education Research Center, 2006
In math, improvements have come both at the “basic” and “proficient” levels on NAEP, both for students overall and for black, Hispanic, and low-income youngsters. But, particularly when it comes to poor and minority students, improvements have been greater at the basic than at the proficient level. Those gains are mirrored across most states. Of the 41 states and the District of Columbia with NAEP data, all have made significant progress in 4th grade math achievement, and all but two have made significant progress in 8th grade math achievement, since 1992.
It’s hard to be as sanguine about reading. The national average in reading barely budged from 1992 to 2005, inching up just 2 points in both grades 4 and 8.
But, even here, somewhat better news lies beneath the surface. The reading scores for black, Hispanic, and low-income 4th graders increased at nearly triple the national average, or about two-thirds of a grade level. The scores for Hispanic and low-income 8th graders also increased, but not significantly.
As Marshall S. Smith, who served as the acting U.S. deputy secretary of education during the Clinton administration, points out in a paper, “The Future of Systemic School Reform,” the results are more solid when one considers how much the composition of U.S. public schools has changed over that time. Most notably, the number of Hispanic students increased by 55 percent between the 1993-94 and 2002-03 school years. Their proportion of the total K-12 population rose from 12.6 percent to 17.7 percent. Thus, the scores of Hispanic students, who typically perform less well than non-Hispanic whites, weigh more heavily in computing the national average in 2005 than they did in 1992.
“Had the proportion of all groups remained constant over the decade, the average national gain would have been greater than 10 points” in reading in grade 8, writes Smith, who is now the director of education programs at the William and Flora Hewlett Foundation, in Menlo Park, Calif.
Of course, it’s precisely the change in the student population that makes the need to improve the performance of traditionally underserved groups more important than ever. Given that, what can be said about the nation’s progress in closing achievement gaps between poor and minority students and their more affluent and white peers?
Dissecting the Gap
Beneath a modest national improvement in 4th grade reading, an analysis reveals considerable variation in patterns of change for individual states.
Gain Patterns in 4th Grade Reading (1992-2005)
Achievement Gap Patterns in 4th Grade Reading (1992-2005)
Note: Accommodations were not permitted for students with disabilities and English-language learners in 1992. Gap analysis is based on average scale scores and examines poor-nonpoor, white-black, and white-Hispanic differences.
SOURCE: Educational Testing Service analysis of U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress, 1992, 1998, and 2005 Reading Assessments
Nationally, the achievement gaps have narrowed somewhat for all groups in both reading and math and at both the 4th and 8th grade levels—except between black and white 8th graders in reading. The gap-closing was significant between black and white students in math at both grades, and between Hispanic and non-Hispanic white students in 4th grade math.
Progress is even more mixed at the state level, although the picture is complicated by the fact that many states either did not participate in the state-level NAEP during the periods examined, or did not have enough poor or minority students generated by the NAEP samples to permit comparisons of change over time.
“I think that the bigger picture is that we were losing ground over the course of the 1990s, and we’ve turned that around,” says Ross Wiener, the policy director of the Washington-based Education Trust, which works to raise the achievement of poor and minority students. But he adds that the recent slowdown underscores the need for vigilance.
An analysis of 8th grade math performance on NAEP by Braun of the ETS and his colleagues found “profound, persistent, and pervasive” achievement gaps between black and white students even within schools serving predominantly nonpoor students.
What’s most striking, in looking at the achievement picture, is how some states stand out.
Some States Stand Out
For example, Delaware gained 12.9 points in 4th grade reading between 1992 and 2005, including gains of 17.1 points for blacks and 25.7 points for poor children. In 8th grade, Delaware students gained 12.3 points in reading, including an 18.1-point gain for African-American students between 1998 and 2005.
Florida’s black 4th graders gained 17.8 points in reading between 1992 and 2005, and 35 points in math. New York state saw the average reading scores of its 4th graders jump 24.2 points for Hispanic students and 13.9 points for low-income students. And in Texas, black and Hispanic youngsters gained more than 25 points in math at grade 4 and in excess of 20 points in math at grade 8.
Performance in the Southern states, which typically started well below the rest of the country, has surged. “The South’s progress has been greater than the nation’s and needed to be,” says Mark Musick, the president emeritus of the Atlanta-based Southern Regional Education Board. Last year, for the first time, 62 percent of 4th graders in the South scored at or above “basic” in reading on NAEP, the same proportion as in the country as a whole.
“Twenty-five years ago,” says Musick, “if you said the South would help pull up the national average, people would have thought you were joking, or smoking something.”
Yet those gains on NAEP have not been reflected in the South’s staggeringly low high school graduation rates. In states such as Florida, Georgia, and South Carolina, just over half of high school freshmen graduate four years later, a rate that is worse for African-American and Hispanic teenagers. Nationally, about seven in 10 students graduate four years after starting high school.
The more controversial question is whether state policies adopted over the past decade have made any difference in raising student achievement or closing achievement gaps.
What’s Going On?
As the chart on Page 10 illustrates, Quality Counts found a steady increase in the number of states embracing state content standards and tests linked to those standards over the course of the past decade. Fewer states adopted all of the accountability and teacher-quality indicators tracked for this report during the same period.
To examine the relationship between standards-based education and gains on NAEP, the EPE Research Center conducted a series of regression analyses for Quality Counts 2006. The independent variable was the changes in the strength of states’ standards-based policies between 1997 and 2005. The dependent variable was the changes in NAEP math achievement between 1996 and 2005, and in NAEP reading achievement between 1998 and 2005. The center conducted separate analyses for math and reading in grades 4 and 8. To avoid biasing the results, the analyses controlled for states’ initial NAEP performance at the start of the period, and for the initial strength of their standards-based policies.
One way in which the research center’s analysis differs from other studies is that it considered the overall effect of states’ standards-based policies—across the areas of standards, assessments, accountability, and efforts to improve teacher quality—and looked at which of those policy areas, relatively speaking, independently contributed the most to changes in achievement.
Initially, the analyses found a moderate positive relationship between states’ overall embrace of standards-based education and gains in student math achievement. A slight negative relationship was observed for reading.
Further analyses exploring the relative contributions of the four major policy components revealed that the implementation of policies related to teacher quality was negatively related to achievement gains in both reading and math. In a second analysis, the teacher-quality policies were eliminated from the overall measure of standards-based-reform implementation in order to focus specifically on the contributions of policies related to standards, accountability, and assessment. Once teacher quality was taken out of the picture, the relationship between states’ efforts to carry out standards-based improvements and their gains in student achievement became much stronger. Improvement for math in both grades 4 and 8 became statistically significant, while more modest but positive effects emerged for reading. (Regression Analysis, this report)
Read related story, Making the Connection or download “Making the Connection: A decade of standards-based reform and achievement”—a special web-only report with a more comprehensive analysis on achievement and policy.
Preliminary analyses also found no relationship between state resource and equity indicators and student-achievement gains, after controlling for states’ initial NAEP performance.
The findings from the exploratory analyses for Quality Counts generally agree with results from other studies.
A Blunt Instrument
In a forthcoming ETS study, “The Black-White Achievement Gap: Do State Policies Matter?,” Braun and his colleagues find a positive, but modest, relationship between states’ overall policy rankings in five areas—governance, education finance, curriculum and standards, teacher quality, and assessment and accountability—and their success in raising black students’ NAEP math scores in grade 8. While no single policy component accounted for the differences in outcomes among the 10 states studied, efforts to improve teacher quality and to institute assessments and accountability appeared to matter.
“State policy is a blunt tool, but a tool nonetheless,” concludes Braun.
Perhaps the bluntest of those tools, but the one that also has gotten the most attention, are states’ efforts to hold schools accountable for results in student achievement.
In 2000, RAND Corp. researcher David W. Grissmer and his colleagues suggested that states such as North Carolina and Texas that had implemented standards and standards-based tests in the early 1990s were showing large positive gains in NAEP math scores between 1992 and 1996. But the authors cautioned that more research across all states was needed before drawing any firm conclusions about the cause of the gains.
Since then, a number of scholars have weighed in with studies that have found a positive relationship between states that adopted strong accountability systems for schools and gains on NAEP. A study by Martin Carnoy and Susanna Loeb of Stanford University in the winter of 2002, for example, found that stronger accountability systems were associated with considerable increases in the percent of students scoring at the basic level or better on NAEP math tests in grade 8, with the largest gains by black and Hispanic students.
In 2004, a study by Eric A. Hanushek and Margaret E. Raymond of the Hoover Institution, also based at Stanford, found that accountability systems introduced during the 1990s had a clear positive impact on student achievement on NAEP. But they concluded that strong accountability systems alone did not lead to any closing of the black-white achievement gap.
And in a 2005 study, economist John H. Bishop of Cornell University found a similarly positive relationship between states that had implemented strong school-level accountability systems between 1997 and 2003, based on Quality Counts’ ratings, and gains on 8th grade NAEP math scores. Bishop also found a positive relationship between states that adopted high school graduation or end-of-course tests and gains on NAEP.
Smith of the Hewlett Foundation also observes that most of the gains on NAEP occurred between the mid-1990s and 2002, consistent with the emerging maturity of state standards-based reform efforts. The greater improvements in math than in reading scores, he argues, are in line with research suggesting that math scores tend to be more sensitive to and reliant upon what is taught in schools. “These fragments of evidence by no means create an ironclad case, but they are suggestive of the reforms’ having a positive force with respect to improving test scores,” he concludes.
The gains in math also may reflect a national push for improvement in that subject, starting with the release of voluntary national standards by the National Council of Teachers of Mathematics in 1989 and promoted by such federal investments as the National Science Foundation’s systemic initiative in math and science, starting in 1990.
Not everyone has found a positive relationship between state accountability systems and test-score gains.
In a 2002 study, researchers Audrey L. Amrein and David C. Berliner at Arizona State University suggested that state efforts to tie student test results to serious consequences for students and schools had yielded no real achievement gains on NAEP. They also found dropout rates increased after states made passing tests a requirement for high school graduation, though other researchers have disputed that finding. Also, a growing body of evidence suggests that test-based accountability is encouraging schools to focus on tested subjects at the expense of nontested subjects.
The overall national graduation rate has improved slightly in recent years after decline and stagnation during the 1990s, but large gaps between groups remain.
SOURCE: EPE Research Center, 2006
Quality Counts found that high school graduation rates remained relatively stable from 1992-93 to 2001-02. They also remain abysmally low, particularly for poor and minority students.
While state accountability systems appear to be linked with gains in student achievement, according to Hanushek of Stanford University, “the only way we are going to get the substantial improvements in student performance that we want is through upgrading the quality of teaching and the quality of teachers that we have in our schools. And that is a much more difficult policy issue than accountability.”
Do Teacher Policies Matter?
Quality Counts’ indicators for efforts to improve teacher quality have changed markedly since 1997, in part to reflect the policy debate. Specifically, the report has added more details about teacher education and testing requirements, alternative routes into the profession, and accountability for teacher quality. During that period, there’s been an increase in the number of states that require prospective teachers to demonstrate knowledge of the subjects they will teach, either by holding a subject-matter major or by passing subject-matter tests.
But the EPE Research Center’s exploratory analysis did not find any relationship between state efforts to strengthen teacher quality and gains in student achievement on NAEP.
One problem, says Hanushek, is that any policy “has to allow for the fact that we have this huge overhang of existing teachers, and you cannot think of simply changing the entire stock of teachers overnight. So to me, that suggests you have to have a much longer planning horizon.”
Another problem, though, is legitimate disagreement about how to measure teacher quality. Hanushek, for example, is convinced that current licensing requirements are almost totally unrelated to teachers’ actual performance in the classroom. “As long as we resist making judgments about who is doing a good job and who isn’t,” he says, “it’s going to be very hard to think of getting any changes in achievement.”
A report released last year by the American Educational Research Association highlighted the lack of research on the impact of teacher-testing, -accreditation, and -certification polices on pupils’ learning. State levers for improving the quality of the teaching force are “pretty weak,” says Robert B. Schwartz, a professor of education at Harvard University. “The action is really much more at the district level here.”
Given what the United States has learned about standards-based education, where does that leave policymakers and educators?
What’s Next?
“We shouldn’t give up on standards-based reform,” says Smith, the Hewlett Foundation and former Clinton administration official. “We should stay the course; we should make it better, improve it.”
To some, that means moving from 50 sets of state standards to one set of national standards.
For many people, that means getting beyond state policy to efforts that actually make a difference to teaching and learning in classrooms.
“Any initiative or any goal to reduce the achievement gap substantially must do something to affect what goes on inside the school,” Braun of the ETS says. “And that’s hard to do from the state level all the way down.”
To others, that means conceding standards-based education has its limits, and that legislators need to look beyond the traditional system of public schools if they are serious about raising achievement, particularly at the high school level.
