Shining a Spotlight On Results

It’s a very American set of ideas: Take responsibility for your actions. Focus on results. And reap--or rue--the consequences.

And these days, it can all be summed up in one word: accountability.

After decades of focusing on such “inputs” as how many books are in the school library and the number of computers in the classroom, American education is shining a spotlight on results. In more and more states, policymakers are moving to reward success and punish failure in an effort to ensure that children are getting a good education and that tax dollars aren’t being wasted.

“Accountability for student performance is one of the two or three--if not the most--prominent issues in policy at the state and local levels right now,” says Richard F. Elmore, a professor at Harvard University’s graduate school of education.

The assumption seems to be that if performance is the problem, what’s missing is the will: Find the right combination of carrots and sticks, and effort and achievement will follow.

You don’t have to look far for evidence that accountability is here to stay: Forty-eight states now test their students, and 36 publish annual report cards on individual schools.

But an analysis of statewide efforts shows that most are skirting the edges of a serious accountability system. For Quality Counts ‘99, Education Week conducted an exhaustive, 50-state survey of state policies on accountability. Among the major findings are that in 1999:

• Fewer than half the states (19) publicly rate the performance of all schools or at least identify low-performing ones.
• Only 16 states have the power to close, take over, or overhaul chronically failing schools in a process\ known as reconstitution, which generally means replacing all or most of a school’s staff and reopening it under new management.
• Only 14 states provide monetary rewards for individual schools on the basis of their performance.
• Only 19 require students to pass state tests to graduate from high school; only six have laws that will link student promotion to test scores in the future.
• And only two have attempted to tie the evaluation of individual teachers to how well students perform.

Yet just the threat of sanctions appear to be focusing educators more squarely on achievement than ever before.

“The one thing we have found, as we’ve gone out to the schools and districts in each state, is schools are making much more data-driven decisions,” says Margaret E. Goertz, a professor of education at the University of Pennsylvania and a co-director of the Consortium for Policy Research in Education, or CPRE, a federally funded research organization.

Not all the results, though, have been positive. “We spend an absolutely inordinate amount of time teaching to the test,” says Richard Belcourt, a 4th grade teacher in Charlotte, N.C., in a complaint echoed by educators around the country.

Indeed, as the stakes attached to performance rise, so does the criticism. Among the most common are that too much class time must be devoted to test-taking skills, and that sanctions fall disproportionately on schools attended by poor and minority students. Of the 1,024 schools identified by states as low-performing, four in 10 have minority enrollments that exceed 90 percent.

In theory, accountability sounds wonderful. In reality, it raises a host of knotty challenges for states that, if left unchecked, could cause a public backlash.

What is the best way to measure student performance? Should young people be denied a diploma if they don’t pass state tests? What should be done with schools that, year after year, don’t improve? And when is it time to pull the plug? Those are just some of the challenges that will be explored in this edition of Quality Counts, which focuses on state efforts to design workable accountability systems.

In general, such systems include six basic elements to hold schools accountable: tests that measure whether students have met state standards; report cards that summarize the performance of individual schools; ratings systems that determine whether a school’s performance is adequate; targeted assistance to help schools improve; rewards for schools based on their performance; and the authority to close, take over, or reconstitute schools that don’t get better over time. States also are requiring consequences for individual students based on their achievement.

How--or even whether--states address each piece varies widely. Often, the various parts of a state’s accountability system do not line up in a coherent fashion. Many states have committed more to paper than they’ve put into practice.

“If you look at the 50-state picture, we’re at a very, very perilous stage,” argues Elmore, “because the progress at the state and local levels is very uneven, and the systems are much slower to develop and much more complicated than a lot of reformers expected them to be.”

Despite such variability, a few things hold true:

• Most states are relying heavily on test scores to help determine rewards and sanctions.
• Most rewards and sanctions are focused primarily on schools and their performance, rather than on individual educators.
• Although 19 states now identify low-performing schools, there is no agreed-upon strategy for fixing them.
• Despite their threats to impose severe penalties, few states are ready or willing to use them.

“All this stuff is almost brand-new from the perspective of most of the states,” says Paul T. Hill, the director of the Center on Reinventing Public Education at the University of Washington. “My guess is that the first attempts will not work, in most cases.”

It’s hardly surprising that states have latched onto accountability as the next phase in the school reform movement. The early 1990s saw states setting standards for what their students should know and be able to do and devising tests to measure whether students and schools meet the standards.

Inevitably, the question arose: What happens when students or schools fall short? The increasing involvement of the business community in education, and the growing proportion of state resources devoted to schooling, also have shifted the focus to results.

“This is the necessary capstone of the entire school reform program,” says William J. Moloney, the state commissioner of education in Colorado. “Standards and assessments are meaningless if there are no consequences.”

But if the sequence sounds logical in theory, it has proved far messier in practice, as states scramble to put standards, assessments, curricula, and accountability provisions into line.

“The political reality is that very few states can afford to put their accountability systems on hold until they’ve done all of the other steps,” says Susan Traiman, the director of education programs for the Business Roundtable, an association of corporate executives that promotes standards-based education reform. “So the sequence in which states are doing things isn’t following this nice, logical progression.”

District and state efforts may also be out of sync. In 1996, Chicago officials put 109 schools on academic probation because of low scores on the Iowa Tests of Basic Skills. The state of Illinois came out with its own early-warning list the following year, based on a different test. Chicago adopted academic standards in 1997-98; the state adopted its standards one year later. Both the state and the district are now in the process of producing tests that mirror their respective standards.

“The way I look at it,” says Phil Hansen, the chief accountability officer for the Chicago schools, “if we had waited for our accountability system to develop the standards and assessments first, we still wouldn’t have an accountability system in place.”

At least some observers divide states into two broad camps in their approaches to accountability: Those that think schools and students will improve if they are given enough resources, support, information, and encouragement; and those that think they need a substantial, external push.

“Typically, if educators are in charge, you get the soft version,” says Chester E. Finn Jr., the president of the Thomas B. Fordham Foundation in Washington. “If businessmen or elected officials are in charge, you get the hard version. And, in the real world, you end up with peculiar hybrids.”

Texas epitomizes the hard-line approach. Schools and districts can receive cash awards for exemplary student performance but are subject to intervention, and ultimately takeover, if achievement falls below a minimum standard. High school students must pass state tests to graduate. Education schools lose their accreditation if too many of their graduates fail teacher-licensing exams. And, for the first time last year, an evaluation system links teachers’ appraisals to schoolwide test scores.

Connecticut relies on a more low-key approach. The state publishes report cards on every school, including their performance on statewide tests. It also gives grants to districts that have shown sustained progress over time. But there are no explicit sanctions for schools that fail to make progress.

The state has put more of its energy into honing the skills of its teaching force: drafting new standards for licensure, paying beginning teachers more, and financing a mentoring program for novices. Some experts refer to such policies as “professional accountability,” and argue that it should be a major part of any state system.

Both Connecticut and Texas point to their strong showings on the National Assessment of Educational Progress, the congressionally mandated program that tests a sampling of U.S. students in key subjects, as proof that their respective approaches work. They were two of six states whose 4th and 8th graders performed significantly better on NAEP mathematics tests in 1996 than they did in 1992 and 1990, respectively.

Whether states have taken a soft or a hard approach to accountability, most experts agree that few have attended to all the necessary components.

Few states, for instance, have given schools all the authority they need to be held responsible for results, such as the ability to hire staff members or spend money as they see fit. Many accountability systems focus on the extremes--rewarding good schools and punishing bad ones--while ignoring incentives for those in the middle. Some states have no real consequences for students. Others have high stakes for students, without ensuring access to the curriculum and teaching that would enable young people to succeed.

Finn argues that accountability ideally should come from two directions: from the state and from the marketplace, in the form of school choice.

States such as Wisconsin are experimenting with market-driven approaches to accountability, including public school choice, private school vouchers, and charter schools. The argument is that if parents can freely choose among schools, over time they will let the low-performing schools die and flock to those that are doing a good job. And, meanwhile, the competition will spur all schools to get better.

“If people are, in fact, able to vote with their feet, and if you have a system where dollars follow students, that can be a very powerful accountability measure,” argues Howard Fuller, a former superintendent of the Milwaukee schools and a distinguished professor of education at Marquette University. But, so far, the research on whether choice programs improve student achievement is mixed. Typically, such mechanisms have not played a prominent role in state accountability systems.

Experts also agree that most states have paid too little attention to “capacity building": providing schools with the help they need to get better. International studies, such as the Third International Mathematics and Science Study, suggest that teaching in many American classrooms is far from ambitious.

“States are certainly not budgeting for the remedy part of this,” says Susan Fuhrman, the dean of the graduate school of education at the University of Pennsylvania and a co-director of CPRE. “We have to disabuse people of the simple-minded notion that you just have to get the accountability structure right, and people will know what to do.”

Others say that accountability has been a one-sided affair, with states failing to live up to their end of the bargain. “There has been a tendency to think of accountability as hierarchical, one-way accountability,” says Peggy McMullen, the project director of the Tools for Accountability Project at the Annenberg Institute for School Reform, based in Providence, R.I. “It’s our view that we’re all accountable for student learning, and that means we all have to take our part of the responsibility.”

At the state level, that might mean providing adequate operating funds for schools, ensuring that they’re staffed by qualified teachers, and setting aside money to repair crumbling buildings.

“Accountability is a big word, and in many ways, it’s a cop-out,” argues Theodore R. Sizer, the founder of the Coalition of Essential Schools and a co-director of a charter school in Massachusetts. “There’s a lot of evidence that there are some very difficult things that need to be done in education, and rather than do them, people say: ‘Well, let’s test the kids again. Or let’s test the teachers.’”

He believes accountability needs to be more broadly defined. “Everybody should be accountable, starting with the testers, starting with the governor, starting with those who are responsible for the funding systems,” he explains. “The accountability is very narrowly focused in most states. It avoids the people who hold the power.”

Many states approach accountability primarily as a statistical exercise. They reduce their ultimate judgment about a school to a single numerical index. Such indices provide the appearance of complete objectivity. But even determining which factors go into the calculations is an exercise in human judgment.

Moreover, critics argue that such indices ignore the distinctive qualities and circumstances of individual schools. They don’t differentiate between a school that has no idea of how to improve and one that is struggling but getting better. In addition, such statistics may not give schools enough guidance about what they need to change to improve.

New Mexico and Rhode Island are among the states that have tried to remedy such shortcomings by combining a statistical profile of a school’s performance with an on-site visit by a team of educators and others to examine school practices.

Others worry that, when push comes to shove, states will back down from the threat of attaching real consequences to student performance. “I know of no place that’s able to manifest both high standards and serious consequences,” says Paul G. LeMahieu, the state schools chief in Hawaii. “Every time you get serious about your consequences, you start rethinking your standards.”

Arkansas dropped a mandate that students pass a new high school exit test to graduate after too many students failed the exams. Faced with the choice of penalizing 22 Michigan schools with consistently low performance, state Superintendent Arthur E. Ellis backed off and opted instead to give the schools the expert help they needed to shape up.

At the district level, the picture is much the same. After an initial flurry of activity, most districts that moved to reconstitute low-performing schools have also backed off. San Francisco, a pioneer in such dramatic measures, announced in October that it will no longer reconstitute schools as a matter of policy.

“I think the real question is going to be, do we really have the will to apply consequences, both positive and negative?” Fuller of Marquette University says. “And I think the jury is really out on whether or not people are willing to do that.”

“In many states,” Hill of the University of Washington agrees, “there’s nobody who’s going to push this over the hard parts. Frankly, I don’t know what that means.”

Teachers, meanwhile, are finding accountability as it is practiced in many states a bitter pill.

While the public supports a number of high-stakes measures to increase accountability, teachers do not, according to a nationally representative survey conducted by the New York City-based research organization Public Agenda in connection with Quality Counts ‘99. Among its findings:

• Seventy-seven percent of employers and 70 percent of parents think it’s a good idea to have principals work under contracts that could be terminated if their schools failed to reach specific goals. In contrast, 64 percent of teachers say that’s a bad idea.
• Sixty-six percent of employers and 62 percent of parents support overhauling persistently failing schools. But 68 percent of teachers disagree.
• Sixty percent of employers and 53 percent of parents believe it’s a good idea to tie student performance to financial incentives for teachers and principals. But 76 percent of teachers do not.

Education Week found a similar split between educators and members of the public in three electronic focus groups conducted for Quality Counts as part of a special project on school report cards. One problem is that teachers do not trust the measures being used. A majority of the teachers surveyed by Public Agenda viewed widely publicized ratings of schools as “unfair and inaccurate” portrayals of their schools. Yet unless teachers, parents, and other taxpayers believe that the criteria and processes employed in state accountability systems are legitimate, fair, consistent, and understandable, such systems are unlikely to survive.

Teachers have every reason to be skeptical. Whether introducing more accountability in education ultimately will improve learning remains to be seen.

“I think we know from our experience in the ‘80s that simply putting new tests in place and tying rewards and sanctions to the test scores does not improve student learning,” argues Linda Darling-Hammond, a professor of education at Stanford University.

“Given the dramatic inequalities in funding, staffing, and knowledge that exist among U.S. schools,” she says, “punishments like failing students, firing teachers, or taking over schools are not likely to produce success where it does not now exist.”

But others are more optimistic. “New accountability systems seem to focus attention on performance, causing school personnel to attend to student achievement,” Fuhrman of CPRE says. “Our conclusion is that, contrary to the views held by some commentators, money bonuses are valued by teachers and can be motivating, and that sanctions such as school reconstitution or identification as a school in decline are also valued, though negatively, and can also function to motivate. Both get teachers’ attention.”

Research by Elmore and his colleagues at Harvard, however, suggests that individual schools and teachers may differ in their response to state accountability systems. Based on research in 20 public, parochial, independent, and charter schools around the country, the team found that schools’ internal conceptions of accountability vary a lot. Some have a strong, collective sense of responsibility. In others, accountability resides with individual teachers and expectations may be low.

In such schools, argues Elmore, strong external systems of accountability are unlikely to yield systematic improvements. “These new accountability systems presuppose a level of capacity and organizational coherence in schools which probably isn’t there,” he says. “It will produce an effect that we’ve seen from a lot of studies for a long time, which is really extreme variability in how schools respond to this kind of external pressure.”

“The irony,” he adds, “is that the introduction of performance standards and assessments in the short term could actually increase inequalities in the system rather than decrease them.”

For now, experts caution, if a state claims it has an accountability system, look at the fine print. The details could make all the difference. And don’t assume that what works in Texas will work in Oregon, or even that what works in Chicago will work in downstate Illinois.

“None of these systems is perfect,” LeMahieu of Hawaii says. “But one of the things that does seem to happen is that when they run into difficulties, the political maneuvering swells, and the debate centers on whether or not to even have the accountability system at all.

“I think a much smarter approach is to craft structures that expect these difficulties and can adapt.”