In 1997, Congress established a hefty state grant program to encourage poor schools to invest in proven, off-the-shelf models for improving learning. By 2006, though, the Comprehensive School Reform Demonstration program—or CSRD, as it’s commonly known—had effectively, and quietly, died. Though still on the U.S. Department of Education books, the program made no new grants to states in that year or since.
Now, along comes a study offering a fitting, if somewhat ironic, epitaph for the program: Models matter.
“It wasn’t simply the fact of adopting the practices that seemed to make a difference on students’ achievement,” said Kerstin Carlson LeFloch, a study co-author. “It’s the whole bundle of practices.”
The federal study, which was conducted by researchers at the American Institutes for Research, or AIR, a nonprofit research group based in Washington, tracked progress in 650 elementary and middle schools, half of which were implementing one of eight different, packaged school improvement models, such as Success for All or Modern Red Schoolhouse, and half of which were using no formal schoolwide program to boost student achievement, between 2001 and 2004.
View the papers on comprehensive-school reform, posted by the Journal of Education for Students Placed at Risk.
Funded in 2000 when President Clinton was still in office, the $7.5 million study was completed last year. But the Bush administration, which does not count comprehensive school reform among its priorities for K-12 education, did not publicize it. The findings were published last fall in a report by the research group and again last month in a series of articles in the Journal of Education for Students Placed at Risk.
What the results show, somewhat surprisingly, is that schools not using the reform models did not look, in practice, all that different from the comprehensive- reform schools. Three-quarters of schools in both groups used practices that were very close to those that model developers saw as ideal for their programs. In other words, if the treatment school in the same district was using a model that emphasized applying test data to make instructional decisions or involving parents more in their children’s schooling, the comparison school tended to have the same sorts of practices in place.
“I think what schools perceive as good ideas in comprehensive school reform spread quickly to other schools,” said Daniel K. Aladjem, the study’s lead author and a principal research scientist at the AIR.
Time to Ripen
Possibly because the two groups of schools looked so much alike, the researchers found no differences between them, overall, in the gains students made on mathematics and reading tests over the five years of the study. Upon closer examination, though, the pattern became more complex: Between the third and the fifth year of implementation, schools that had stayed true to their school improvement models experienced achievement gains at a rate outpacing those of the comparison schools. The boost in achievement was all the more notable because the experimental schools had started out lagging behind the control schools. In the experimental schools that adhered less closely to the program guidelines, on the other hand, test-score gains were no different from those in other schools.
“In short,” one of the journal papers on the subject concludes, “CSR ‘works’ when external models are implemented faithfully and consistently for three to five years.”
The learning gains also varied, depending on the program that schools used. Success for All, a program developed by researchers from Johns Hopkins University in Baltimore, appeared to produce the largest academic improvements of any of the models studied, the study found, and to have schools that adhered closely to its program.
Some other models, though, stood out for their progress in building “social capital” in schools—fostering, in other words, an atmosphere in which teachers expressed a feeling of collegiality, collaborated in lesson planning, took part in extensive professional-development activities, and shared a sense of mission for their schools.
According to the report, two models produced the most gains in that regard: Authentic Teaching Learning and Assessment for All Students, or ATLAS, and the Accelerated Schools Project. Because some of the practices that go hand in hand with the development of social capital, such as professional development, were linked to long-term achievement gains, Mr. Aladjem speculated that gains in those schools might catch up, over time, to those in the Success for All schools.
“By definition, if you’re going to build a professional-development community, it’s going to take longer,” he said.
NCLB Catalyst
In a five-year federal study, researchers from the American Institutes for Research tracked the life cycle of comprehensive-school-reform programs in hundreds of elementary and middle schools across the country. Featured programs were:
Accelerated Schools Project
Developed in 1986 by Henry M. Levin, then an education and economics professor at Stanford University, the program is known best for its focus on providing accelerated instruction to all children, including those labeled “at risk.” The national program is now based in Storrs, Conn., and has satellite training centers throughout the country.
ATLAS Communities
Short for Authentic Teaching Learning and Assessment for All Students, ATLAS Communities is the 1992 brainchild of four prominent school improvement thinkers: Yale University psychologist James P. Comer; Howard Gardner, a Harvard University psychology professor; Theodore R. Sizer, now a Brown University professor emeritus; and Janet Whitla, a former president and chief executive officer of the Education Development Center, a Newton, Mass.-based research group. The national office is based in Cambridge, Mass.
Co-nect
Established in 1992, Co-nect is a data-driven schoolwide-improvement program. The model is now overseen by Pearson Achievement Solutions, an arm of Pearson Education, a for-profit educational media company based in Saddle River, N.J.
Expeditionary Learning/Outward Bound
Though the Outward Bound outdoor-adventure program has been around for 60 years, it did not evolve into a formal whole-school-improvement program until 1992. The program, based in Garrison, N.Y., makes real-world instruction and community-service projects a key feature of school curricula.
Modern Red Schoolhouse
This K-12 design was founded in 1992 by the Hudson Institute, a think tank based in Washington. The program is now housed at the Modern Red Schoolhouse Institute in Nashville, Tenn.
Success for All
Perhaps the nation’s best-known school improvement program, the Baltimore-based Success for All was founded in 1987 by husband-and-wife researchers Robert E. Slavin and Nancy A. Madden, both of Johns Hopkins University in Baltimore.
Turning Points
Based at the Center for Collaborative Learning, in Boston, Turning Points is a middle school reform model developed in 1999. It grew out of a landmark national report of the same name that was published by the Carnegie Council on Adolescent Development.
Urban Learning Centers
This model for the pre-K-12 grades was a collaboration of the Los Angeles Unified School District and United Teachers of Los Angeles.
SOURCE: Education Week
The study’s findings are based on longitudinal surveys of principals, teachers, and district administrators in the participating schools, successive waves of test-score data, and case studies conducted at 24 of the experimental schools. Because the study was not a strict scientific experiment, the authors note, the findings are not definitive. But they mirror, to some degree, those of others suggesting that schoolwide changes take three to five years to bear fruit and that fidelity to the program is key to success.
“When teachers are completing surveys under confidential circumstances, and they’re being asked about behaviors and not their perceptions, that data is pretty valid and reliable,” said Laura M. Desimone, an assistant professor of public policy and education at Vanderbilt University in Nashville, Tenn., who critiqued the study. “The real strength of this set of studies is that they were able to look at the full process, from adoption to effects, to sustainability,” she said.
Two-thirds of the experimental schools, the researchers found, sustained a relationship with the developers of their program models for more than three years; some endured more than a decade. One-third of the schools had ended their formal program relationships by the end of the 2003-04 school year.
Yet, even though they had dropped their programs, those schools carried on business as usual, albeit to a lesser degree. They continued many of the same practices they had picked up from the brand-name improvement programs they dropped, according to the study.
Another surprise in the study: The advent of the federal No Child Left Behind Act, which was signed into law in January 2002, prodded a few of the schools in the study sample to pick up the pace at which they were implementing their program models.
Among the smaller sample of case-study schools, Ms. LeFloch found that when schools were identified to be in need of improvement under the law, they speeded up implementation of their reform programs rather than choosing to discard them.
“I think it was the case that they already had a strategy to turn to,” said Ms. LeFloch, a principal research analyst at the AIR, “and I think that’s in keeping with NCLB’s theory of action.”
Forthcoming Studies
Schools in the comprehensive school- reform study sustained their efforts to varying degrees.
Nonreform never takes hold and ends quickly.
Momentary reform flourishes briefly but quickly dies or is overtaken by another reform.
Nominal reform establishes itself but in name only and is eventually abandoned.
Resident reform establishes itself but persists in name only.
Transient reform establishes itself, changes the system, and then passes away, leaving little evidence that it ever occurred.
Temporary reform establishes itself, changes the system, but gradually gives way to the forces of inertia and persists in name only.
Sustained implementation overtakes whatever preceded it so completely that it is institutionalized as the status quo and ceases to be “reform.”
Sustained implementation without a single sustained reform occurs when schools maintain their model, yet continue to adjust it, after the formal relationship with the developer has ended.
SOURCE: American Institutes for Research
The AIR study is among several federally financed research projects on comprehensive school reform coming out after the program effectively ended.
Researchers at WestEd, a San Francisco research group, have been tracking 1,000 schools across the country since 2002 as part of an evaluation of the $1.6 billion federal program. The findings from that $8 million study, however, have been in federal review for more than a year, according to Naida Tuschnet, the program director for evaluation research at WestEd.
Researchers at the University of Michigan in Ann Arbor have also been conducting a long-term study of various schoolwide models.
All those findings will fill out a research track record for comprehensive school changes that has been fairly spotty until now. While some programs boast strong evidence of success, most have yet to muster definitive proof they can turn schools around. The movement also suffered from widely publicized failures in some districts, such as Memphis, Tenn., that had been implementing the schoolwide programs on a wide scale.
In the meantime, many reformers have shifted their sights to the district, rather than schools, as the locus for change, sometimes and sometimes not incorporating comprehensive-improvement models into their reform mix. And the No Child Left Behind law, with its emphasis on discrete parts of the school curriculum, particularly reading and math, operates in some respects at cross-purposes with some of the most popular change models for schools.
Still, Mr. Aladjem said, “I do think there has been an indelible impact from the program, … and some of that would be lost in the demise of CSR.”
