In a quiet corner of the University of Chicago, Shazia R. Miller is putting the finishing touches on what she calls her “little people.” The small, human- shaped figures on a bar chart represent students as they go through high school. Those at risk of academic failure are orange, while those who graduate are green. Dropouts are red.
The simple but striking illustration, created by sifting through a mountain of data on student mobility and performance, shows what actually occurred to students from a real Chicago school. Here at the Consortium on Chicago School Research, producing high-quality studies to meet the needs of local educators and policymakers is the guiding creed.
“You could send something out to a journal and reach a small sphere of people,” said Ms. Miller, an associate director with the consortium. “But it’s not the same as having this kind of impact.”
Amid a new push in education for more scholarship focused on the immediate concerns of practitioners, the work of the consortium represents an unusual example of “usable research” focused on one place. In a steady stream of reports, its researchers dissect Chicago district policy and paint detailed pictures of the conditions inside schools.
From teacher professional development to magnet schools, little that’s been done in the name of school improvement in the city has escaped the consortium’s scrutiny. Researchers have pointed out the shallowness of the content taught in many Chicago classrooms. And they’ve shown the important link between students’ academic success in their freshman year and whether they graduate from high school.
“They are not ivory-tower researchers,” said Arne Duncan, the chief executive officer of the 437,000-student Chicago district. “These are people who roll up their sleeves and get out to the schools and conduct research that is applicable to real situations.”
Accomplishing that goal has meant casting aside the usual relationship between researchers and their subjects. For the consortium, the local school community is both the object of analysis and its closest partner. Its members regularly cull the advice of education advocates and school officials in deciding what to investigate. And they keep them in the loop throughout the study process.
At times, it’s an uneasy marriage. The education leaders who help shape the consortium’s research agenda don’t always agree with the results. Its studies have fueled heated debate on a host of questions, such as how the city’s dropout rates should be calculated, and the wisdom of Chicago’s policy of requiring students to repeat grades if they don’t meet certain standards.
During the administration of former CEO Paul G. Vallas, who is now the head of the Philadelphia public schools, tensions ran so high that some feared a breakup with the consortium.
Anthony S. Bryk, the University of Chicago education professor who led the push to form the consortium, called that friction “the price of relevance.” What matters to him, he said, is not whether everyone agrees about the consortium’s conclusions, but that its studies raise the debate on school improvement in the city by grounding it in hard evidence.
“We’re informing the reform conversation,” said Mr. Bryk, who is now on sabbatical at Stanford University. “We’re constantly thinking about the big ideas that people can take away from our work that would make them think better about what they do.”
Setting the Agenda
In large measure, the Consortium on Chicago School Research owes its existence to the 1988 state law that sought to greatly decentralize the governance of the city schools. The measure gave broad new authority to decide issues of personnel, budget, and instructional programs to a local council of parents and educators at each school.
To many observers, it seemed obvious that such dramatic change warranted a major research effort to gauge its effects. Such a project might help schools use their new powers wisely, the thinking went, by providing new data on what worked and what didn’t.
The challenge was to find a way to give various stakeholders in the city the chance to help shape and inform the research agenda, while still producing rigorous, credible studies that didn’t appear biased toward any group’s perspective.
The answer was the consortium’s steering committee, forged in a series of meetings that Mr. Bryk, at the request of district officials, convened in 1990 with local school leaders, education advocates, and professors from area universities.
With 24 members, the panel includes academics, district leaders, union officers, and representatives of many of the city’s school-related interest groups. The body reflects a wide array of views.
Donald R. Moore, the executive director of the group Designs for Change, for instance, has long criticized the city’s efforts to end “social promotion” by holding back students based on test scores.
Meanwhile, Barbara Sizemore, a retired dean of DePaul University’s education school, has been one of the strongest supporters of the approach.
“There have been times when passions have been very high,” said Alfred G. Hess Jr., an education professor at Northwestern University who also sits on the panel. “But the steering committee has weathered those.”
While members of the panel offer advice on which studies to carry out, and even on how, the researchers retain ultimate control over the reports they produce.
The committee also doesn’t hold the consortium’s purse strings. With a staff of about 20, the consortium gets most of its $1.8 million annual budget from local philanthropies, primarily the John D. and Catherine T. MacArthur, Joyce, and Spencer foundations.
Still, the steering committee plays a crucial role in making sure the research staff stays focused on what matters most to Chicago’s education community, said Penny Bender Sebring, the consortium’s most veteran research associate.
“It ensures that the research we do is going to be a lot more useful, and that it’s going to be some answer to questions that people locally have,” she said. “It just keeps our feet on the floor.”
Mining Tools
Teaming up with the district gives consortium researchers access to an unusually rich source of information. To date, the group has amassed records on some 1.4 million current and former Chicago public school students, including test scores, attendance rates, and grade transcripts. That trove of data is what allows the consortium to follow what happens to students as they go through the system.
Dorothy Shipps, a former consortium researcher who is now a professor at Teachers College, Columbia University, points to a consortium study on truancy to explain why keeping close track of students is important. Mining the extensive database showed that, in fact, many students considered truant at one school had actually moved to another. So part of the truancy problem was really a mobility one.
“We came up with very interesting findings you could never get if you didn’t have this longitudinal database,” Ms. Shipps said. “Kids were moving among schools within a two- or three-mile radius.”
Some of the consortium’s most powerful findings have emerged by linking Chicago students with the district’s own survey results. Since 1991, the research group has carried out periodic polls of Chicago principals, parents, teachers, and students. Items on the questionnaires are meant to probe the quality of a school’s climate, leadership, staff training, and instruction.
Those surveys are what allow the researchers to track how major education policies play out in the schools. For instance, three years after the 1988 decentralization law, a consortium report showed that some 40 percent of the city’s elementary schools had well-conceived plans to raise student performance. But many others had just tacked on an assortment of unrelated programs—what the study memorably called “Christmas tree” schools.
Mr. Hess of Northwestern University says the snapshot made two important points: Despite some earlier predictions to the contrary, large numbers of schools were making good use of the new authority granted to them. Others clearly needed more guidance.
“Here was something that said some schools are doing better, and if some are, why aren’t the rest?” Mr. Hess said. “It kind of created the possibility for improvement.”
To help individual schools see what they most need to work on, the consortium has also created a kind of report card based on its surveys. Each school that returns enough questionnaires to make for a statistically valid sample gets a document that includes tallies of its own answers.
From the responses, the consortium calculates a series of indices meant to show the school’s strengths and weaknesses in certain critical areas, such as parental involvement, student discipline, and professional development.
The reports aren’t used for accountability purposes. In fact, to ensure candid responses, they aren’t even given the district’s central office. Instead, they’re used by school leaders and staff members to pinpoint problems.
Barbara J. Martin, the principal at Holmes Elementary School, for instance, said the survey responses helped her see the need for improving communication among her staff. When she received her first consortium report, she said, “we were just shocked to find that we didn’t do a lot of talking to each other.”
‘In the Middle of It’
By holding a mirror up to the district and its schools, however, the consortium often finds itself at the center of controversy. Said John Q. Easton, who was named executive director after Mr. Bryk left for Stanford: “We’re sort of in the middle of it, all the time.”
Especially contentious has been the consortium’s work focusing on the accountability measures put in place under Mr. Vallas, the district’s former chief executive.
Sent to run the schools by Mayor Richard M. Daley after the state gave the mayor control of the school system in 1995, Mr. Vallas established cutoff scores on standardized tests to determine whether students could go on to the next grade. The policy required students in danger of being held back to attend summer school, after which they’d be retained if they still didn’t meet the standard.
In general, the consortium’s studies have shown that the strategy has had mixed results. Although the summer school program has helped many students meet the requirements for promotion, the boost in academic achievement appears short-lived. And while the dropout rates of students who were held back haven’t spiked—as some critics of the effort had said would happen—they haven’t plummeted.
The findings contributed to a public debate on whether Mr. Vallas’ highly publicized plan was working, straining relations between the district and the research group, according to many members of the consortium’s staff and steering committee. Some even worried that the district might cut the consortium off from the student data that is its lifeblood. But it didn’t.
“There were concerns about the relationship breaking off,” said Paul D. Goren, the acting president of the Spencer Foundation. “There were efforts internally and externally to patch things up.”
Mr. Vallas now downplays the tension. Often, he said, his dispute was more with what he saw as a particular spin in the consortium’s press releases and the commentary in its reports—not with its findings. In fact, he has plans to replicate the consortium in his new district, Philadelphia, by starting a similar independent, privately supported research group dedicated to studying that city’s schools.
“I always viewed the consortium as an asset more than a liability,” Mr. Vallas said recently, “even during our own version of the ‘days of rage.’ ”
There is no doubt that the consortium enjoys good relations with the Chicago schools’ current leadership. Many of the policy initiatives of the current district chief, Mr. Duncan, appear to address concerns raised by the research group.
Mr. Duncan, for example, has broadened the criteria used to hold schools accountable for their results, adding factors meant to gauge students’ improvement, not just their performance.
And when he unveiled a major report last summer outlining new policy directions for the school system, it stressed many consortium findings, such as the importance of instructional “program coherence” to ensure that students aren’t taught the same things grade after grade.
The Chicago consortium is a rare example of scholarship that has been able to shape policy while also studying it, said Michael W. Kirst, a professor at Stanford who has observed the research group from afar.
“They’ve got to be critical and influential,” he said. “And they’re able to do that because of the depth of the research they do.”
Coverage of research is underwritten in part by a grant from the Spencer Foundation.
