Concerned about the credibility of federally financed education studies, Congress passed a law in the fall of 2002 that replaced the U.S. Department of Education’s top research office with the Institute of Education Sciences.
More focused and politically independent than its predecessor agency, the new institute also had a new mission: to foster “scientifically based” research that might offer definitive insights on how to improve schooling.
Now, four years later, all the pieces of that overhaul are in place, and critics and outside experts are giving it mixed, but mostly positive, grades.
The shift has meant that it’s taking a long time for federally funded studies to produce findings, but their methodological quality is better, analysts say. And while some researchers worry that certain kinds of studies—those that address messy policy questions, for instance, or that use different kinds of research designs—are getting short shrift, others see the agency’s newfound focus as long overdue.
“We’re in a better place than we were,” said C. Kent McGuire, who headed the research office under President Clinton and is now the dean of Temple University’s education school. “But it takes a while for the cake to bake, and what does the field do while it’s waiting for those studies to come out of the oven?”
Just as important, observers ask, will practitioners and policymakers know—or care—how to make use of the findings once they’re “baked”? Not unless the institute’s leaders pay as much attention to finding ways to make research useful as they do to making it more rigorous, some suggest.
“The quality of the research is improving,” said Gerald E. Sroufe, the director of government relations for the 22,000-member, American Educational Research Association. “Whether it will have more impact on schools remains to be seen.”
Dramatic Changes
The dawn of the Institute of Education Sciences marked the fourth overhaul of the federal educational research apparatus since the 1972 creation of the National Institute of Education. But longtime experts in the field say the changes this time around were especially sweeping.
To buffer the research operation from political interference, Congress said the agency should have a separate director serving a six-year-term—long enough to span more than one presidential term.
President Bush chose Grover J. “Russ” Whitehurst for the job, a former university-based researcher with a strong methodological bent. He had already been leading the agency, then known as the office of educational research and improvement, or OERI, for a year when Congress passed the Education Sciences Reform Act, the law that established the new institute.
Mr. Whitehurst quickly initiated a series of changes to the 185-person agency. For one, he de-emphasized the competitive grant program for field-initiated studies, which let researchers across the field propose their own study ideas, and replaced it with more-focused grant programs in cognitive science, early-childhood education, and other areas.
He shut down the far-flung Educational Resources and Information Center, or ERIC, clearinghouse system and created a leaner, more centralized ERIC to replace it. Mr. Whitehurst complained that the system, known as the nation’s electronic education library, had become like an old attic: creaky and inefficient. Instead of exhaustively archiving every piece research ever presented or written on a topic, the revamped system contains only published studies.
The federally funded regional education laboratories were revised, and the national research-and-development centers likewise were trimmed and given new missions and more-specific research tasks. The IES also put in place a new peer-review process that looks more like the procedure used at the National Institutes of Health to screen study proposals.
To nurture a generation of researchers capable of doing the kinds of studies Mr. Whitehurst had in mind, he established interdisciplinary pre- and post-doctoral fellowship programs. Begun in 2004, the programs so far have underwritten training for 177 researchers around the country. “There is hardly anything in place now that is like it was in previous regimes,” said Mr. Sroufe.
Yet the institute’s emphasis on scientifically based research, part of a wider movement in the federal government to transform education into an evidence-based field akin to medicine, was perhaps its most disputed innovation.
While “scientifically based research” encompasses a range of study designs, experts agree that pure experiments or randomized-control studies are the gold standard for determining whether an intervention really works. Though common in medicine, such studies are rarely done in education—partly because they cost so much, and partly because they are difficult, and sometimes unethical, to do.
Striking a Balance
From the start, researchers worried that randomized-control trials would squeeze other forms of research out of the federal-funding pie. But Mr. Whitehurst calls such concerns “outdated and uninformed.” While the percentage of experimental projects funded by the institute has increased since 2002, such studies account for just a third of its overall research portfolio, he says.
In comparison, nearly half of the grants the institute has awarded since 2002 pay for development activity—projects that synthesize research for use in real-world contexts.
“That’s a reasonable ratio from my point of view,” said Robert C. Granger, the chairman of the national board for education sciences, which Congress established to guide the institute’s work. “What the staff at the institute has been able to do is get very focused around a set of questions that leave some things in and some things out.”
Of all the grant proposals to the Institute of Education Sciences, nearly half concerned program development. The four categories of applications are:
*Click image to see the full chart.
SOURCE: National Board for Education Sciences
Given the Education Department’s small share of federal research dollars, it could never generate solid findings without a focused funding strategy, other experts say.
“If there’s not enough productivity coming out of the thing, it’s always going to be seen as a backwater,” said James W. Kohlmoos, the president of the National Education Knowledge Industry Association, a Washington trade group known as NEKIA.
Funded at $575 million a year, the institute ranks near the bottom among federal agencies in funding for research and development, according to the Washington-based American Association for the Advancement of Science. Of 16 large federal agencies that receive such funding, only the Agency for International Development, the Smithsonian, and the Nuclear Regulatory Commission receive less from Congress.
Still, some of the department’s concentrated research investments are poised to bear fruit, observers say. A case in point is its program of cognitive-science research, which is starting to yield insights on ways to help students retain more of what they learn. (“Cognition Studies Offer Insights on Academic Tactics,” Aug. 30, 2006.)
At the same time, though, some experts—Mr. Kohlmoos included—continue to believe the institute’s methodological and topical focus is too narrow.
“I am very much a proponent of what’s called scientific research in education,” said Ellen Condliffe Lagemann, a professor of history of American education and former dean at Harvard University’s graduate school of education. “On the other hand, I don’t think it’s the only way of knowing about education. I think we need support for other kinds of research, too.”
Ignoring Policy?
One such gap in the institute’s research portfolio may be in analyses of policy initiatives such as the federal No Child Left Behind Act, said Christopher T. Cross, a former assistant secretary for educational research and improvement under President George H.W. Bush.
“I’ve been on meetings on the Hill where people are very anxious to know how No Child Left Behind is being implemented, and they can’t get a handle on it because almost no research has been done,” said Mr. Cross, now the chairman of Cross & Joftus, an educational consulting firm based in Bethesda, Md., and Danville, Calif.
“We all want to know how various things are working, and why they are,” said Susan H. Fuhrman, the president of Teachers College, Columbia University.
“What’s happening to the big high schools left over when small high schools are taken out?” she added, citing another current initiative by way of example. “Now that’s a big policy question, and it’s hard to think of where in IES you could propose this.”
The Center for Policy Research in Education, or CPRE, a five-university consortium that Ms. Fuhrman helped found, was one of the long-standing federal research centers that lost out when the institute reshaped the centers’ missions.
Mr. Whitehurst, the IES chief, noted that the institute has continued to underwrite studies and national research centers, such as the Center on Teacher Incentives, based at Vanderbilt University in Nashville, Tenn., that address policy-related issues.
“It is true, however, that the challenges, both methodological and logistical, of doing policy work are formidable,” he said. “Our ability to address broad policy perspective with our $2 million to $3 million grants is limited, … so we nibble around the edges.”
Other experts suggest Congress and the Institute of Education Sciences missed a chance to build leadership, expertise, and outreach capabilities in key topics when they cut the size of the 10 national education research centers and refocused their missions on new topics and targeted research questions. Now funded at roughly $2 million to $5 million a year, the centers are on average about half the size they were under the old research office. “That’s not a conception of what a center ought to be,” said Frederic A. Mosher, a senior consultant for CPRE. “That’s a conception of what a big research project ought to be.”
Mr. Mosher and others also say the institute has not paid enough attention to studying how to repackage what researchers have already learned in forms that educators and policymakers can use.
“What Russ [Whitehurst] is missing is the full realization of how little capacity there is to do good design and development work in the education systems,” he said. “I think there’s even less capacity for that than there is for randomized trials.”
In the research office’s previous incarnations, that job largely fell to the regional educational laboratories, which are located across the country. Now, the labs, too, are charged with undertaking experimental work. But they also continue to serve the field by producing “fast response” analyses intended to give policymakers in those regions quick answers to pressing questions.
The What Works Clearinghouse, an $23.4 million federal project to create an electronic version of Consumer Reports for education research, was supposed to be the institute’s main vehicle for making research more useful and user-friendly.
Read the related story,
‘One Stop’ Research Shop Seen as Slow to Yield Views That Educators Can Use
But observers agree that the clearinghouse has not yet lived up to that promise. After four years in development, the clearinghouse as of last week had only highlighted eight interventions with “positive” or “potentially positive” evidence of effectiveness.
Mollifying Critics
But, among the agency’s three goals of rigor, relevance, and real-world applicability, rigor had to come first, Mr. Whitehurst says. Still, he said, “I’ve been acutely aware that a What Works Clearinghouse that identifies nothing is not particularly relevant for a school superintendent who is trying to make a decision tomorrow.”
To fill in the gap, the IES over the past six months or so has been stepping up its efforts to be more relevant to practitioners, tinkering with the What Works Clearinghouse, making plans to develop practical policy guides, and forming task forces on urban education that put researchers in touch with educators from the trenches.
The efforts appear to be mollifying some critics. One is Michael D. Casserly, the executive director of the Council of the Great City Schools, a Washington group made up of large urban districts. Last year, in an appearance before the national board for education sciences, he said the Education Department’s research arm produced studies that were “narrow and irrelevant” and lacked the answers that practicing educators needed to meet the demands of the No Child Left Behind Act.
“I’m increasingly hopeful that education research will prove useful to practitioners,” Mr. Casserly said. “It’s very clear to me that IES has listened hard to our concerns.”
Others have been harder to convince. “Each new version of what used to be called the NIE has become more and more political,” said Bruce Hunter, the chief lobbyist for the Arlington, Va.-based American Association of School Administrators. “They are more serving of the purposes of the political administration, and less serving of the purpose of American education.”
But most observers, in contrast, say Mr. Whitehurst’s management of the IES has been remarkably independent, given the context in which the agency operates.
They point out, for example, that Mr. Whitehurst was among a handful of federal officials who offered a dose of scholarly caution last year when U.S. Secretary of Education Margaret Spellings trumpeted historically high results for 9-year-olds on national reading and math tests as “proof that No Child Left Behind was working.”
“I think Whitehurst has done an admirable job of trying to stay out of making political statements when he has been pressed to,” said NEKIA’s Mr. Kohlmoos.
Mr. Whitehurst said the department’s top officials so far have not interfered in the institute’s work. “It is a negotiation,” he said, “and sometimes issues come up for which there are no rules, but I think that it has worked out OK.”
