Experts worry about the influence of advocacy-based education studies.
The Education Policy Studies Laboratory, a study center at Arizona State University in Tempe, seemed to be gearing up for a fight when it circulated a report last fall attacking the work of the Mackinac Center, a Michigan-based think tank.
“Mackinac Center work is often of low quality, and because of this it should be treated with considerable skepticism by the public, policymakers, and political leaders,” concluded the study, which was written by two academics. The authors based their conclusion on an examination of 11 years of education-related studies by the think tank, which was founded to advance free-market principles.
The words were harsh—especially in the generally cautious and genteel world of education scholarship. But the mission of the policy-studies lab, founded by a feisty education professor at the university, is to provide a counterpoint to what some see as a proliferation of education reports coming from think tanks, like Mackinac, that were founded to advocate particular points of view.
Not to be outvoiced, the Mackinac Center countered with a statement pointing out that the Arizona center was not without some potential sources of bias of its own. For one, the study was paid for by a foundation created by Michigan’s largest teachers’ union, a longtime foe of the Michigan think tank. What’s more, the Mackinac Center noted, the authors had failed to uncover any outright errors in the center’s work. And, while it’s true that the Mackinac Center’s reports do not undergo the same degree of peer scrutiny as articles written for academic journals, neither did the critique issued by the Arizona center.
Though unusually pointed, the back-and-forth between the two centers laid open a common—and, some say, growing—dilemma for policymakers and educators trying to puzzle out the mounting heap of reports and studies on education that cross their desks. Whose advice is right and whose is wrong, the more determinedly neutral among them might ask. Who is the dispassionate scholar, and who is the advocate?
Education research that may be colored by advocacy—and the difficulty that readers of research face in distinguishing it— are causing concern for some researchers.
“To bring about educational change, I believe advocacy is required. And I would hope that advocates would look to educational research as one source of the basis for their advocacy,” says Andrew Porter, the president of the American Educational Research Association, a Washington-based group representing 22,000-plus researchers worldwide. “In education research, however, I think there’s no room for advocacy.”
Porter was concerned enough about the issue to call for a panel on the subject at the AERA’s annual meeting in New Orleans this spring.
‘What these conservative think tanks have done ... is coupled their documents with a very sophisticated publications and media strategy.’
Researchers disagree over whether education studies with a point of view are growing in number. After all, they say, a vein of advocacy has always run through the social sciences— and even in the so-called hard sciences, such as medicine.
“There’s no such thing as being value-free,” says Bruce J. Cooper, an education professor at Fordham University in New York City who has written on the subject. “Why would researchers choose a topic unless it was something they felt strongly about?”
Scholars agree, however, that advocacy research is not just a product of think tanks or organizations with a cause. Professors can also become advocates when they become convinced that they have hit upon a policy idea or a program that can make a difference for schools and children.
Experts say the problems arise when that bias begins to elbow out objectivity in researchers’ work, or when studies that are skewed to a particular outcome masquerade as dispassionate analyses.
“I think we’re on contested ground in public education now,” says Peter W. Cookson Jr., an associate professor of education at Teachers College, Columbia University, and the main author of the study on the Mackinac Center. “I think there are a number of organizations that are writing from their own perspective, and that’s OK. The issue arises when that advocacy raises the level of the publication to some definitive study that ought to be relied on for policymaking.”
“What happens is a particular fact gets into the press and then becomes kind of the conventional wisdom,” he adds, “and that affects the money flow.”
The effectiveness with which many think tanks are getting their points across to policymakers on education issues has, in fact, prompted some of the attention that researchers are giving the subject now.
In Michigan, for example, the Mackinac Center is widely credited with influencing the debate that led to new laws allowing charter schools and easing restrictions on school boards’ privatization of some school-related services, such as transportation or janitorial work. The center receives most of its funding from private foundations.
“Before the Mackinac Center researched and published about the benefits of allowing parents to choose the safest and best schools for their children, charter schools weren’t even on the table,” says Joseph Lehman, the executive vice president of the center, which is based in Midland, Mich. For all its work, the center gets an average of 1,200 mentions a year in the state’s print media, Lehman says.
“Policymakers don’t read academic journals,” he says. “If they did, we would publish our work there.”
Whose advice is right and whose is wrong, educators ask.
One reason the more advocacy-oriented research organizations are making a difference, observers of the field say, is that they can feed much- needed information to legislators, governors, and other policy leaders in a fraction of the time it takes for traditional researchers to publish their studies in academic journals. For some journals, the process of reviewing, editing, and publishing a study can take up to a year, while think tanks can get a report into print in a few months.
“One of the huge problems for education research is that it takes time,” says Ellen Condliffe Lagemann, the president of the Spencer Foundation, a Chicago-based philanthropy that supports education research, including coverage of the subject in Education Week. “You cannot produce instant knowledge.”
Often, what slows the publication channel in an academic journal is a lengthy peer-review process designed to provide assurance that the study meets scholarly standards. Typically, researchers say, articles being vetted for publication in serious academic journals are reviewed by three scholars who have either worked in the same field or used the same research techniques. The reviews are also “blind” in the sense that the critics don’t know whose work they are reading. For reviewers working in the same field, though, it’s often easy to guess.
The review processes at think tanks, by comparison, tend to vary, although “blind” reviews are much rarer. Books being considered for publication by the Brown Center on Educational Policy at the Brookings Institution, a long-established Washington think tank, are reviewed by five scholars, according to Tom Loveless, the center’s director. Two of the reviewers are resident scholars, and three are nonresident scholars who also sit on the center’s board.
At the Mackinac Center, says Lehman, the reviewers come from a 40-member board of scholars, some of whom are affiliated with Michigan universities and some of whom may be lawyers or business people.
Reports published by the Washington-based Thomas B. Fordham Foundation are scrutinized by Chester E. Finn Jr., the foundation’s president, and Diane Ravitch, one of its founding scholars.
“I don’t have much use for peer review in education research,” says Finn, who served as the U.S. Department of Education’s assistant secretary for research in the Reagan administration. “By selecting the peers, you’re preordaining the outcome of the review—and that’s the short, dirty secret of education research.”
“We’re engaged in an argument,” he adds. “We’re not refereeing an argument.”
Think- tank research in education has also generated concern in some quarters because of what some see as a growth in studies with right-of-center viewpoints.
Jack Jennings, who served as a top Democratic staff member for the House Education and Labor Committee for three decades, from 1967 until the Republican takeover of the House in the 1994 elections, says he has seen the ideological shift in the hundreds of reports and studies that cross his desk.
In the 1960s and 1970s, he says, conservatives complained that many education-related reports smacked of liberal biases. Now, “the latest development is that the very conservative elements of society have organized to create their own think tanks and cadres of researchers and evaluators who issue reports that will represent more of their point of view,” says Jennings, who now directs the Center on Education Policy, a Washington think tank that tries to hug the ideological center.
‘Policymakers don't read academic journals. If they did, we would publish our work there.’
It’s hard to tell if Jennings is right. Mackinac, for example, belongs to a network of 40 regional think tanks around the country that share a free-market approach to public policy. And a number of conservative-leaning or free-market think tanks with more of a national audience, such as the Heritage Foundation, the Manhattan Institute, the Cato Institute, and the Fordham Foundation, have all ratcheted up their focus on education issues since the late 1980s. But some more liberal-minded education groups have grown in influence, too.
“What these conservative think tanks have done over the last 20 years is coupled their documents with a very sophisticated publications and media strategy,” says Alex Molnar, the education professor who founded the Education Policy Studies Laboratory at Arizona State. University professors, in comparison, have no well-organized means for injecting their findings into the public debate, says Molnar, who co-wrote the report on the Mackinac Center.
The policy-studies lab is part of the university but some of its projects are financed by grants from various groups, including the Michigan Education Association, Consumers Union, and the Rockefeller Foundation.
To counter the work of ideological-leaning think tanks, Molnar is drawing together scholars from around the country who can respond to studies quickly, while they are still being played in the news media.
For the Mackinac study, Molnar and Cookson rated 22 center studies against social science research guidelines drawn up by the American Psychological Association for publication in its journals. The studies were divided into two groups: original studies and interpretative analyses of existing work. The former was given an average rating of 1.80 on a scale of 3; the latter scored an average of 2.09 on the same scoring system.
But Lehman of Mackinac points out that producing research for publication in academic journals was never the center’s point.
“We wouldn’t get published in poetry magazines, either,” he says. “We could perhaps make some marginal improvements in our peer-review process, but the cost of that would be risking the timeliness and relevance of our publications.”
A Fine Line
While the Mackinac Center is upfront about aiming to promote free-market answers to problems, advocacy is often harder to discern when it comes from individual scholars.
“There’s always a fine line between believing in one’s scientific results and wanting to disseminate those results because the evidence is there, and starting out with a strong preference and using research creatively and selectively in support of that preference,” says Michael J. Feuer, the executive director of the Center for Education at the National Research Council. The council is the main operating arm of the National Academy of Sciences, which was chartered by Congress in part to sort out competing scientific advice.
One education researcher who has encountered some criticism from colleagues in the field for his efforts to promote his own work is Robert E. Slavin, the co-developer of Success for All, a comprehensive improvement program used in roughly 1,800 elementary schools. Slavin, the director of a federal education research center based at Johns Hopkins University in Baltimore, and his wife, Nancy Madden, formed a nonprofit foundation to support dissemination of their popular program.
Researchers have taken criticism for taking their work straight to the press, rather than letting it percolate more slowly through the traditional academic channels.
“We’re doing dissemination because we feel we’ve developed something effective,” Slavin says. “We feel we have a responsibility, if we’ve found something effective, to make it available to large numbers of children who are in risky situations, and that exposes us to criticism from people who are criticizing our motives.”
“They feel that the research is window dressing for the foundation,” he says, “rather than our point of view, which [is] the research led us to disseminate something genuinely beneficial.”
Other researchers have come in for criticism by taking their work straight to the press, rather than letting it percolate more slowly through the traditional academic channels, or by making broader interpretations than their data warrant.
As public intellectuals, education researchers also get asked for their opinions by reporters, Loveless of the Brookings Institution observes. “In order to communicate with reporters, students, and the outside world, scholars have to simplify their findings, and sometimes they oversimplify,” he says. “Those oversimplifications sometimes seem biased.”
Yet not all advocacy is bad, scholars say, and debate between experts who forcefully advocate their positions can be a good thing for education research.
No Neutrality
“I don’t think any researcher goes into research questions completely neutral,” says David C. Berliner, an Arizona State education professor. “If research is done well and meets peers’ requirements, then it’s fine to take it seriously. But what I’m saying is ‘take it seriously,’ rather than take it as truth.”
“My rule of thumb is that the more controversial the issue, the more likely people are to take sides,” says Cooper of Fordham University. “When I see that, I know it’s important research.”
Part of the problem, he says, is that politicians may unwittingly bring on academic debates when they they fail to build mechanisms and money into new programs so that they can be properly evaluated from the beginning.
“It’s too late three years later to evaluate something like Head Start, because no control groups were built in,” he says.
Even for fellow academics, however, it can be hard to tell when bias has crept into a study.
“It’s like pornography,” quips Feuer of the National Research Council. “You just know it when you see it.”
The difficult work of sorting out the substantive study from the biased report may be too much to expect from policymakers, says one professor.
One way to tell, says Berliner, is to find out if the researchers have released their data so that others can check and reanalyze their work.
Nel Noddings, the current president of the National Academy of Education, an elite group of scholars in the field, also says serious researchers are careful to address all the arguments that cast doubt on their findings.
“Sometimes, researchers just pick out the goofiest arguments on the other side, address them, and say their critics don’t have anything,” she says.
A look at the research question that a study poses, Loveless says, might also show that the study was meant to have a predetermined outcome.
Molnar says a hallmark of high-quality research is whether the methods used are appropriate to the questions asked.
But doing the difficult work of sorting out the substantive study from the biased report may be too much to expect from policymakers and practitioners right now, he says.
“My view is that it’s already a step forward when they know there’s an argument,” Molnar says. He says his own goal in the short term is just to provide decisionmakers and readers of research with another side.
“What I can say is, ‘You have this document, and now you have our documents,’” he says. “Use your own good judgment and common sense, and sort it out as best you can. Right now,” he adds, “there ain’t much sorting out.”
The Research section is underwritten by a grant from the Spencer Foundation.
