At a national conference in 1994, two prominent education researchers--Howard Gardner and Elliott W. Eisner--started a lively debate that would go on for two more years. The question was: Should novels count as doctoral dissertations in education?
It’s the kind of question that might be dismissed out of hand in medicine, physics, or any of the other “hard sciences.” But in education, a field in which alternative forms of research proliferate like gnats in springtime, the debate has serious implications.
At its heart are differing views about what constitutes good education research. And how do you know it when you see it?
“There are so many alternative paradigms in education research that we’re not really agreed upon what knowledge counts and what’s good research,” says Penelope L. Peterson, the dean of the school of education and social policy at Northwestern University in Chicago.
Such questions are taking on a new urgency now, as education research and the federal system that feeds it come under the microscope at the national level. A host of commissions, advisory groups, former U.S. Department of Education officials, and other interested parties this year have sought to spell out what researchers ought to be studying, how they ought to conduct those studies, how the federal government can best support the whole enterprise, and how to ensure that findings get used in real schools and classrooms.
The implications of this soul-searching are important. How education scholars decide what ''counts’’ could ultimately raise or lower the field’s credibility with the teachers and policymakers increasingly hungry for advice. If education researchers can’t determine for themselves what good scholarship is, how can anyone else?
Mixing of Traditions
To some degree, all the social sciences, tied up as they are in measuring complex and unpredictable qualities of human behavior, wrestle with the same issues. Even medical researchers--often held up as exemplars by critics of their education counterparts--use different methodologies to point them to conclusions. They might, for example, draw on both lung-tissue samples and epidemiological studies to study the effects of smoking, notes Lorrie A. Shepard, the president of the American Educational Research Association, a Washington-based group representing 23,000 researchers.
But in education, the growth of so many different forms of research may be more pronounced because the field itself is an amalgam of academic traditions. Anthropologists, psychologists, economists, political scientists, sociologists, and historians--to name a few--all engage in education research. And the tools they use in their work range from questionnaires to field notes to videotapes.
Education researchers do surveys, longitudinal studies that track students’ progress over time, and meta-analyses--a statistical technique that enables researchers to summarize effects found across many studies. They might delve into national databases, or into schools themselves with strictly controlled experiments or quasi-experiments to find out if an educational intervention really makes a difference. They might sit for months in classrooms and produce narrative descriptions of teachers, schools, classrooms, and thinking processes at work.
They might even write novels.
Psychology an Early Model
The Gardner-Eisner debates were no pie-in-the-sky intellectual exercises. By 1994, the first year the two scholars took up the topic, Hofstra University in Hempstead, N.Y., had already accepted a novel as a doctoral dissertation in education, according to one scholar who took part in those debates.
“In a sense, this flowering of methods is a healthy consequence of the fact that we’ve outgrown some of the methodological straitjackets we lived in 30 or 40 years ago,” said Alan H. Schoenfeld, the AERA’s immediate past president and an education professor at the University of California, Berkeley. “But now I think there’s probably too much, in the sense that the number of methods we have now will either turn out not to be robust or will be eclipsed by other things.”
In earlier decades, education research predominantly modeled itself on psychological studies. Experiments were often conducted in laboratories, rather than real schools, and the results were always neatly quantifiable. Gradually, however, some researchers came to believe that the traditional methodologies weren’t giving them the whole story. The study of schooling, they reasoned, needed to take place in schools and other real-life contexts.
What is more, statistical studies examining simply whether or not an intervention worked only scratched the surface. If a new program or classroom technique succeeded, statistics couldn’t answer the big question: Why?
Notions of objectivity also came into question. Even though standard empirical or quasi-scientific studies may give the appearance of objectivity, some academics pointed out, the outcomes could easily be biased by the researchers’ choices of measures or comparison groups.
Thus was born a move, which became almost a flood during the 1980s, toward more descriptive studies, known as qualitative research. At the same time, interest revived in research done by teachers and in “action research"--where researchers themselves often are active participants in the changes they are studying. Why bother to try to be a dispassionate observer, some proponents of those approaches may have reasoned, when objectivity may be an impossible goal?
Good vs. Bad
Now the pendulum is swinging again.
“There’s beginning to be some serious skepticism about the movement in education toward qualitative analysis,” Marshall S. Smith, the Education Department’s acting deputy secretary, told researchers last month at a meeting in Cambridge, Mass., organized by the American Academy of Arts and Sciences. “People are beginning to realize that case studies are only useful when they’re well-grounded in theory.”
Part of the problem has been that descriptive studies, some of them involving a single school or classroom, don’t carry much weight with policymakers. And while some qualitative studies have drawn accolades for their elegance, their detail, and their ability to shine a spotlight on what really goes on in some classrooms or in the minds of learners, others add little to the field’s collective knowledge.
“It’s not qualitative versus quantitative,” Henry M. Levin, an education professor at Teachers College, Columbia University, told the Cambridge gathering. “It’s good research versus bad research, and the qualitative field opens up a lot more possibilities for bad research.”
Even when qualitative research is good, some scholars note, there is no mechanism that enables such findings to accumulate so that they become more than anecdotes and isolated stories.
An example of the type of solid, experimental work that policymakers are demanding is a noted Tennessee study on the effects of smaller classes. The study, begun in 1985, is also significant in that it set out to examine a question with direct and far-reaching policy implications.
With more than $12 million in total funding from the state legislature, researchers from the Student/Teacher Achievement Ratio, or STAR, project conducted a classic experiment in which thousands of students from 79 schools across the state were randomly assigned to either small classes of 15 to 18 students or classes of 22 to 25 students.
The researchers found that students from the smaller classes outscored their counterparts in every year of study. Those students held on to their academic edge years after returning to larger classes. ( “Tenn. Class-Size Study Finds Long-Term Benefits,” May 5, 1999.)
All but ignored when it was first published widely in 1990, the study has since drawn the eye of state legislators and President Clinton. And the findings have become powerful ammunition in the movement to reduce class sizes in the early grades.
Cost a Factor
What made the STAR study so influential, experts say, was its use of a random-assignment research methodology that reduced the risk of bias and made it possible to look across different school populations.
If the Tennessee researchers could pull off that kind of rigorous, scientific experiment on such a large scale, proponents of such experimental methods say, why can’t other education researchers?
But conducting reliable random-assignment studies also presents some practical challenges. A big one is cost--a major concern in a field that is widely considered to be underfunded.
“The Tennessee study is a good example of the expense involved in doing [such research] on a scale that’s credible,” says the AERA’s Ms. Shepard, who is a professor of education and research methods at the University of Colorado at Boulder. “It’s clear that doing this in education means bigger-scale investments.”
Another practical obstacle is that school districts sometimes find it hard to refuse parents’ demands to include or exclude their children in experimental groups. Accommodating such wishes can dilute a study’s strength.
“The chances for controlling for what we should be controlling for are pretty slim,” says Lauren B. Resnick, a co-director of the Learning Research and Development Center at the University of Pittsburgh. “There are problems with that approach, but there are certain questions that can be addressed that way.”
And certain questions that can’t. One example Mr. Schoenfeld, the former aera president, cites is studies on “metacognition,” or people’s awareness of their own learning strategies.
Standards for Research?
The bigger issue for many researchers is defining and maintaining quality in the midst of all the ferment in their profession. And they are divided over whether setting common standards for education research would help.
“If anyone right now were to say, ‘These are the standards, and they’re carved in stone,’ they’d wind up setting the field back rather than moving it forward,” Mr. Schoenfeld says. But, he adds, within particular academic disciplines, “there are serious questions to ask about what kinds of claims you can make and on what grounds.”
To some extent, research standards already exist. Academic journals act as gatekeepers when they send prospective articles out for peer review. Reviewers also screen the funding proposals that come before the Education Department’s office of educational research and improvement.
But one recent federal report questions whether the OERI’s peer-review panels are doing an adequate job. The report by the National Educational Research Policy and Priorities Board notes that while most peer reviewers appear to be qualified for the task, a few panels that screened proposals for research competitions held in 1996 and 1997 had few or no members with any expertise in research. (“Panel Urges Tighter Review of Research-Grant Proposals,” March 24, 1999. )
The Education Department is considering creating standing review panels whose members would work together over longer periods of time, learn from one another, and reach consensus on some common evaluation standards.
“There’s no getting away, at the end of the day, that quality is to some considerable degree in the eyes of the beholder,” says C. Kent McGuire, the department’s assistant secretary in charge of the OERI. “But I think these things need to be publicly discussed, and the field needs to worry about quality.”
Different Notions of ‘Research’
So, is a novel a valuable enough piece of research to qualify as a dissertation? Or is it just a piece of fiction, with little value for those trying to build better schools?
After two years of talking, neither Mr. Gardner nor Mr. Eisner had moved any closer to agreeing on whether a novel could be a dissertation.
Mr. Gardner remained skeptical."Not only is art not true, it makes no effort toward truth,” the Harvard University scholar, best known for his “multiple intelligences” theory, said the last time the two debated the topic, in 1996. “It seems to me the essence of research is effort, however stumbling, to find out as accurately as you can what’s happening and then to report it accurately.”
For his part, Mr. Eisner, of Stanford University, conceded that while a novel might not be an appropriate vehicle for every sort of research imaginable, it could create deeper, more empathic understandings in its readers and convey some things that facts cannot reveal.
“If you want to know what it feels like to be an associate professor when you’re 54, ‘Who’s Afraid of Virginia Woolf?’ is a good way to find that out,” he said in that exchange three years ago. “Exploring new forms of inquiry is part of trying to create an intellectual climate in schools of education where those contributions are not excluded because they don’t match the existing categories.”
