Is ‘Evidence’ Enough?

—Dave Cutler

There is much talk these days about “evidence based” policy in education and elsewhere. But empirical evidence alone, valuable as it is, can never be enough to determine what policy is best. One reason for this is that in the social sciences, even the best available evidence almost always leaves room for doubt and uncertainty. More fundamentally, however, the right thing to do always depends on values as well as facts, and values need systematic study just as much as facts.

Consider some of the ways in which evidence underdetermines policy. Although there is a movement for school choice, another for standards and accountability, and another for small schools, each of those movements unites around a broad idea, not a single specific reform. School choice comprises charter schools, vouchers, and open enrollment; “small schools” comprises small schools, schools within schools, and the Harry Potter-ish “smaller learning communities.” Each state has its own charter legislation; the Milwaukee, Cleveland, and Florida voucher systems are all quite different from one another. My guess is that there are as many different variants of smaller learning communities as there are schools adopting them.

But the success or failure of any reform depends on a myriad of contextual factors. Small-scale studies of particular instances of a reform idea can be sensitive to context, but finding out that a reform produces certain results in one place doesn’t tell us what results it will produce elsewhere. Large-scale studies of several variants of a reform may tell us whether to bet on it when it is introduced in a new context. But the results range over a wide variety of reform designs in a number of different contexts to which the study is not sensitive. Knowing that 10 variants of some reform have a particular consequence on average doesn’t tell you whether it will have that consequence in your circumstance.

In addition, depending on the level at which you are contemplating reform, you face the “scaling up” problem. Suppose you are a state superintendent, and the research tells you that pretty much wherever districts have invested heavily in professional development, it seems to have paid off. Should you do the same? What the microlevel studies do not usually tell you is the extent to which the payoff is the result of improved quality in the existing teachers, and the extent to which the investment has simply attracted and retained the better teachers in the regional labor market. To the extent that the former mechanism is at work, the strategy is likely to work for the state; to the extent that it is the latter, the prospective payoff at the scaled-up level is low.

So at best the would-be reformer is deciding under uncertainty. All these problems could be addressed by more and better (and more expensive) empirical studies; and I’m sure my colleagues in schools of education would welcome more of those.

But suppose that you had perfect information about the prospective effects of every reform option. Even this would not tell you what to choose, absent reflection on the values that should inform schooling. Suppose, for example, that a reform raises test scores for the whole population, but raises the higher test scores by more than it does the lower test scores; a rising tide that increases the gap. Is this better than an alternative that raises the lower scores but lowers the higher scores (an uneven tide that reduces the gap)? Or compare a reform that improves math scores across the board, but at the cost of schools’ de-emphasizing pedagogical goals in which success is less easy to measure, like responsible citizenship, or a lifelong love of literature. A nation of highly numerate but economically self-interested voters is not clearly preferable to a nation of good-willed and reasonable deliberators who have difficulty with calculus.

Skeptics might think these are “philosopher’s examples”—interesting in the abstract, but irrelevant to the real world, where there are no real trade-offs. I doubt this. A recent study of schooling in Amsterdam found that low-income Turkish and Moroccan students achieved better academic results when concentrated into the same schools than when integrated into schools with native Dutch students—a prima facie case of a trade-off between closing the achievement gap and improving social solidarity. Closer to home, consider Nathan Glazer’s comment in his review of Jonathan Kozol’s 2005 book The Shame of the Nation:

“To be sure, the case for both [racial] integration and equality of expenditure is powerful. But the chief obstacle to achieving these goals does not seem to be the indifference of whites and the nonpoor to the education of nonwhites and the poor. … Rather, other values, which are not simply shields for racism, stand in the way: the value of the neighborhood school; the value of local control of education, and, above all, the value of freedom from state imposition when it affects matters so personal as the future of one’s children.”

Glazer claims that in our world, when considering real reform ideas, these values come into conflict, and it is hard to believe that he’s wrong, even if you ultimately side with Kozol’s way of making the trade-off.

Surely policymakers just apply their own values to the trade-offs? Well, maybe they do, but doing so is rather like making empirical judgments on the basis of their own experience. These are values that really matter, and there are better and worse ways of adjudicating between them in the real world of conflict. Careful philosophical reflection is needed, and this is something that requires research. Indeed, there is a good deal of philosophical research on the liberty-equality trade-off, and my reading of it is that it points strongly in favor of Kozol’s way of deciding, if feasibility constraints allow.

So policymakers need to think carefully about the values in play when they approach reforms, and to take a particularly careful look at trade-offs between values. Philosophical research can help.

Now, a warning. Philosophical research does not determine how a policymaker should act, any more than empirical research does. It is almost useless in the absence of the relevant empirical research. Even in combination with the empirical research, it is not usually fully determinate, because a policymaker has to weigh all the relevant value considerations in the circumstances, and it is a rare piece of philosophical research that considers all things. A policymaker who slavishly took up a particular philosophical perspective would be as bad as one who considered only the empirical research that supported his case.

But I’d hazard a guess that fewer policymakers use philosophical research than use empirical research in making their decisions. The contrast with health policymakers is interesting. Health policymakers and medical practitioners are keenly aware not only that their choices involve making trade-offs between different values, but also that they are not able to pull a value off the shelf. They understand their obligation to scrutinize rigorously the ethical issues in question. Every hospital has a staff ethicist and an ethics board. The National Institutes of Health and the World Health Organization have several ethicists pursuing original research and consulting with decisionmakers.

Despite the fact that ethical questions arise all the way down the educational structure—from the federal government choosing whether to allocate budgets to children from low-income backgrounds or with disabilities, through school districts deciding whether to institutionalize acts of patriotic observance, to teachers deciding whether to inform parents of their suspicion that a child is experimenting with drugs—there isn’t a public school, public school district, or state department of public instruction with a staff ethicist.

Educational policymakers bear almost none of the blame for this state of affairs. Social-science academics view questions of value with suspicion; quantitative scholars try to avoid value judgments, while qualitative scholars treat value judgments as inevitable biases. Philosophers, by contrast, have by and large failed to involve themselves in the real world of education policy and research. They are trained to ignore questions of feasibility, which bedevil well-willed policymakers and constrain the extent to which important values can be realized. So their findings are rarely presented in ways that are readily interpreted and usable by social scientists and policymakers.

Medical and health-policy ethics did not emerge as a discipline relevant to and informative for policy and practice simply because it was wished up by the profession. Philosophers had consciously to learn about the medical world, and, while continuing to do philosophical work at an abstract level, simultaneously deliberate about the concrete problems that the profession faced, in a way that would be accessible to nonprofessional philosophers.

Policymakers need resources to help them think through the questions of value as rigorously as possible, and they need empirical research that enables them to choose the variants of the reforms in play that have the best prospects for realizing the most important values. These resources do not come out of thin air, but, as the case of medical and health-policy ethics suggests, through an ongoing dialogue involving all three parties.