I heard the other day that Andreas Schleicher had been invited to address the NAEP governing board. Long overdue, I thought, but better late than never. The design of PISA makes it a much more valuable tool than NAEP for sorting out which policies and practices are associated with high student performance and equity.
Oops! I just said “sorting out,” and that clearly means that I am guilty of the classic beginners’ error in statistics: mistaking correlation for cause. Sure, PISA was designed to make it possible to correlate a wide range of background variables with student performance. But, surely, Marc, you know that correlation is not cause. The sun rises every morning and you have breakfast every morning, but the sun does not cause you to have breakfast. So what good is correlational data?
Well, we all “know” that more time on task yields greater student achievement. Suppose I were to tell you that the PISA data show that that more time studying is, in fact, negatively correlated with performance? Would you dismiss that finding because no one had proved that the increase in time caused the diminished achievement?
There’s a corollary to that point. Since we all “know” that more time on task means more learning, would you be surprised to learn that one of the top performers--Finland--expects its students to be in school for a good deal less time than is customary in the United States. Would that pique your curiosity? Would you, perhaps, begin to wonder whether time is the issue, or whether how that time is used might be more important? Or would you say that because no one had proven that less time causes better learning, we should assume that more time causes better learning? That would be a very strange conclusion.
How about money? We all “know” that more money means higher student achievement. Suppose I were to tell you that the PISA data show that when nations spend less than $50,000 on a student’s entire education from first grade through the end of high school, there is a very close correlation between the amount spent and that student’s achievement, but, when more than that is spent, there is very little correlation between spending and student achievement? One could conclude that how the money is spent is more important than how much is spent. Or would you say that, because no one has ginned up a study to show beyond a shadow of a doubt that, above the $50,000 mark, how the money is spent is more important than how much is spent, the PISA data mean nothing? Wouldn’t you begin to wonder whether you really “know” that more money “causes” higher achievement?
It may be “obvious” to you that investments in instructional technology pay off in better student performance. What if I were to tell you that there is hardly any correlation at all in the PISA data between amounts of money invested in instructional technology and student performance in literacy and mathematics? What if I told you that the U.S. invests more than any other country on earth in instructional technology but a recent OECD study shows that U.S. millennials are among the worst in the world at using technology to solve problems, when compared to workers of the same age in the other surveyed countries? Would you dismiss the results because they did not show that investment in instructional technology caused American millennials to perform poorly in a test of technology-enabled problem solving ability? Or would you scratch your head and begin to look more closely at the way our schools are using the technology they are buying?
In countless cases of this sort, the correlations made possible by the PISA data should make you wonder whether what you “know” for sure is really true. For the most part, American education policy is made not on the basis of what research has shown about what causes what, but on the basis of what policy makers AND researchers BELIEVE to be true. Over and over again, the data from PISA challenges those beliefs and makes the observer question eternal verities of the kind I have just described.
The idea that NAEP should not collect this sort of background data because it does not establish cause is just plain absurd. I hope I have persuaded you that, provided you know that correlation is not cause, correlational data can be extremely useful.
If I were a member of Congress, I would file legislation to provide to every state the money needed to do a state sample for all the PISA surveys. I might even be tempted to make participation in PISA a condition for the receipt of federal funds. Why? Because PISA would turn out to be the most important and highest impact investment in education research we have ever made. Just as it is, PISA could do for our states exactly what it does for the highest performing nations in the world: provide a wealth of longitudinal data that would enable researchers everywhere to invalidate bad ideas based on wrong assumptions and open their minds to guesses about what might work that they would not otherwise have thought of. Using the same questions and protocols as all the other nations that support PISA would mean that states could compare the performance of American states not only to the performance of other states, but to the performance and background variables in other countries, opening our eyes to variances in policies and practices rarely seen in the United States, but common elsewhere, variances that might just hold the key to big advances in student performance.
But, if I failed to get that through the Congress, I would take a shot at requiring NAEP to collect the same kind of background data that PISA collects, for all the curricular arenas that NAEP is required to assess. We would not get the international comparisons that would add so much to a solely domestic picture, but we would at least be able to open the door to the rich possibilities inherent in correlational data that I described above.
In an ideal world, I would do both: turn NAEP into a PISA-style monitoring system and require every state to participate in PISA to provide policy makers at every level meaningful data with which to make national and international comparisons and better inform their decision making. Time for us to join the family of nations.