This week, Andreas Schleicher and I respond to charges from Tom Loveless that the PISA results for Shanghai are suspect.
Following the release of the results of the latest PISA survey, showing that Shanghai was again leading the world’s league tables, Tom Loveless of the Brookings Institution wrote two blogs (blog one, blog two) attacking these findings. He asserted that a number of Chinese provinces had taken the assessment, that OECD and China had some sort of secret deal to release only the Shanghai scores (presumably to make China look as good as possible) and that the Shanghai PISA results looked impressive only because the authorities used the Hukou system of household registration to keep the 15-year-old children of poor migrant workers out of the Shanghai high schools and therefore out of the PISA results, creating a very distorted picture of the achievements of the Shanghai education system. But Loveless has his facts wrong.
First, there is no secret deal. Shanghai is the only province in mainland China that volunteered to fully participate in the last two rounds of PISA. Other provinces have recently agreed to participate, but they won’t be surveyed until 2015. Sure, other cities and regions in China have been experimenting with the PISA instruments, as is the case in many countries, but none of these entities have sought to participate in PISA, which would have involved drawing representative samples, compliance with the international technical standards, and collaboration with the OECD and the contractors charged with the implementation of PISA. Even though results from these experiments have been widely discussed among researchers and educators in China, it would have been irresponsible to report results from those provinces in the PISA findings.
The Loveless blog continues with assertions about the representativeness of the Shanghai sample. The fact is that the sample for the province of Shanghai was drawn in full accordance with the international standards established for PISA and similar international surveys. The adherence to these standards was validated by Westat, a U.S. based company contracted by the OECD to oversee the sampling and test administration in the countries and economies taking part in PISA. A close examination of the Shanghai sample does not reveal anything that suggests that PISA’s target population has not been correctly sampled. If 15-year-olds are in public or private schools in Shanghai, including schools catering for migrants, then they are covered by the PISA samples in exactly the way they would be in other countries.
Loveless’s case depends in part on his assertion that 15-year-olds (the age group surveyed by PISA) are in Shanghai high schools and migrant children are kept out of those high schools by the fact that the Hukou system requires the children of migrant children to return to the high schools in the province in which they are registered to take their college entrance exams. If all that was true, then it would be true that there would be few if any migrant children in the Shanghai PISA sample.
But it is not true. Forty-four percent of Shanghai’s PISA sample comes from middle schools; only 56 percent comes from high schools. Those figures correspond exactly to the respective proportions in the PISA age group.
By 2010, there were 8.9 million migrants in Shanghai from outside the province and the proportion of migrant children in that age group is around 24 percent. Loveless says that these children are kept out of the regular public schools and are forced to spend what little money they have to educate their children in shoddy private schools. It is correct that migrant children in Shanghai still face high barriers to academic high schools and it will only be as of 2014 that the rules governing access will be relaxed. However, all migrant children attend junior high school. Shanghai has been working hard to accommodate more and more children in the regular government schools. In suburban areas the authorities have outsourced the provision and pay a fixed amount per pupil, with local governments increasing the per capita payment. Certainly, those schools leave lots of room for improvement--and the OECD has written extensively about that--but the anecdotal evidence provided by Loveless about their low-grade infrastructure and quality just doesn’t match reality. In fact, when we compare the performance of the 10 percent most disadvantaged schools in the United States with the 10 percent most disadvantaged schools in Shanghai, we find that the disadvantaged students in the Shanghai schools far outperform those in the United States. And Shanghai’s middle school students - migrants included - outperformed the high school students in Massachusetts on PISA.
The 2012 OECD Survey of China recommended that entry to senior high school should be open to all children. The coverage of 15-year-olds in Shanghai is 79 percent. That coverage is low by OECD standards (for most OECD countries the coverage rate would be around 90 percent, although OECD countries like Mexico and Turkey have lower coverage rates than Shanghai). Even the United States, the country with the longest track record of universal high-school education, covered only 89 percent of its 15-year-olds in PISA - and it did not include territories like Puerto Rico. Not least, the coverage rate in Shanghai is quite typical for the non-OECD countries and economies that participate in PISA or other similar international assessments. Importantly, the coverage figure extends to all 15-year-olds physically resident in Shanghai for at least 6 months, and different from what Brookings claims, irrespective of where they are registered under the Chinese Hukou system. Those 15-year-olds not enrolled in schools, and thus not covered by the PISA sample, are certainly the most disadvantaged ones, but again, that is no different from the United States or elsewhere.
Much of Loveless’s case rests on his analysis of census figures, which, he says, contain a “gaping hole” suggesting a huge number of missing migrant children, the children he says are not allowed into Shanghai regular public schools. But he fails to take into account the exceptionally low birth rate in Shanghai. He also fails to take into account the fact that the migration rate is low for children but high for the active labor force. This is not unique to China, but is typically the case in developing countries with internal migration, where people are moving to seek work.
Loveless, as we said, charges that Shanghai runs the Hukou system so as to make sure that migrant families are forced to send their high school age students home so they won’t drag down the high school PISA scores. If that were true, then you would expect a precipitous drop in the numbers of high school age children. But the opposite is true. The census figures for 2010 showed that the number of children who are 15 years old exceeded the numbers at each age level for children ages 6 to 14. That is hardly what one would expect to find if there were a massive exodus of 15-year-old migrant children from the city.
We are not trying to paint a Panglossian account of the Shanghai schools. Loveless is right to point out that Shanghai is a sort of education Mecca in China, that it is likely to represent the peak of Chinese education performance, not the average. He is right, too, to call attention to the unfairness of the consequences of the Hukou system for the millions of immigrants who have helped to make Shanghai one of the world’s great modern cities.
But migration, both within and between countries, is one of the world’s more intractable problems. Many Americans want low-wage immigrants to work in their kitchens and fields for very low wages, but do not want to confer on them the rights of citizenship (including, by the way, the right to go to school and college at public expense). Many American wealthy communities want low-wage workers in their homes as domestics and in their gardens mowing the lawn, but refuse to build low-income housing that would enable these workers to go to their schools. No one, we think, has the right to throw stones on this issue.
Years ago, the Shanghai Municipal Education Commission started working to include the children of migrants in the regular government schools, notwithstanding the fact that they were not required to do so. They did that because they realized that the migrants were not going home, and were therefore going to constitute a very large fraction of Shanghai’s future work force.
During 2008 to 2010 alone, Shanghai invested over US$1.6 billion to build 363 elementary schools, middle schools, and kindergartens in suburban and rural areas of Shanghai where many of the migrants live. Those improvements are reflected in the PISA samples. Between 2009 and 2012, the share of migrant students in the PISA samples increased from below 20% to over 26%, and is therefore close to the 28% of residents without Shanghai Hukou. It is very hard to reconcile these statistics with Loveless’ assertion that the government has made it a matter of policy to deny migrant children access to Shanghai government schools.
The Chinese government knows that the stability of the country and the future of its economy depend in no small measure on solving the Hukou problem. If we consider the speed with which the Chinese government earlier increased access to compulsory education, greatly reduced illiteracy in the adult population and improved the quality of Chinese education, and the scale on which they did it, we would be ill-advised to bet against the government as it addresses this goal.
Though China has very high enrollment rates in its compulsory schools, it is still building out its high school system. But, at the rate it is going, it is entirely possible that it will exceed high school enrollment rates in the United States before many more years have gone by. But that would be par for the course, since, as we just pointed out, the rate of improvement in the nation’s education system since Deng Xiao Ping took over in 1978--including, by the way, the education of migrant children--has been stunning.
Since PISA released the results of its first survey in 2001, the reaction of some Americans has been to denounce PISA by trying to discredit the methods PISA uses or by arguing that it is just not possible to compare the United States to any other countries. Nothing could be more dangerous to the United States.
International comparisons are never easy and they are never perfect. But ignoring the success of East Asian education system will be a major mistake. The world has become indifferent to tradition and past reputations, unforgiving of frailty and ignorant of custom or practice. Success will go to those individuals, institutions and countries which are swift to adapt, slow to complain and open to change. And the task for governments is to help citizens rise to this challenge. PISA can help to make that happen.