Frustrated by the lack of significant progress in school reform, the nation’s governors met at the National Education Summit in Palisades, N.Y., last March to recommit themselves. Accompanied by business leaders from their states, the governors declared an “urgent need for schools to improve and for academic performance to rise.” They invited the voters to hold them accountable for progress made in their states toward improving student achievement.
Although we began our study before the National Education Summit last spring, this report responds to the governors’ declaration that they should be held accountable for the progress of their schools and their call for “an external, independent, nongovernmental effort to measure and report each state’s annual progress in setting standards, improving the quality of teaching, incorporating technology, supporting innovation, and improving student achievement.”
Quality Counts is a serious effort to create something useful with the limited tools available. Viewed as a whole, the information presented in the following pages provides a snapshot of the condition of public education in the 50 states-the efforts being made to improve schools, and what remains to be done.
We hope this report will contribute to the national discussion of what count in education and how that should be measured. As the title implies, we believe that quality counts, and that the focus of policymakers and practitioners should shift from process to substance.
What We Don’t Know
Hardly a week passes without some federal agency reporting on one or more key economic indicators: unemployment, interest rates, wholesale prices, new housing starts, trade balances, orders for durable goods, and on and on.
Based on these government-provided numbers, millions of Americans make important economic decisions every day. Corporate plan are shaped, state and national policies are formed, and global markets move.
If the data we depend on to monitor the economy were as incomplete, as unreliable, and as out of date as the data we depend on to monitor education in the United States, we might as well have the economy of a Third World country.
A state can improve its education system only if it has a clear sense of its weaknesses and mechanisms in place to chart progress.
Public education is a vast enterprise that directly touches the lives of most Americans. Its success is clearly linked to the welfare of the nation and the future of our children.
We spend more on education than on anything else except national defense and health care. It is the largest or second largest single item in the budget of every state.
More than 44 million students attend about 85,000 public schools in 14,400 districts, staffed by about 2.5 million teachers at a total national cost exceeding $244 billion a year. Yet we do not know in any but the crudest way how well our education system is performing.
States know roughly how many students attend their schools, but little about what, or even whether, they are learning. States know how much they spend on education each year but have only a vague idea of whether they’re getting their money’s worth. They have elaborate bureaucracies to run their systems but don’t know what effect most of their policies and practices have on teaching and learning.
There are encouraging signs nationally that more students are taking more rigorous courses, and there is national evidence that the dropout rate has declined over the past two decades. But in neither case, do we have good information at the state level.
States can point to “break the mold” schools here and there, policies that hold real promise for positive change, and research that may offer new solutions to old problems. But they lack solid data to measure their progress and, overall, they cannot say explicitly what the billions of education dollars spent over the past decade have bought.
Finding the Right Indicators
The first and most difficult challenge in assessing the condition of education in the states is to find reliable indicators--data that are accurate and up-to-date, are comparable among states, and, most important, that actually tell us something important about the condition of education.
Aware of this challenge, Congress in 1988 authorized a study of indicators that could be used to measure the health of the education system.
Three years later, that study reported: “If the broad reform movement is to succeed, the United States must develop a comprehensive education-indicator information system capable of monitoring the health of the enterprise, identifying problems, and illuminating the road ahead. Without such a system, reform cannot be sustained because, lacking a reliable means of charting progress, it will have to rely on inadequate data and poorly conceived analysis.”
Because that recommendation remains largely unfulfilled, no assessment of public education in the 50 states can be as comprehensive and precise as it ought to be.
We spent the better part of a year searching for reliable indicators. There are mountains of statistical information available, but, for our purposes, much of it fails the tests of timeliness, comparability, and, most important, usefulness.
Typically, efforts to evaluate and rank organizations, like the various rankings of American colleges, gather hard numbers--such as how many faculty members have doctorates, the number of books in the library, how many admission applications are received, and the number of students finally selected-and assign values to them to come up with a composite score.
That is neat and precise, but library books, doctorates, and selection rates do not say much, per se, about the quality of education. To evaluate public schools, we need much more.
If it is vital to have enough useful indicators, it is equally crucial to avoid using the “wrong” ones. Indicators that focus attention on the wrong things can undermine progress in improving schools.
Nationally normed standardized tests are a good example.
Many states and school districts report student test scores on various commercial multiple-choice tests. The media often publish these as “report cards” on the quality of the public schools. And parents and taxpayers base their opinions of schools on how they compare and whether their scores are rising or falling.
At best, these norm-referenced standardized-test scores are poor indicator of quality, and, at worst, they are misleading. They do not measure a student’s progress against the curricula in a state but rather against a sample of students used to set a norm. And they are d signed to ensure that half the students will score above average and half below.
Our principal goal was to measure what really matters, and we relied on research to determine that. This report focuses on the conditions that are most likely to improve teaching and learning, according to the most convincing studies we could find.
When research findings were contradictory, we chose the research we found to be most persuasive. Researchers disagree, for example, over whether there is a correlation between class size and student achievement. We are convinced that class size matter.
When there was no research to link an indicator with better schools or higher achievement, we relied on common sense. It is too soon, for example, to prove that an increase in the number of teachers certified by the National Board for Professional Teaching Standards will improve teaching and learning.
But because teachers who receive board certification must meet very high standards, we believe they are likely to be better teachers. So we reward states with policies that encourage and help teachers attain board certification.
An Effective System of Public Education
Research of the past few decades suggests that a successful public school system very likely would have the following characteristics:
1. High standards in English, math, science, and history for all children and assessments that align with those standards.
The standards movement is too new to predict with any certainty whether it will succeed.
While there are examples linking higher standards to higher student achievement, there has been too little research at this point to guarantee the success of the standards-based strategy. But experience and common sense tell us that children do better when they know what is expected of them and when those expectations are high. Standards are intended to clearly spell that out for them and their teacher and parents.
Standards alone are not enough to improve student performance. They must serve as the basis for specific, challenging curricula. They must be incorporated into the programs of teacher-training institutions. Instruction, textbooks, and other learning materials must be keyed to the standards.
Several industrialized countries with which we compete economically have education systems based on standards for what students should know and be able to do, along with assessments that measure student mastery.
Unlike the secrecy that surrounds standardized testing in the United State, countries with standards-based systems make clear at the outset the content and skills that student will have to demonstrate in their examinations--even to the extent of publishing the exam questions in newspapers.
Students can’t cheat by memorizing the right answer in multiple-choice exams. They don’t know which questions they will have to answer, so they must be prepared to exhibit the knowledge and reasoning required to answer all of them.
The norm-referenced standardized tests that most state have long relied on are not adequate to assess student performance in a standards-based system because they are not aligned with a state’s standards. New and different assessments will be required.
Once standards and aligned assessments are in place, all students must have the opportunity to learn, and their performance should have consequences for them and their schools.
2. Teachers whose primary focus I on student learning and who possess the knowledge, skills, and commitment to teach to higher standards.
In the end, the success of a school system depends on its teachers. They must be adequately prepared and motivated. They must know well the subjects they teach, and they must be competent to help students meet the standards.
Teachers need the time and opportunity to be true professionals and to continue their own learning.
States can increase the quality of teaching by establishing policies that improve teacher recruitment and education, raise licensing and certification standards, link compensation and incentives to school improvement, and provide ongoing substantive professional development for teachers.
3. Schools that are organized and operated in a way that encourages and supports teaching and learning.
Schools should be humane institutions where the virtues and highest values of a democracy are modeled. They should be safe and orderly, well-maintained, and adequately equipped. Their highest priority should be on children learning.
Teachers should play a central role in decisions about education, and those decision should be made as close to the students as possible.
Those conditions are state or grade them, but we more likely to be present in schools and classes that are small enough for teachers to know their students well and to work closely with their colleagues.
Unfortunately, for economic and educational reasons, the trend for the past half century was toward larger schools, more bureaucracy and centralization, more teacher-proof curricula, and more distractions from teaching and learning.
None of the reform activity matters much if it does not change the way school are organized and operated.
4. Adequate funding distributed equitably to all children and focused on the functions that matter.
Some research finds that higher spending doesn’t correlate with higher achievement; other research finds that it does.
Common sense tells us that money does matter especially when there isn’t enough of it to sustain high-quality education.
After a point, research suggests, the way funds are spent probably matters as much as the amount that is spent.
Resources, therefore, should be concentrated on teaching and learning, for both students and teachers. And more of those financial resources should be concentrated on the neediest children, so that the appalling and long-standing disparities between public education spending for the rich and the poor are finally eliminated.
5. All students achieving at high levels and engaged in challenging intellectual work.
Some people may never be convinced that “all children” can learn at high levels, but at the very least, they should behave as though they believe it. Policies and practices based on the premise that not all children can learn become a self-fulfilling prophecy.
In an effective education system, students accumulate knowledge and develop skills that help them grow into successful and productive citizens and fulfilled adults. Standardized tests can be indicators of achievement, but the real measure of whether students are learning is the quality of their work.
The best measure of a successful education system, therefore, would be assessments that evaluate student achievement on the basis of their performance and what they produce. But such assessments are not yet available in most states.
Question of Commitment
In addition to these five major characteristics of a successful system, we looked at commitment--the willingness of policymakers, taxpayers, parents, and business leaders to work for better schools and higher student achievement.
We could not find enough comparable data to compare state or grade them, but we cannot stress enough the importance of public involvement and support.
The home is the child’s first school, and parents are a child’s first teachers. Parental involvement in children’s education can have a significant and lasting effect. One major research study concludes that 75% of students’ achievement--positive and negative--is the result of the home and family, factors that schools are relatively powerless to change. But that means that schools have the potential to raise student achievement by as much a 25%--and that would be an enormous accomplishment.
Policymakers and business and community leaders can help make that possible through their interest in and support of public education. An enterprise as vast and expensive as public education can succeed only if it has the understanding and commitment of the larger public.
Report Card for the States
The objective of the Education Week/Pew annual report is to determine how close the 50 states are to having such a system of public education. Toward that end, we have compiled a number of indicators that bear on each of the desired characteristics described above.
Some indicators tell little about the quality of a system; some have more impact on student achievement than others; some are inconclusive because the information is not as reliable or complete as we would wish; and some reflect the situation as it was several years ago.
But we use these indicators because they are the best and most up-to-date information now available. While one indicator may not tell much by itself, when considered with other related indicators, it can be very informative. And all together, the indicator when combined with the accompanying text summary of each state provide the best available evaluation of the condition of education in the states.
Reports that use such indicators in an attempt to measure various aspect of the education system usually strive to be totally objective, free of all human judgment and opinion. This one does not.
Assessing, evaluating, and drawing conclusions are all actions that inherently involve making judgments. Without those judgments,Quality Counts would be a sterile exercise in number crunching. There are enough of those available.
It is not our intent to advocate, except in the broadest sense of supporting better education for all children. We make no explicit recommendations, but we realize that our values are embedded in our decisions and conclusions.
As the introductory text with each set of indicators describes, we based our judgments as often as possible on research studies that were most persuasive and made the most sense to us.
And although our choice of indicator was a subjective one, we found, in an Education Week/Pew survey conducted last summer, strong agreement with many of them on the part of superintendents, principals, and teachers.
Finally, in assigning values to indicators, setting benchmarks for performance, and giving letter grades to the states on each of our major characteristics, we exercised our best judgment in interpreting and analyzing the data.
We also want to stress that:
- We are not assessing the effectiveness of individual schools in a state or the quality of its teachers. The data do not exist to permit that. Instead, we seek to determine and measure what the state has done and is doing to ensure that it will have effective schools staffed by competent teachers working with high-achieving students.
- When we refer to the state, we mean the entire state: the governor, the legislature, the department of education, the education establishment, the business community, the media, and the taxpayers. All bear responsibility for the condition of public education in their state. All can take credit for the grades and rankings.
Grading and Scoring
Beginning on page 26, we present each of five sets of indicators. The introductory text in each case explains our reason for choosing the indicators and the method we used to evaluate and score them.
In most instances, the numbers under the indicators are percentages on a 100% scale. Where we set a benchmark lower than 100%, that is noted and explained.
The tables of indicators list the states by rank, according to the overall score each state received on the indicators.
We converted these percentage scores on each set of indicators into the letter grades on the individual report cards that appear with each state’s summary beginning on page 63.
We did not give each state a single, composite grade on all five sets of indicators. Nor did we grade states on commitment. Rather, as teachers do with deportment on children’s report cards, we included a comment on the states’ report cards based on 50 summaries written by Education Week’s reporting staff.
Why Rankings and Letter Grades?
We rank states because millions of Americans move across state lines every state would meet some reasonable standard.
If not, they should at least know what sacrifice they might make by moving to a particular state. Readers should understand that the rankings are based on data that are not always as reliable as we would wish. There are extenuating circumstances sometimes. Critical nuance and extremes are sometime obscured, causing the snapshot to be somewhat blurred.
A handful of states--Iowa, North Dakota, Nebraska, New Hampshire, for example--rank at the top in National Assessment of Educational Progress scores in reading and math, in part because they are largely rural with homogeneous populations and do not have to cope with the problems of large cities.
Ironically, these states do not fare as well on our indicators as lower-achieving states. That is mainly because their students do reasonably well, and there is not a lot of pressure for state-level reform policies, which are the main focus of this report.
But we should also note that these states do not do as well as they could. None has even half of its students at the proficient level in reading and math, and there are glaring inequities among schools. They are doing relatively better than other states, but that is little cause for celebration given how far they have to go.
Some argue that it is unfair to compare states because they differ significantly from each other economically and demographically. How can we fairly compare California, with its large immigrant and minority enrollment, to North Dakota, which is mostly rural and white? We recognize the impact of such diversity in California. But we also note that if California’s challenges are bigger than North Dakota’s, so are its resources.
More important, adjusting the methodology for comparing states does not change the fact that the children in the public schools are real children who deserve a high-quality education whoever they are and wherever they live. Factoring out the demographic and social problems to make comparisons more fair would only further obscure the failure of the system to deal with those problems.
As it is, the columns of numbers as they appear in the body of the report conceal the critical and worrisorne difference among students from diverse racial and ethnic backgrounds. Poor children--many of them members of minorities--and immigrants who have not mastered English suffer most from the failure of public schools. Until their needs are effectively met, they and the larger society will pay a heavy toll.
In an article on page 10, we draw from a revealing report on the achievement gap between minorities and whites released last December by the Education Trust in Washington. That article also briefly addresses demographic and social factors that significantly influence student achievement. By controlling for the effects of poverty, mother’s education, and family size on students’ NAEP scores, for example, we can more fairly see what impact the states’ education system have on student achievement--which are doing better or worse than might be expected given their particular circumstances.
Because letter grade are not very informative and even sometimes misleading, we debated using them. But parents demand letter grades (and schools cheerfully provide them) because they want a quick, easy picture of how their children are doing. The public deserves the same opportunity to see a simplified snapshot of how the states are doing. Complicated tables with columns of percentages, after all, are dense and hard to read.
We agree with the spirit of a comment made by Gov. Tommy G. Thompson of Wisconsin. “If you can’t read or write, if you can’t calculate, if you don’t know the difference between Elroy and Argentina, you’re not going to get a diploma in the state of Wisconsin,” he says. “And if a school district is failing to teach those skills, the taxpayers and parents deserve to know about it.”
Some will take issue with the judgments in this report. Some will differ with our findings. And some undoubtedly will object to the grades we give to their states. That is to be expected.
Education Week is a newspaper, and Quality Counts is clearly a journalistic endeavor. We gathered the best information available, analyzed it as carefully as possible, and drew some reasonable conclusions from it. And in doing so, we tried to bring a high level of intellectual effort, objectivity, and conscientiousness to the task.
We hope this effort will have positive results. We also hope the data we gather to monitor progress will get better with each passing year, as states and agencies improve their collection and analysis of information.
Finally, we realize that this report can be misused by those who want to advance one point of view or another. That is always a danger in ventures like this, but it is not a sufficient rea on not to proceed. We believe that risk is far outweighed by the potential benefit of helping illuminate the problems of public education so corrective action will be demanded and taken.
