It started with a simple question from a parent: Why was his 11th grade son being scheduled for Spanish when he still couldn’t read competently in English? Wouldn’t it be better to enroll him in a reading class?
The query sent Superintendent Richard A. Ross off in search of more data. How many students in grades 7-12 in the Reynoldsburg city schools, he wondered, were reading below the 4th grade level?
Within 15 minutes, the Ohio superintendent had his answer: Eighty-four students fit that description. Based on the information, the district designed a more targeted reading program for those students. Now, almost a year later, “we’re seeing grade-level growth, on average, in excess of a year and a half in those kids,” Mr. Ross said.
“Would I have had that information if I had to go through 3,000 paper files to see who was reading at the 4th grade level?” speculates the superintendent of the 6,300- student district. “It might have taken weeks and weeks and weeks.”
Instead, thanks to a Web-based data-analysis and decision-support system, known as Schoolnet, “in 15 minutes, we were able to drill down and get those names,” he said.
Such experiences are making Mr. Ross and other school leaders like him believers in the power of analyzing data to improve public education. Although the practice goes by a variety of names—business intelligence, data warehousing, decision-support systems, data-based or evidence-based decisionmaking—the idea is the same. By knitting together the thousands of data elements now collected by districts and states into a centralized computer bank, educators can look for patterns that could help improve both the management and productivity of schools.
“I think people within education have realized that there’s a tremendous amount of value they can apply to helping kids learn that’s embedded in the data they already have, once they can look at it all in one place,” said Shawn Bay, the founder of eScholar, a data- warehousing company.
Mr. Bay, who was a management-information director at the Procter & Gamble Co. before starting eScholar, said businesses have been using data warehousing and data-based decisionmaking for more than a decade to enhance their bottom lines. But the practice is just now taking hold in education. His client list has grown from about 10 districts to more than 700 in the past two years.
A similar transformation is taking place at the state level, where states ranging from Georgia to Michigan to Mississippi are creating huge warehousing systems to link everything from student test scores and enrollment figures to teacher information and financial data. Michigan and Pennsylvania, meanwhile, have contracted with Standard & Poor’s School Evaluation Services to provide the public with Web-based access to information about the performance of individual districts and schools.
The passage of the “No Child Left Behind” Act of 2001 significantly increases the pressure on states, districts, and schools to collect, analyze, and report data in a timely fashion. Under the federal law, for example, states must report annually on student performance by race, income, gender, English fluency, migrant status, and disability. They also must report on the percentage of students not tested, by the same categories. And they must track the professional qualifications of teachers, including the percentage of classes taught by those lacking certification in their subjects.
Chester E. Finn Jr., the president of the Washington-based Thomas B. Fordham Foundation, describes the law’s “earnest efforts to force better information from the education system” as perhaps its greatest virtue.
What’s Working, What’s Not
Beyond the requirements in the revised Elementary and Secondary Education Act, observers point to a number of factors that are fueling the hunger for data-based decisionmaking. The decades-long push for accountability, for one, has focused attention on test scores and other results.
“I think what’s happened is the accountability demands are now forcing us to look much more deeply into the granular data and to do more sophisticated analyses,” said Philip A. Streifer, an associate professor for educational leadership at the University of Connecticut in Storrs. “That’s because just looking at how School A versus School B did, in the aggregate, was not revealing enough to tell us what to improve.”
Studies of high-performing schools and districts have found that they rely heavily on data to inform their practices. Increasing state expenditures on education also have prompted calls for more and better information about what that money is buying and whether it’s being used effectively.
Georgia lawmakers mandated the creation of a comprehensive education-information system in 2000 as part of the A-Plus Education Reform Act. While the system could cost $100 million to develop, estimates Thomas M. Wagner, the executive director of the State Data and Research Center at the Georgia Institute of Technology, which has responsibility for the program, “we’re spending in excess of $10 billion a year on public education in Georgia.”
Equally important, noted Ted Sanders, the president of the Denver- based Education Commission of the States, “with the new electronic technologies, it becomes more and more possible to have at your fingertips useful data to answer important questions.”
Last year, the ECS joined with the University of Texas at Austin and Just for the Kids, an Austin-based nonprofit organization, to form the National Center for Educational Accountability. The center’s goal is to improve states’ knowledge of how to use data to monitor, analyze, and improve student and school performance.
The center currently has data for six states—Arkansas, Florida, Minnesota, Tennessee, Texas, and Washington—that together represent 19 percent of the student population of the United States. It will help states compare schools with similar populations and identify the high fliers so that others can learn from them. The Council of Chief State School Officers, the National School Boards Association, and the American Association of School Administrators also have launched projects to work with districts on the collection, analysis, and use of data.
‘Hundreds of Pools’
While many see huge potential in mining data to improve public education, most states and districts have a long way to go before the rhetoric matches the reality.
“Very few states really have the kind of data that would enable ease of reporting either at the federal level or for policymakers at the state level,” said Brad Duggan, the president of the national accountability center and the executive director of Just for the Kids. “Probably no more than 10 states have a substantial amount of the information necessary.”
Only 16 states, for example, maintain a unique identification number for each student that enables them to link student-test data to other student and school characteristics or to follow the progress of individual students over time. Only 17 include breakdowns of information on student achievement—such as by race or family income—on school report cards.
The same holds true at the district level. “My experience is that probably the large majority, if not the supermajority of districts, have very little capacity to do this,” said Mr. Streifer of Connecticut, who is also the president of EdSmart, a company that provides decision-support systems to some 20 districts, primarily in New England.
“They don’t have the technology,” he said. “They don’t have the money for the technology. They don’t have the human resources to get this done.”
For instance, a report this spring on New Jersey’s management of education data, commissioned by the Institute on Education Law and Policy at Rutgers University, lambasted the state for its lack of a coherent information-management system. The report, prepared by education consultant Philip E. Mackey, found that the state education department “has no centralized repository—indeed has no record—of all the data it has available.”
Instead, Mr. Mackey found, mid-level managers tended to set up their own small-scale, project-specific databases, which numbered in the thousands. “That was the revelation to me,” he said, “when I realized it’s not that they’ve got information that they’re not sharing with us. They can’t find the information themselves, and they don’t even know what they’ve got. They’re expressing just as much frustration as the users out in the field.”
Since the release of the report, a new state administration has formed a task force to work on the development of a student-level information system.
Similarly, in California, a report this spring by the Center for the Future of Teaching and Learning and SRI International decried the lack of a coherent data system about the teacher workforce. “We frequently have been asked important questions about the teacher workforce that simply cannot be answered due to the inadequacy of state-level data,” the authors write.
“When we started doing this work,” said Margaret Gaston, a co-director of the center for teaching and learning, “there were these hundreds of pools of information, and they were so scattered and so diffuse that policymakers just couldn’t get to them.”
Often, the most basic information in states is either missing or reported in ways that make it only minimally useful—information such as detailed attendance and discipline data, graduation rates, student-coursetaking patterns by race and ethnicity, and teacher qualifications.
“I think the reason is that the way the public school system has been created over time, data were collected for compliance purposes, not for diagnostics, and it’s an entirely different mind-set required to look at data in a way that illuminates what’s actually going on,” said Leslye A. Arsht, the president and co-founder of StandardsWork, a nonprofit organization based in Washington. The group last year produced “The Results Card,” which compares student performance and nontest indicators across Arizona, Colorado, Georgia, Florida, Maryland, Ohio, Pennsylvania, Texas, and Virginia.
‘Very, Very Messy’
Even when data do exist, observers note, such information is often of poor quality, because too little attention has been paid to validation.
“School data is generally very, very messy,” said Alvin Crawford, the vice president for Internet sales and marketing for Schoolnet, a New York-based company whose products are now used in about a dozen districts in five states. “Typically, the lowest-paid employee in the district is the one who’s responsible for adding and updating the data.”
“Having the data open and available to users makes districts realize how bad it is and to do something about it,” he argued. “Just having a system that lets them analyze the data means districts are now working toward fixing it.”
In 1997, for instance, the Oregon legislature mandated a central database for schools that includes funding, staffing, and student information. Previously, the state’s 200 districts had submitted in excess of 200 forms to the state education department, allowing numerous opportunities for redundancy and errors, said Michael J. Hess, a managing director with KPMG Consulting, which helped set up the system. The new system now includes fewer than 100 indicators, as well as a detailed process for validating the data.
Developing a robust system of data collection, analysis, and use in education can be a long, complicated, and costly process.
Ohio, for instance, spent an estimated $79 million on its education management-information system between 1989 and 1999. Georgia lawmakers have appropriated $50 million since 2000.
The various data-warehousing, -reporting, and -analysis systems marketed to districts can also be expensive. The products offered by eScholar range from $5 to $15 per pupil, said Mr. Bay, depending on the size of the district and the scope of services offered. Schoolnet costs between $2 and $5 per pupil.
Broward County, Fla., one of the first districts to establish a data warehouse, did so with a $2 million grant from IBM’s Reinventing Education program, along with $2 million of its own money.
“Companies clearly see an opportunity,” said Richard J. Wenning, the president of the education performance network for the Arlington, Va.-based New American Schools, a nonprofit group that promotes comprehensive school reform.
“There are probably over 100 companies that provide these types of services,” he noted, “but there are only a handful of providers that have the ability to work at any scale. And many of them are relatively young companies as well.”
“We really think a very careful review of the marketplace and understanding of the different offerings is essential,” said Mr. Wenning, whose organization is hoping to position itself to provide such advice to clients and, potentially, handle some data management itself.
Peter J. Stokes, the executive vice president for EduVentures Inc., a Boston research company that specializes in the education industry, said the best measure of the business sector’s growing interest in data-based decisionmaking in education “is the fact that some of the largest and best-established education companies are thinking more seriously about getting into the decision-support and data-warehousing business.” That includes such companies as the publishing giant NCS Pearson and Chancery Software, a leading publisher of information-management systems and Internet learning communities.
“It’s not the most glamorous or sexy technology market going,” Mr. Stokes said, “but it has the long-term potential to be, in some ways, the über-market. It will help administrators realize which technologies really impact education and which technologies really improve student outcomes. And that’s the real benefit.”
Companies like Schoolnet, for example, offer a suite of applications for data-based decisionmaking that, in addition to providing analysis and reporting, enable teachers to align their instruction with state and district academic-content standards.
“There’s a tremendous opportunity here, and we’re going to discover lots and lots of competition,” said Denis P. Doyle, the vice president and chief academic officer of Schoolnet. “It’s the one area in which the private sector can make a significant contribution to lasting school reform.”
Others worry, however, that in the short run, there could be more data than clarity.
“What I’m seeing is a growing interest in the public reporting of data,” said Brian Stecher, a senior social scientist at the RAND Corp., a think tank based in Santa Monica, Calif. “There seems to be the assumption that those to whom it’s reported will be able to make wise use of it. But it isn’t clear to me that anyone has a specific plan for use in mind.”
Ultimately, the hope is that the push to data-based decisionmaking will also help teachers and principals. This school year, for example, every teacher in the 262,000-student Broward County district received a detailed analysis of how each of their students performed on every subtest of the state assessment system. If a student performed below the passing grade in an individual area, such as fractions or statistics, the report flagged it.
“The whole idea is that all this information is there,” said Phyllis Chasser, the senior data-warehouse analyst for the Florida district, which includes Fort Lauderdale, “but if we don’t get it out to teachers, who can then change the way they teach, ... it’s useless to have the information.”
In fact, one of the hardest tasks, observers say, is not technical but cultural. “You have to build a trust and comfort level with schools,” said Mr. Wagner of Georgia. “We’re dependent on them to input a large amount of this information. It’s really, I’ve told superintendents, about providing them with tools they need to be effective.”
“I think what it does, with the ease of use, is allow us to move away from voodoo education, where we’re just shooting in the dark about what the issues are with students,” said Superintendent Ross of Ohio’s Reynoldsburg district, “and to become much more professional in our delivery of instruction, assessment of instruction, and description of interventions for kids.”
