A number of services have cropped up over the years to help schools answer a question that, in an age of information overload, scarce resources, and new technology, is becoming central to how they move forward: What works?
Whether it’s aggregating the most trustworthy studies on education or culling user reviews on products, as Yelp does with restaurants, none of these available services, both privately and publicly financed, has gained national scale. But a new proposal from two economists is perhaps the most ambitious—or, if you ask some of its potential users, flawed—attempt at creating a Consumer Reports for education technology to address the lack of independent evidence available on such products’ efficacy.
The proposal is called Edu Star, a technology tool that would allow schools to conduct rapid, randomized evaluations of education products, collect and analyze the results, and publish the data to the public. By doing so, schools would make better-informed purchasing decisions, and entrepreneurs would have evidence that their products worked, the economists hope.
“Not surprisingly, when no one knows what works, schools are unlikely to buy, and innovators are unlikely to create,” the economists, Aaron Chatterji, a professor at Duke University, and Benjamin Jones, a professor at Northwestern University, write in a discussion paper sponsored by the Hamilton Project, an economic-policy initiative of the Brookings Institution. The paper was discussed at a Sept. 27 forum hosted by the Washington think tank.
A new proposal from two economists would create a Web-based system to rapidly conduct randomized trials among students on the efficacy of education technology products.
SOURCE: The Hamilton Project, “Harnessing Technology to Improve K-12 Education”
Innovate Faster?
Mr. Chatterji and Mr. Jones came up with the idea while both were on the White House Council of Economic Advisers two years ago. Edu Star is a convergence of three factors they believe drive much of the economy’s health: education, innovation, and “building things,” as Mr. Jones put it in an interview.
“If there is a weak link in this interconnected chain, it really seems to be K-12 education,” he said.
K-12 education accounts for just 0.2 percent of the research and development expenditures in the United States, according to a 2010 report by the President’s Council of Advisers on Science and Technology, despite $600 billion in total annual K-12 expenditures. A low-cost way to conduct randomized trials could help education technology innovate at the same speed as other fields, where products are rolled out and tested at a much faster rate, Mr. Jones said.
To understand how Edu Star’s trials would work, Mr. Chatterji and Mr. Jones suggest thinking of Google’s rapid product development rather than, say, the U.S. Food and Drug Administration’s trial studies. Education technology companies would offer their products to be tested, and schools would agree to test the products. After logging in to Edu Star, students would take a pre- and post-assessment on a skill tied to the Common Core State Standards.
In between, students would be assigned exercises in either one of two products built to teach that skill or a placebo product. All of the results would be recorded in Edu Star and reported on its website.
Schools would have an idea of whether a product improves academic outcomes, proponents of the plan say. Entrepreneurs without major sales forces could shift focus from wooing district leaders to sign contracts to proving a product’s value to its end user, the teacher, they say.
“It’s not an enterprise sale, it’s the bottom-up approach,” Eric Westendorf, the chief executive officer of LearnZillion, a Washington-based video-lesson-sharing tool, said at the recent forum. “That, working in collaboration with an Edu Star-type platform, is really exciting.”
But early responses to the proposal suggest Edu Star takes a limited view of what’s required to evaluate educational products. Factors such as professional development, ease of implementation, and long-term academic performance are also important to purchasing decisions, said Mark Edwards, the superintendent of the 5,500-student Mooresville, N.C., school district.
“In my experience, there’s a level of complexity around using digital resources that would not align with a quick and efficient means of evaluation,” he said.
Other Services Emerge
A commonly referenced barrier for companies in the education market is the country’s approximately 14,000 school districts, which essentially make up 14,000 distinct buyers. But many of those barriers are reduced because technology companies can enter the market cheaply.
The common-core standards, adopted by all but four states, are likely to both simplify the market for vendors and create confusion for school-level customers.
“Almost everything you see now says, ‘aligned with common core.’ I don’t believe that,” Dennis Van Roekel, the president of the 3.2 million-member National Education Association, said at the Washington forum. “But everyone says so because it’s a nice marketing line.”
Some services have cropped up to help make sense of the market, and most are geared toward teachers, indicative of their increasing role in the marketplace.
EdSurge, a Silicon Valley startup news service that covers the education technology industry, offers selective reviews of digital learning tools from educators who have used them. EdShelf, another Silicon Valley-based company, offers a directory, reviews, and videos of digital learning tools.
PowerMyLearning, a K-12 learning platform from CFY, or Computers for Youth, a New York City nonprofit organization, rates free digital educational content using reviews from teachers, students, and parents. The organization is in talks with Edu Star about potential partnerships, said CFY co-founder Elisabeth Stock. Using both anecdotal user reviews and empirical randomized trials would help give a fuller picture of the education technology marketplace, she said.
“You can have a teacher who uses a product and has great results, but that doesn’t mean that teacher has tried that product against three other products,” Ms. Stock said. (The Bill & Melinda Gates, Eli and Edythe Broad, and W.K. Kellogg foundations contributed a total of $7 million to CFY. The Gates Foundation provides grant support for Education Week‘s coverage of K-12 business and innovation.)
Another effort, the Learning Resource Metadata Initiative, funded by publishing associations and foundations, including Gates, aims to categorize the open learning materials available on the Web for easier search.
The What Works Clearinghouse, a website funded by the U.S. Department of Education, evaluates the credibility of education research. But there are few up-to-date items on education technology on the site and, overall, it’s been criticized for a lack of user-friendliness.
‘The Right Answers’
But would Edu Star itself work?
Right now, it is only a proposal, though it’s one that Mr. Chatterji and Mr. Jones intend to pursue, by raising $5 million in foundation funding and hiring a small staff. (Neither plans to work on Edu Star full time.)
The idea has drawn early critics, meanwhile.
Betsy Corcoran, EdSurge’s chief executive officer, doesn’t believe the Edu Star ratings would work for modern technology products constantly in flux. “It won’t accurately capture the usefulness of young and emerging tools because those tools will be in transition a lot,” she said.
Mr. Westenberg of LearnZillion said it is “scary” that a rating of an early version of his product could be published.
There is also criticism that the rapid assessments the proposal envisions aren’t the best way to judge a product, let alone student learning in general.
“Ah, educational research. Ah, test scores. Ah, common core. Ah, what a very limited definition of ‘learning,’ ” wrote education technology blogger Audrey Watters.
Mr. Jones, the Northwestern economist, argues that because Edu Star would test students on the common standards, what was being assessed would have legitimacy and would provide an apples-to-apples way of judging disparate groups of students.
But in general, he acknowledged, there is still a lot to figure out, including how to provide incentives for participation for entrepreneurs and schools, and how to navigate the Family Educational Rights and Privacy Act, which protects the confidentiality of students’ education data.
Mr. Jones also stressed that the proposal doesn’t aim to reinvent learning or the education marketplace; it’s just another tool for helping produce better decisionmaking.
“The real innovation isn’t Edu Star,” he said. “It’s not about knowing what the right answer is, it’s about creating a system where the right answers can emerge.”
