Evaluating the usage of ed-tech products is tricky, complicated, and oftentimes confusing. But it can be done.
Consider the case of the Granite County school district in Utah. It partnered with a company called LearnPlatform to measure whether time spent on three particular pieces of software led to a bump in student achievement.
The district found that one program had great results for English-language learners and Native American students. Another seemed to get results when students used it as often as the manufacturer suggested, but going beyond that didn’t lead to better outcomes. A third was barely used at all, and the district is considering nixing it.
But most districts aren’t nearly as sophisticated as Granite.
Report after report cites low usage numbers for software, a problem driven largely by districts not thinking through how an ed-tech product should be used prior to buying it and then setting realistic expectations for usage, experts say.
Complicating matters is the fact that there isn’t a clear consensus on just what constitutes a “good” usage rate: Is it 25 percent, 50 percent, or 100 percent? And experts and educators are divided on whether education technology leaders should even be worried about low usage rates.
This special report—the first in a series of three special reports for the 2019-20 school year that Education Week is producing for K-12 ed-tech leaders—examines how schools track tech usage and what steps they should take to make better use of educational technology tools. Read the full report here.
To solve this problem, experts say the best way to get the best bang for your education technology buck is to set expectations for how often a particular piece of software should be used. Then keep careful track of how close you come to your goal. If a program isn’t coming close to meeting that goal, it’s probably time to jettison it.
But that’s easier said than done.
There’s no silver bullet when it comes to evaluating which software to keep and which to ditch, said Joshua Patchak, the executive director of education technology and innovation for the Green Bay Area public school district in Wisconsin.
“If it is magic, I would love for someone to teach me those spells,” Patchak said. “I don’t know the secret to it. It’s difficult, unless you have a very disciplined staff from the classroom all the way up to the top.”
‘Waste of Resources’
Most software licenses districts buy never get used, according to a November 2018 report by BrightBytes, an education data-management and -analytics platform. In fact, the study found a median of 30 percent of ed-tech licenses are never used at all.
What’s more, nearly all—98 percent—of licenses aren’t used intensively, meaning a student spends 10 hours or more with the product between assessments. And most teachers aren’t following vendors’ recommendations when it comes to “dosage"—how often and for how much time a student uses a particular program or product.
We clamor all the time about how we don’t have enough resources to meet the needs of our children, and yet if we’re investing in resources and they go unused, I think that’s unconscionable.”
Those findings are based on a set of data from 48 school districts, 393,000 students, and 177 apps—which BrightBytes defines as browser-based online curricular and learning tools. The districts studied were of different sizes, with enrollments from more than 30,000 students to fewer than 1,000.
Similarly, a study released earlier this year by the ed-tech company Glimpse K12 that looked at $2 billion in school spending found that, on average, 67 percent of educational software product licenses go unused.
Glimpse K12 tracked 200,000 curriculum-software licenses purchased by 275 schools during the 2017-18 school year. The analysis found educational software was the biggest source of wasted spending in K-12 districts. And it estimated that, overall, districts are losing a whopping $2 million each on these products every year.
Those data depress Anton Inglese, the chief financial officer for Batavia Public School District 101, in a Chicago suburb.
“I think it’s a waste of resources,” he said. “We clamor all the time about how we don’t have enough resources to meet the needs of our children, and yet if we’re investing in resources and they go unused, I think that’s unconscionable.”
But Richard Culatta, the CEO of the International Society for Technology in Education, said looking just at how often a particular product is used is the wrong way to think about the problem.
“I actually don’t like using usage as a metric,” he said. “It implies that we want everybody to use the software that we bought. What we should be thinking about is: Are the right kids and the right teachers using the right software at the right time? There’s rarely a case where the answer should be 100 percent, or having more is better.”
Instead of asking how often a particular piece of software is being used, Culatta said, district officials should ask: “Is it meeting the learning need?”
Reasons for Low Usage
One potential reason for low usage rates: District leaders in charge of buying software can’t always consult with every teacher to figure out what meets his or her needs.
I actually don’t like using usage as a metric. It implies that we want everybody to use the software that we bought. What we should be thinking about is: Are the right kids and the right teachers using the right software at the right time?”
In many cases, “school districts have to make one-size-fits-all decisions,” said Dan Carroll, the chief product officer of Clever, an education technology company that has helped track usage rates in the past. Given that reality, it “actually feels a little bit inevitable that you’re not going to get 100 percent of people using your software.”
Of course, in his years working with districts, Carroll, a former tech director, said he’s also “seen a lot of dysfunction” where software or an application is purchased sometime over the summer and not a single teacher uses it for the entire school year, sometimes because they don’t know it even exists.
But many education technology leaders will allow teachers or principals to buy software that not everyone may be interested in because they don’t want to close the door on experimentation with new applications.
“That’s obviously a tricky situation when the tech evolves so fast. If you lock the door and don’t let anything in, you miss opportunities,” said Hal Friedlander, the founder and CEO of the nonprofit Technology for Education Consortium and a former chief technology officer for the New York City schools. “It’s a tough job.”
Another contributing factor that district officials point to: aging workers who might not be quick to adapt to technological advances and some who might resist using them altogether.
“We have a much older workforce than a lot of other organizations and industries,” said Matthew Lentz, the chief financial officer for the Upper Moreland district, near Philadelphia. That means some educators aren’t as “nimble” on “how you can adapt in a changing environment.”
What Products to Jettison
Part of the Batavia district’s strategy: classifying curriculum materials into three categories, including core materials that get widespread use, supplemental materials for remediation or enrichment, and materials that might just meet an individual school, student, or teacher’s needs. The latter category gets the most scrutiny, with the goal of shifting materials in it to one of the other categories.
Districts spend millions on educational technology products and services. But many of those products and services are not used by teachers or students as much as you might expect. Here’s some advice from ed-tech experts and educators on how to pump up those usage rates:
1. Give educators meaningful opportunities to offer feedback on potential ed-tech purchases.
2. Listen carefully to educators’ critiques of the ed-tech products and services currently used in the district.
3. After you buy an ed-tech product or service, establish a detailed plan for showing all educators why you purchased it and how to use it. Have a usage plan or goal in place and continually check in and measure progress against the goal.
4. If you expect the software or application to be used for a certain amount of time, or for a specific purpose, make sure that expectation matches what’s actually going on in the classroom. For instance, if a teacher is only going to use a piece of software as an option in a lesson rotation and some students won’t have access to it for weeks, but the vendor is recommending an hour of use per week for each student, it might not be the best fit.
5. Pilot software and structure those pilots in a way that district leaders and teachers can see what the student outcomes are likely to be, perhaps by using a control group and a treatment group. Districts should make sure that they are testing out a particular piece of software or service in an environment similar to where it will eventually be used, since some tools might work well with some types of students but not so well with others.
There’s no secret sauce in trying to decide what to jettison, Inglese, the district’s CFO, said. “There aren’t universal criteria you can apply to every situation. You have to delve into the particulars of each situation to make decisions about whether or not these things work.”
And then there’s giving school leaders direct responsibility for deciding which technology will work best for their schools. In the IDEA charter network in Louisiana and Texas, principals can choose to buy software using discretionary funds. But that money doesn’t have to go to technology—it could go to other priorities.
That means if a particular product isn’t being used or isn’t effective, principals are going to hear about it.
“They would get that feedback from their teachers and the staff,” said Cody Grindle, the senior vice president of information systems at IDEA, which serves 53,000 students in 97 schools. “That’s a lot easier and [more] successful than me sitting in my ivory tower saying, ‘You’re only using 20 percent.’ We do share data around that stuff, but we’re not the ones slapping the wrist.”
What’s more, the amount of time a student spends on a particular application may not actually say much about whether it is effective, said Karl Rectanus, the co-founder and CEO of LearnPlatform. For instance, one district that partners with LearnPlatform found its students were getting much better results by using a product once a week, then supplementing with other materials, as opposed to the five times a week recommended by the vendor.
Some products are designed to save time, so you don’t necessarily want students spending huge amounts of time using them, he said. In his mind, “good usage is the usage that matches instruction and gets the best outcomes for students.”
