School & District Management

How Schools Can Identify ‘Evidence-Based’ Programs That Could Actually Work

Federal law urges educators to use evidence-based interventions for school improvement. What does that actually mean?
By Sarah D. Sparks — September 03, 2024 4 min read
Animated graphic showing the concepts of research, data, and analysis
  • Save to favorites
  • Print

Amid shrinking budgets, multiplying federal and state requirements for school improvement, and rapidly changing educational technology, it’s critical for education leaders to identify whether a given program will really improve learning for their students.

While leaders have more access than ever to education program evaluations and research clearinghouses, experts say it’s easy to overlook red flags in studies and evaluations of particular programs and interventions.

“It’s very challenging, and we’re asking educators whose job isn’t evidence to apply this to making their work more effective,” Nancy Madden, co-founder of the Success for All Foundation and an education professor at the Center for Research and Reform in Education (CRRE) at Johns Hopkins University, said during an Education Week webinar last week. “Educators are just now beginning to get down into the weeds enough to figure out how to make that evidence matter.”

Madden; Robert Peters, the senior vice president and chief academic officer for the nonprofit education research clearinghouses the Successful Practices Network and the National Dropout Prevention Center; and Amanda Neitzel, CRRE’s deputy director of evidence research, highlighted ways education leaders can find evidence-based interventions more effectively.

What does ‘evidence based’ mean?

The Every Student Succeeds Act formally defines three tiers of research evidence indicating how promising researchers have found a particular intervention to be. (Some states also allow a fourth tier for early studies exploring a new program with a strong rationale.) These tiers are used mostly to identify appropriate programs for school improvement under the law. To fit in any tier, the study must be well-conducted and completed by a researcher who is independent of the program, and it must show statistically significant findings.

  1. Tier 1 (strong) studies track the outcomes of a large group—350 or more students, teachers, or other participants. These studies randomly assign participants to the intervention or a control group in an effort to show cause and effect. This practice is considered the gold standard for causal research.
  2. Tier 2 (moderate) evidence comes from a study that also includes 350 or more participants tracked over time, but may use non-randomized, quasi-experimental designs in situations in which it is not ethical or possible to randomly assign participants.
  3. Tier 3 (promising) evidence may have a smaller sample group, use correlational versus causal designs, or be retrospective (with analysis performed after the intervention) rather than follow the sample over time.

Madden noted that terms like “research-based” and “evidence-based” are often used interchangeably to mean the same thing. However, different research groups and clearinghouses often disagree about exactly what is needed to prove that an intervention is effective.

The federal Institute of Education Sciences’ What Works Clearinghouse and CRRE have adopted the ESSA tiers when evaluating research. Neitzel said the highest-quality studies also follow a standardized intervention for at least 12 weeks, and include treatment and control groups that are similar and have had relatively few participants drop out.

Will an evidence-based intervention actually work in the classroom?

Many programs and interventions also sound great on paper or in the lab, but fizzle when they get to the classroom.

It’s important, experts said, to look for red flags in a study that suggest it could be hard to implement in a new class or school.

For example, no intervention will work if students don’t receive it, so an evaluation should state how much time students actually participated in a program that produced particular results.

“We see often, especially in cases of education technology, studies will compare students who used the program a lot to students who didn’t use it at all, and drop out all those kids in the middle who had low usage,” Neitzel said. “While that will tell you how this works for kids who use it five days a week, consistently for the whole year, it doesn’t necessarily tell you how it’s going to work for every one of your students.”

Even if a program shows strong evidence of effectiveness, it’s important for leaders and educators to consider how easily they could replicate the model with their own students.

“Choosing something with great evidence that doesn’t address the problem in your situation is not going to make the progress that you want to make,” Madden said.

She recommended that education leaders try to look for insight into the following questions when reading education research.

  • What training do teachers or other intervention staff need to implement the program effectively?
  • What aspects of the program may be harder to implement in your district? For example, rural districts may have less access to staff or resources used to implement a program in an urban school.
  • How will the school monitor the effectiveness of the intervention with its own students?

Peters, who works with districts on improvement plans, said the highest-performing school systems also analyze the cost-effectiveness of their programs at the end of each school year to understand how to prioritize them.

“A lot of times we see large school districts across the country spend millions of dollars on programs that are not even connected to their strategic planning,” Peters said.

Events

This content is provided by our sponsor. It is not written by and does not necessarily reflect the views of Education Week's editorial staff.
Sponsor
Science Webinar
Spark Minds, Reignite Students & Teachers: STEM’s Role in Supporting Presence and Engagement
Is your district struggling with chronic absenteeism? Discover how STEM can reignite students' and teachers' passion for learning.
Content provided by Project Lead The Way
Recruitment & Retention Webinar EdRecruiter 2025 Survey Results: The Outlook for Recruitment and Retention
See exclusive findings from EdWeek’s nationwide survey of K-12 job seekers and district HR professionals on recruitment, retention, and job satisfaction. 
Jobs Virtual Career Fair for Teachers and K-12 Staff
Find teaching jobs and K-12 education jubs at the EdWeek Top School Jobs virtual career fair.

EdWeek Top School Jobs

Teacher Jobs
Search over ten thousand teaching jobs nationwide — elementary, middle, high school and more.
View Jobs
Principal Jobs
Find hundreds of jobs for principals, assistant principals, and other school leadership roles.
View Jobs
Administrator Jobs
Over a thousand district-level jobs: superintendents, directors, more.
View Jobs
Support Staff Jobs
Search thousands of jobs, from paraprofessionals to counselors and more.
View Jobs

Read Next

School & District Management 3 Big Challenges School Lunch Programs Face as They Feed Students
School nutrition directors report problems with costs, supply shortages, and staffing.
4 min read
Students wrap up their lunch break at Lowell Elementary School in Albuquerque, New Mexico, on Aug. 22, 2023.
Students wrap up their lunch break at Lowell Elementary School in Albuquerque, N.M., on Aug. 22, 2023. Rising costs and staff shortages are squeezing school nutrition programs.
Susan Montoya Bryan/AP
School & District Management Superintendents Say Public Schools Can Compete With School Choice. Here's How
The four finalists for the National Superintendent said schools have to get creative to attract students.
4 min read
011425 SOY Finalists BS
The four finalists for the 2025 National Superintendent of the Year speak at a Jan. 9 panel discussion at the National Press Club in Washington. From left to right: Debbie Jones, Walter B. Gonsoulin Jr., Sharon Desmoulin-Kherat, and David K. Moore
Courtesy of AASA
School & District Management Classroom Interruptions Add Up Quickly to Lost Learning Time
During a typical school year, teachers contend with potentially thousands of interruptions to classroom time.
3 min read
Image of a clock on supplies.
Laura Baker/Education Week via Canva<br/>
School & District Management Are Snow Days Making a Comeback?
While some school districts use remote learning days when wintry weather strikes, others are reverting to—or sticking with—snow days.
4 min read
Rosie Henson, from left, Charlotte Hall and Jaya Demni play around in the snow near Schifferstadt Museum in Frederick, Md.,on Monday, Jan. 6. 2025.
Rosie Henson, from left, Charlotte Hall and Jaya Demni play around in the snow near Schifferstadt Museum in Frederick, Md.,on Jan. 6. 2025.
Ric Dugan/The Frederick News-Post via AP