Disadvantaged students who regularly attend top-notch after-school programs end up, after two years, academically far ahead of peers who spend more out-of-school time in unsupervised activities, according to findings from an eight-state study of those programs.
Known as the Promising Afterschool Programs study, the new research examined 35 programs serving 2,914 students in 14 communities stretching from Bridgeport, Conn., to Seaside, Calif. The programs, all of which had been operating at least three years when the study began, were selected because of a record of success.
For advocates of after-school programs, the results offer a counterpoint to a controversial 2005 evaluation of the 21st Century Community Learning Centers program, a federal initiative that finances after-school enrichment programs for 1.3 million elementary and middle school students nationwide.
Conducted by the Princeton, N.J.-based Mathematica Policy Research Inc., the earlier study found that the federally funded programs provided no special learning boost and may even have led to a slight statistical increase in some negative behaviors.
“My hope is that this research can really put to rest the research by Mathematica and really show that after-school programs are making a difference for the children that are participating,” said Jennifer Rinehart, the vice president for policy and research for the Washington-based Afterschool Alliance, which is working to circulate the new results.
But some scholars caution that the latest study, which was underwritten by the Charles Stewart Mott Foundation of Flint, Mich., could run up against criticism, too, in ongoing debates over federal spending on after-school programs. One problem cited was that researchers used as a comparison group students who attend after-school programs sporadically, suggesting that those students might have been less motivated at the outset than students who regularly attend such programs.
“What it told me is that kids who are different at baseline are even more different two years later,” said Mark Dynarski, who led the Mathematica evaluation. “That’s not evidence of effectiveness.”
Focus on Quality
In the new study, researchers divided students into three groups: a “program only” group of students who attended their after-school program two to three days a week and did nothing else outside of school; a “program plus” group who visited the after-school programs two to three days a week and also took part in sports, church programs, music lessons, or other extracurricular activities; and a “low supervision” group who dropped in on a mix of after-school activities from one to three days a week.
The researchers found, over the course of the three-year project, that the more engaged students were in supervised after-school activities, the better they did on a range of academic, social, and behavioral outcomes.
For instance, 3rd and 4th graders in the “program plus” group tallied gains on standardized mathematics tests that were 20 percentile points higher than those of the children who rarely went. The frequent attenders also made more progress in developing sound work habits, task persistence, and better social skills, and in reducing negative behaviors, such as skipping school or fighting.
The 6th and 7th grade pupils who regularly attended after-school programs outpaced the math learning gains their “low supervision” counterparts made by 12 percentile points by the end of the study period. The “program” and “program plus” groups also reported reduced rates of drug and alcohol use, compared with students with spottier attendance.
“What makes these findings interesting, and maybe surprising to some people, is that the math gains are occurring in programs that are not specifically targeted to academic skills,” said lead author Deborah Lowe Vandell, the chairwoman of the education department at the University of California, Irvine. “Children were developing persistence, focus, and engagement, and we believe those are the kinds of skills that maybe children take to school with them and that may contribute to their math gains.”
In designing the Promising Programs study, Ms. Vandell said, she set out to address some of what critics saw as shortcomings in the Mathematica report. Critics and advocates complained, for example, that the earlier study fell short because it involved programs that were young or varied in quality.
So Ms. Vandell and her research partners—Elizabeth R. Reisner of Policy Studies Associates Inc. of Washington and Kim M. Pierce of UC Irvine—zeroed in on the best programs. Not all the programs were funded under 21st Century Schools or based in schools.
The research team selected 35 programs serving low-income elementary and middle-school students from among 200 that were nominated for their successful track records.
“Other work that I have done looking at the association between varying quality programs and child development was suggesting that higher-quality programs were where you expected better outcomes,” said Ms. Vandell. “We do see gains there that we don’t see otherwise.”
Ms. Vandell acknowledged that the comparison-group issue that Mr. Dynarski raised posed a “limitation” for her study, but said her team took steps to minimize potential bias. One way was to compare students’ gains with their own previous performance, rather than looking at the overall achievement levels of the various groups.
Funding at Issue
The Mathematica study began in 2000, four years after President Clinton launched the 21st Century Community Learning Centers program and two years after the program was repurposed to provide a more academic focus.
The study became a target of criticism, in part, because President Bush used its early findings to justify his call in 2002 for a 40 percent cut in that program, which at that time was funded at about $1 billion. Congress later agreed to provide $981 million for the program and it has remained at that annual funding level ever since.
The findings from the new study are in keeping with a growing body of research linking after-school programs to gains in social and emotional outcomes for students in organized activities after school.
For instance, a review of 73 studies published this year by the Chicago-based Collaborative for Academic, Social, and Emotional Learning looked specifically at programs aimed at developing youths’ personal and social skills and found that such programs could be linked to a wide range of improvement in students.
Compared with their control-group counterparts, that analysis concluded, program participants experienced greater increases in self-esteem and self-confidence, more decreases in problem behaviors, and improved grades and test scores.
But the sizes of the effects in the Promising Practices study are especially dramatic. The 20-percentile-point gain that “program plus” elementary students made, relative to the “low supervision” group, works out to an effect size of .73. That’s more than three times the learning boost that educators get by reducing an elementary school class by eight students, according to Ms. Vandell’s research. And that, in Mr. Dynarski’s view, may be cause for skepticism.
“How do you get dramatic increases in effects from something that’s not even school?” he asked. “You’re in it a couple of hours a week or a couple of hours a day, but the activities you’re experiencing are very fragmented.”
Ms. Vandell said those effect sizes look large in part because the gap was widening over time between the program students and the “low supervision” group. “You have to remember, these were students who were either hanging out with friends or home alone with siblings for two to three days a week,” she said of the latter group.
Though the final report from the new study has not yet been published in an academic journal, researchers last month visited Capitol Hill to share the data with federal lawmakers.
The spending measure for health and education programs that President Bush vetoed earlier this month included a $100 million increase for after-school programs, the first such proposed increase in years, according to advocates. They credit the results emerging from the new study for helping to turn the tide in funding prospects for the federal program.
