What do we expect after-school programs to achieve? Unfortunately, there's no single answer.
Over the last half-decade, after-school programs have moved from the periphery to the center of the national education policy debate. It happened very quickly. Between 1998 and 2002, federal funding for the 21st Century Community Learning Centers program grew from $40 million to $1 billion. State support has also grown. In a 2002 ballot initiative, Californians voted for a six-fold increase in funding for after-school programs. Now, the heat is on to demonstrate that after-school programs are effective.
But what do we expect after-school programs to achieve? Unfortunately, there’s no single answer. Over the last century, they have served several distinct purposes. Some programs have been designed to supervise youths and prevent them from being the victims or perpetrators of crime. For others, the main goal has been general youth development, helping young people explore new talents and interpersonal skills. Many programs serve as a child-care bridge between the end of school and the end of parents’ workdays. More recently, the focus on test-based accountability in elementary and secondary education has led to an increased emphasis on academic performance in after-school programs.
After reviewing the results from several recent evaluations, we have four recommendations for policymakers, evaluators, and those operating after-school programs:
1. Programs must increase attendance or they will not achieve their goals. The most consistent finding among these studies is that many young people attend sporadically and for a short period of time. In the typical program, the average participant in elementary and middle school programs attended between one and two days per week. No program can make a difference if it does not change the daily experiences of young people, and it cannot do that if attendance is poor.
From polling data, we know that the general public feels that much of the responsibility is the parents’. In effect, the argument is that they should “make the kid go.” We are sympathetic to the argument, but as parents we are also realistic. These are voluntary programs, and if they are not engaging, it is hard to enforce participation, particularly as young people age and have more alternatives.
After-school programs can help young people explore and deepen interests, make decisions, try out different roles, and get involved with people, ideas, and activities. Within each study, the researchers saw that when such experiences occurred, young people were more likely to participate. Unfortunately, in too many programs, the activities did not provide the rich, age-appropriate experiences that lead to sustained attendance and engagement.
In a few of the studies, attendance rates were higher when staff members tracked down nonattendees or when programs required five-days-per-week participation. The former is the more promising approach, even though both may produce the appearance of higher participation rates. Participation mandates (three absences and you’re out) may simply weed out young people who have fewer resources, when even limited participation would have been worthwhile. But when staff members hit the streets or the phones to pull kids into a program, they can boost participation among the very young people who will benefit most. Participation is also helped when programs are located in neighborhoods where the young people have fewer alternatives.
2. We need to be more realistic about what it takes to create discernible effects on achievement-test scores. In the national samples used to norm the Stanford Achievement Test-9th Edition, 5th grade students scored only one-third of a standard deviation higher than 4th graders on reading, and one-half of a standard deviation higher on math. This reading-score difference is about as large as the difference in moving from 1000 to 1070 points on the combined SAT,or 100 to 105 on an IQ test. In other words, everything that happens to a student between the end of 4th grade and the end of 5th grade--a while school year of full-day classroom instruction, interactions with family, conversations with friends, and homework--is associated with an important but not huge gain on an achievement test.
With this as a backdrop, consider the typical after-school program, with youths attending one to two days per week for two to three hours per day. While it is reasonable to expect that after-school activities can affect performance as measured by achievement tests, it is likely that such effects will be small. This is particularly true for reading scores, since they are traditionally less responsive than mathematics scores to instruction.
Therefore, even if the programs are helping, effects on achievement tests are likely to be hard to detect statistically. We should balance a focus on test scores with an examination of intermediate effects-- more parental involvement in school-related activities, more diligent homework completion, more school attendance, and better grades, for example--which may pay off in improved test performance over time. Although the studied programs often did not have statistically significant effects on achievement-test scores, some programs did have promising effects in other areas.
3. Programs need to reach vulnerable kids who would otherwise be on their own after school. It seems reasonable to expect that so-called “latchkey” children would benefit the most from after-school programming. Unfortunately, only one of the four recent evaluations kept track of students’ after-school care arrangements. Perhaps surprisingly, the researchers found that many of the program participants would not have been on their own, but with a parent or sibling, if the programs were not available. Reducing sibling care is perhaps a good thing, but it is less obvious that time spent in an after- school program would be more worthwhile than time spent with a parent after school. Future evaluations should focus on whether the programs have had an impact on the number of children under adult supervision after school.
The country is making an investment in after- school programs that warrants asking whether they are delivering.
As we have noted, it will be difficult to discern the impact of after-school programs on academic achievement. It may be even more difficult to identify the impacts they have on hard-to-measure outcomes such as the risk of committing or being the victim of a crime. But if programs truly are reaching a large number of young people who would otherwise be on their own after school, policymakers and the public will be more willing to give the programs the benefit of the doubt.
4. Build on examples that are demonstrable winners. In all the studies, there are individual programs where kids are safe, engaged, and attending consistently. Some programs may only appear to be succeeding because they are starting with highly motivated children. But it is hard to believe that this is the full story.
Within the after-school field, there is reasonable agreement on the key ingredients required for success: interesting activities, supportive relationships, and the capacity to deliver such things. As with public schools, almost everyone would agree on how an effective program looks and feels. There is much less agreement on how to get there. Progress will come when evaluators learn to ask the right questions by listening to policymakers, advocates, and practitioners, and when people in those roles learn from evaluation results.
Indeed, the most recent wave of evaluations offers a number of valuable lessons for all the interested parties. Future programmatic reforms should focus on raising participation rates, particularly among children who would otherwise be on their own after school. In doing so, practitioners should be assertive and creative in how they recruit and retain students. Similarly, evaluators and policymakers need to be clear about the nature and magnitude of expected effects and be sure studies are prepared to measure them.
When federal and local tax dollars are combined with the value of time and resources donated by volunteers, the country is making an investment in after-school programs that warrants asking whether they are delivering. The unequivocal summary from these recent reports is that some are, but the average program needs to get better. Americans need effective programs to ensure that children are spending their time after school safely and productively. Our advice is to turn the political and scholarly agenda toward making that happen.
A more complete review of four recent after-school-program evaluations is available on the William T. Grant Foundation’s Web site at www.wtgrantfoundation.org.
Robert C. Granger is the president of the William T. Grant Foundation, in New York City. Thomas Kane is a professor of policy studies and economics at the University of California, Los Angeles.
