The latest lackluster results from a study on the impact of the Reading First program offer little insight into which parts of the program are worth saving and which need revamping—vital questions, experts say, that could help guide any future federal endeavor to address the nation’s reading woes.
One of the largest and most rigorous studies ever undertaken by the U.S. Department of Education found that the $6 billion funding for Reading First has helped more students “crack the code” to identify letters and words, but it has not had an effect on reading comprehension among 1st, 2nd, and 3rd graders in participating schools.
But the study’s limited scope—the impact of the federal funding on reading comprehension—does not offer enough information to craft better plans or policies aimed at reform.
“My one complaint about this evaluation is that its design was powered to ask a limited question (Does RF work on average?) rather than the contextual question of: Under what conditions does RF work and why?” Barbara R. Foorman, a prominent reading researcher at Florida State University in Tallahassee, wrote in an e-mail.
Ms. Foorman, who served as commissioner of education research at the Education Department’s Institute of Education Sciences in 2005 and 2006, said she is reassured by the findings that Reading First schools are spending more time teaching and practicing basic skills such as decoding, and seeing improved student achievement in those areas. She is disappointed, however, in the lack of improvement in students’ understanding of text—outcomes, she said, which are to be expected given the program’s priorities of teaching the fundamentals.
The final impact study does not provide much hope to Reading First advocates at a time when Congress has slashed funding—from $1 billion a year since the program was launched in 2002 to $353 million in fiscal 2008—and has proposed eliminating it altogether. Proponents of the program point to other findings from state data that show greater gains on several measures among participating schools.
“It’s not such a bad piece of work, but there are no conclusions you can draw from it,” Stanford University researcher Michael L. Kamil said of the study. “But as you look at the whole set of data that we have on Reading First, it is much more encouraging. There’s lots of evidence that this has worked.”
Those state-reported testing data and surveys, Mr. Kamil acknowledges, were not gathered using rigorous methods or compared against results at similar schools that are not in the program.
The impact study, released by the IES Nov. 18, is the only scientifically rigorous review of the grant program. Some of its findings are consistent with those state reports and independent surveys. More time is spent on structured reading instruction and teacher professional development, for example, in schools that received Reading First grants than in comparison schools.
The $36 million study may have left more questions than answers, many observers say.
“There is very little in the report that’s useful,” said G. Reid Lyon, who as then-chief of the reading-research arm of the National Institute of Child Health and Human Development helped draft the Reading First legislation and the requirements for evaluating the program under the No Child Left Behind Act. “The only way for it to be helpful was in an effectiveness study, which was supposed to look at which kids did respond [to Reading First-style instruction] and why, and which kids did not respond and why.
“This evaluation,” he added, “did not help us to improve the program in any way.”
Missing Elements
The study was reconfigured to fit a reduced budget for the evaluation, and also because it was started after implementation of Reading First began, making it unfeasible to assign schools randomly to the treatment or comparison groups, according to federal officials.
Broader questions, however, are the ones that policymakers and researchers say could help refocus a federal reading initiative. It is unclear whether Reading First will survive—Congress’ fiscal 2009 budget proposal provides no funding, but the program will receive continuation funding at least through next March—or if Congress or the new administration will push for a new plan for addressing the nation’s reading woes.
Congressional staff members debated future directions for reading policy last month at a briefing on the impact study, according to Grover J. “Russ” Whitehurst, the director of the institute, the Education Department’s research arm, until Nov. 21.
Those present at the briefing, he said, were not suggesting that the federal government abandon its efforts, but rather try to discover what is missing from the current reading program. Stronger pre-K literacy programs, adolescent literacy, efforts to improve students’ background knowledge, and teacher preparation were all discussed, Mr. Whitehurst said.
The impact study shows clearly, he said, that the program needs improvement.
“Advocates for the program will be pleased that it’s shown a positive correlation on [improved] decoding skills ... the focus of the program,” said Mr. Whitehurst.
Among both students in the Reading First and non-Reading First schools examined in the study, reading achievement was low and had not improved significantly over the course of the three-year study. Fewer than half of 1st graders and less than 40 percent of 2nd and 3rd graders showed grade-level proficiency in their understanding of what they read. Yet, on a basic decoding test, 1st graders in Reading First schools scored significantly better than their peers in the comparison schools.
“I don’t think anyone should be celebrating the fact that the federal government invested $6 billion in a reading program that has shown no effects on reading comprehension,” Mr. Whitehurst added.
Critics of Reading First say the lack of results are because of the program’s narrow focus on explicit skills and the inadequate attention to the complex set of knowledge and skills children need to learn to read.
“You can’t get a much more concentrated effort to make something positive happen than this program has,” said David Reinking, an education professor at Clemson University in Columbia, S.C., and the vice president of the National Reading Conference, a research organization based in Oak Creek, Wis. “The whole [approach] rested on the assumption that improved reading fluency and decoding would translate somehow directly into gains in comprehension.”
The study, he added, “certainly isn’t a ringing endorsement for Reading First, and by extension ... of scientifically based evidence, as being the magic bullet.”
Similar or Not?
Some observers argue, though, that the study’s complex design makes it difficult to draw positive or negative conclusions overall. It compares Reading First schools with similar ones in the same districts that are not part of the program to determine the impact of the extra funding on instruction, reading proficiency, and the relationship between instruction and students’ comprehension.
Between 30,000 and 40,000 students in grades 1, 2, and 3 were given a reading-comprehension test four times from fall 2004 to spring 2007. The students attended nearly 250 schools in 17 districts and a statewide jurisdiction.
The study also includes extensive classroom observations to identify the instructional practices in both types of schools, as well as surveys of teachers, principals, and reading coaches.
An interim report on the findings, released in May, drew scathing criticism from supporters of the program, who suggested that the design of the study was flawed because it did not consider the likelihood that Reading First principles and practices had spread to schools that were not in the program. (“Reading First Doesn’t Help Pupils ‘Get it’,” May 7, 2008.)
Other studies have found that a significant proportion of schools serving struggling students have incorporated explicit instruction in the basic reading skills found to be essential in learning to read: phonemic awareness, phonics, fluency, vocabulary, and comprehension.
But Mr. Whitehurst dismissed those claims, saying that although there may be some “bleed over” into non-Reading First schools, the classroom observations and survey data show that the schools are not so similar.
“The schools were not doing the same thing,” he said. “There were differences in professional development, there were differences in their use of reading coaches, ... and there were significant differences in classroom practices.”
Some experts, however, disagree.
“The schools in the study started out as different as one could expect,” Mr. Lyon said. “But as time went on there was more districtwide adoption of Reading First methods. I’m not at all persuaded.”
