Two states may have received unfair boosts in their scores on a 1998 National Assessment of Educational Progress reading exam because they removed a higher percentage of disabled students from the test than in previous years, according to a preliminary review of results from the federal program.
Assuming that the 4th graders removed from the sample would have scored at the lowest end of the assessment spectrum, the gains made by Kentucky and Maryland would not have been statistically significant, as had been previously reported, according to the review by the Educational Testing Service.
Its findings, which had been awaited with intense interest by testing experts and state officials, were released here last Friday at a meeting of the National Assessment Governing Board.
One state emerged in a more favorable light from the ETS analysis. Based on the review, Minnesota’s increases would be considered statistically significant--earlier, they were not--because the state excluded fewer disabled students from its testing in 1998 than it did in 1994, Pascal D. Forgione Jr., the national commissioner of education statistics, told the governing board.
All other states participating in the test would have remained in the same one of two categories: either no statistical change, or a statistically significant improvement in performance. No state showed a statistically significant decrease from 1994 to 1998.
In unveiling the report, Mr. Forgione added that he would not change the results that he released in March from what is often referred to as the nation’s report card. While the exclusion rates may have affected the scores in Kentucky, Maryland, and Minnesota, other factors are just as likely to have influenced the results, he said.
“The results are what they are,” Mr. Forgione said.
“You could ask about the impact of poor children in a state,” he added. “That fluctuates,” just as exclusion rates do.
Results Raised Questions
When Mr. Forgione, who heads the U.S. Department of Education’s National Center for Education Statistics, released the 1998 NAEP reading results in March, Kentucky and Maryland were two of nine states that posted statistically significant gains on the 4th grade test compared with their students’ performance in 1994. The others were Connecticut, Colorado, Delaware, Louisiana, South Carolina, Virginia, and Washington. (“A Glimpse at the States With Big NAEP Gains,” March 10, 1999.)
The assessment was based on a sampling of achievement of 31,000 students in 1,400 public and private schools.
After the results’ release, some states’ gains were called into question because they excluded a higher percentage of students with disabilities or limited English proficiency in 1998 than they had in previous years.
Kentucky, for example, held out 10 percent of its potential sample of 4th graders last year, compared with just 4 percent in 1994 and in 1992--the only other years the current NAEP reading test has been given. Most of Kentucky’s growth in its exclusion rate occurred because of the increase in the number of disabled students requiring test-taking accommodations spelled out in their special education plans. (“Education Dept. Examining Rise in Students Excluded From NAEP,” March 24, 1999.)
For the 1998 exam, the assessment governing board, which oversees NAEP, issued new rules, which were expected to include more students than under the previous policy. Under the new rules, any child with an individualized education plan, or IEP--which is required for special education students by federal law--that explicitly says he or she needs accommodations to be tested fairly is supposed to be removed from the NAEP sample.
Under the rules followed when the reading exam was given in 1992 and 1994, students were to be included in the NAEP testing if they spent at least half their time in a mainstreamed classroom. Principals also could pull students from the test.
In reviewing all 36 states with student samples from both 1994 and 1998, the ETS analysts assumed that students excluded from the 1998 tests would have scored about the same as students who scored the worst in earlier years.
However, Mr. Forgione said, the excluded students might have scored higher than that if they had been tested, even if they hadn’t received the accommodations listed in their IEPs.
The 1998 NAEP exam faced the same puzzling question that most state assessment programs are trying to address: How should states include or accommodate disabled students?
Changes in policies in particular states make it hard to compare their performance from year to year, said Mark D. Musick, the chairman of the NAEP governing board.
The board is only beginning to review ways of addressing those variations, Mr. Musick said. One way to highlight a state’s change in exclusion practices, he suggested, would be placing an asterisk next to its scores, just as NAEP does when a state does not recruit enough participants for a statistically valid sample.
False Assumptions?
Even under what Mr. Forgione called a “harsh model” of interpreting the results in light of the higher exclusion rates in some states, only Kentucky, Maryland, and Minnesota would have shifted categories.
Other states, such as Connecticut, Louisiana, and South Carolina, had sharp rises in exclusion rates, but the demographic makeup of the student samples there did not contribute to the statistically significant score increases, the ETS found.
Last Friday, Kentucky’s commissioner of education questioned whether his state’s gains should have been called into doubt.
“We would challenge the assumption ... that the students who were excluded were the lowest-performing students,” Wilmer S. “Bill” Cody, who is also a member of the assessment governing board, said after Mr. Forgione’s presentation. “It’s not true in Kentucky.”
Mr. Cody said he estimates that most of the additional special education students excluded in 1998 in his state perform above the lowest levels on other exams. These students, he said, might have a learning or other disability that doesn’t diminish their academic abilities.
Mr. Forgione said he has contracted with the National Institute of Statistical Sciences to evaluate the ETS study and conduct its own analysis. The institute may even create a model that assumes the excluded students would have scored better than the lowest levels.
Mr. Forgione said he expects to have the institute’s research available later this year, perhaps as early as this summer.
