Perhaps no topic has as thoroughly vexed officials who oversee the nation’s leading test of academic progress as the wide variation among states and cities in the proportion of students with disabilities and limited English proficiency whom they exclude from taking the exam or provide with special accommodations for it.
The board that sets policy for the National Assessment of Educational Progress has been wrestling with the issue for roughly a decade and has made a number of changes in an attempt to close those gaps.
Yet today, the broad discrepancies in the exclusion and accommodation rates of individual states and cities that take part in the heavily scrutinized exam, known as “the nation’s report card,” continue to spark complaints from those who believe those factors skew the results.
Now, an ad hoc committee of the board is once again looking at exclusions and accommodations in the hope of bringing more consistency to those policies. In addition, federal officials have arranged two ongoing studies of the topic, one to determine whether they can identify a model exclusion rate for states, given their student populations, the other to see whether the official reporting of NAEP scores should be changed to better publicize the effect that exclusions are having on different jurisdictions’ scores.
Just four years ago, members of the National Assessment Governing Board, which sets policy for NAEP, approved a measure they hoped would bring more consistency in exclusions and accommodations. Known as a “decision tree,” the policy was aimed at giving local officials clearer guidance on whether students with disabilities or limited English ability should take the test and under what circumstances.
But exclusion and accommodation rates continue to vary widely today, leading some board members to call for more serious changes.
“I would not keep things as they are now,” said Andrew C. Porter, a governing-board member who is serving on the ad hoc committee. That panel is expected to produce recommendations for the full board by next year. State and city policies currently “differ in ways that are problematic,” he said.
“We know what we want. We want valid state-by-state comparisons” that take exclusions and accommodations into account, Mr. Porter added. “How to get them is a little less clear.”
Participation Rising
Accommodations were first allowed on NAEP in 1996. That change came amid a nationwide push among state and local officials to identify and include more students with disabilities on tests and to provide them with special accommodations, such as extra time or the ability to take a test in a more isolated environment. Changes in federal disabilities law encouraged that movement.
Since then, federal officials have sought to include as many students with disabilities and limited proficiency in English as possible on NAEP, with the goal of providing a more complete picture of academic progress within the student population of participating jurisdictions.
Peggy Carr, the associate commissioner of the National Center for Education Statistics, which administers NAEP, noted that federal officials have succeeded in increasing the percentage of students tested in both categories. In 1992, for instance, only 40 percent of students identified as being in those categories participated in the 4th grade reading portion of the test, while today, more than 70 percent do. This is a significant accomplishment, she said, given the demands of testing those populations.
Unofficial estimates, known as “full-population estimates,” show how jurisdictions would have fared on the National Assessment of Educational Progress in 4th grade reading from 2005 to 2007 if excluded students were included.
Source: Institute of Education Sciences
Full tables for 4th and 8th grade reading and math are available here.
“In the future, I fully anticipate we’ll continue to improve,” Ms. Carr said.
Today, states must participate in NAEP reading and mathematics every two years to receive federal Title I funds. In addition, 11 big-city school districts voluntarily take part in a separate NAEP Trial Urban District Assessment. Seven additional cities will join that test soon.
Still, recent test results have revealed wide gaps in the percentages of students that states and districts exclude from NAEP, or provide with special accommodations. For instance, on the 2005 urban tests in 4th grade reading, two Texas districts, Austin and Houston, excluded 20 percent and 23 percent of students with disabilities or limited English skill, respectively, compared with just 4 percent in Atlanta and 6 percent for Los Angeles and New York City. Federal officials made special note of the Texas districts’ higher exclusion rates in reporting the data.
Those gaps also have been evident on NAEP state results in reading and math. For example, Massachusetts excluded 9 percent of students in both categories combined, in grade 8 math in 2007, while California excluded only 2 percent of them.
IEP Conflicts
But federal attempts to bring more uniformity to exclusion and accommodation rates are complicated by the mishmash of state and local policies governing the issue.
One major obstacle, for instance, is that there are no uniform policies across states or cities on which students are classified as having a disability or lacking English-language skill. As a result, the percentage of students identified as belonging to either category on NAEP varies across jurisdictions.
Another complicating factor: Decisions about what special testing accommodations students with disabilities receive are left to local education officials, who rely heavily on students’ individualized education programs, or IEPs. Those plans, mandated by the federal government for students who receive special education services, are written by school officials and parents.
Sometimes, a student’s IEP calls for a testing accommodation that is allowed on state tests but not on NAEP. One of the most common examples of this disconnect occurs in reading, federal officials say. Some states allow students with certain disabilities to have passages on state reading assessments read aloud to them. But federal officials do not permit that accommodation on NAEP, reasoning that such a practice amounts to a test of students’ listening skills, rather than their reading comprehension.
In those cases, local officials may simply choose to exclude those students from taking part in NAEP. Federal officials have encouraged local officials to consider allowing those students to take part in NAEP anyway, even without the accommodations specified in their IEPs. They also note that individual students’ test scores are not reported on NAEP, as they are on individual state assessments.
Yet local officials are often reluctant to stray from the recommendations of the IEP and permit students to take part in NAEP. That wariness was evident in a 2007 federal study, which found that local decisionmakers typically believe that administering NAEP without accommodations would “constitute a violation of the student’s rights.”
Full-Population Estimates
In an attempt to bring more consistency to exclusion and accommodation rates, the National Assessment Governing Board in 2004 approved the use of a “decision tree” designed to help state and local officials determine which students should participate on NAEP with or without accommodations.
The decision tree asked local officials to consider a series of questions, such as whether a student with disabilities took his or her state’s assessment with or without accommodations, or took a modified test, in making a final decision about NAEP participation. But that process is still contingent on dissimilar state and local testing policies.
“It’s not solved the problem,” said Mr. Porter, the governing-board member, who is also dean of the graduate school of education at the University of Pennsylvania. “We wouldn’t be dealing with this if it had stopped the problem.”
One option would be to have federal officials produce and report statistical estimates for how states’ and cities’ NAEP scores would change if all their students—including those excluded—had participated, or hadn’t been accommodated.
Federal officials already publish a version of those calculations for excluded students, known as “full-population estimates,” on the NAEP Web site. For most states and cities, the impact of incorporating their excluded students in test results would have been relatively small, those estimates show. For instance, in 4th grade reading in 2007, the scores of only three states, plus the District of Columbia, would have risen or fallen by 2 or more points, on a 500-point NAEP scale, from the previous testing year, on the basis of estimates of how their excluded students would have fared.
But the impact of exclusions would have altered scores in some jurisdictions. The District of Columbia’s 2007 gains in 4th grade reading on the state NAEP would have gone from a 6.3-point increase, which was statistically significant, to a 3.1-point increase, which was not. Louisiana’s reading score, on the other hand, would have swung from a 1.8-point decrease to a 2-point gain, though neither change was statistically relevant.
The full-population estimates were developed for federal officials by Donald McLaughlin, formerly a chief scientist at the American Institutes for Research, in Washington, who is now retired. Mr. McLaughlin argues that the full-population estimates, which are based on assumptions about how excluded students would have performed, should be presented as the official NAEP scores, rather than simply shown in online tables.
While the effect of state and city exclusion rates varies with each administration of NAEP, those differences undermines the assessment’s credibility, he argued.
“The impact varies from state to state, grade to grade, subject to subject, and year to year,” he said in an e-mail. “The only way to get rid of the problem, and not to have the uncertainty of exclusion hanging over NAEP’s head, is to use the full-population estimates.”
Studies Under Way
But while Ms. Carr of the NCES said she believes online estimates are statistically sound, she also notes that they are only “hypotheticals about what students might have done,” rather than the scores of actual students who took NAEP. Therefore, federal officials should be cautious before presenting them as official results. To accept the scores, she said, “you have to be able to swallow the premise” of the estimates.
Federal officials have commissioned a study to determine whether it makes sense to include those estimates in the official NAEP reports. That work is being conducted by an independent research organization, the National Institute of Statistical Sciences, in North Carolina.
The second, ongoing federal study is attempting to determine whether it is possible to set reasonable NAEP exclusion rates, for individual states, given their accommodation policies and populations of students with disabilities, Ms. Carr said. A similar study could be performed later for urban districts, NCES officials say.
Mr. Porter and some other governing-board members would like to see federal officials attempt to establish a uniform exclusion and accommodation policies for NAEP across jurisdictions. But others question whether that is feasible and whether states would agree to such measures.
One way of getting around such reluctance would be to have states agree to voluntary exclusion and accommodation policies, said board member Alan J. Friedman. “I’d much rather see the states come together themselves,” said Mr. Friedman, a New York-based consultant on science education issues. “It would be better than it is now.”
