For more than 30 years, the National Assessment of Educational Progress has provided information on what American students know and can do in the core academic subjects. The congressionally mandated project is so widely respected that it’s nicknamed the nation’s “report card.”
Now, a proposal by President Bush could permanently change the nature of the testing program, by using NAEP results to confirm a state’s own testing data before determining federal rewards or penalties for states based on student achievement.
Although many assessment and policy experts support Mr. Bush’s proposal in concept, they caution that it must be executed carefully to preserve the credibility that has made NAEP so valuable. Others, meanwhile, worry that proposals to expand NAEP could inch the United States closer to a national curriculum and a national test.
“NAEP is a precious thing that we have,” said Lauren B. Resnick, the director of the Learning Research and Development Center at the University of Pittsburgh, “and it works as a barometer of student achievement because there isn’t any motivation to teach to it in a direct way.”
The more incentives there are for states to care about NAEP results, to align their tests and curricula with the assessment, and to get their scores up, the more NAEP’s role as a neutral barometer will be lost, she warned. “It’s delicate. I basically think we should do it, but we should do it with great care,” Ms. Resnick said.
‘Willing To Take the Chance’
In 1969, when NAEP began assessing a nationally representative sample of students, the idea of using the results to monitor state testing information would have been politically unthinkable. So much concern arose about the federal government’s role in tracking student performance—and potentially interfering in state and local curricula—that Congress explicitly prohibited the reporting of any state or local NAEP results.
But by the late 1980s, governors and state legislators had become so worried about academic performance that they were anxious to find ways to compare student achievement across state borders. Since 1990, states have had the option of surveying representative samples of their students to yield state-by-state NAEP scores in addition to the samples used to monitor student performance nationally and produce long-term trends.
President Bush’s proposal—part of his broad package of school initiatives unveiled in January—would expand the state NAEP to test a sample of 4th and 8th graders in each state every year in reading and mathematics. Currently, those subjects are tested once every four years at the state level, and state participation is voluntary.
Under the plan, a new fund would be set up to provide financial rewards to states that narrowed the achievement gap between groups of students, such as disadvantaged and nondisadvantaged students or minority and white students, and improved overall student learning based on the results of their state tests. Progress on state assessments would need to be confirmed by state NAEP results before any rewards were given. Similarly, the secretary of education could reduce administrative aid for states that failed to make progress in the achievement of their disadvantaged students, as measured by the state tests and confirmed by NAEP.
“I’m willing to take the chance,” said Chester E. Finn Jr., the president of the Washington-based Thomas B. Fordham Foundation and a former assistant secretary of education in the Reagan administration. “I do think some kind of external audit is needed. I can’t think of a better one.”
A ‘Second Snapshot’
Two big questions are what “confirmed” means and who would make that decision. Studies by the National Research Council have pointed out the difficulties in directly comparing or “equating” the results from NAEP with those from state testing programs. One reason is that NAEP was not designed to reflect a particular state’s curriculum, so the tests may measure students’ mastery of different material. In addition, the testing format, the population of students assessed, and the motivation of students to do well on the exams may differ.
“In terms of thinking about NAEP as a validation check, it’s likely to be difficult to make a direct connection between the two,” said Pasquale DeVito, the director of the NRC’s board on testing and assessment.
At a recent meeting here, Sandy Kress, an education adviser to Mr. Bush, said the administration did not intend to use NAEP for the technical validation or verification of state test scores. Rather, he explained, NAEP would provide a “second snapshot” of student progress. Under the proposal, he said, the secretary of education would look to see whether a state was making yearly gains in closing the achievement gap as gauged by its own measures, and whether that was also true on NAEP.
“In reality, a state shouldn’t get a reward unless you see progress on both barometers,” argued Mr. Kress. “We think it’s important, for the sole purpose of these rewards and sanctions, to have two snapshots.”
As outlined, the proposal would have far less impact on the national assessment than would, as former Vice President Al Gore suggested during the presidential campaign, the use of NAEP results as a direct trigger for financial rewards and penalties.
At its winter meeting last month, the National Governors’ Association endorsed the idea of using NAEP to confirm state test results, but advised that “work must be done to examine questions and issues that may arise to validate this new role for NAEP.”
The governors also stressed that rewards and punishments in any state “should not be based solely on NAEP results but should rely on the state’s own accountability system.”
“I actually was prepared to say this whole thing was stupid, but it isn’t as onerous, as currently stated, as I thought it was going to be,” said H.D. Hoover, a professor of education at the University of Iowa and the president-elect of the National Council on Measurement in Education.
Even so, he pointed out, it’s more technically complicated than simply eyeballing the two types of assessments to see if the trends are headed in the same direction.
“As far as I’m concerned,” he said, “you’re looking at numerical information that changes, and then the question is, ‘Is it reliable? Is it valid?’ and all those things. So, it is still something that somebody has to worry about.”
For example, any test has built-in measurement error, which means there could be cases when state test scores were advancing and NAEP scores were not, or vice versa, and a way would be needed to confirm whether the magnitude of the changes was real.
“I’ve done some of that myself, comparing how Kentucky has done on their tests versus NAEP, so I couldn’t argue against NAEP as a second check,” said Robert L. Linn, a professor of education at the University of Colorado at Boulder. But, he added, “you have to be careful how it’s done because if you start saying, ‘Oh, we’re going to completely discount state trends if they don’t show up on NAEP,’ then, in effect, you’re making NAEP the first cut.”
Conservative Concerns
So far, the biggest resistance has come from some conservative lawmakers and groups that worry that proposals to expand NAEP and make it mandatory for states could lead to a national curriculum.
In hearings before the House Education and the Workforce Committee last week, Rep. Judy Biggert, R- Ill., questioned Secretary of Education Rod Paige about whether the proposal would create a “national standards test.” She said the House bill would likely permit states to use NAEP or an equivalent “nationally recognized test” to check on student progress.
On the Senate side, where the Health, Education, Labor, and Pensions Committee was working on the reauthorization of the Elementary and Secondary Education Act, Sen. Judd Gregg, R-N.H., had drafted an amendment that would allow states to opt out of the state NAEP. Those that did would no longer be eligible for federal rewards.
Participation in the assessment “can be a hardship in some states,” said Mr. Gregg, who ended up not offering the amendment for consideration, “and it’s also a national-testing issue.”
“We do not want the federal government mandating a test,” said Douglas Domanech, the executive director of the national center for home education at the northern Virginia-based Home School Legal Defense Association. “We don’t think Bush’s plan does that,” he added. “Our concerns are more what the indirect effect will be.”
Paul E. Barton, an independent testing expert, said: “It’s all just a matter of degree. I think it’s just a factual matter to say that if you hold up this test as something you should strive to do well on, then by doing well on it, you are doing well on the basis of a national set of objectives.”
Another consequence, Mr. Barton argued, is that people would pay more attention to the NAEP curriculum objectives and frameworks, which guide the development of test items, than they have in the past. “I think people will be looking much more closely at those objectives, and that may create more disagreement and polarization in their formulation than we have been used to,” he said, “simply because they’ve become more important.”
But Lisa Graham Keegan,the state superintendent of education in Arizona, said she was “very comfortable” with the proposal. “I think it’s a good audit for us, and it’s helpful,” said Ms. Keegan, a Republican. “I don’t believe the president will implement the NAEP test in such a way as to be a test of the state’s test.”
Jeanne Allen, the president of the Washington- based Center for Education Reform, a nonprofit group that promotes school choice, said she doesn’t “see how the administration could have the program they’re proposing, and use state tests to really hold states’ feet to the fire, unless they have one consistent standard against which to judge how the states are doing because the state tests are so uneven right now.”
‘A Lovers’ Quarrel’
One option suggested by Mr. Barton is to create a separate test to monitor trends in state assessments, by drawing on already released NAEP items and questions drafted for the system of voluntary national tests proposed by President Clinton but effectively blocked by Congress. “I think that if you want to use NAEP as part of a treatment, you’re better off creating another instrument for that,” Mr. Barton said.
The cost of creating and administering such a new assessment in two subjects at two grade levels would be about $68 million a year, he estimated. “But that’s not ready now and takes time, and NAEP is in the here-and-now.”
Last month, the National Education Goals Panel approved recommendations to give the state NAEP annually and to pay states to do so. One option, suggested by the panel, would be to test in reading and writing one year, and math and science the next.
A more expensive option would be to test in three subjects each year: reading and math every year, and science and writing in alternate years.
The panel recommended more regular testing in science and writing as well, said Emily O. Wurtz, the acting executive director of the goals panel, because the subjects are more likely to be taught and attended to if they’re assessed regularly.
Referring to the differences between the panel’s recommendations and Mr. Bush’s proposal, Ms. Wurtz said: “It’s a lovers’ quarrel, I would say.”
The bill marked up by the Senate education committee last week would authorize $110 million for NAEP in fiscal 2002and such funding as needed for the next six yearsto administer the state NAEP in every state annually in reading and math in grades 4 and 8.
Incentives To Participate
The goals panel also proposed monetary and nonmonetary incentives for schools to participate in NAEP and other federal data-collection efforts. While states volunteer for the state NAEP, ultimately it’s individual schools and districts that decide whether their students will take the assessment.
In the 1999-2000 school year, for instance, 48 states signed up for NAEP, but only 40 had their students take the tests because the others could not persuade enough schools to volunteer.
The goals panel recommended budgeting approximately $1 million per state or other jurisdiction for schools to participate in NAEP and other data-collection efforts, at a cost of $55 million. It also recommended providing schools that participated in NAEP the option of receiving feedback on their students’ performance. Another enticement, it said, might be the production of a teacher kit of NAEP-related material to support instruction.
In a statement adopted last month, the National Governors’ Association said the responsibility of financing NAEP testing in all states should fall on the federal government, with incentives provided to states to encourage districts to take part. The Council of the Great City Schools also is seeking federal underwriting for a pilot program that would allow urban districts to test a sample of their students and receive district-level NAEP results.
Ms. Resnick of the University of Pittsburgh voiced concern that the more NAEP results are provided at the school and district levels, the more NAEP is likely to influence the curriculum. “I think those are getting in the danger zone,” she said of such steps.
Yet many observers agree that persuading enough schools to participate in NAEP is an issue. Moreover, if NAEP results are to be used to confirm disaggregated test scores at the state level, such as the results for poor and minority students, the sample sizes in most states will have to grow, further exacerbating the problem of inadequate participation.
Meanwhile, some states have such small populations of students from minority groups that NAEP may not be able to offer anything conclusive about whether those states are closing achievement gaps between minority and white students, one of the goals of President Bush’s plan.
One option would be to test in reading and math every other year, which many researchers said would still yield adequate trend data. “I don’t see why it needs to be done every year,” said Mr. Linn of the University of Colorado. “Trends, by their nature, take a while to be able to see much. If you have a schedule where NAEP is done every two years, that would seem like plenty to me.”
Others have suggested giving states the option of using commercial norm-referenced tests to verify trends in state testing programs. “That discussion has come up,” said the White House’s Mr. Kress. “We’re open to flexibility there.”
It’s not clear, though, whether the Bush administration would support using federal money to support such alternatives.
“NAEP is there, and that’s fine, but there are other tools that are available and that work just as well,” argued Stephen D. Driesler, the executive director of the Washington-based school division of the Association of American Publishers, which includes leading producers of commercial tests.
Many states already use such norm-referenced tests, in addition to their standards-based exams, to see how their students are performing compared with a nationally representative sample. The tests already yield individual results for students, Mr. Driesler noted, and they cost far less to administer than NAEP.
Legislative Proposal
In draft legislative language sent last week to Rep. John A. Boehner, R-Ohio, the chairman of the House education committee, the AAP proposes that states have the flexibility to confirm improvements in student achievement by using NAEP “or any other assessment that is available nationwide and can provide valid and reliable information for such a purpose.”
In Iowa, the percentage of students who score above the national average on one such test, the Iowa Tests of Basic Skills, and the percent who score above the national average on NAEP are virtually identical, according to Mr. Hoover, the senior author of the ITBS. “I feel quite comfortable that as long as similar populations of kids are being tested, you wouldn’t get very different answers,” he said.
But others said using such off-the-shelf tests was less desirable for a number of reasons: It would fail to produce a uniform measure against which state trends could be compared. The national norms for the tests are different and, in some cases, quite low. The tests are less secure than NAEP, and thus more easily compromised by potential cheaters. And they do not measure the same breadth and depth of knowledge and skills as the national assessment.
In addition, some states now use the ready-made exams for accountability purposes.
“The trade-off, in my mind, is purely political, to avoid making NAEP absolutely mandatory,” Mr. Finn of the Fordham Foundation said of that idea.
Others worry that if states had the option of using an off- the-shelf test, instead of NAEP, it would be even more difficult to recruit enough schools to participate in the national assessment.
The National Assessment Governing Board, which oversees NAEP, has discussed some of the technical issues related to the president’s proposal, many of which have yet to be resolved. Even so, several members expressed support for the idea.
“Of course, only by trying this out will we know how well it works,” said Michael T. Nettles, the vice chairman of the board and a professor of education at the University of Michigan in Ann Arbor. “But it just seems like a logical and reasonable approach.”
