The governing beard of the National Assessment of Educational Progress has fired a team of researchers that prepared a critical evaluation of the beard’s process for setting achievement levels for use in reporting results of NAEP’s 1990 mathematics assessment.
In a letter to researchers and policymakers who had received a draft copy of the final report, officials of the National Assessment Governing Board called the draft “so thoroughly flawed as to be unsalvageable.”
In addition to numerous “egregious errors and misstatements of fact,” the officials stated, there is a “purposeful lack of objectivity permeating the entire document.”
The officials also charged that the reviewers had committed a “political act” by submitting the draft to policymakers, such as Congressional aides and Gov. Roy Romer of Colorado, rather than just to technical experts who could make informed comments.
“It was clear from the beginning they didn’t believe in achievement levels, they didn’t want levels set,” Roy E. Truby, executive director of the N.A.G.B., said in an interview. “It was even more clear they don’t like a N.A.G.B.-type board.”
He said the beard was terminating the evaluators’ services because it was unwilling to invest additional funds in editing the final report.
Richard M. Jaeger, a professor of education at the University of North Carolina at Greensboro and a member of the evaluation team, responded that the board’s charges were “absurd,” and he took offense at the charge that the team was biased and had failed to follow the standards of education-program evaluation.
The chairman of the review panel, Daniel L. Stufflebeam, director of the evaluation center in the college of education at Western Michigan University, wrote the evaluation standards, Mr. Jaeger noted.
The charges are “code words for saying they didn’t like our conclusions,” Mr. Jaeger said. “This is a case of not liking the message and acting to kill the messenger.”
Other researchers familiar with the incident voiced outrage at the beard’s action, and warned it could threaten the credibility of NAEP.
But Governor Romer, who just stepped down as chairman of the National Education Goals Panel, said the controversy would not deter the goals panel from including the disputed data in its forthcoming report.
“My judgment is, we will continue to use the material, and label it the best judgment we have,” he said. ‘“We are aware of the judgmental nature of this, and we will use these figures with appropriate conservatism.” John F. Jennings, an aide to the House Education and Labor Committee, said the committee has asked the General Accounting Office to look into the matter.
“We want them to make a judgment about whether the Stufflebeam team was correct on their points, or whether N.A.G.B. was correct in their rebuttal,” he said.
At issue in the dispute is the NAEP governing board’s decision last May to change the way it reports NAEP results. Unlike in the past, when the congressionally mandated project simply reported how students performed on the assessment, the board agreed to compare student performance with agreed-upon standards for “basic,” “proficient,” and “advanced” levels of achievement.
As part of its effort to set standards, a technically complex procedure, the beard had agreed last year to hire a team of experts to evaluate the process. In addition to Mr. Stufflebeam and Mr. Jaeger, the team included Michael Scriven, a professor of education at Western Michigan University. Although the team’s formal $11,000 contract expired last December, the board asked the researchers to continue their work and to produce other reports. The board has paid the researchers an additional $7,000, Mr. Truby said.
In its draft final report, the evaluation team concluded that, despite numerous positive aspects, the process the board used to set achievement levels “must be viewed as insufficiently tested and validated, politically dominated, and of questionable credibility.”
“If used at all, these achievement levels should be applied with caution,” the report states. “Their psychometric properties are suspect to the degree that they should not be used to infer the quality of 1990 mathematics achievement of grades 4, 8, and 12 students in the United States, and certainly should not be used to form the baseline of a time series representing U.S. students’ mathematics achievement.”
The team also urged the Congress to reconstitute the N.A.G.B. to include a greater number of technical exports, and to invest in a research-and-development project to investigate standards-setting issues.
In a 22-page rejoinder, the board took strong issue with the researchers’ assertions, and concluded that ! their report was “a document riddled with inaccuracies, articulated in unsubstantiated generalizations, and devoid of what most evaluators look for in such a report--data.”
Achievement Levels Defended
“The board believes,” the rejoinder states, “that the achievement levels it has adopted for reporting results of the 1990 NAEP mathematics assessment are reasonable, defensible, and useful; that, despite the rhetoric and bias of the evaluation consultants, the facts argue for release and evaluation of the achievement levels; and that releasing the achievement levels will help to inform the ongoing discussion related to national education standards.”
Mr. Truby denied that the board was attempting to squelch the evaluators’ report, and noted that he was making available the draft report, along with the board’s rejoinder, to all interested parties.
Mr. Jaeger said that, despite the firing, the panel has submitted its final report to the N.A.G.B., and added that he hoped others would read it and come to their own conclusions.
“I’m hopeful people who are less passionate about the argument will evaluate the evidence on its merits,” Mr. Jaeger said.
