Harcourt Educational Measurement has not had a stellar year. California slapped the testing company with a $1.1 million fine for administrative errors committed last summer. Then this winter, Vermont received rebates from Harcourt because of mistakes in scoring state exams in 1998 and 1999.
Despite such problems, the San Antonio-based company recently landed a $76 million contract to direct Massachusetts’ assessment program next fall.
At a time when state officials are under increasing pressure to implement high-stakes assessments, they typically turn to the same pool of testing contractors again and again—regardless of some embarrassing glitches—because of the dominance of a few large companies. “The fact that we got three substantial bids was actually great. Some states face the problem of putting out proposals and not getting any,” said Jeff Nellhaus, the Massachusetts state testing director.
Growing Pains
As measured by the number of state contracts, Harcourt, CTB/McGraw-Hill of Monterey, Calif., and Riverside Publishing of Itasca, Ill., dominate the school testing market. The three traditionally have made off-the-shelf tests. But, following the demands of the current market, they also have moved into customized, standards-based testing.
They continue to be the big players in this newer market because, in large-scale assessment, size matters. The enormous amount of paperwork involved and quick turnaround of results needed means that large companies can underbid most up-and-comers.
Yet the transition into making customized, high- stakes tests has not been easy for the big companies.
“There are a lot of growing pains,” said Jennifer Vranek, the director of benchmarking and state services for Achieve Inc., a nonprofit group formed by state and business leaders aiming to raise student achievement. “The level of expertise isn’t up to delivering high- quality, rigorous, standards-based exams.”
Meanwhile, the costs—and potential profits—are rising. In 1996, the 50 states spent about $165 million on tests, according to Achieve. The organization, based in Cambridge, Mass., expects the total to double to $330 million this year. To some degree, the leap in costs reflects the higher price of standards-based tests that go beyond multiple-choice questions. By Achieve’s estimates, an off-the-shelf, multiple-choice exam runs $1 to $2 per student. The price jumps to $5 to $6 per student for a test based on a state’s own academic standards that includes performance items—questions that require a student to show what he or she knows beyond just filling in the correct bubble on an answer sheet.
And with the accountability systems that many state legislatures have put into place that rely chiefly on tests, the cost of mistakes by test- makers can extend beyond the price of the exams. Students might not graduate or be promoted to the next grade. Teachers, principals, and their schools may not receive the financial bonuses due them. In some states, low test scores could cost some educators their jobs.
Once errors in scoring come to light, a flurry of lawsuits could follow. Such cases would most likely be aimed at the states and not the testing companies, experts say."Tests provide a vehicle on which decisions can be made. What we provide is accurate data,” said Michael H. Kean, the vice president of public and governmental relations at CTB. He acknowledged that tests are designed for a limited purpose, and states and districts misuse them when they use exams as the only indicator of a student’s success or failure. “High- stakes decisions should not be made on a single measure,” he said.
George F. Madaus, a professor of education and public policy at Boston College and an expert on testing, does not know of any state having been held liable in a testing-product-liability case.
‘None Is Without Sin’
Testing errors have not been limited to Harcourt; almost all the big testing companies have had their share. “None of the testing companies have gotten it right this year,” Wesley D. Bruce, the testing director in Indiana, said. “None is without sin.”
While Harcourt had numerous problems in California last year, including the late delivery of tests and incorrect scores for almost 200,000 students, its mistakes were not the headline grabber of the year. The biggest blooper belonged to CTB/McGraw-Hill.
CTB erred in comparing students’ scores with national rankings. Among other consequences, 3,500 New York City students were mistakenly sent to summer school.
Mr. Kean acknowledged that his company made an error, but he said that “the CTB test was not designed to make those decisions” on whether students had to attend summer school.
Corporate admissions of mistakes and apologies for them have done little to placate Rose Fairweather-Clunie, the principal of Middle School 321 in the Bronx. Based on the incorrect scores, her school was identified by the state as one that needed to be closed.
The erroneous results from CTB showed 22 percent of the students meeting the national benchmark in 7th grade mathematics on the company’s standardized exam. When the correct results were calculated, 32.7 percent of the students scored at or above the standard—just three-tenths of a point less than the school needed to stay open. Nonetheless, the district held firm in its decisions to shut the school.
Had the school received the correct scores at the beginning of last summer, Ms. Fairweather- Clunie believes it would not have been identified as a school that needed to be closed. “Once we were declared ‘phased out,’ the state was set,” she said.
“The attendance rate is up, the suspension rate is down. ... [This is] blatantly unfair,” the principal added. The school has even been identified by the district as an exemplary school because of its improvements on test scores.
Error-Free?
The testing companies acknowledge some of the difficulties. Said Eugene Paslov, the president of the Psychological Corp., a division of Harcourt: “As tests are used for high-stakes purposes, the demands on the [testing] contractors have increased.
|
In the past, if you had an error or two, it wasn’t being reported to the state legislature or the public. Timing wasn’t all that important. The demand for accuracy is much greater now.”
In response to the errors in California, Harcourt has made considerable changes to improve the quality of its product, according to Mr. Paslov. For example, the company has improved its quality-control system and made some personnel changes.
Some observers, however, claim the companies make so much money from state testing contracts that errors are easily written off.
Testing companies find it more profitable to have contracts with many states and make mistakes than to work with just a few and have a flawless performance, contended Julia M. McMillan, a senior project associate with the Council of Chief State School Officers. “Large companies take on so many contracts at once that they make careless errors,” she said.
That leaves states in a precarious position, she and others say. Tight budgets—and in some cases, state laws—often demand that state education agencies select the company that offers the lowest bid. But state officials know that the companies offering the lowest bids often are the same ones making frequent mistakes. As a result, state testing directors are keeping a close eye on the testing companies."The stakes are too high for students, schools, and districts to allow the testing companies to police themselves,” said Mr. Bruce of Indiana.
|
Massachusetts’ Mr. Nellhaus acknowledged that one of the major reasons his state chose Harcourt was its low price tag. Harcourt outbid Advanced Systems, the previous contractor, by $25 million and also promised to provide a quicker turnaround on the results of the exams, which will be used until 2003.
Mr. Nellhaus also said that the state hopes to add a stipulation to the contract that Harcourt pay financial penalties to the state if the company makes any errors.
Building penalties into testing contracts is something more states are interested in, some observers said. But states have also seen such provisions backfire.
For instance, California penalized Harcourt for errors in administering the Stanford—9th Edition. Yet, when the state sent out a proposal to create a high school exit exam, not a single company submitted a bid because of the stipulations on payment for mistakes.
So, the verdict on such penalties is still out, according to Ms. McMillan. For states, she said, “it’s a game of watching to see what the leaders get out of it.”
Breaking the Grip
States have attempted to break the grip of the triumvirate of top testing companies. Ten states, including Massachusetts, have banded together to try to create their own assessment. Known as the Mathematics Achievement Partnership and headed by Achieve, the project will offer states an internationally benchmarked 8th grade math assessment in 2003, if all goes smoothly. States will share the cost of making the test, and the exam also will allow them to compare results across states.
“States collectively are spending hundreds of millions of dollars each year on their testing programs,” Gov. Tommy G. Thompson said last year in announcing Wisconsin’s participation in the partnership. “Our effort will build on the momentum of states’ efforts to raise standards and accountability while taking advantage of new information about academic expectations in the rest of the world.”
Indiana recently took matters into its own hands when its testing contractor, CTB/McGraw-Hill, miscalculated test scores. The state, which is also part of the math partnership, decided to require the company to undergo an external audit by an accounting firm. The firm examined CTB’s quality-control program and its process for administering and scoring the state exam.
The state-requested audit is believed to be the first of its kind in the country, the state education department’s Mr. Bruce said. The audit did not turn up any evidence that the test was unreliable, nor did it show that CTB had committed any egregious errors. The state, however, will require all future test contractors undergo similar audits, Mr. Bruce said.
“The bottom line,” he said, “is that there is a good deal of evidence that the controls in any of these companies are not sufficient.”
