Testing Computerized Exams

It’s 8:15 a.m. on a Monday in early spring and about two dozen teenagers at Century High School here are taking the Oregon Mathematics Performance Assessment, the state-required test to determine their progress toward meeting the state’s academic standards.

Standardized tests are old hat to most of these 10th graders. They’ve put No. 2 pencils to the salmon-pink grids of 0’s on the Scantron sheets since 3rd grade. But today is different: They’re gazing into computer monitors and using a mouse to select their answers, clicking a black dot into the little circles next to their choices on the screen.

Slouched in front of computers in a brightly lit lab on the south end of this 1,800-student suburban campus, these students are part of a trial of a new online testing system that Oregon officials plan to roll out this coming fall to 300 schools, and—assuming that the legislature will foot the bill— eventually make available to every school in the state. Beyond that, the students are part of a growing national trend toward using computers to administer tests. Virginia finished its own pilot study of online testing this spring, and the South Dakota legislature recently voted to give its academic assessments online beginning in the spring of next year. Other states are considering similar steps.

Such efforts are among the baby steps in a “complicated evolution” toward online testing throughout education, says Randy E. Bennett, the online-testing expert at the Educational Testing Service, the world’s largest private testing organization, based in Princeton, N.J.

“This evolution is going to happen; it’s going to take a while, but it’s going to happen,” Bennett says.

Yet in the same breath, he concedes that “there are going to be setbacks,” and that “it’s going to be costly.”

“Simply giving assessments on computers in a school setting is a big jump,” he says.

The ETS administers some graduate-level standardized tests by computer at special testing centers, and a pilot study is under way to evaluate the strengths and weaknesses of offering the SAT—the widely used college- entrance exam—online.

One of the biggest lessons learned by those who have ventured into this new way of testing, Bennett says, is “the way to make it happen is to do it in relatively small steps, not bite it all at once.”

On this spring day at Century High, the students have just completed a 10-minute tutorial for the online exam and are now encountering some of the potential pitfalls of online testing. More than anything else, they are having trouble settling into a rhythm.

The questions themselves are the familiar sorts—word problems, geometric figures and diagrams, algebraic equations, and word definitions, each followed by four possible responses. But on computer screens, some students say, the standardized questions look unfamiliar, eliciting a strange feeling that they might not know what they’re doing.

One girl shifts in her chair as she gazes at the first question on the test, which asks her to find the least amount of money needed to purchase 45 cookies, based on a table with different prices for a single cookie, a half-dozen, and a dozen. She moves her mouse arrow across the on-screen calculator and clicks in a few numbers. But then she stops, releases the mouse, and retreats to the familiar comfort of a pencil and starts scratching equations on a sheet of paper.

The paper and pencil had been passed out by the two test proctors at the start of the test, along with regular hand-held calculators and reference sheets of common mathematical formulas—even though the testing system has on- screen equivalents of the calculators and reference sheets.

Other students in the room also are moving back and forth between trying problems on the screen and on paper, seemingly unable to decide which is the better approach.

One frustrated boy clicks the “Next” button repeatedly, searching for a test problem to his liking. Then he starts repeatedly clicking the “Back” button to go back to questions he skipped over and did not answer. Still, the online test seems a natural fit for other students.

Among them is a girl with short red hair, who uses both pencil and paper and the computer calculator in an orderly fashion to figure the cost of flooring to cover an L-shaped corridor in a house plan. After writing down equations in neat rows on paper, she plugs the numbers into the on-screen calculator.

Only one aspect seems unnecessarily difficult for her: The screen doesn’t show the entire floor plan at once. As a result, she must click the mouse to move the scroll bar up and down to view different sections of the diagram, which makes it harder to visualize the whole floor plan than if it were on paper. In response to that, she and other students draw the diagram themselves on scratch paper.

Despite the technical quirks and their unfamiliarity with taking tests online, several students, after they finish taking the two 30-question, multiple-choice tests, say they prefer the online format.

“I think it’s better on the computer,” says Tina Leung, 15, because “it’s easier to go back [to earlier questions].”

Leung—who, like most other Century High students, has a computer and Internet access at home—adds that the paper tests seem outdated. “If we have [computers], we should use them,” she says.

The trial has placed these students in an important role—to help work out the kinks so the state’s Technology Enhanced Student Assessment system, as it is called, lives up to its name—by enhancing Oregon’s system of academic assessment, rather than making it harder to give a test than it is with the time-tested paper-and-pencil format.

About 4,000 students at 28 elementary and high schools in Oregon are participating in the trial of the online versions of mathematics and English assessments; their test scores and critiques will be used to improve the program.

Initially, it will cost the state nearly $600,000 a year to pay a testing company to run the online tests. Within three or four years, state officials predict, the computerized format will save Oregon money on its assessment program by cutting printing and document-handling costs, for example. Still, a net benefit is far off because of the additional costs of ironing out technical wrinkles and providing adequate technology.

Moreover, in Oregon and elsewhere, there are still many questions to answer and concerns to address about online testing, says Michael K. Russell, an education researcher at Boston College. Russell, who works in the college’s Center for the Study of Testing, Evaluation, and Assessment, has conducted several studies with other center researchers comparing schoolchildren’s performance on standardized writing tests when using paper vs. a computer.

Their main conclusion: students who are accustomed to writing on computers tend to do better on computerized tests than on paper exams. Conversely, students who don’t use computers often to write tend to do better when they complete their tests on paper.

Bennett of the ETS says the findings of Russell and other researchers are a predictable result of students’ switch to high-tech modes of learning.

“It’s very clear that as kids become used to and routinely do writing on computers, paper-and-pencil tests don’t do a fair job of determining their skills,” Bennett says.

In response, he says, educators and policymakers must revamp tests to reflect those changes. “Tests that are delivered in a mode that’s different from the one students are learning in will eventually become indefensible,” he argues. “They won’t be credible to parents, teachers, and students. In end, they won’t be credible to the testing community either.”

But Russell cautions that the move to online testing risks introducing new inequities into state assessments. “As soon as you enter technology into the picture,” he says, “despite the problems of testing in its current form of paper and pencils, you automatically have to start thinking about the haves and the have-nots and the digital divide.”

The Oregon trial is getting at those questions and concerns by giving students one test section on paper in addition to the two sections on computer. The idea is to compare the scores on several overlapping questions between the online and the paper versions. State officials want to determine if the academic knowledge and skills of less technologically skilled students would be underestimated by a computerized test.

Several Century High students, however, pointed out a flaw in the state’s computer vs. paper comparison. They said they easily remembered the answers from the online tests and simply put those down on the paper test, which they completed during the same 90-minute session.

The fear among some experts that students who have less exposure to technology will score lower is based on several studies that conclude that people who are unfamiliar with computers are at a disadvantage on computer-based multiple- choice tests. But Russell says that research was focused on subjects who were age 16 or older. Younger children, he points out, may perform differently because they were exposed to technology earlier in their lives and have not developed test-taking routines. Beyond that, Russell’s list of problems to be solved includes several mundane but essential ones, such as how to avoid the loss of data when computer systems crash, how to keep the costs of using online tests at an affordable level, and how to ensure that all students—including children with disabilities—can participate. Those are all problems that Oregon, South Dakota, and Virginia officials are grappling with.

State and local education officials are also worried about the security of online test questions and results. The Oregon tests, delivered via the Web, can be made available anywhere in the school building. For now, though, Century High School administrators have limited computerized test-taking to two computer labs. And the online assessments won’t be accessible from homes.

In those two computer labs, under the watchful eyes of proctors, students log on to the system with their student-identification numbers, followed by a “daily access key,” a number that is changed every day, and a test key for the specific test they are authorized to take.

Such measures address some of the concerns people have about the security of online tests.

“It’s very often the case that people confuse Internet delivery with lack of security,” says Bennett of the ETS. “The two are unrelated—you can have highly secure online tests.”

School officials here in Hillsboro say that the new system will strengthen the district’s ability to keep ahead of its assessment needs at a time when its student population is being transformed by economic and demographic changes.

Hillsboro County, a short drive west of Portland, has grown up with the sprawl of that city. Its school enrollment has increased by 14 percent, to 18,000 students, since 1996, when it was formed by merging seven school districts. It is now the third-largest school district in the state.

The district has a highly mobile population, thanks to the influx of employees who move here to work at the Intel and Nike corporations, which have their headquarters nearby. Seasonal farmworkers come and go with their children, and a growing number of Cambodian and Vietnamese families are moving into the area.

Student mobility, therefore, is exceptionally high—with about 30 percent of Century High’s student population turning over each year, says Steve Callaway, the Hillsboro district’s director of communications. And that makes placing students in the right classes a difficult task.

Administrators hope online testing will help make some of those tasks—especially placing students in the appropriate academic classes—much easier because the school will be able to test students soon after they walk in the door and then review the results of those tests immediately.

They also want to use data on students’ performance to modify the academic program and teachers’ professional development, says Kathi B. Robinson, Century High’s assistant principal for assessment. “It would give us the opportunity to modify the curriculum and beef up programs in some places,” she says.

As it is, the assessment data that Century High students generate through hundreds of hours of paper testing currently gives almost no help to the school, officials here admit.

“It doesn’t drive instruction at all,” Robinson says.

The problem is teachers simply don’t get the paper-test results returned in time to use them. Because it takes so long to grade and sort paper exams, the teachers don’t see their students’ results from the winter or spring testing periods until July or August, well after the students have finished their classes.

And forget about giving the test scores to the students’ teachers the following year, school officials say. Summer vacation erases the diagnostic value of the tests—especially for students who take a double block of English in the fall semester, and don’t take English again until the following fall.

Meanwhile, Virginia state education officials say they are making the move to online testing because of similar problems. “A key factor in all of this is that we want to get scores reported back to the school divisions as quickly as we can,” says Mark J. Schaefermeyer, the associate director for Web-based assessment in the Virginia Department of Education. “The electronic-testing environment should let students know how they did virtually instantly.”

Beginning next year, the online tests in Oregon will return students’ exam scores to teachers immediately.

Getting those scores right away would be a “huge change” and “an enormous advantage for everybody,” says Gail Young, who teaches 10th grade advanced language arts at Century High School and helped proctor the trial of the English assessment.

“If I had it, I’d look at the composite [scores], and if everybody’s down on drawing conclusions or reading charts, I’d adjust my lessons,” she says.

The fast results would also allow schools to wait longer before administering the assessments in the semester, so that students would have completed a greater share of the coursework.

As it stands now, the Century High students who take the paper assessments in January—just a week before the semester’s end—have completed about 90 percent of their instruction. But students who take the same test in March have barely finished half their instruction, and they often complain that they are tested on skills they haven’t been taught, teachers and administrators say.

What’s more, Shawna Dehler, who teaches 10th grade advanced algebra at Century High, says the online system would be much more flexible than the paper assessments, which are administered to students in groups at just a few designated times and locations. The online system could easily be used with a student or two—new transfers to the school, for example—in the computer lab at almost any time.

And if state officials make the online assessments available at all times, as they plan to do, teachers potentially could have students take the exams as pretests at the beginning of the year and multiple times thereafter, making such assessments a tool to measure progress during the year.

“It’s a huge benefit to be able to pre-test and post-test,” says Dehler.

Her colleague, Linda Butler, who teaches algebra and geometry at the school, adds that “having results within a reasonable time would give us the advantage of retesting on a timely basis.”

Not only would teachers see how their students measure up individually against the state academic standards, but the students would, too, Young says.

State officials say that when the final versions of the online exams roll out in the fall, the last question students will see on the tests will be: “Would you like to know your score?”

“Students ... want to know their scores,” says Young, who has taught for 29 years, including four at Century High. “Many of them, especially the advanced kids, they really care. [But] by the time they get back in the fall, they don’t care.”

The Oregon trial is revealing a number of additional problems with online testing—some easily correctable and others seemingly harder to fix in the immediate future.

Some text passages, for instance, were presented in a tiny typeface. And scroll bars often appeared in the middle of questions.

“The questions were very poorly formatted,” says Barbara Ternus, a math teacher who proctored some of the math assessments.

On the English assessment, problems with scrolling apparently were worse, according to Young. “At some points, kids were having to scroll horizontally to read the text passages, and scroll vertically to see the answers"—undoubtedly a glitch that could raise their test anxiety.

The more serious problems have to do with students’ computer skills and how computers are used by teachers in their classrooms to help students learn.

To be sure, the teenagers here have spent a lot of time with computers, but that might not be enough, Ternus suggests. “They’re not used to critical thinking on the computer,” she says.

Almost none of Century High’s teachers use computers in math instruction, Ternus says. And just a couple of teachers use software that teaches students how to use mathematical spreadsheets.

Ternus acknowledges: “I think as a teacher I have to do more math on the computer, just to make them comfortable thinking on the monitor.”

Having all students take the test several times a year would eat up far too many class days, Ternus concludes. But to be able to use it at will in special cases—such as when a student transfers into the school—"would be fantastic,” she says.

Two weeks ago, a student who transferred to Century High in the middle of the term had told a guidance counselor that she had been enrolled in geometry at her previous school. It took more than a week for school officials to realize that she had been in a pre-algebra/pre-geometry combination course, and was far from ready for the material she was being given. Such scenarios might be avoided with the quick diagnostic features of online tests, administrators here believe.

Meanwhile, students who were in the trial have additional insights into the strengths and weaknesses of the online exams.

“It was easier because you don’t have to fill in all the bubbles [with a pencil],” says Danielle Hadley, 16.

The teenagers’ complaints about the test centered around the presentation on screen.

On the math assessments, some graphics were too small and blurry. And almost everybody had trouble using the on-screen calculators, which were available in four-function, scientific, and graphing varieties, depending on the test question. In response to those troubles, most of the test-takers used the hand-held calculators the school provided.

The hand-held calculator “is not as frustrating,” says James Olson, 16. He explains that the computer calculator had a totally different feel from the hand-held device’s number pad, and that operating the computer version demanded too much precision with the mouse.

On both the math and the English assessments, some questions required a great deal of scrolling to read a question or find an answer, students report. That hampered sizing up figures with a glance and skimming text—test-taking skills that students have learned to use on paper exams.

Another problem that cropped up was more frustrating: For about 20 minutes, the system slowed down significantly, making students wait for problems to appear on their computer screens. Then the system shut down completely—though in the five minutes it took for the school’s computer technician to phone the testing company, it was working again.

Identifying real and potential problems is what the trial is all about, says Dave Parker, the Century High assistant principal who was put in charge of the project.

Robert M. Olsen, the team leader of the state education department’s Technology Enhanced Student Assessment initiative, adds that the online system will become more powerful as the formatting and other start-up bugs are ironed out and as several new features are added.

For instance, Olsen says, the online system will eventually be able to handle short-answer and essay questions in addition to multiple-choice questions. And, as early as next year, the online assessments will have adaptive features that adjust the difficulty level of the questions while the tests are in progress, depending on how an individual student is performing.

In fact, one day the online tests, with their adaptive features, could be combined with a system for delivering online instruction tailored to students’ individual strengths and weaknesses, Olsen says.

The adaptive-testing component that Oregon officials plan to add next year would be as close to the cutting edge of testing as any state has ventured so far, experts say.

The computer would switch a student who had missed several items to a series of questions that were less advanced and which, it is hoped, would isolate weaknesses in skills. A student who completed a string of questions without errors would be switched to a more advanced track of questions.

The system must be planned carefully to make the final scores comparable for all students.

Oregon can add an adaptive feature quickly, state officials believe, because the paper version of the state assessment already uses three difficulty levels, which teachers hand out to students based on their knowledge of students’ abilities. However, students don’t move from one level to another once they begin the test, as would happen with an adaptive test.

Despite the potential benefits of adaptive testing, Virginia education officials, meanwhile, look at such testing with a bit of trepidation.

Virginia, which ran a competitive trial of four online versions of its state assessment in about 20 schools this spring, chose not to include adaptive testing in its plans, says Cam Harris, the assistant superintendent for assessments in the state education department.

“We’re looking at things from a very traditional and conservative perspective, looking at the statistical things we need to do to have fair and valid tests,” she says, noting that one potential pitfall of adaptive testing is that students can pass or fail—in a “high stakes” test like Virginia’s—based on fewer test items in a particular skill than if they were taking a test that was not adaptive. The state doesn’t want to get too far ahead of public perceptions about what is fair, Harris says.

But at Oregon’s Century High, some teachers believe the potential benefits of adaptive testing far outweigh its drawbacks.

Ternus says adaptive testing could isolate weaknesses in students’ skills. She explains that if a student missed a geometry question, for example, a subsequent pattern of questions might determine whether he or she had missed building-block concepts.

Looking ahead at the future of testing, Ternus says “the adaptive portion would be fantastic.”