NRC Panel: Rethink, Revamp Testing

The tests that are playing an increasingly central role in education need to be changed substantially to reflect new knowledge about how people think and learn, a report released last week asserts.

The report, by a committee of the National Research Council, argues that the combination of new research about cognition and advances in technology and measurement provides an “opportune time” to rethink the theoretical underpinnings of assessments.

“One of the central dilemmas in testing policy in the United States has been the collision between tests used for accountability purposes and the same tests’ being used to inform instruction,” said Michael J. Feuer, the executive director of the NRC’s Center for Education. “What this report gives us is a road map that would go a long way to connecting tests that can be useful for accountability but that can also provide information for teachers and students.”

Many existing tests, the report notes, focus on discrete bits of knowledge and skill rather than on the most complex aspects of student achievement. They do not chart students’ progress over time, nor do they provide adequate insights into students’ thinking strategies, the nature of their misunderstandings in particular subjects, or the types of help that could best improve learning.

The council’s 17-member Committee on the Foundations of Assessment proposes a “major program of research” that would synthesize current thinking about cognition and measurement in order to design new tests that could yield fairer and more accurate information about students.

It also recommends that states and districts sample a broader range of student competencies and understandings by using a variety of testing techniques or multiple measures of student performance.

“One of the things that we’re pushing for is more coordinated systems of assessments,” which might include such alternative measures as portfolios and tasks that students complete during the course of ongoing classroom instruction, said Naomi Chudowsky, the study’s director.

As part of that effort, the report advocates shifting more attention and resources toward the classroom assessments given by teachers and away from high-stakes, large-scale exams.

“The current educational assessment environment in the United States assigns much greater value and credibility to external, large- scale assessments of individuals and programs than to classroom assessment designed to assist learning,” the report says. “A vision for the future is that assessments at all levels—from classroom to state—will work together in a system that is comprehensive, coherent, and continuous.”

Although creating such coherent assessment systems would require more time and money, Mr. Feuer said, doing so is not unrealistic. He noted that the U.S. Senate has proposed $400 million in fiscal 2002 to help states create new tests. “Even on Capitol Hill, they know if they’re going to get this next round of testing right, it’s going to require a sizable investment,” he said.

Bits and Pieces

The National Research Council’s report, “Knowing What Students Know: The Science and Design of Educational Assessment,” argues that in the past four decades, studies of the human mind have provided considerable insight into how children develop conceptual understanding, how people acquire expertise in particular subjects, and which thinking processes are associated with competent performance.

At the same time, advances in measurement technology mean it’s now possible to characterize student achievement in terms of multiple aspects of proficiency rather than a single test score, to track students’ growth over time, and to analyze which factors contribute to student learning.

Despite such advances, the report says, most tests remain mired in the past, reflecting earlier theories of learning characterized as the step-by- step accumulation of facts, procedures, and definitions.

“What we have is an approach which literally takes bits and pieces of information and treats them as largely substitutes for one another, and doesn’t do justice to the nature of knowledge and understanding,” said James W. Pellegrino, a co-chairman of the NRC committee and a professor of cognitive studies at Vanderbilt University.

Although current tests do a reasonable job of measuring students’ knowledge of basic facts and procedures, the report says, they provide very limited information that educators can use to identify why students do not perform well or how teachers can best modify instruction.

A New Generation

The committee, whose work was supported by the National Science Foundation, suggests crafting a new generation of assessments founded on advances in cognitive research, technology, and measurement.

“If we would spend more time on careful, thoughtful crafting of the assessments and getting them to the point where they could answer the kinds of questions we want answered,” said Mr. Pellegrino, “then ultimately, these instruments would be far more valuable, and the investments would have a far greater return.”

The committee recommends basing assessments on an underlying theory of how student understanding develops in a content area. That theory, in turn, should be based on empirical studies and models of how people learn and exhibit expertise in a particular subject, such as geometry.

Such assessments should focus less on isolated facts and skills and more on how students organize factual and procedural knowledge in a subject, the report argues. Tests also should probe how students monitor or reflect on their own thinking—a process known as “metacognition” that is one of the hallmarks of expertise. And tests should examine the specific strategies that children use to solve problems.

To make assessments fairer, the committee also urges that students’ opportunities to learn the material being tested be taken into account both in designing tests and in interpreting children’s responses.

Too often, the report says, teachers’ classroom-based assessments mirror the same testing formats and scoring practices found in large-scale tests. The report advocates that instruction about assessments and about how students learn become a required part of teacher-preparation and professional-development programs. Such education should be linked to actual experiences in classrooms, both in assessing students and interpreting their development of competence, the report says. And it suggests that states require such teacher education through their licensing and accreditation standards.

In addition, the report calls for new ways of reporting test results that focus less on ranking students and schools and more on tracking students’ development of competence within a subject. Such reporting would include specific benchmarks and examples of where students stand on a continuum of learning.

Harnessing Technology

The panelists argue that alternatives to on-demand, standardized testing of every child already exist. If individual scores are needed, their report says, much richer information about children’s learning could be extracted from classroom work produced during the course of instruction, through such means as portfolios and tasks that students complete while learning the curriculum, but that are common across schools or districts.

If the primary purpose of the tests is program evaluation or accountability, the committee says, it may not be necessary to test every student.

The report is particularly optimistic that new advances in technology could help remove some of the constraints that have limited testing practices in the past. Sophisticated computer-based programs, for example, can track the sequence of actions students take to solve problems, analyze patterns of correct and incorrect reasoning, and provide rapid and informative feedback to students and teachers, while also serving accountability purposes.

One of the best examples, Mr. Pellegrino said, is “intelligent tutoring” systems, such as those used in algebra. Students answer questions on a computer, which matches their responses to a cognitive map of how student thinking in algebra develops over time.

When a student asks for help, the computerized “tutor” can estimate where the student is in his or her development and provide hints that are tailored to the student’s particular approach to the problem.

Bringing together scientists, educators, task designers, and psychometricians to work on such efforts, the report argues, could result in a “significant leap forward in the field of assessment.”