Researchers Play Catch-Up in Gauging Beyond-School Effects

Emerging research shows the science school-age children learn in informal settings—from museums and clubs to online communities and television shows—can have a big impact on their lives. Yet the open format and distinct structures of informal science make it next to impossible for researchers to evaluate the quality of those experiences in the same way they can gauge formal schooling.

School assessments generally focus on cognitive measures, such as what a student knows and can demonstrate about particular content. In contrast, informal learning is dominated by noncognitive measures such as motivation, interest, and identity, according to Larry E. Suter, the National Science Foundation’s program director for informal science education. Moreover, traditional “gold standard” research methods such as randomized controlled trials can be detrimental to activities that base their strength on people’s choosing to participate, rather than being assigned.

Such research raises the risk, Mr. Suter said, that “if you touch it, you’re going to kill that thing you’re trying to study.”

That has led to some creative research alternatives. Alan J. Friedman, a former director and chief executive officer of the New York Hall of Science and the editor of the NSF’s 2008 framework for evaluating informal science education, recalls judging the effectiveness of an astronomy exhibit by the number of visitors who chose an astronomy poster over a different prize. Barbara N. Flagg, the director of the Multimedia Research consultant group in Bellport, N.Y., said she has used smudged museum walls indicating where visitors have touched exhibits, changes in Google and Amazon search terms over time, and phone interviews with parents and children.

“If you’re comparing this to other education research, you’ve got to turn back the clock 30 or 40 years,” said Kevin J. Crowley, the director of the University of Pittsburgh’s Center for Learning in Out-of-School Environments and an associate professor of education and psychology. “We are just now in the Wild West frontier, and people are just starting to gear up the longitudinal studies on how this will coalesce into a coherent narrative of how people learn science.”

Now, the tools being born of that creativity show the potential to link children’s education across a lifetime of different experiences, and in the process uncover more of how and what children learn than has ever been measured in a school test alone. Children, after all, spend more than 80 percent of their waking hours outside the classroom.

“The research has evolved,” Mr. Crowley said. “In the past, the great flaw of the informal-learning science was we looked at it in situations; we weren’t really looking at a lifelong trajectory in science learning. For the first time, we’re asking questions about how learning and participation are moving across place and across time.”

“Brenda’s” school performance, for instance, didn’t really show her scientific achievements or level of interest in the subject. Teachers reported the first-generation Haitian daughter of a single mother in Seattle had little interest in science and consistently “failed to engage” with the chemical-mixing tasks in her school lab.

Yet because researchers at the University of Washington, in Seattle, had been observing the 4th grader across formal and informal settings for more than 2,000 hours, they knew that school didn’t tell Brenda’s whole story. Not only did she regularly measure and mix chemicals and record the results for her perfume-making hobby, but she also had told the researchers she was considering becoming a chemist when she grew up.

“School science underrepresents her developing expertise,” Philip Bell, an associate professor of learning sciences at the university and the director of ethnographic and design-based research at the Everyday Science and Technology Group there, said during a recent lecture. “Just in terms of how people learn, our literatures don’t do justice to the varied pathways that people take through their experiences to make progress on things they care about.”

Interest vs. Grades

That’s a dangerous disconnect, experts say, because mounting evidence shows that early engagement, even through informal pathways, eventually can lead to careers in the STEM fields of science, technology, engineering, and mathematics more surely than top grades in school.

In a 2006 study published in the journal Science, Robert H. Tai, an associate professor at the University of Virginia’s Curry School of Education, in Charlottesville, tracked thousands of students via the National Educational Longitudinal Study. He found that students who had only average grades in middle school but expressed interest in science were two to three times more likely to earn bachelor’s degrees in a science or engineering field 12 years later than high-achieving students who did not voice interest.

The landmark 2009 study that Mr. Bell co-wrote, “Learning Science in Informal Environments,” for the National Research Council, synthesized a vast array of research, including Mr. Tai’s, to conclude that “abundant evidence” exists that informal experiences can effectively support learning. What’s more, it found that those free-choice, low-stakes pathways can engage children from groups underrepresented in science fields: students in poverty, those from ethnic and language minorities, and women.

Still, while the number of research proposals submitted to the NSF about informal education has doubled in the past five years, from 300 to 600 annually, Mr. Suter said, the vast majority have been and continue to be from practitioners evaluating their individual programs, clubs, and exhibits, often using self-made assessments.

Better Alignment

In 2008, the Program in Education, Afterschool, and Resiliency at Harvard University and McLean Hospital in Boston found that of 64 widely used informal science evaluation tools, none met all five of the NSF’s five domains of informal learning: engagement and interest, attitude toward science and behavior, content knowledge, competence and reasoning, and career knowledge and acquisition.

Program founder Gil G. Noam and Rick Bonney, the director of program development and evaluation at the Cornell University Ornithology Lab, in Ithaca, N.Y., and the creator of several nationwide citizen-science projects, are working to change that. Mr. Noam has established the Assessment Tools in Informal Science database of existing tests. He said that he and Mr. Bonney are about a year away from using the tests to develop a single, unified assessment of informal science learning.

“Our big goal in all this is to bring into this field some proof that the informal science work makes a difference,” Mr. Noam said.

Likewise, Mr. Crowley’s organization, CAISE, has gathered five years’ worth of evaluation reports from NSF informal science projects—many never published—and has started to cull best practices and other lessons learned from across the many types of informal learning.

“Only recently has the learning theory caught up with those hunches, and now we’re at a place where we really can test design hypotheses based on what we know about the science of learning,” he said.

Ultimately, John H. Falk and Lynn D. Dierking, both professors in free-choice learning at Oregon State University, in Corvallis, hope the field will move toward more “synergy” studies, like Mr. Bell’s longitudinal work, that can explore how people learn science across the formal and informal experiences of their lives.

The Oregon State researchers now are monitoring how children in Portland, Ore., come to learn scientific concepts across the city’s many formal and informal science offerings during a four-year period. Portland provides a perfect microcosm for science learning, Mr. Falk said, because it has one science center, zoo, and public-broadcasting station and a few school districts that will allow the researchers to track individual students over time.

“In order to see real systemic improvement, you have to have a whole greater than the sum of its parts, and now the whole is less than the sum of its parts because there’s no alignment among these pieces,” Mr. Falk said. “We need to develop ways to better integrate practitioners across formal and informal education. We don’t understand how people navigate, make sense of, and make use of the resources in their community.”