Although sharing is one of the first lessons children learn in school, a few prominent education scholars are suggesting some of their colleagues ought to take a refresher course in the subject.
The issue: Investigators are often reluctant to share data from their studies so other researchers can reanalyze the information, replicate the studies, or expand on the findings.
“We need cooperation and collaboration and sharing in a way we haven’t in the past,” said Barbara Schneider, a sociologist at the University of Chicago. “We’re in the heart of education reform, and we have tremendous amounts of resources being funneled into a lot of experiments.”
|
Ms. Schneider, who is a senior social scientist at the university’s National Opinion Research Center, has led some of the calls for researchers to be more forthcoming with the data they collect—particularly when the federal government is paying for their work.
She has been joined by other scholars, foundation officials, and policymakers, all of whom contend that their fellow researchers need to “share and share alike” for the field to amass a credible store of knowledge on what works in education.
The calls for collaboration still do not represent a consensus, however. Some researchers have cautioned against pushing too hard and fast to make data-sharing a standard practice. That’s especially true, they add, when research subjects’ privacy is at stake.
“It would absolutely kill ethnography,” said Mary Haywood Metz, a University of Wisconsin-Madison researcher who specializes in qualitative studies looking at the internal social and emotional workings of school communities. While it’s admirable that some individual researchers have taken it upon themselves to make their data public after their papers are published, she said, “to set that up as a standard for everybody to follow is quite another matter.”
Privacy High Wire
A panel organized by the National Research Council, an arm of the National Academies of Sciences, is exploring ways to make education research more “scientific.” The group, known as the Committee on Research in Education, was interested enough in the issue of data-sharing to include it in a workshop last summer on accumulating a knowledge base in education.
“They spend many years collecting their data, and they want to make sure they have first crack at it,” Kay Dickerson, a Brown University professor of community health and member of the panel, said of researchers’ attitudes. “On the other hand, sometimes researchers take a long time getting to it, and that’s frustrating.”
Sometimes, too, information goes unused because researchers’ grants do not cover the cost of “cleaning up” the data for public use. That means stripping away any information that could be used to identify study subjects and adding in enough context to make the data understandable.
Protecting subjects’ privacy, though, remains the main reason most researchers avoid opening up their files to outside inspection. In qualitative studies and small-scale quasi-experiments, researchers say, it’s hard to disguise everyone’s identity. A school might have just one art teacher, for instance. Or a small city might have a single high school principal.
Also, if subjects know that their views and actions will be widely shared, the University of Wisconsin’s Ms. Metz said, they might be less forthcoming—or refuse to participate in the study altogether.
“People won’t tell you what’s going on unless they can trust you,” she noted.
On the other hand, she said, when researchers remove too much identifying information, they risk losing some of the context that outside scholars need to understand what’s being studied.
Such debates are not unique to education.
Just last year, the federal National Institutes of Health, after vetting the opinions of researchers in the health fields, issued new rules requiring grant recipients to make all their research materials available to the public once their studies are accepted for publication.
Likewise, the Washington-based American Sociological Association—a group that also includes education researchers— revised its ethics code in the mid-1990s to explicitly encourage its members to share their data with colleagues.
|
“I don’t necessarily see educational research as behind in this,” said Felice Levine, the executive director of the American Educational Research Association, which also is located in Washington. Ms. Levine was the executive director of the sociological association when its ethics code was updated, and she served on a National Science Foundation panel on data-sharing.
In her current organization, she said, such policies are implicit in the “normative fabric’’ of the ethics code. “Could a code be more explicit?” she said. “It might be.”
Publication Pressures
Regardless of the status of data-sharing in other fields, Ms. Schneider of the University of Chicago and others suggest that the practice could be more ingrained in education research than it is now.
They offered this case in point: Researchers who publish in the journals Nature, Science, the American Sociological Review, the American Economic Review, and Demography all must agree to share their work when their articles are published. Researchers said few, if any, education research journals have similar requirements.
Similarly, the U.S. Department of Justice’s National Institute of Justice requires the researchers it supports to make their data public.
The U.S. Department of Education has no such blanket requirements for its researchers. That’s not to say that no department-financed studies are made widely available. All the data collected by the department’s National Center for Education Statistics or one of its contractors have long been open to the public. And the department’s Institute of Education Sciences, newly required by federal law to do the same, is making plans to follow NCES’ example.
Some of those federal data sets, though, are still a year or two from being widely available online or on CD-ROMs, according to Grover J. “Russ” Whitehurst, the institute’s director.
Likewise, Ms. Schneider is spearheading a project to develop a common database from all the information gathered from a federally funded, cross-agency research initiative on education.
Mr. Whitehurst said he doubts, however, that the Education Department will ever be able to demand that the individual researchers it subsidizes through the field-initiated-studies grant program or other smaller-scale research efforts put up their data for all to use.
While data-sharing may be a good way to get the “biggest bang for the buck” out of federal research expenditures, he said, “there are, in most cases, severe confidentiality issues that constrain distribution of that data.” But when it comes to informing public policy, some researchers say timely sharing of data becomes a moral obligation.
“If researchers want to hoard data, that’s fine, but I think the rest of the policy community should ignore the results until someone else has had the opportunity to analyze the data,” said Alan B. Krueger, a Princeton University economist who has sought access to data from school voucher experiments in New York City, the District of Columbia, and Dayton, Ohio.
Mr. Krueger said his own attempts to reanalyze the data from the three-city school choice experiments have been “eye-opening.” Although results from those experiments were released last year, the Princeton economist could not persuade Harvard University researcher Paul E. Peterson and his colleagues to part with their data from Dayton and Washington.
But Mathematica, the Princeton, N.J.-based research group that had worked with Mr. Peterson on his New York study, complied. Mr. Krueger reanalyzed the data using a different method for categorizing students by race. And he came up with some distinctly different results. (“Study: No Academic Gains From Vouchers for Black Students,” April 9, 2003.)
In an e-mail message, Mr. Peterson explained that he had declined to share the data because members of his research team are still publishing more papers from the results.
In its earlier analysis, the Mathematica-Peterson research team found that the privately financed tuition vouchers had led to sizable learning gains for the African-American students who used them to transfer to private schools. They outperformed a similar group of disadvantaged students who had also sought the vouchers, but didn’t win them in the lottery. Mr. Krueger’s refiguring, however, found those gains statistically insignificant.
That reanalysis did not end the debate on the benefits of school choice. Mr. Peterson continues to dispute Mr. Krueger’s conclusions. Experts say the example does, however, point up the need to cast a second eye on controversial studies that might be used to inform public policy.
Stepping up cooperation among researchers is important, they argue, because it offers another possible route for improving the quality of educational research.
One such proponent is Marshall S. Smith, the director of educational programs for the Menlo Park, Calif.-based William and Flora Hewlett Foundation, who was a top education official in the Clinton administration. The foundation has been pushing to make research data and other information resources from higher education more widely available.
“It introduces transparency, which creates a level of accountability which hasn’t existed before,” Mr. Smith said.
When someone is looking over their shoulders, he added, “you make people much more rigorous about carrying out their analyses.”
Coverage of research is underwritten in part by a grant from the Spencer Foundation.
