Advances in our understanding of the neurological underpinnings of learning, memory and behavior have the potential to improve education, but findings so far have been extrapolated far beyond the strength of their studies, conclude researchers in the latest edition of Nature Reviews Neuroscience.
In “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience,” researchers from the University of Bristol and the University of Oxford in the United Kingdom, the University of Virginia, Charlottesville and Stanford University in Palo Alto, Calif., warn that most brain imaging and other neuroscience studies base their conclusions on too small of samples, leading to a strong risk of results being do to random luck of the draw rather than a real effect. (The satirical IgNobel neuroscience award for 2012 went to a great example of the dangers of statistical noise, in which researchers identified “brain activity” in a dead salmon.)
Along the same lines, the authors of the article analyzed the statistical power and potential sampling biases of 48 neuroscience studies published in 2011 (ironically, itself a somewhat small sample). They found studies routinely use sample sizes too small to generate reliable results, and also frequently over-inflate the importance of positive findings, in light of their relatively weak effects.
“We looked at neuroscience literature and found that, on average, studies had only around a 20 percent chance of detecting the effects they were investigating, even if the effects are real,” said lead author Katherine S. Button, a psychiatrist at the University of Bristol’s School of Social and Community Medicine, in a statement. “This has two important implications—many studies lack the ability to give definitive answers to the questions they are testing, and many claimed findings are likely to be incorrect or unreliable.”
This can lead to overblown implications of those findings for educators, doctors and other practitioners—as is often seen in the persistence of educational neuro-myths such as “left-brain” and “right-brain” learners.
Button and her co-authors urged researchers to give clearer disclosures of their research methodology and work to make their experiments easily replicable by other researchers, to build up reliable results. The field does seem to be moving in the direction of using bigger and more collaborative datasets that could help ease the problem. The Roanoke Brain Study, for example, is looking at the brains of 10,000 people, including more than 200 children, with multiple measurements over the course of many years. And President Obama’s recently launched $20 million Brain Research through Advancing Innovative Neurotechnologies (BRAIN) Initiative is intended to develop more accurate tools to measure neurological processes as well as mapping connections in the brain.
In the mean time, the Cognitive Neuroscience Society’s annual meeting, which starts this weekend in San Francisco, will feature a symposium on the mixed results and potential future research of one of the most popular—and controversial—interventions to come out educational neuroscience: training to improve working memory.
Want more research news? Follow @SarahDSparks on Twitter for the latest studies, and join the conversation.