Lately, 60 Minutes and other news outlets have been doing stories about fMRI (functional magnetic resonance imaging) brain scans -- saying things like "Ooh, scary. Can they use this to figure out what we're thinking?" Lesley Stahl's story on Reading Your Mind says "Neuroscience has learned much about the brain's activity and its link to certain thoughts.... [I]t may now be possible, on a basic level, to read a person's mind."
At least NPR is doing a good job of cutting through the hype. Their recent story, False Signals Cause Misleading Brain Scans, explains that fMRI "images appear amazingly crisp and precise. But scientists say the truth behind them is a little fuzzier." fMRI detects oxygen being carried to various parts of the brain. Since more oxygen indicates more brain activity, they are using the images to pinpoint when people think different types of thoughts. But the signals are really weak, and the signal-to-noise ratios are very low. Every time your heart beats, or you move your head, or you think about something randomly, a signal can occur.
You, sir, must correct the scientific record immediately! Jon Hamilton of NPR spoke with Chris Baker with the Laboratory of Brain and Cognition at the National Institute of Mental Health. "Baker says that when you're searching an ocean of noisy data, it's possible to find patterns produced purely by chance. 'There are situations in which you could apparently produce a signal out of just noise,' he says."
Baker is a co-author of the article Circular analysis in systems neuroscience: the dangers of double dipping in the journal Nature Neuroscience. Another researcher, Hal Pashler, a professor at the
University of California, San Diego, is also questioning fMRI findings: He's co-authored the article Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition that appears in the journal Perspectives on Psychological Science -- the abstract uses relatively plain English to describe why they believe many published fMRI studies are flawed (see abstract at end of this post). (Pashler's web site says the article was formerly titled "Voodoo Correlations in Social Neuroscience".)
Double-dipping? NPR's Hamilton explains that "Pashler says he and his co-authors wrote the paper because they were seeing results from fMRI studies that seemed just too good to be true. The team began to ask why and zeroed in on a statistical practice sometimes called 'double dipping,' a kind of circular reasoning. Pashler and Baker say many fMRI studies do something a lot like this when they dip into the same data twice: once to pick out which parts of the brain are responding, and again to measure how strong the response is. The scientists say that's probably made a lot of results appear stronger than they really are."
Legal implications? The Neuroethics and Law blog has a discussion of Pashler's work, and references the recent Wall Street Journal article The Brain, Your Honor, Will Take the Witness Stand, subtitled "Researchers Probe How the Mind Determines Crime and Punishment, but the Science Isn't Beyond a Reasonable Doubt". Neuroscientists have used fMRI to understand how the brain might function in judicial and jury decisions: "The fMRI monitored the blood flow and oxygen demand associated with neural activity as each subject made two distinct legal judgments about blame and punishment in 50 hypothetical scenarios ranging from simple theft of a music CD to rape and murder." But if scientists' interpretations of fMRI results are statistically flawed, then maybe these scans are scary after all -- just not in the mass-media, Lesley Stahl sort of way.
Research implications? Recent emphasis on fMRI might be leading to misplaced priorities. NPR's Jon Hamilton points out "Funding agencies and other researchers are also influenced. That could mean some areas of the brain are getting attention they don't deserve, while other potentially important areas are being ignored."
Abstract from Vul, Winkielman, and Pashler's Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition (in press). Perspectives on Psychological Science.
"The newly emerging field of Social Neuroscience has drawn much attention in recent years, with high-profile studies frequently reporting extremely high (e.g., >.8) correlations between behavioral and self-report measures of personality or emotion and measures of brain activation obtained using fMRI. We show that these correlations often exceed what is statistically possible assuming the (evidently rather limited) reliability of both fMRI and personality/emotion measures. The implausibly high correlations are all the more puzzling because social-neuroscience method sections rarely contain sufficient detail to ascertain how these correlations were obtained. We surveyed authors of 54 articles that reported findings of this kind to determine the details of their analyses. More than half acknowledged using a strategy that computes separate correlations for individual voxels, and reports means of just the subset of voxels exceeding chosen thresholds. We show how this non-independent analysis grossly inflates correlations, while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that other analysis problems likely created entirely spurious correlations in some cases. We outline how the data from these studies could be reanalyzed with unbiased methods to provide the field with accurate estimates of the correlations in question. We urge authors to perform such reanalyses and to correct the scientific record."
Comments