Evidence Soup
How to find, use, and explain evidence.

« Does frequent stop-and-frisk reduce crime? Police in Brownsville, Brooklyn say yes. | Home | Evidence really is like a bowl of soup. »

Tuesday, 13 July 2010

If they're going to talk about evidence, journalists and pundits need to show it to us.

On Boston.com, Joe Keohane writes about How facts backfire: Researchers discover a surprising threat to democracy: our brains. There's some good stuff about how people resist 'facts' that contradict their beliefs. (We've seen findings like this before - it's a fact that we like to ignore the inconvenient facts.)

Keohane's piece agrees with science research I wrote about last week: He mentions a study showing that "politically sophisticated thinkers were even less open to new information than less sophisticated types. These people may be factually right about 90 percent of things, but their confidence makes it nearly impossible to correct the 10 percent on which they’re totally wrong." (Sounds like grad school to me.)

Show us the evidence! But it struck me how difficult it was to figure out exactly what research Keohane is referring to. (Contrast this with work by Miller-McCune, who reliably provides specifics and links, as I wrote about recently.) Apparently much of the 'facts' discussion centers on studies done by Brendan Nyhan at the University of Michigan. But Keohane doesn't give us article titles or publication dates. And he goes back and forth, first referencing "a series of studies in 2005 and 2006", then mentioning "New research, published in the journal Political Behavior last month" - followed in the very next sentence with more references to a 2005 experiment by Nyhan.

I'm not asking for footnotes at the end of every op-ed. But the world is (or should be) more transparent now: Journalists and pundits need to throw us a link or two. (Other references are also fuzzy, such as a "study done in the year 2000, led by James Kuklinski of the University of Illinois at Urbana-Champaign". If Kuklinski gets lots of things published, it's going to be difficult to follow up on this if we want to see the evidence.)

Sprinkled throughout the story are some fun findings: "Facts, they found, were not curing misinformation. Like an underpowered antibiotic, facts could actually make misinformation even stronger." And "the more strongly the participant cared about the topic — a factor known as salience — the stronger the backfire". It would be great if Keohane directed us to the source.

By the way, the column mentioned one of my favorite Onion articles: Area Man Passionate Defender Of What He Imagines Constitution To Be. That just about says it all.


Thank you, Tracy! I've been meaning to write a letter to Wired on this very issue. Sometimes they'll give us the reference, other times it's something like "researchers at Harvard found" or "as published in [journal name] -- but no direct reference or credit to the individual authors. I take my students to task for lack of cites, both because they need to give credit and for their own knowledge management.

The comments to this entry are closed.