Evidence Soup

I've got some questions about the evidence from the latest faux shock therapy (Replicating Milgram).

In the so-called Milgram experiment, people famously administered electric shocks to others as part of a 1960s psychological study. The shocks weren't real, but participants weren't told that. When the experiment was recently replicated*, again people followed instructions to shock "learners" if they made mistakes. Jerry Burger of Santa Clara University did the research; Replicating Milgram appears in January's American Psychologist (download the 11-page pdf here).

Hmmm. I have some questions. I've read Burger's research, and can follow how he arrived at his evidence. But at the risk of sounding cynical, isn't it possible that some of the participants were just playing along -- suspecting that the electric shocks couldn't possibly be real in this day and age? In my graduate courses that involved role-playing games, I always thought it was fun to take an extreme stand, to make things interesting. Hear me out:

• If someone offered you $50 to participate in an experiment, and then instructed you to shock people, wouldn't you suspect something was up? I'll bet some of us would laugh out loud. Lots of people have heard about the Milgram experiment -- they wouldn't know it by name, but many are aware of various controversial studies.

• Participants received a sample shock, and were shown a "Shock Generator, Type ZLB, Dyson Instrument Company, Waltham, Mass. Output 15 volts – 450 Volts." Aren't most people at least *aware* of The Wizard of Oz, or The Roadrunner, or Punk'd, or The Simpsons, making them less likely to fall for the research stunt?

• And if participants did in fact believe the shocks were real, where have they been all these years? Do they represent a valid sample of human behavioral traits?

Go ahead: One more can't hurt. Among other things, this evidence is supposed to reflect people's tendency to obey an authority figure, particularly in situations where behavioral norms aren't well-established. By gradually increasing the voltage in the faux shocks, both studies show that people are more likely to comply if an activity starts small and then escalates. Burger told the San Jose Mercury News: "The conclusion is not: 'Gosh isn't this a horrible commentary on human nature,' or 'these people were so sadistic'.... It shows the opposite — that there are situational forces that have a much greater impact on our behavior than most people recognize."

Some researchers expected different study results because they believe fewer people will blindly follow authority today than in the 1960s -- but Burger disagrees, saying they're making a fundamental attribution error: He claims that the influence of the situational forces in the experiment design has not changed over time. Maybe so, but I'd like to hear his response to my questions.

Not surprisingly, media coverage was heavy, including stories by Science Daily and CNN.

*Burger revised the methodology to avoid the most controversial aspects of Milgram's experiment. At least medical ethics have evolved over the past 40 years.

Blaming bed lunch for weight gain? Blaming sugar when your kids are hyper? The evidence says think again.

I love the term bed lunch. While a friend of mine was growing up in North Dakota, this is what they called a piece of pie, or other snack, people enjoyed right before bedtime. Now there's good news about bed lunch -- it doesn't contribute disproportionately to weight gain.

Rachel C Vreeman and Aaron E Carroll are pediatrics researchers fighting the good fight to bring more evidence into family life. Previously they've busted myths about drinking 8 glasses of water/day, fingernails continuing to grow after death, and other common beliefs. Now they've published new evidence challenging conventional wisdom about typical wintertime activities:

"In the pursuit of scientific truth, even widely held medical beliefs require examination or re-examination. Both physicians and non-physicians sometimes believe things about our bodies that just are not true. As a reminder of the need to apply scientific investigation to conventional wisdom, we previously discussed the evidence disputing seven commonly held medical myths. The holiday season presents a further opportunity to probe medical beliefs recounted during this time of the year."

Sugar makes children hyperactive. "At least 12 double blind randomised controlled trials have examined how children react to diets containing different levels of sugar. None of these studies, not even studies looking specifically at children with attention-deficit/ hyperactivity disorder, could detect any differences in behaviour between the children who had sugar and those who did not. This includes sugar from sweets, chocolate, and natural sources.... Scientists have even studied how parents react to the sugar myth. When parents think their children have been given a drink containing sugar (even if it is really sugar-free), they rate their children’s behaviour as more hyperactive. The differences in the children’s behaviour were all in the parents’ minds."

Bed lunch makes you gain more weight. There is some correlation between overweight and late-night eating, but "just because obesity and eating more meals at night are associated, it does not mean that one causes the other." Research shows "taking in more calories makes you gain weight regardless of when calories are consumed."

Poinsettias are toxic. (I blogged about this in 2006, doing my part to salvage the reputation of the beleaguered flower in No, Virginia, there really aren't any poisonous poinsettias).

Hatless people lose 1/3 of their body heat through their heads. Actually, you lose heat from any exposed area of your body -- proportionally. This myth resulted from a flawed army field study that just won't die.

Suicides increase over the holidays. Nope: "The epidemiological evidence just does not support that the holidays are a time of increased risk."

Permit me an Ebenezer moment: Among many others, our local CBS News covered these medical myths. I'm pleased the mainstream media picked up the story, though I still cringe when they say things like "highly scientific". And I wish their web coverage included links to the journals publishing the research, or at least the names of the published articles.

Vreeman and Carroll's article, Christmas 2008 Seasonal Fayre: Festive medical myths, can be cited as BMJ 2008;337:a2769.

Some claim that Buyology missed significant evidence about cigarette warnings.

I wrote last week about the book Buyology, and its author's claims that cigarette warning labels actually make people want to smoke. But some remain unconvinced: Folks are pointing out weaknesses in Martin Lindstrom's conclusions, which were based on a functional MRI study. In response to Lindstrom's Dec 11 New York Times Op/Ed piece, Inhaling Fear, in which he discussed his research, some insightful letters to the editor (Dec 18) included these observations:

• Lindstrom didn't acknowledge evidence from studies showing that graphic warnings contribute to smoker's knowledge of health risks.
• The study didn't include an equal number of non-smokers. Doing so would allow researchers to determine if the cigarette warnings triggered the "disapproval" areas of the brain -- which could indicate whether the warnings help prevent non-smoking people from picking up the habit.

New book can help you develop effective, evidence-based presentations.

It's not every day I get to review a book offering chapters with names like "Marshalling your evidence" and "Assembling the anecdotes that will illustrate your evidence". But I've got my hands on a copy of Advanced Presentations by Design: Creating Communication that Drives Action by Andrew Abela. In October I wrote about Abela's work in my post What does the evidence say about presenting evidence? (He's an ex-McKinsey consultant who is currently a marketing professor.) Now he's published a book on the subject, analyzing the building blocks of effective presentations in a way I've never seen.

There's lots of good guidance in this book, even if you choose not to follow his steps per se. Abela provides specific evidence of what's most likely to work, and why, when you want people to *act* on something you present to them. But the book is more than a recap of scientific findings: He writes from the perspective of a marketer and business manager, offering practical, evidence-based advice about how to focus on a problem your audience has, and how to show them you can help solve it. Two of his key topics are:

• Structuring stories. Abela presents a SCORE method for sequencing evidence: Situation, Complication, Resolution, Example. The aforementioned "Assembling anecdotes" chapter talks about three types of stories we can use: 1) One that relates directly to our situation, 2) a hypothetical story, or 3) a more metaphorical one.
• Using graphics. The book provides numerous examples of charts and other graphics, explaining which can help you best present your data. I've heard plenty of discussions of visual presentation -- Abela goes into more detail than most, staying focused on interpreting the hard evidence about how to engage an audience.

I would be proud to have written this book. At $40.50 (US$) from amazon.com, it would be a wise investment.

Evidence from functional MRI shows cigarette warning labels make smokers want to light up.

This is one of the findings reported in Martin Lindstrom's new (and brilliantly titled) book, Buyology: Truth and Lies About Why We Buy. The book recaps how functional MRI (fMRI) was used to conduct a multi-year, multi-million dollar "neuromarketing" study. This is a very different type of evidence than I'm familiar with.

Evidence about warning labels. From the book: "In short, the fMRI results showed that cigarette warning labels not only failed to deter smoking, but by activating the nucleus accumbens, it appeared they actually encouraged smokers to light up. We couldn't help but conclude that... [labels] had become a killer marketing tool for the tobacco industry" (page 15). Assuming the evidence is valid, this is one tragically funny public policy lesson.

Evidence re: product placement. The book talks about advertising and product placement on the American Idol show. Coca Cola and Ford (automotive) were two key companies evaluated, and Coke was evidently the winner. Lindstrom claims that people actually remembered *less* about Ford after the study -- saying that the "Coke-saturated show actually suppressed" memories about Ford (page 49). He explains these results by saying that a product needs to play an integral part in the narrative -- you can't just plunk random products on the screen. I like his example about the movie E.T.: "Elliott didn't just pop those Reese's Pieces into his mouth during a thoughtless bike ride; they were an essential part of the storyline..." (page 51). If you're planning to blizzard people with brand after brand, he says you might as well set fire to those advertising dollars.

Is this real? Lindstrom was interviewed on NPR this week. He presents his findings enthusiastically, and I enjoyed listening to him. Typically, when I see someone touting a book about how companies market to the consumer's subconscious, I keep on clicking. But this is interesting. I'll confess, I know next to nothing about fMRI, so don't know if the Buyology conclusions are valid. But it certainly seems plausible.

Evidence shows people rarely use online resources when selecting a healthcare provider.

Yesterday I wrote about the tendency for hospitals to remain open despite evidence they provide sub-par care. I asked "Is 'buyer beware' enough? ...I wonder how many people actually know about the available ratings services." Today I heard from Maribeth Shannon, with the California Healthcare Foundation. She said "If you didn’t see the piece released last week from the Center for Studying Health System Change you may find it interesting. We have been trying to answer the question: why don’t people use these sites?" (Thanks, Maribeth.) Alas, there's lots of evidence to support my hunch.

The report, Word of Mouth and Physician Referrals Still Drive Health Care Provider Choice, found that much of the available evidence isn't being used very extensively: "Self-reported use of quality information was low among specialist and procedure shoppers but was higher (23%) for primary care physician shoppers." My first reaction was "Oy." Then it occurred to me that I'm sometimes guilty of the same thing -- asking a friend without also checking an authoritative online resource.

If you build it, will they come? When considering the implications of their findings, the authors concluded that "The consumer-directed health care vision of consumers actively shopping is far removed from the reality of how most consumers currently choose health care providers. Few consumers make use of Internet information sources or price and quality information. Most still rely on the traditional sources of physician referrals or word-of-mouth recommendations from friends and family. "

So why don't people use evidence when it's available? The authors believe that "consumers would likely feel more compelled to use quality information if they believed that quality differed substantially across providers, and that these differences can have concrete, serious -- even life-or-death -- consequences. For policy makers seeking to engage consumers in provider shopping and quality improvement efforts, a critical challenge is to educate consumers about the existence and the serious implications of provider quality gaps." As in other efforts to promote evidence-based management, we've got to make the evidence more attention-getting.

Evidence about poor hospital quality isn't enough to make change happen.

Even when there's substantial evidence that a particular hospital delivers an unacceptable level of care, that doesn't necessarily mean it will be improved or closed: Other forces often outweigh the medical evidence, including the jobs the hospital provides and the community it serves.

Is "buyer beware" enough? There are ways to dig up information about a hospital's performance, though I wonder how many people actually know about the available ratings services.

 

HealthGrades.com provides ratings based on Medicare data. Here's a snapshot of search results on Gastrointestinal Surgeries and Procedures in Syracuse, New York. Both Crouse and University Hospitals are rated 'Poor" for Inhospital Mortality. The site uses a statistical method to predict how many deaths would typically be expected, and compares that number to actual outcomes. (Though as you can see here, the evaluation is based on only 61 Medicare cases.)

The consumer recommendation site Angie's List has added reviews of physicians, hospitals, and other medical providers. But compiling detailed quality evaluations is really difficult: I wrote about one of the most impressive sites doing this, CalHospitalCompare.org, last year (see Comparing hospitals: Lots of evidence. But who is it helping?).

The United States doesn't have a federal agency providing hospital oversight, though I'm not convinced that would make much difference: States have various ways of compiling information already. None of this bodes well for evidence-based medicine: As explained in today's Evidence Gap column in the New York Times, University Hospital in Syracuse, New York "provides a stark example of how hard it can be — not just in New York, but around the nation — to close or shrink hospitals, even when there is evidence they are providing costly and below-average care.... In 2006, patients at University were three times as likely to develop infections stemming from hospitals as were patients at the average New York hospital. HealthGrades, a company that rates hospitals using data from Medicare, ranks University among the least safe hospitals in the United States — although the hospital’s executives strongly dispute that assertion.... [The United States] relies on a patchwork of state health departments and a nonprofit group called the Joint Commission that sets basic quality standards for the nation. Hospitals are rarely closed or hit with significant financial penalties for hurting patients. One of the reasons is that even troubled hospitals are major employers, and communities generally rally behind them when they face the threat of cuts, as Syracuse did for University."

Do we really need hard evidence that human beings have souls? In other news, Debbie Downer is still not the life of the party.

Happy Fun-with-Evidence Friday. This week's edition of Westword, Denver's so-called alternative newspaper, ran an ad saying New Book Offers Concrete Proof of the Soul (though amazon lists a May 2007 publishing date). Let's assume for the sake of argument that we do want/need concrete evidence that people have souls. Here's how the author, Manya Long, explains this research: "This philosophical book proves through concrete evidence that the soul actually and absolutely exists. Included in this new work is a picture of a person's soul, taken by means of an aura camera. I have studied and arranged the data that composes this book for over twenty years. This book not only provides proof that the soul does in fact exist, but it also renders the scientific and mathematical substantiation to back up my claim.... What my hypothesis says is your 'soul' or conciousness [sic] can be seen through the electromagnetic fields created by your body." OK, if we could actually get this type of evidence, what would be the purpose? To assess different souls and recommend ways to improve them?

Update: Just as I posted the following writeup of the "happiness" study, the radio program Science Friday came on, interviewing one of the researchers. Arrrgh! James Fowler said that people in the center of the network are happier than people on the fringes (duh). Evidently those people are more likely to catch "waves" of happiness, so we're supposed to connect with more happy people, and help our friends become happy.

Misery loves company, but company does not reciprocate. There's lots of social-network analysis happening these days, though some is pretty light. Social scientists continue creating colored graphs showing people's interconnectedness, but much of this evidence fails to help us manage our organizations, or our lives, any better. A recent example comes to us from researchers Nicholas A. Christakis & James H. Fowler. Quoting from their blog post about it, "We found that social networks have clusters of happy and unhappy people within them that reach out to three degrees of separation. A person's happiness is related to the happiness of their friends, their friends' friends, and their friends' friends' friends — that is, to people well beyond their social horizon. We found that happy people tend to be located in the center of their social networks and to be located in large clusters of other happy people. And we found that each additional happy friend increases a person's probability of being happy by about 9%." (Yes. They actually said that.) And they finished with this gem: "It thus seems to be the case, online as well as offline, that when you smile, the world smiles with you." We need a guy from Harvard Medical School to tell us this?

I do believe that the people we surround ourselves with have a big influence on our attitudes and achievements. I just don't know if this research tells us more than we all knew already. And there's always the problem of defining "happiness" --- it's a lot like "knowledge". We all want it, and know it when we see it, but can't really manage it.

These conclusions are based only on survey data: "On three separate occasions between 1984 and 2003, the participants filled out a questionnaire designed by the Center for Epidemiological Studies to assess depression and emotional health. To measure happiness, Christakis relied on people's answers to four questions in the survey, including: 'How often during the past week would you say: I enjoyed life? I felt hopeful about the future?'" Yet this managed to get wide coverage in the mainstream U.S. media, who are grasping for happiness-related news these days (see Wall Street Journal blog, New York Times, and National Public Radio).

Although I don't see this social survey analysis as very useful, it certainly address an important topic: Lots of people wish they were happier. (One tip: Evidently happy people watch less TV, based on research reported recently in the New York Times' What Happy People Don't Do.) There's even an article on WikiHow called How to be happy. As if.

Digging for evidence? New guide offers 100 tips & tools for researching the deep web.

Next time you're digging for evidence, it's worth taking a look at the new guide Useful Tips and Tools to Research the Deep Web on the Online College Blog, which explains that "Experts say that typical search engines like Yahoo! and Google only pick up about 1% of the information available on the Internet. The rest of that information is considered to be hidden in the deep web, also referred to as the invisible web."

They've compiled a handy directory of meta-search engines (such as DogPile), semantic tools (such as FreeBase), search engines and databases (academic, scientific, etc.), plus custom search engines (more info at sites like CustomSearchEngine.com) and collaborative ones (such as delicious and StumbleUpon). Also included are search tips and strategies, plus links to resources, such as Wikipedia's Deep Web page. Here's a snapshot of what they've put together:

Kaiser searches for evidence of health gadget usefulness.

Where medically related gadgets, tools, and instruments are concerned, their contribution to evidence-based medicine depends heavily on their usefulness to doctors, nurses, and patients. This is the idea behind so-called persuasive technology intended to help people promote their health.

But there are many types of technology that could improve health, not all of them persuasive in nature. As explained on SiliconValley.com, "With tech companies in Silicon Valley and elsewhere offering a growing assortment of health-related gadgets for hospitals and patients, Kaiser Permanente executives have developed what they say is a unique laboratory to test the products' usefulness. By simulating how health care personnel and the general public might use the devices, the executives say they hope to identify technologies that can help patients, while also reducing hospital-related expenses by making medical services more efficient."

The lab is located in San Leandro, California.