Evidence Soup

Evidence of paralysis-spider-cure is greatly exaggerated.

News outlets recently reported that a wheelchair-bound California man, David Blancarte, regained the ability to walk after a spider bite. But the evidence of this "cure" is questionable at best -- and sadly, this story appeared on reputable networks including CNN, ABC, NBC, and CBS. Christopher Wanjek, the Bad Medicine Columnist for LiveScience.com, put together a nice analysis of the evidence gaps and journalistic gaffes: "None of the news reports, however, included a doctor or scientist commenting on the possibility of a spider curing paralysis, let alone confirming Blancarte's recollection of the medical facts."

How did I miss this? Wanjek included a wonderful Carl Sagan quote that I somehow overlooked all these years: "Extraordinary claims require extraordinary evidence." Amen to that. Alas, it seems that only the claims were extraordinary this time around. And Wanjek makes a good point about the kind of impact this can have: "Yet more than just another example of lousy broadcast journalism, such stories bring false hopes and even danger to those desperate enough to experiment with venom to cure their paralysis."

I'm as pleased as anybody to hear good news these days -- anything, for Pete's sake*, other than more details about our economic crisis. But it sounds like journalists were extraordinarily sloppy with the evidence this time.

*My dog's name is Pete. I love this phrase.

Sometimes evidence-based, data-driven design doesn't make sense. Take Google, for instance.

Google is well-known for being a data-driven company... nothing wrong with that. Their focus on data-gathering is well-chronicled: They have so many users that they can run all sorts of usability tests to improve their software interface designs. Google's home page is famously clean, with a very low "word weight". I think the most interesting evidence they've published is based on eye-tracking studies, where they follow people's eyes as they look at web pages, to see what patterns emerge. Good stuff.

Evidence can be an excuse for taking chances. But data is good for testing things you've already thought of, not for creating beautiful, innovative things from scratch. In Goodbye, Google, one of the company's (former) designers, Douglas Bowman, explains exactly where the data-driven approach has gone too far. "When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions."

Fiddling with pixels while Paris burns. Bowman says "I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that. I’ve grown tired of debating such minuscule design decisions. There are more exciting design problems in this world to tackle."

Meanwhile, I'd love to point out some design choices for Google to consider, especially concerning their online Office-docs clone.The company's Google Docs product is a rather uninspired (but, thankfully, light) version of what Microsoft has been producing for years. Why aren't they incorporating clean, effective ways for people to present evidence?

Evidence-based medicine? Maybe we should start with common-sense-based medicine.

The New York Times' reliably good Gina Kolata wrote a piece Monday showing how far evidence-based medicine (or maybe common-sense-based medicine) has to go. In Good or useless, medical scans cost the same, she talks about a woman whose MRI scan was blurry and useless: "Her insurer refused to pay for another scan, but the doctor said he was sure she had torn cartilage that stabilizes the knee and suggested an operation to fix it. After the surgery, [the patient] ...received a surprise: the cartilage had not been torn after all." Why should an MRI provider be paid for an "uninterpretable" scan? How much money, time, and other resources were wasted on that surgery, and how much unnecessary pain did it cause? 

Where's the incentive to do better? Kolata says "Insurers pay the same for a scan done on a 10-year-old machine as one on the latest model, though the differences in the images can be significant. Insurers do not distinguish between scans that are done poorly or done well or read by less- or more-qualified doctors." If insurers won't provide an incentive for people to improve the quality of the medical evidence they are producing, who will?

Once again, the evidence shows dieting success depends on counting calories.

If wishing could make it true, that low-carb, eat-all-the-bacon- you-want diet would work. Or the Diet of 1,000 Smoothies. Etc., etc. But in what is being called the most rigorous and far-reaching study of various schemes, scientists have found -- once again -- that dieting success depends on simply consuming fewer calories. As explained in the Los Angeles Times, the findings tell us "any diet that is low in calories and saturated fats and high in whole grains, fruits and vegetables -- and that an individual can stick with for a lifetime -- is a reasonable choice for people who need to lose weight."

Comparison of Weight-Loss Diets with Different Compositions of Fat, Protein, and Carbohydrates was published this week in the online New England Journal of Medicine. Time quoted the lead author, Dr. Frank Sacks, hitting the nail squarely on the head: "'We have a really simple and practical message for people: it's not so much the type of diet you eat,' says Sacks. 'It's how much you put in your mouth.'" (Weren't we all secretly hoping for a different conclusion?)

I love transparency. As research abstracts go, this one is written in relatively plain English. Under Background, they simply say that "The possible advantage for weight loss of a diet that emphasizes protein, fat, or carbohydrates has not been established, and there are few studies that extend beyond 1 year." The Methods section isn't horribly obscure, telling us they randomly assigned 811 overweight adults to one of four diets that meet guidelines for cardiovascular health. And the conclusion says only that "Reduced-calorie diets result in clinically meaningful weight loss regardless of which macronutrients they emphasize." The abstract also clearly identifies the study's registration with ClinicalTrials.gov, where NIH-funded research is identified *before* the results are in -- one (long overdue) way of keeping people honest.

The Biggest Loser is on to something. Included in the results is a finding that "Satiety, hunger, satisfaction with the diet, and attendance at group sessions were similar for all diets...." But Weight Watchers and reality TV appear to have gotten it right by getting people together: The research showed that "...attendance was strongly associated with weight loss (0.2 kg per session attended)."

On this story, journalists did a reasonable job of bringing the evidence to the people. In What's the best diet? Eating less food, Time magazine online presented some research details without going overboard. Ditto for the Los Angeles Times. And both publications actually told us where we could find the published study (NEJM, beginning Feb 26).