Evidence Soup
How to find, use, and explain evidence.

Monday, 21 July 2014

The Data-Driven vs. Gut Feel hyperbole needs to stop.

Smart decision-making is more complicated than becoming ‘data-driven’, whatever that means exactly. We know people can make better decisions if they consider relevant evidence, and that process is getting easier with more data available. But too often I hear tech advocates suggest that people’s decisions are just based on gut feel, as if data will save us from ourselves.


We need to put an end to the false dichotomy of 'data-driven' vs. 'human intuition'. Consider the challenge of augmenting the performance of a highly skilled professional, such as a medical doctor. Investor Vinod Khosla claims technology will replace 80%+ of physicians’ role in the decision-making process. “Human judgment simply cannot compete against machine-learning systems that derive predictions from millions of data points”. Perhaps so, but it’s really tricky to blend evidence into patient care processes: Research in BMJ reveals mixed results from clinical decision support technology, particularly systems that deliver alerts to doctors who are writing prescriptions.

Data+People=Better. One tech enthusiast compares IBM’s Watson to a hospital CEO. Ron Shinkman asks if it could “be programmed to pore over business cases, news clippings, algorithms and spreadsheets to make the same recommendations?” Actually, that’s what Watson does. But Shinkman overlooks the real opportunity: To supplement, not replace, a CEO’s analytical skills. (Note: This is an excerpt from a research paper I recently wrote  at Ugly Research.)

Why IT Fumbles Analytics. In an excellent Harvard Business Review analysis of how decision makers assimilate data, Donald Marchand and Joe Peppard explain that

management lacks “structure. Even when an organization tries to capture their information needs, it can take only a snapshot, which in no way reflects the messiness of their jobs. At one moment a manager will need data to support a specific, bounded decision; at another he’ll be looking for patterns that suggest new business opportunities or reveal problems.”

Here's another example of a claim that new technology will replace human intuition with fact-driven decision-making.


Source: Business analytics and optimization for the intelligent enterprise (IBM).

You’re not the boss of me. There’s a right time and a wrong time to look at data. As Peter Kafka explains, Netflix executives enthusiastically use data to market TV shows, but not to create them. Others agree data can interrupt the creative process. In The United States of Metrics, Bruce Fieler observes that data is often presented as if it contains all the answers. But “metrics rob individuals of the sense that they can choose their own path.”

However, people could do better. Of course decision makers frequently should ignore their instincts. Andrew McAfee gives examples of algorithms that outperform human experts, and explains why our intuition is uneven (we need cues and rapid feedback).

The Economist Intelligence Unit asked managers “When taking a decision, if the available data contradicted your gut feeling, what would you do?” Most preferred to crunch some more numbers. Only 10% said they would follow the action suggested by the data. The sponsors of Decisive action: How businesses make decisions and how they could do it better concluded that while “many business leaders know they need to make better use of data, it’s clear that they don’t always know how best to do so, or which data they should select from the enormous quantity available to them. They are constrained by their ability to analyse data, rather than their access to it.”

How do you challenge a decision maker? When data is available to improve a result, it must be communicated so it challenges people to apply it, not deny it. One way is to provide initial recommendations, and then require anyone who takes exception to enter notes explaining their rationale. Examples: Extending offers to customers on the telephone, or prescribing medical treatments.

Excerpted from: Data is Easy. Deciding is Hard.

Wednesday, 29 January 2014

Enough already with the Ooh, Shiny! data. Show me evidence to explain these outcomes.

I love data visualization as much as the next guy. I'm big on Big Data! And I quantify myself every chance I get.

But I've had my fill of shiny data that doesn't help answer important questions. Things like: What explains these outcomes? What do the experts say? How can we reduce crime?

Crime data viz Source: Tableau.

Does new technology contribute nothing more than pretty pictures and mindless measurement? Of course not. We can discover meaningful patterns with analytics and business intelligence: Buying behavior, terrorist activity, health effects.

But not all aha! moments are created equal. Looky here! There's poverty in Africa! People smile more in St. Louis! Some of this stuff has marginal usefulness for decision makers. A recent New York Times piece underscores the apparent need for arty manipulations of relatively routine data. In A Makeover for Maps, we learn that:

  • “It doesn’t work if it’s not moving.” (Eric Rodenbeck of Stamen Design)
  • "No more than 18 colors at once. You can't consume more than 18." (Christian Cabot, CEO of wildly successful Tableau Software)

I dare say these aren't the aha! moments strategic decision makers are looking for. This seems like a good time to re-visit the Onion's classic, Nation Shudders at Large Block of Uninterrupted Text.

Crime research forest plotShiny objects are great conversation starters. But many of us a) are busy trying to solve big problems, and b) don't need special effects to keep us interested in our professional lives. We need explanations of causes and effects, transparency into research findings, analysis of alternatives. Take the forest plot, for instance, described very effectively by Hilda Bastian. Here you don't just see crime stats: You discover that some tax-funded social programs might actually increase crime.

Decision makers need presentations that are better suited to them. That's the real data story.

Other examples of gee-whiz visualizations that signal a worrisome trend: The Do You Realize? dashboard, winner of a QlikView BI competition, as reported by Software Advice. And Have you ever wondered how fast you are spinning around earth's rotational axis? Probably not, but now you can find out anyway!

On a brighter note, the very talented Douglas van der Molen is quoted in Makeover for Maps, saying he is “looking for ways to augment human perception to help in complex decision making.” Maybe today's sophisticated tools will lead to something game-changing for problem solvers. Or maybe we'll keep manufacturing faux aha! moments.

Wednesday, 30 October 2013

Don't show me the evidence. Show me how you weighed the evidence.

Sometimes we fool ourselves into thinking that if people just had access to all the relevant evidence, then the right decision - and better outcomes - would surely follow.

Calculator for decision makingOf course we know that's not the case. A number of things block a clear path from evidence to decision to outcome. Evidence can't speak for itself (and even if it could, human beings aren't very good listeners). 

It's complicated. Big decisions require synthesizing lots of evidence arriving in different (opaque) forms, from diverse sources, with varying agendas. Not only do decision makers need to resolve conflicting evidence, they must also balance competing values and priorities. (Which is why "evidence-based management" is a useful concept, but as a tangible process is simply wishful thinking.) Later in this post, I'll describe a recent pharma evidence project as an example.

If you're providing evidence to influence a decision, what can you do? Transparency can move the ball forward substantially. But ideally it's a two-way street: Transparency in the presentation of evidence, rewarded with transparency into the decision process. However, decison-makers avoid exposing their rationale for difficult decisions. It's not always a good idea to publicly articulate preferences about values, risk assessments, and priorities when addressing a complex problem: You may get burned. And it's even less of a good idea to reveal proprietary methods for weighing evidence. Mission statements or checklists, yes, but not processes with strategic value.

Boxplots-D3 libraryThe human touch. If decision-making was simply a matter of following the evidence, then we could automate it, right? In banking and insurance, they've created impressive technology to automate approvals for routine decisions: But doing so first requires a very explicit weighing of the evidence and design of business rules.

Where automation isn't an option, decision makers use a combination of informal methods and highly sophisticated models. Things like Delphi, efficient frontier, or multiple criteria decision analysis (MCDA); but let's face it, there are still a lot of high-stakes beauty contests going on out there.

What should transparency look like? Presenters can add transparency to their evidence in several ways. Here's my take:

Level 1: Make the evidence accessible. Examples: Publishing a study in conventional academic/science journal style. Providing access to a database.

Level 2: Show, don't tell: Supplement lengthy narrative with visual cues. Provide data visualization and synopsis. Demonstrate the dependencies and interactivity of the information. Example: Provide links to comprehensive analysis, but first show the highlights in easily digestible form - including details of the analytical methods being applied.

Level 3: Make it actionable: Apply the "So what?" test. Show why the evidence matters. Example: Show how variables connect to, or influence, important outcomes (supported by graph data and/or visualizations, rather than traditional tabular results).

On the flip side, decision makers can add transparency by explaining how they view the evidence: Which evidence carries the most weight? Which findings are expected to influence desired outcomes?

How are pharma coverage decisions made? Which brings me to transparency in health plan decision-making. Here you have complex evidence and important tradeoffs, compounded by numerous stakeholders (payers, providers, patients, pharma). When U.S. pharmaceutical manufacturers seek formulary approval, they present the evidence about their product; frequently they must follow a prescribed format such as AMCP dossier (there are other ways, including value dossiers). Then the health plan's P&T (Pharmacy and Therapeutics) committee evaluates that evidence.

Recently an industry group conducted a study in an effort to gain deeper understanding of payer coverage decisions. Results appear in “Transparency in Evidence Evaluation and Formulary Decision-Making” (Pharmacy and Therapeutics, August 2013).

“Right now, there is a bit of a ‘black box’ around the formulary decision-making process,” said Robert Dubois, MD, PhD, NPC’s chief science officer and an author of the study. “As a result, decisions about treatment access are often unpredictable to patients, providers and biopharmaceutical manufacturers. We sought to identify ways to clarify the process.”

Whose business is it, anyway? Understandably, manufacturers want to clarify what factors influence the level of access their products receive. And patients want more visibility into formularies: What coverage and co-pays can they expect from their health plan? How is safety weighed against effectiveness? Now that U.S. healthcare is becoming more consumer-driven, I expect something to change.

Transparency in Evidence Evaluation and Formulary Decision-Making
The process. Put simply, the project sponsors were asking payers to explain how they balance the evidence about drug efficacy, safety, and cost. Capturing that information systematically is a big challenge. In scenarios like this, you'll often end up with a big checklist, which is sort of what happened (snippet shown above). An evidence assessment tool was developed by surveying medical and pharmacy directors, who identified key factors by rating the level of access they would provide for drugs in various hypothetical scenarios. 

And then sadness. The tool was validated, then pilot-tested in real-world environments where P&T committees used it to review new drugs. However, participants in the testing portion indicated that "the tool did not capture the dynamic and complex variables involved in the formulary decision-making process, and therefore would not be suitable for more sophisticated organizations." Once again, capturing a complex decision-making process seems out of reach.

Setting expectations. Traditional vendor/customer relationships don't lend themselves to openness. If pharma companies want more insight into payer expectations, they'll have to build strong partnerships with them. That's something they're now doing with risk-sharing and value-based reimbursement, but things won't change overnight. Developing the data infrastructure is one of the challenges long-term, but it seems to me - despite the unsuccessful result with the formulary tool - that more transparency could happen without substantial IT investments.

Friday, 18 October 2013

The Illustrated Book of Bad Arguments.

It's a glorious Fun-with-Evidence Friday. Because I've discovered The Illustrated Book of Bad Arguments. The author is Ali Almossawi (@alialmossawi), a metrics engineer in San Francisco. It's fantastic. Available online now, and soon in hardback.

Illustrated Book of Bad Arguments

Besides the fun illustrations, you'll find serious explanations of logical fallacies, plus definitions of key terms:

"Soundness: A deductive argument is sound if it is valid and its premisses are true. If either of those conditions does not hold, then the argument is unsound. Truth is determined by looking at whether the argument's premisses and conclusions are in accordance with facts in the real world." (BTW, I did not know premise is also spelled premiss.)

Almossawi says "I have selected a small set of common errors in reasoning and visualized them using memorable illustrations that are supplemented with lots of examples. The hope is that the reader will learn from these pages some of the most common pitfalls in arguments and be able to identify and avoid them in practice."

Happy weekend, everyone. 


Thursday, 17 October 2013

Got findings? Show us the value. And be specific about next steps, please.

Lately I've become annoyed with research, business reports, etc. that report findings without showing why they might matter, or what should be done next. Things like this: "The participants biological fathers’ chest hair had no significant effect on their preference for men with chest hair." [From Archives of Sexual Behavior, via Annals of Improbable Research.]

Does it pass the "so what" test? Not many of us write about chest hair. But we all need to keep our eyes on the prize when drawing conclusions about evidence. It's refreshing to see specific actions, supported by rationale, being recommended alongside research findings. As Exhibit A, I offer the PLOS Medicine article Use of Expert Panels to Define the Reference Standard in Diagnostic Research: A Systematic Review of Published Methods and Reporting (Bertens et al). Besides explaining how panel diagnosis has (or hasn't) worked well in the past, the authors recommend specific steps to take - and provide a checklist and flowchart. I'm not suggesting everyone could or should produce a checklist, flowchart, or cost-benefit analysis in every report, but more concrete Next Steps would be powerful.

PLOS Medicine: Panel Diagnosis research by Bertens et al

So many associations, so little time. We're living in a world where people need to move quickly. We need to be specific when we identify our "areas for future research". What problem can this help solve? Where is the potential value that could be confirmed by additional investigation? And why should we believe that?

Otherwise it's like simply saying "fund us some more, and we'll tell you more". We need to know exactly what should be done next, and why. I know basic research isn't supposed to work that way, but since basic research seems to be on life support, something needs to change. It's great to circulate an insight for discovery by others. But without offering a suggestion of how it can make the world a better place, it's exhausting for the rest of us.

Wednesday, 09 October 2013

Could we (should we) use evidence to intervene with compulsive gamblers?

If you've stopped for gas in Winnemuca, you've likely seen a down-and-out traveler in quiet conversation with Max Bet, the one-armed bandit. There's no shortage of heartbreaking gambling stories.


Now some researchers claim thay can identify compulsive gamblers. Advocates say we should follow that evidence to intervene before they suffer devastating losses. For me, this raises a number of questions: It's always complicated when we try to save people from themselves.

The evidence. Casinos know a great deal about customer behavior. The Wall Street Journal cover story Researchers Bet Casino Data Can Identify Gambling Addicts describes the work of Sarah Nelson PhD, who has developed the Sports Bettor Algorithm 1.1 [paywall]. Crunching data from casino loyalty programs, the SB algorithm pinpoints "risky betting patterns such as intensive play over long periods of time, significant shifts in behavior, or chasing losses". Dr. Nelson cautions that the system is not yet very accurate, though it has established some correlation between certain behaviors and addiction.

Focal Research Consultants also works in this area. Tony Schellinck, PhD has spearheaded the creation of algorithms and assessment tools, including the FocaL Adult Gambling Screen (FLAGS), a multi-construct instrument specifically designed to measure risk due to gambling. He is quoted in an ABC story about gambling analytics.

"You've got to learn that a mark is going to give it away to somebody, kid. There's no way to stop a real mark. So when he's ready, you just try to be first in line." 
-John D. MacDonald, The Only Girl in the Game

The question. Two questions, actually. What's a 'compulsive gambler'? And what could (should) casinos do to help them? Understandably, the industy is concerned about exposure to liability (similar to bartenders who are expected to intervene with intoxicated patrons). Some casino executives object, saying this compares to asking a nonprofessional to "diagnose a mental health disorder" (not sure I buy that argument).

  •  Like evidence-based medicine, intervention is easy to imagine, but not so easily achieved. If Caesar's (or another big casino) cut off an addicted customer, wouldn't they simply gamble elsewhere? Or find a different addiction?
  • Where does this end? With health screening at the entrance to every McDonald's? Holding Nordstrom accountable for binge shopping?
  • Speaking to ABC News, Schellinck said some casinos fear algorithms "will show that some of their best customers are addicts, and that the casino's bottom line will suffer if management intervenes with troubled high-rollers." But he has discovered that "The vast majority of problem gamblers are not big spenders. They're people who are spending $200 a month on their habit but can't afford to do it."

Critics want casinos to do more; Jen Miller in Philly Mag vehemently makes that argument. The BlackJack episode of This American Life tells the sad story of a woman who gambled away her inheritance - and then sued the casino, blaming them (she lost).

Clearly this isn't my area. At least theoretically, maybe there could be a pooled service, where alerts went to a central intervention group. But then we'd need a Unique Gambler Identifier to track people across multiple casinos -- sheesh, this is starting to sound like our U.S. healthcare quagmire.

Thursday, 22 August 2013

Major study debunks belief that people in the distance are really tiny.

Evidence shows our eyes have been playing tricks on us. As reported by The Onion, a "five-year study, conducted by researchers at Princeton University, has shattered traditionally accepted theories that people standing some distance away from you are very small, and people close-by are very big."

Wishing you a happy Fun-with-Evidence Friday. On a serious note, though, it seems to be getting tougher to tell the faux research from the silly, funded-with-other-people's-money stuff we often read about. For more on this, take a look at the Annals of Improbable Research.

Monday, 19 August 2013

How can we speed up the adoption of new medical evidence?

Why does some medical evidence gain acceptance quickly, while other findings do not? That is the eternal question. Three recent contributions address this in very different ways: The Journal of Comparative Effectiveness Research, the Trip database, and a New Yorker piece authored by a surgeon/public-health researcher.

But first, I'll pose some questions of my own.

What's "acceptance"? How do you pinpoint acceptance of evidence? Inclusion in a clinical practice guideline? In a formulary? By documenting actual treatments over a sustained time period?

Also, how can you truly know what evidence was used in making a decision? Supporting evidence is sometimes formally ranked, referenced, etc. - but often not. Lack of transparency and consistency are an unfortunate result. (During my PhD research, I attempted to pinpoint the basis of regulatory health decision-making: Bottom line, it ain't easy.)

How much do people consider the source? Sometimes an audience is skeptical about the provider of given evidence; peer-review systems help in this regard (though our current science journal process is plagued with problems). Maybe the findings come from clinical trials, when evidence from real-world patient outcomes is what's preferred. The list goes on.

Our feelings about a particular source overshadow our discussions of the evidence. Just ask Matt Ridley, who evoked a strong response to his views on climate change, especially when writing in the Wall Street Journal. I do admire his efforts to remain objective, saying "it is the evidence that persuades me whether a theory is right or wrong, and no, I could not care less what the 'consensus' says."

How can we speed adoption of evidence? "Diffusion of innovation" is the phrase often applied when people investigate the spread of new findings. A number of factors influence speed of diffusion.

1. Understanding the process. Key influencers are considered in When is evidence sufficient for decision-making? A framework for understanding the pace of evidence adoption (Journal of Comparative Effectiveness Research, July 2013).

The authors (Robert W. Dubois​, Michael Lauer​, and Eleanor Perfetto​) looked at three diverse case studies - statins, drug eluting stents, and bone marrow transplantation for breast cancer – to establish a proposed framework. Five factors stood out: 1) validity, reliability, and maturity of the science available before widespread adoption; 2) communication of the science; 3) economic drivers; 4) patients’ and physicians’ ability to apply published scientific findings to their specific clinical needs; and 5) incorporation into practice guidelines.


This report thoroughly evaluates the case studies and what happened with the associated supporting evidence. I'd like to see a coding scheme to support these qualitative assessments -- though formal codification of such subject matter can get pretty artificial. And the authors directly acknowledge that an "objective application of the framework to a broader and randomly selected set of situations is needed to further validate the findings from the three case studies. [p. 389]" So it's all good.

2. Synthesizing evidence faster. Jon Brassey, developer of the Trip database, has lamented the painfully slow Cochrane systematic review process. He claims "On average a Cochrane systematic review takes 23 months from protocol to publication." and "In an analysis of 358 dermatology questions only three could be answered by a single systematic review, so less than 1%."

So Jon's trying out a provocative idea: Replacing people with machines. "One thing I've been working on recently has been an ultra-rapid review system, based on machine learning and some basic statistics.  In a nutshell can we take multiple abstracts, 'read' what they're about and combine the results to give a 'score' for the intervention? More importantly, will any score actually be meaningful?" These first test results were released August 2.



3. Keeping people talking. In Slow Ideas, Atul Gawande looks at the different adoption patterns for 19th-century surgical anesthesia and antiseptics (New Yorker Annals of Medicine: Some innovations spread fast. How do you speed the ones that don’t?).

Gawande, a surgeon / public-health researcher, observes that "We yearn for frictionless, technological solutions. But people talking to people is still the way that norms and standards change." Amen to that.

What can we do next? [Disclaimer: Shameless self-promotion.] Speed is not always our friend. But simple visualization and evidence synthesis certainly are. That's what I'm working toward at PepperSlice.com.


Friday, 14 June 2013

Fun-with-Evidence Friday: Science works, b!t3^&s!

Sciencetee  Spotted this great t-shirt in Berkeley, California. Evidently the creator made this to summarize his doctoral research: Science: It Works, B!t3^&s.  Sciencetee2







What does "clinically studied" mean? (A cynic might interpret that as "research funds were spent").  I happened to see GNC's Mega Men Sport multivitamin, labeled as a "Clinically Studied Multivitamin^". Labels are small, so detailed findings can't appear on the front of a consumer package. But I wondered about the meaning in this context. Here's what the product page says.

Clinicallystudied"Lutemax 2020™
^In a randomized, double-blind, placebo-controlled study of 112 healthy volunteers, subjects taking the GNC vitamin and mineral blend in this product for six weeks experienced statistically significant improvements in markers of B vitamin and antioxidant status, as well as improvements in SF-36 Vitality and Mental Health scores compared to those taking a placebo."

And there you have it: A little evidence of something or other.

Happy Fun-with-Evidence Friday, everybody.

Friday, 26 April 2013

We've got to stop drooling at shiny data visualizations and keep searching for non-obvious evidence.

Yes, we want people to provide hard evidence supporting their claims. But aren't some things just too obvious to bother with? Shouldn't we use our talents to discover things we don't already know?

Fbfriends_wolfram2I like a shiny data visualization as much as the next guy, but recent details about Facebook analysis made me wonder why some great minds (with great resources) are focusing on some pretty unimportant stuff.

Here's what happened. Stephen Wolfram - who is certainly much smarter than me - released findings from his Wolfram|Alpha Personal Analytics for Facebook project. Among the revelations is that we have a peak number of friends around age 20, with that number slowly dwindling as we reach our 70s.

Sort of like the actual world, wouldn't you say? Doesn't this just confirm what we already know, only with a cool, crowd-sourced, data-donation methodology and attractive data visualization?

Doesn't pass the "six smart people" test. This reminds me of something Rita Gunther McGrath has talked about. She, too, is critical of obvious / we-already-know-that research, saying that if "six smart people in a room" can immediately see it, then it isn't something that needs formalized research. (Rita's a smart one, and I suggest following her at @rgmgrath).


The Daily Mail covered this with the headline "Growing old on Facebook: Search data reveals we talk more about the weather and  politics as we age." (Sadly, reading their story, it's tough to tell if they're being sarcastic or not.) Wolfram found we accumulate clusters of friends as we grow older - with the average 35-year-old having four clusters (again, no kidding: neighbors, work friends, friends from back home, etc.).

Wolfram has done some great stuff, such as A New Kind of Science. I know social network analysis can be important, and reveal useful things about how new knowledge develops. Let's see more of that.

Happy Fun-with-Evidence Friday. Have a great weekend, everybody.