Evidence Soup
How to find, use, and explain evidence.

« Enough already with the Ooh, Shiny! data. Show me evidence to explain these outcomes. | Home | Tech needs to embrace diversity in more ways than one. »

Monday, 21 July 2014

The Data-Driven vs. Gut Feel hyperbole needs to stop.

Smart decision-making is more complicated than becoming ‘data-driven’, whatever that means exactly. We know people can make better decisions if they consider relevant evidence, and that process is getting easier with more data available. But too often I hear tech advocates suggest that people’s decisions are just based on gut feel, as if data will save us from ourselves.


We need to put an end to the false dichotomy of 'data-driven' vs. 'human intuition'. Consider the challenge of augmenting the performance of a highly skilled professional, such as a medical doctor. Investor Vinod Khosla claims technology will replace 80%+ of physicians’ role in the decision-making process. “Human judgment simply cannot compete against machine-learning systems that derive predictions from millions of data points”. Perhaps so, but it’s really tricky to blend evidence into patient care processes: Research in BMJ reveals mixed results from clinical decision support technology, particularly systems that deliver alerts to doctors who are writing prescriptions.

Data+People=Better. One tech enthusiast compares IBM’s Watson to a hospital CEO. Ron Shinkman asks if it could “be programmed to pore over business cases, news clippings, algorithms and spreadsheets to make the same recommendations?” Actually, that’s what Watson does. But Shinkman overlooks the real opportunity: To supplement, not replace, a CEO’s analytical skills. (Note: This is an excerpt from a research paper I recently wrote  at Ugly Research.)

Why IT Fumbles Analytics. In an excellent Harvard Business Review analysis of how decision makers assimilate data, Donald Marchand and Joe Peppard explain that

management lacks “structure. Even when an organization tries to capture their information needs, it can take only a snapshot, which in no way reflects the messiness of their jobs. At one moment a manager will need data to support a specific, bounded decision; at another he’ll be looking for patterns that suggest new business opportunities or reveal problems.”

Here's another example of a claim that new technology will replace human intuition with fact-driven decision-making.


Source: Business analytics and optimization for the intelligent enterprise (IBM).

You’re not the boss of me. There’s a right time and a wrong time to look at data. As Peter Kafka explains, Netflix executives enthusiastically use data to market TV shows, but not to create them. Others agree data can interrupt the creative process. In The United States of Metrics, Bruce Fieler observes that data is often presented as if it contains all the answers. But “metrics rob individuals of the sense that they can choose their own path.”

However, people could do better. Of course decision makers frequently should ignore their instincts. Andrew McAfee gives examples of algorithms that outperform human experts, and explains why our intuition is uneven (we need cues and rapid feedback).

The Economist Intelligence Unit asked managers “When taking a decision, if the available data contradicted your gut feeling, what would you do?” Most preferred to crunch some more numbers. Only 10% said they would follow the action suggested by the data. The sponsors of Decisive action: How businesses make decisions and how they could do it better concluded that while “many business leaders know they need to make better use of data, it’s clear that they don’t always know how best to do so, or which data they should select from the enormous quantity available to them. They are constrained by their ability to analyse data, rather than their access to it.”

How do you challenge a decision maker? When data is available to improve a result, it must be communicated so it challenges people to apply it, not deny it. One way is to provide initial recommendations, and then require anyone who takes exception to enter notes explaining their rationale. Examples: Extending offers to customers on the telephone, or prescribing medical treatments.

Excerpted from: Data is Easy. Deciding is Hard.


Bernadette: Wish I'd included Karl Popper. I'll have to steal your idea and use the 'three worlds' concept in a slide deck sometime soon.

Thanks for your kind words and insightful comments.
Evidence Soup

Well said! As suggested by Karl Popper and Steven Wallis, making better decisions requires a combination of good data, good theory, and meaning to oneself and stakeholders. All of these "three worlds" are important. http://projectfast.org/validation-of-theory-exploring-and-reframing-poppers-worlds/

The comments to this entry are closed.