Evidence Soup

Is the evidence-based management movement dead?

I hope not, but there are many obstacles. Richard Puyt explored this question today on his blog Evidence-Based Management | Skeptical Thinking. He observes that "in management we are still in the middle ages of science, where the alchemists still try to make gold from lead. And by alchemists I mean all types of managers (managers, consultants, coaches, interim-managers, project managers, etc.). One of the reasons why managers still make decisions based on anecdotal evidence, gut feeling or a whim is the fact that management is not a profession.... Management is still treated as a 'skill' and if you have a better story than the next guy, you just found yourself a new career." Puyt makes some good points, concluding that "The Evidence Based Management movement is still there, but progress is really slow. Most developments are exchanged in closed communities and you really have to make an effort to dig up new information and stay in the loop of recent developments."

Here's my quick take on this: The Evidence Soup Guide to Keeping the Evidence-Based Movement Alive.

Part I. How to kill the EBM movement. If people do these things, we'll be writing an obituary:

1. Frown on new evidence. Develop an environment where people are discouraged from challenging tradition, scrutinizing old habits, or asking tough questions. 
2. Oversimplify things that are tremendously complicated. Insist on clearly delineating which decisions are evidence-based, and which are not. Doggedly pursue a set of formal rules for determining precisely whose actions are evidence-based, and whose are not.
   
3. Make evidence exclusive. Behave as if certain insiders (or groups) are the keepers of the evidence, and the rest of us (outsiders) had better sit up straight and pay attention.

Part II. How to pump more life into the EBM movement. Here's how we can nudge EBM into the mainstream:

1. Avoid painting all fuzzy stuff with the same brush. Resist the urge to divide the world into two distinct hemispheres: One where all things are evidence-based, and one where people are just plain wrong. It's not that simple, and we should know better.
2. Accept that we often lack good evidence. It's better to openly acknowledge where solid evidence is missing than to pretend. It sends the wrong message when we try to force-fit or stretch uninteresting evidence where there is none.
3. Set a good example. Encourage people to do things that are evidence-guided (or evidence-informed) every day, to the best of their ability. Create a corporate culture where its okay to ask intelligent questions that challenge authority, myth, and tradition.
4. Use smarter technology. Find better ways to distribute more good evidence to more people. Make evidence easier to interpret so people can appreciate its value and apply it more easily.

My proposal for the Enterprise 2.0 conference: Focus on Quality *and* Quantity.

Thanks to those who voted for me as a speaker at the upcoming Enterprise 2.0 conference (if you haven't yet, you can vote today by going to this survey -- http://bit.ly/E20_Vote -- I'm listed on page 11 as Tracy Altman with Ugly Research. You'll have to page through and click 'Next' to find my proposal on page 11 [How to Get More Enterprise 2.0 ROI], then go to page 16 to submit your vote http://bit.ly/E20_Vote.)

My proposal was inspired by Deming's idea that you can't inspect quality into a product after the fact: Quality is either in a product, or it's not. But I believe many online communities are being managed as if quality can somehow happen later: The focus is on quantity (building bigger networks, establishing connections, and keeping usage levels high). Too often, we allow people to contribute whatever they want, however they want -- and then we use voting, peer review, search tools, and other filtering mechanisms to identify the quality stuff. But there are other ways to handle it: We can design user interfaces that set expectations for how people contribute. This will reduce information overload and improve the quality of the evidence that becomes available.

That's what was on my mind when I submitted this 100-word abstract for Enterprise 2.0:

How to Get More Enterprise 2.0 ROI: Focus on Quality *and* Quantity
Many Enterprise 2.0 initiatives will fail because they emphasize quantity over quality. Initiative owners have invested in creating millions of network connections, but haven’t invested enough in information quality. The result? Information overload and inadequate ROI from social networks and collaboration. This presentation will show how Enterprise 2.0 can support essential conversations that generate more strategic value. Our experience has shown that we can’t improve quality by filtering out the low-quality stuff after the fact – quality must be there from the start. Two key steps: 1) connecting people’s conversations to valuable information, and 2) controlling how people contribute.

New evidence shows that heavy multitaskers have *less* task-switching ability.

A new study offers evidence that heavy media multitaskers actually perform worse at task-switching - according to the researchers, this is "likely due to reduced ability to filter out interference from the irrelevant task set". Makes sense to me. The findings appear in Cognitive control in media multitaskers, published by Eyal Ophir, Clifford Nass, and Anthony D. Wagner in the Proceedings of the National Academy of Sciences.

Stanford's news unit released a plain-English explanation of the evidence, including helpful interviews from the researchers: Media multitaskers pay mental price, Stanford study shows. "People who are regularly bombarded with several streams of electronic information do not pay attention, control their memory, or switch from one job to another as well as those who prefer to complete one task at a time.... High-tech jugglers are everywhere – keeping up several e-mail and instant message conversations at once, text messaging while watching television, and jumping from one website to another while plowing through homework assignments. 'We kept looking for what [multitaskers] are better at, and we didn't find it,' said Ophir, the study's lead author and a researcher in Stanford's Communication Between Humans and Interactive Media Lab."

You know who you are. Test subjects were divided into two groups: People who regularly do a lot of media multitasking, and people who don't. During several experiments, the researchers expected the heavy multitaskers to excel at something compared to the other group. But it didn't work out that way. The experiments asked people to do things like look at flash cards with rectangles or letters and identify whether they changed from one card to another. "'The low multitaskers did great,' Ophir said. 'The high multitaskers were doing worse and worse the further they went along because they... had difficulty keeping them sorted in their brains.'"

The researchers also investigated whether heavy multitaskers excel at switching between tasks. Nope. '"They couldn't help thinking about the task they weren't doing,' Ophir said. 'The high multitaskers are always drawing from all the information in front of them.'"

The research abstract is relatively straightforward, though it tells us less than it could, and is written in typical academic-journal style. It says that "A series of experiments addressed whether there are systematic differences in information processing styles between chronically heavy and light media multitaskers. A trait media multitasking index was developed to identify groups of heavy and light media multitaskers. These two groups were then compared along established cognitive control dimensions. Results showed that heavy media multitaskers are more susceptible to interference from irrelevant environmental stimuli and from irrelevant representations in memory. This led to the surprising result that heavy media multitaskers performed worse on a test of task-switching ability, likely due to reduced ability to filter out interference from the irrelevant task set."

Yikes. Evidently, it's possible to fake DNA evidence.

Gulp. The New York Times piece DNA Evidence Can be Fabricated explains that scientists have demonstrated that it's possible to fake DNA evidence. "The scientists fabricated blood and saliva samples containing DNA from a person other than the donor of the blood and saliva. They also showed that if they had access to a DNA profile in a database, they could construct a sample of DNA to match that profile without obtaining any tissue from that person." Big gulp.

Does this change everything? Findings are presented in Authentication of forensic DNA samples, published in the Elsevier journal Forensic Science International: Genetics. The NYT quoted Dan Frumkin, lead author of the paper: “You can just engineer a crime scene.... Any biology undergraduate could perform this.”

Dr. Frumkin is with Nucleix, a company that has developed a lab test for identifying fake DNA. According to their site, the Nucleix "authentication assay distinguishes between natural and artificial DNA samples. The assay... outputs a decision whether the sample is natural or artificial. An internal validation step also controls for amplification failure due to PCR inhibitors, insufficient template DNA, etc." If their research stands the test of time, I'm guessing this will be very, very good for Nucleix investors and insiders.

Gold star. In a well-written abstract, the researchers do a good job of explaining their approach, and what they found: "the disturbing possibility that DNA evidence can be faked has been overlooked. It turns out that standard molecular biology techniques such as PCR, molecular cloning, and recently developed whole genome amplification (WGA), enable anyone with basic equipment and know-how to produce practically unlimited amounts of in vitro synthesized (artificial) DNA with any desired genetic profile. This artificial DNA can then be applied to surfaces of objects or incorporated into genuine human tissues and planted in crime scenes. Here we show that the current forensic procedure fails to distinguish between such samples of blood, saliva, and touched surfaces with artificial DNA, and corresponding samples with in vivo generated (natural) DNA. Furthermore, genotyping of both artificial and natural samples with Profiler Plus® yielded full profiles with no anomalies. In order to effectively deal with this problem, we developed an authentication assay, which distinguishes between natural and artificial DNA based on methylation analysis of a set of genomic loci: in natural DNA, some loci are methylated and others are unmethylated, while in artificial DNA all loci are unmethylated. The assay was tested on natural and artificial samples of blood, saliva, and touched surfaces, with complete success. Adopting an authentication assay for casework samples as part of the forensic procedure is necessary for maintaining the high credibility of DNA evidence in the judiciary system."

Operations research supports evidence-based management. Such as explaining why governments won't fully eradicate the Taliban by force alone.

Analytics. Business intelligence. Operations (a/k/a operational) research. They overlap somewhat, but each refers to a sophisticated way of analyzing information to support management decisions. You can use these techniques to get valuable evidence about marketing strategies, business performance, or government programs: Not with systematic reviews or experimental research, but with data mining and mathematical modeling. I'd like to see more collaboration between people in these fields and so-called evidence-based management.

Developing evidence. I've been looking at how people are using operations research (O.R.) to manage risk and inform evidence-based decision-making: How is the evidence developed, and how is it communicated? The Wikipedia page says O.R. is an "interdisciplinary branch of applied mathematics and formal science that uses methods such as mathematical modeling, statistics, and algorithms to arrive at optimal or near optimal solutions to complex problems. It is typically concerned with determining the maxima (of profit, assembly line performance, crop yield, bandwidth, etc) or minima (of loss, risk, etc.) of some objective function. Operations research helps management achieve its goals using scientific methods."

O.R. has many practical applications. The Science of Better.com describes it as "the discipline of applying advanced analytical methods to help make better decisions." The site is sponsored by the Institute for Operations Research and the Management Sciences (INFORMS®), a group of O.R. professionals. The Science of Better offers an excellent Executive Guide providing examples and success stories (16-page pdf). They offer practical advice, such as how to work with an O.R. professional (explaining what to expect during an initial assessment followed by analysis, prototyping, and system implementation). Although the guide doesn't actually refer to "evidence", they are describing ways to produce current, best evidence that helps executives make up their minds -- and isn't that what evidence-based management is about?

Fighting insurgents. O.R. helps people understand public policy issues and evaluate military strategies. As described in Operations Research, a journal published by INFORMS®, a recent O.R. study demonstrates why "even under best-case assumptions ...government cannot totally eradicate an insurgency by force." In “Why Defeating Insurgencies is Hard: The Effect of Intelligence in Counterinsurgency Operations – A Best-Case Scenario”, Moshe Kress and Roberto Szechtman of the Naval Postgraduate School discuss what they say is the first study of its kind to "combine military intelligence, attrition and civilian population behavior in a unified model of counterinsurgency dynamics" ($22US; Vol. 57 No. 3, May-June 2009, pp. 578-585). The researchers modeled the dynamic relations among intelligence, collateral casualties in the population, recruitment to the insurgency, attrition, and reinforcement to government forces.

The authors do a good job of explaining their findings in the research abstract, especially considering how complex such a study must be. They say in plain English: "In insurgency situations, the government-organized force is confronted by a small guerrilla group that is dispersed in the general population with no or a very small signature. Effective counterinsurgency operations require good intelligence. Absent intelligence, not only might the insurgents escape unharmed and continue their violent actions, but collateral damage caused to the general population from poor targeting may generate adverse response against the government and create popular support for the insurgents, which may result in higher recruitment to the insurgency." Rather than completely eradicate insurgents by force, the best a military can do is contain it "at a certain fixed level."

Note to the P.R. department: You could do a better job of spreading the evidence. The press release for the insurgency research article links to an obscure page about the Operations Research journal -- not the abstract, a Table of Contents, or a page where you can pay to access the article. I had to look around more than should be necessary.

Epilepsy Foundation includes survey responses as evidence supporting policy recommendations.

Anecdotal evidence can be valuable evidence. The Epilepsy Foundation (in the United States) conducted an online survey to gather input on what people experienced after their drug prescriptions were changed. The purpose wasn't to get a representative sample -- it was to get specific examples to include with a set of policy recommendations. Results appear in the report In Their Own Words: Epilepsy Patients’ Experiences Changing The Formulation of The Drugs They Use to Prevent Seizures: Findings from the Epilepsy Foundation's Survey of Patient Experiences, Nov 2006- Mar 2009 (17-page pdf).

More than 1,000 epilepsy patients (and parents of children with epilepsy) participated in the web survey. They raised numerous issues, including problems that arise when prescriptions are changed without the knowledge of patients -- or their physicians. (Epilepsy and seizures affect millions of Americans, often with debilitating consequences. As a volunteer, I advise the Epilepsy Foundation of Colorado on technology strategy. If you're looking for a cause to support, this one is very worthwhile.)

In their own words. The report says people "have for years reported experiencing seizures and other harmful side effects after switching from one version of an anti-epileptic drug (AED) to another, whether the switch was brand-to-generic, generic-to-brand, or generic-to-generic. By law, the amount of medication delivered by one AED may differ from the amount delivered by another AED that the FDA deems 'equivalent,' and it may deliver the medication at a different rate. Amid growing evidence that these variations, however slight, can mean the difference between controlled epilepsy and breakthrough seizures or other negative consequences, the Epilepsy Foundation asked to hear patients’ experiences."

Combining anecdotes with formal research. Along with survey responses, the report includes numerous references to more traditional evidence that supports the foundation's recommendations. I think it's an interesting approach -- the quotes from patients may be anecdotal evidence, but they *are* hard evidence. Clearly there can be hidden costs, and not-so-hidden consequences.

The survey responses revealed that::

• 15% of those who reported switching from a generic to a brand-name AED reported worse seizures; 18% reported worse side effects.
• 59% of respondents who switched from a brand-name to a generic reported worse seizures; 49% reported worse medication side effects.
• Over 25% of participants reported problems when switching between different generic formulations.

Research and policy recommendations. The report includes several specific recommendations for U.S. policymakers and also for patients, pharmacists, and health care providers. These include:

• FDA should research whether brand-name and generic AEDs are truly therapeutically equivalent. Methodology considerations should include determining whether test groups appropriately represent the patient population, and requiring individual bioequivalence measures.
• FDA should "require drug manufacturers to make all the bioequivalence information of their generic and brand-name AED products available to physicians the scientific community" and to "carry out post-marketing surveillance for all generic and brand-e AEDs, and should take all necessary steps to ensure broader reporting of adverse reactions to switching."
• Patients and pharmacists should discuss with physicians the appropriateness of substitution before switching among different versions of the same AEDs. Patients should provide informed consent prior to any change in medication. Pharmacists should be educated on the unique nature of epilepsy, and the need for caution when switching AEDs.

Why don't physicians adhere to evidence-based guidelines? For starters, the guidelines need to be more transparent and less ambiguous.

Implementation Science has a new article, Why don't physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners. The researchers did a qualitative study: 6 focus groups including 30 physicians, reviewing 56 key recommendations derived from 12 national guidelines. Each group discussed barriers preventing implementation of key recommendations.

Findings. The most commonly perceived barriers were:

• Lack of agreement with recommendations due to lack of applicability or lack of evidence (68% of key recommendations),
• Factors such as organizational constraints (52%),
• Lack of knowledge regarding the guideline recommendations (46%), and
• Guideline-specific factors such as unclear or ambiguous recommendations (43%).

Evidently, we have a ways to go before we'll have decent evidence-based guidelines. The barriers identified by the physician participants differed depending on the particular guideline being discussed. The authors recommend "barrier-driven implementation strategies focusing on key recommendations... to improve adherence in practice. In addition, guidelines should be more transparent concerning the underlying evidence and applicability, and further efforts are needed to address complex issues such as comorbidity in guidelines."

These folks are improvers. The authors of this study -- and essentially all researchers whose work I've reviewed in Implementation Science -- fall into the improver category I described last week when writing about Nick Midgley's essay on problems with evidence-based practice: "Improvers 'support the aims and goals of the EBP movement' and want to see it influence real-world practice. For them, the primary issue is better implementation of research findings. They focus on activities such as diffusion, dissemination, knowledge transfer, translation, and spread – believing the best way forward is to identify obstacles that prevent evidence uptake, and find ways around them."

The authors are Marjolein Lugtenberg, Judith M Zegers-van Schaick, Gert P Westert, and Jako S Burgers. Implementation Science 2009 4:54 doi:10.1186/1748-5908-4-54, 12-August-2009.

Using survey responses as hard evidence: Attensity software analyzes customer feedback.

Surveys fascinate me. They're a great way to interact with people who want to be heard, despite all the drawbacks about inadequate sample size, self-selection, etc. You can get solid evidence, or at least bits of evidence to spark ideas. But slogging through page after page of comments, looking for a gem of an insight, trend, or idea, is time-consuming and difficult to manage.  (This is like digging through public comments on proposed government regulations, which is what I analyzed for my PhD dissertation.)

The Attensity Group is doing something with text analytics that sounds like it could provide valuable evidence for product managers and marketers. Its technology helps make sense of responses by analyzing open-ended customer feedback survey questions known as 'verbatims'. (KM World covered this in And the survey really said.)

According to Attensity, its "award-winning Voice of the Customer application uses semantic analysis to mine the unstructured responses in customer feedback. It combines structured information from surveys with unstructured verbatims to get to the root cause of given scores, identify warning signs and 'cries for help,' and help you take appropriate actions to mitigate issues and ensure customer satisfaction." It can help you "better understand customer sentiment, satisfaction, loyalty and potential product or service issues." The service combines the company's text analysis engine with its patented Exhaustive Extraction technology: "Attensity VoC Survey Advantage automatically identifies facts, opinions, requests, trends and trouble spots from unstructured text and transforms the data into structured, actionable intelligence." If they can do that, it sounds like a good way to support evidence-based decision-making.

Scientists get training on knowledge translation. Excellent way to spread the evidence.

There's a movement to require scientists to participate more fully in knowledge translation and improve the impact of their research. Later this month is a Scientist Knowledge Translation Training Course (Aug 31- Sept 1) in Toronto, Ontario. According to the announcement, "A well-developed knowledge translation (KT) plan is emerging as a proposal requirement for health research in Canada and abroad, and there is greater attention to research utilization and research impact." The course was "developed on the premise that scientists are agents of change in creating research impact, promoting research utilization, and ensuring their research findings reach the appropriate audiences. There is a skill set surrounding KT practice, and it is these competencies that the training course was designed to impart."
More info at  http://bit.ly/StrLL.

Not sure about "evidence-based"? Try eminence. Or providence. Or eloquence, even.

It's not every day I see something offering insight into "evidence-based ___" that also makes me laugh. Seven alternatives to evidence based medicine is a tongue-in-cheek analysis that's worthwhile reading whether you're in health care, business, government, etc. The authors, "two humble clinicians... asked colleagues what they would do if faced with a clinical problem for which there are no randomised controlled trials and no good evidence." They encountered "personality based opinions" and categorized their findings "on the basis of no evidence whatsoever."

As shown here, you can opt for several other flavors of action, each with its own way of assessing effectiveness: The authors elaborate on each approach. David Isaacs and Dominic Fitzgerald are with the Departments of Education and Medicine at New Children's Hospital, Westmead, Australia. All kidding aside, they do a nice job of recapping a key problem with evidence-based management, saying: "Clinical decisions should, as far as possible, be evidence based. So runs the current clinical dogma. We are urged to lump all the relevant randomised controlled trials into one giant meta-analysis and come out with a combined odds ratio for all decisions. Physicians, surgeons, nurses are doing it -- soon even the lawyers will be using evidence based practice. But what if there is no evidence on which to base a clinical decision?