Evidence Soup

Garry Trudeau agrees: It's an evidence-based world now.

I can't think of a more promising development than Doonesbury's declaration that "it's an evidence-based world now." Indeed!
Here's an excerpt from the cartoon for 30-Jan-2009:

Thanks to Drew for spotting this. Happy Fun-with-Evidence Friday, everybody!

Announcing the Ebby Awards: Honoring evidence-based excellence.

Evidence-based action can make a big difference in people's lives -- and sometimes it saves people's lives. But it's not easy finding evidence, figuring out what it means, and using it to make change happen. At Evidence Soup, we think these efforts should be better recognized. So we're delighted to announce the first annual Ebby Awards, honoring excellence in all things evidence-based.

The Ebbys will recognize the people, products, services, web sites, and management programs that do the best job of helping us find, understand, and apply meaningful evidence. Winners will be awarded in several categories -- including (but not limited to) medicine, energy, publishing, enterprise software, technology development, strategic planning, human capital, teaching, sales, and marketing.

Want to nominate somebody? Details are coming soon. Stay tuned.

Ebby Awards? Not to be confused with the Emmys. Or the Webby Awards. The Ebbys belong to those of us who dedicate ourselves to evidence-based action.

Ski resort wants Colorado tourism officials to show them the evidence.

Evidence-based marketing (or any type of evidence-based management, for that matter) shouldn't happen only in the private sector. So I like what Vail officials are saying about the management of Colorado tourism dollars. According to today's Rocky Mountain News, they want to "hold the state's tourism office more accountable for the marketing money it spends by setting up an industry panel that would advise and oversee the government agency. The state budgets almost $20 million a year for tourism marketing. In a five-page proposal obtained by the Rocky Mountain News, the Broomfield-based company told the Colorado Tourism Office it should be held to 'the same strict return on investment requirements that corporate marketing groups are held to.'"

Good idea. President Obama is pushing for more government transparency. This sounds like a good place to make that happen.

What does the evidence say about good decision-making?

One would hope that you'd benefit from routinely following a good decision-making process -- but what would be the characteristics of that process? The McKinsey Quarterly recently addressed that in a global survey and published the results in How companies make good decisions. The article opens by asking "Which decision-making disciplines really make a difference?" and "Do strong decision-making processes lead to good decisions?"

All this had me wondering how they define "good decision". Any decent manager knows a good decision isn't simply one with a good outcome: Wagering your entire 401(k) at the Bellagio in Vegas is probably a bad decision, regardless. But investing in promising R&D innovations is a good decision, even though a significant portion of those bets will be financial losers.

What does the evidence say? McKinsey provides evidence to support their conclusions. As shown above, their survey "highlights several process steps that are strongly associated with good financial and operational outcomes.... The results highlight the hard business benefits — such as increased profits and rapid implementation — of several decision-making disciplines. These disciplines include ensuring that people with the right skills and experience are included in decision making, making decisions based on transparent criteria and a robust fact base, and ensuring that the person who will be responsible for implementing a decision is involved in making that decision."

Try to get the odds in your favor. As my middle school Latin teacher used to say, you can make your own choices, but you can't make your own consequences. By choosing a reasonable decision process, you will usually avoid the ugliest consequences -- the evidence shows that taking the right steps should improve your success rate. So, what goes into a "good" decision?

• A "good" decision means that the "right" variables were considered, and the "right" people consulted.
• McKinsey found that "good decision making involves avoiding some basic mistakes. Decisions initiated and approved by the same person generate the worst financial results — indicating the value of good discussion. And decisions made at companies without any strategic planning process are twice as likely to have generated extremely poor results as extremely good ones — more than a fifth of them generated revenue 75 percent or more below expectations."
• The survey also found that "One relatively straightforward finding is of strong relationships linking financial success, clarity about who is responsible for implementation, and the involvement of that individual in the decision-making process...."

But outcome isn't the only evidence. I like what McKinsey is doing here, but I wish they'd emphasized the importance of resisting the urge to equate good outcomes with good decision-making. It's easy to see weak results and question the ability of the person running the show, and vice versa. Basic financials are frequently poor analytical benchmarks: The petroleum executives reporting astronomical profits two quarters ago have the same skills today -- ExxonMobil is still considered a well-managed company, and certain others, not so much. To McKinsey's credit, they did include some other meaningful indicators in the survey, such as time to project completion.

Like it or not, sometimes you'll collaborate like crazy, and analyze your brains out, and still overlook a crucial relationship, side effect, or possible consequence (as explained very well by Edward Tenner in Why Things Bite Back). I'll let the U.S. economy from 2000-2008 speak for itself.

Evidence for probiotics is slowly developing.

If you've ever lost much of your so-called flora -- the various bacteria in your gut that help you process food -- I'll bet you never forget that experience. I'll spare you the details and just say that I'd go to tremendous lengths to avoid reliving it.

So should I be consuming the new breed of probiotic functional foods and dietary supplements? There's some evidence that says yes, though many questions remain unanswered, and some scientists claim we don't yet know which bugs to add to our diet.

Pasteurization has improved food safety by removing bad bugs from our food supply. But a healthy human gut contains a few pounds of good bugs (thousands of species and strains of bacteria). We know that things like antibiotics can disrupt flora for months. Now lots of products are being introduced into the market to help maintain or replenish it. However, it's not clear which products make a substantial difference for most people.

A gray area, evidence-wise? The World Health Organization defines probiotics as "live microorganisms that when administered in adequate amounts confer a health benefit on the host." WHO has issued guidelines on evaluating probiotics in food. The U.S. National Institutes of Health (NIH) National Center for Complementary and Alternative Medicine talks about the science behind probiotics, summarizing recent research and listing references.

This week's Aches & Claims column in the Wall Street Journal (behind paywall) details a few products backed by scientific evidence, identifying which bacteria they include and what benefits are associated with them. For example, the dietary supplement Florastor uses Saccharomyces boulardii yeast to help people avoid harmful effects from antibiotics. (However, in keeping with WSJ tradition, they don't name, or link to, the supporting studies. So if you want to see the evidence, you'll have to go looking on your own.) Other promising evidence was recently reported by Science Centric.

As explained in a 2008 L.A. Times article, most probiotics contain bacteria isolated from milk products, typically species of Lactobacillus or Bifidobacterium. The article mentions a clinical study of 135 patients published in the British Medical Journal that showed 22% improvement for people who had been taking antibiotics -- they drank a probiotic containing live Lactobacillus and Streptococcus. I'll award a few points to the Times for at least telling us what journal published the findings (even though they don't link to the article). In the BMJ I found a study concluding that, in a randomised double blind placebo controlled trial, people drinking a probiotic containing L casei, L bulgaricus, and S thermophilus experienced fewer negative effects. So, if I ever muster the courage to take antibiotics again, I'm washing the pills down with this stuff!

The jury is still out on much of this. Skepticism remains, however. Hundreds of new probiotic products are hitting store shelves, virtually all of them promising better digestion and/or overall health. And some of those claims are being challenged in court, including those for the Dannon product Activia.

How useful are pre-employment screening tests? Evidence shows they can help retailers save money and make money... even if some people cheat.

The Wall Street Journal ran a recent story about pre-employment personality tests (behind paywall). In Test for Dwindling Retail Jobs Spawns a Culture of Cheating, they talk about someone who found answers online prior to taking a test called Unicru (now known as Kronos Workforce Acquisition). This test is frequently used to screen applicants for hourly jobs, such as retail positions.

Show me the evidence. Reading that, I wondered if such tests are really very helpful. Does this cheating erode the validity and value? Kronos, the sponsor of the test mentioned in the Wall Street, offers some impressive evidence that their workforce program can pay off for employers. They provide a case study for one of their customers, the sports retail chain Finish Line: 

"Finish Line has seen continuous year-over-year improvement in critical metrics like employee turnover, time spent per interview, and sales per associate since implementing Workforce Acquisition. In a one-year period, over 300,000 candidates applied for employment at Finish Line, with 75 percent of them applying online. Using Kronos’ assessment scores, the company was able to eliminate approximately one-third of applications that did not meet the basic hiring criteria. Finish Line estimates that it saved 60,000 store manager hours by automating the administrative aspects of the hiring process and not having to interview those candidates who did not meet the basic hiring criteria. Furthermore, those who do get hired are staying on the job longer.... Finish Line has also seen a nearly 10 percent improvement in retention companywide. This translates into hard cash savings as the costs associated with sourcing, interviewing, and hiring more employees has dropped. Cost savings are a tangible and important benefit, but even more important are those benefits that help drive business results. This is evident in the increase in sales per associate by approximately 20 percent."

No one can say for sure how many of these improvements can be directly attributed to the Kronos employment testing, but based on these numbers, it's clearly part of a worthwhile management initiative.

Hey, at least cheaters are proactive. As for the cheating? I suppose someone who goes to the effort of finding test answers online is showing initiative -- so maybe that's evidence suggesting they'll be a more engaged employee. According to the Wall Street, "The producer of the test, called Unicru, says it believes the incidence of cheating is low, because there's no decline in the benefits it brings retailers: lower employee turnover, better safety and improved sales performance. 'We see absolutely no evidence of any significant cheating taking place in the use of our assessments or that the cheating is substantially affecting the validity of the assessments,' says David Scarborough, who helped develop the test and works for its owner, Kronos Inc."

Kronos seems to be doing something right: They say their test is deployed in 38,000+ distributed locations nationwide (more than anyone else), and that more than 100,000 daily users are processing, on average, one hourly application every two seconds.

(Call for submissions.) Science you can use: Evidence-based principles & practices.

I've written about the Evidence-Based Management Collaborative before. They're developing ways to structure research findings and other validated evidence for consumption by decision-makers, business managers, and other practitioners. Now they're requesting proposals for an interesting project called Science You Can Use. I've reproduced their call for submissions below. Submittals can go to Denise Rousseau at Carnegie Mellon (denise AT cmu.edu).

 

Call for Submissions

Science You Can Use: Evidence-Based Principles and Practices

 

This is a Call for Submissions to a new SIOP/APA series Science You Can Use.

 

Science You Can Use is an annual series of SIOP to be published by APA Press. Each chapter provides an answer to a practical question relevant to practitioners, including managers, HR professionals, consultants and policy makers. Science You Can Use is a key part of SIOP’s initiative on Evidence-Based Management, making important and useful scientific evidence accessible to practitioners.

All chapters in this series will be based on quality evidence and written in a practitioner-friendly fashion. 

 

The basis of each chapter is a question (or set of questions) of concern to contemporary practitioners.  For example, How to train people quickly? How to manage long-standing conflict between two workgroups? How to build commitment in a distributed team? How to select team members for high performance?  How to effectively use pay as a motivator? When is pay not an effective motivator? Each chapter reflects the state of our science regarding evidence-based organizational and psychological practices and principles. 

 

As an annual series, Science You Can Use chapters will be revised over time as the body of evidence shifts and as knowledge on effective application grows.  Once the first volumes are published, updates for previous chapters based on new knowledge and applications will be very welcome. 

What is Quality of Evidence?

Each chapter is supported by systematic scientific evidence of high quality. Systematic evidence takes one of three forms:

1.      Meta-analysis, statistical summaries of quantitative results and effect sizes;

2.      Systematic review, comprehensive reviews of quantitative and qualitative research relevant to a specific question--usually practice-related; or

3.      Good empirical evidence is sufficient basis for a chapter where evidence #1 or #2 don’t yet exist. Good empirical evidence exists when an important relationship is found in a minimum of three studies from at least two different research groups/teams where no other existing evidence contradicts this finding (cf., Chambless & Hollon, 1998).

           

High quality evidence means that the findings of 1, 2 or 3 above reflect reliable and valid data with appropriate controls for bias and alternative explanations. Specifically, high quality is attained where causal analysis exists based on longitudinal studies excluding plausible alternative hypotheses or on randomized experiments. Outcome measures are from sources different from independent variables. Last, all measures are reliable and valid.

 

          How are Chapters Structured?

All Evidence Summaries Follow the Same Format. Use of common format makes them easier to read and use. Our goal is to encourage thoughtful practitioners to access and use the entire set of chapters.¹

 

Chapter Outline

1.   Use: How do I …?  A key question(s) motivates each chapter. Recall, the examples mentioned above: How to train people quickly? How to manage long-standing conflict between two workgroups? How to build commitment in a distributed team? How to select team members for high performance? Etc.

The uses or circumstances related to the question should be described in ways that enable me, a practitioner, to imagine how I might use the information the chapter provides. 

2.    Basic psychological and management principle(s)

A basic principle states an important, general causal relationship. X increases Y under Z conditions: e.g., “Clear Task Goals enhance Learning especially for Inexperienced Learners.”  A statement of the basic psychological or management principle should be accompanied by a brief plain English summary or short explanation of the core idea on which the chapter is based.

3.    Mechanisms

 What mechanisms underlie the core principle? Explain why I can get this result?

            Conditions in which connections are stronger/weaker

(How important is experience? Prior training? What if I only hire one person a year? Have limited authority?)

4.   Detailed Explanation

Summarize the scientific evidence on which this principle is based. If a meta-analysis or systematic review is the basis, a brief statement of its findings or conclusions is sufficient. A more descriptive summary is needed in the case of evidence #3 above, so that I as a practitioner have a good reason to be confident of your conclusions and recommendations.

Additional details here might help me the user tailor my own applications. Help me link the principle and its related practices other scientific principles and interventions I might already be using.

5.   Validation/Testimony

            Validation: quality of evidence from Sources 1, 2, or 3 above is summarized in clear English.

Testimony (expert responses, see epicurious.com) Practitioners like me place an enormous weight on examples. Any evidence of the principles being used well (“Exemplary Uses”) would help me envision what to do.

  6.   Implementation:

A.    What Problems Can Acting on this Principle Solve?

        Specify the General Class of Problem to which This Principle Applies.

Review each general circumstance where the principle can be applied. In discussing specific uses, the reader should be able to realize the following: “I am acting against the evidence when my practices don’t follow this principle.”

    B.    How to Apply the Principle? Issues in Application

Evidence-based Procedure(s) for Solving Problem --Keep in mind that practitioner judgment and experience play important roles at each step. General action guides and features of implementation are the focus here.

a.       Diagnosis: problems to solve?

b.      What actions? By whom?

c.       Scope: level/people/system

d.      Time frame: how long do I need to wait for the mechanisms to take effect?

e.       Decision Supports: checklists, action guides

f.        Context and Limiting Condition: 

Moderators--Enabling Conditions, Disabling Conditions

            Practical limits: Time, dollars, level of effort and attention.

            Potential ethical dilemmas

g.       Effects/consequences (effect size/statistical confidence)

i.             Possible side effects (reactions to the action)—counter-indicators (how will I know if it might not be working)

ii.            Practitioner Guidelines: What are indicators that principle is working? Lead? Lagged? 

      Repeat for each distinct problem where principle can be applied.

    7.     References

 

What is a Reasonable Chapter Length?

Approximately 20-25 manuscript pages total would be reasonable and ‘aspirational’.  If you can produce a well-written chapter in fewer than 20 manuscript pages, wonderful!

Why Should I Contribute a Chapter(s) to Science You Can Use?  

First and foremost, your chapter would put a validated principle regarding organization and management practices into the hands of thoughtful practitioners. You’re helping disseminate knowledge practitioners can use and trust. Doing so places a plank on the bridge that helps to span the research-practice gap.  And, we hope, this effort encourages bridge building from the practitioner direction too. Second, per the experience in evidence-based movements, authors of evidence summaries become well-recognized experts in their subject domain.  It is a two-fer.  

What’s the Submission Process?

 Send a 2-3 pg proposal to Denise M. Rousseau (Carnegie Mellon U), Editor (denise AT cmu.edu) outlining the principle(s) you’ll cover and the quality of the validating evidence (#1-3 above). Include a supporting bibliography--a single reference is sufficient in the case of meta-analysis or systematic review, a minimum of 3 empirical studies for #3 “good quality evidence.” The editorial team will evaluate proposals. The Editor will inform authors of target dates for upcoming Annual publication.  Target dates are scheduled to allow time for editorial review of each chapter prior to publication.

Accepted proposals will be publicized on the Science You Can Use website to avoid redundant efforts.  General summaries of completed chapters will be available on-line with author contact information to promote public awareness and use of these evidence summaries. 

Can I Write a Chapter for the First Volume of Science You Can Use?

Proposals received by April 15, 2009 are in line for publication in the 2010 issue.

Completed chapter drafts are due August 15, 2009 for editorial review prior to publication in Volume 1. Please contact Denise M. Rousseau (denise AT cmu.edu) with any questions.

Consulting Editors

John Boudreau, USC Rob Briner, Birkbeck College Wayne Cascio, U of Denver Debra Cohen, SHRM David Denyer, Cranfield Don Ferrin, Singapore Management U Mark Fichman, Carnegie Mellon Michael Frese, U Giessen, London Business School Andy Garman, Rush Presbyterian Medical Jody Goodman, U Connecticut Jerald Greenberg, U Texas-Arlington Robert Greene, RewardSystems Inc. Leaetta Hough, Dunnette Group Timothy Judge, U of Florida Steve Kerr, Goldman Sachs

Anthony Kovner, New York University Carrie Leana, U of Pittsburgh Ed Locke, U Maryland Anne O’Leary-Kelly, U Arkansas Jone L. Pearce, UC Irvine Craig J. Russell, U of Oklahoma Sara Rynes, U of Iowa Lise Saari, IBM Eduardo Salas, U Central Florida Ben Schneider, Valtera Sim Sitkin, Duke Univ Jayne Speicher, Pradco Inc. Vicki Vandaveer, Vandaveer Associates Rolf van Dick, U of Frankfurt John Zanardelli, Asbury Heights

 

¹ Chapter format is based on the evidence-summary format developed by the Evidence-Based Management Collaborative.

Reference

Chambless, D.L. & Hollon, S.D. 1998 Defining empirically supported therapies. Journal of Clinical and Consulting Psychology, 66, 7-18.

Evidence available to help people select a nursing home.

I can't think of a better use for evidence than to help people select a nursing home for a loved one. In the U.S., Medicare certifies some facilities to participate in their program, and gathers lots of evidence along the way. Using that evidence, they've developed a set of resources for the general public, available at Nursing Home Compare. Overall, I believe NHC does a reasonably good job of presenting useful information, and helping people find evidence for themselves: This includes a search tool, checklists, and plain-English descriptions of rating criteria.

I've written before about the difficulty of gathering evidence to compare hospitals in a meaningful way -- and the difficulty of getting people to use that evidence once it becomes widely available. I'm guessing the same holds true for efforts to rate nursing homes. But the NHC search tool is easy to use. Within the search results are ratings such as these, based on a 5-star system:

Evidence for everyone. To evaluate a nursing home, Medicare provides lots of hard evidence that one could consider, such as health inspection results or number of nurses/resident. But anyone who's stayed in a hospital or other care facility knows many things influence the success of a stay -- so I'm glad to see they're encouraging people to consider a wide range of factors, including the # of miles from family members, the noise levels in dining rooms, and lighting throughout the facility. Here's a sampling of the checklist provided:

Self-reported evidence. Some of the 5-star rating criteria, such as quality measures, are based on data that's self-reported to Medicare by the nursing homes themselves. That could be problematic. The NHC site explains it this way:

The NHC database is available for download: "You have the option of downloading the data that is used by the Nursing Home Compare tool onto your computer. The data will be downloaded as zipped Microsoft Access databases. This function is primarily used by health policy researchers and the media." These data are grouped into five categories, providing details about nursing homes, inspection results, residents, the nursing home staff, and facility ratings.

Not all facilities are included, however. Those listed on Nursing Home Compare 1) are certified to participate in Medicare and/or Medicaid, and 2) provide so-called "skilled" care (skilled nursing or rehabilitation staff). Other homes may be licensed by state agencies. The site also identifies and explains some long-term care alternatives to nursing homes.