Evidence Soup
How to find, use, and explain evidence.

Tuesday, 06 October 2015

Superforecasting, hot hand redux, and junk science.

This week's 5 links on evidence-based decision making.

Hear me talk on communicating messages clearly with data. Webinar October 14: Register here.

1. Good judgment → Accurate forecasts → Better decisions Jason Zweig (@jasonzweigwsj) believes Superforecasting: The Art and Science of Prediction is the "most important book on decision-making since Daniel Kahneman's Thinking Fast and Slow." Kahneman is equally enthusiastic, saying "This book shows that under the right conditions regular people are capable of improving their judgment enough to beat the professionals at their own game." The author, Philip Tetlock, leads the Good Judgment Project, where amateurs and experts compete to make forecasts - and the amateurs routinely win. Tetlock notes that particularly good forecasters regard their views as hypotheses to be tested, not treasures to be guarded. The project emphasizes transparency, urging people to explain why they believe what they do. Are you a Superforecaster? Find out by joining the project at GJOpen.com.

2. Better evidence → Better access → Better health CADTH (@CADTH_ACMTS), a non-profit that provides evidence to Canada's healthcare decision makers, is accepting abstract proposals for its 2016 Symposium, Evidence for Everyone.

3. Coin flip study → Surprising results → Hot hand debate The hot hand is making a comeback. After a noteworthy smackdown by Tom Gilovich, some evidence suggests there is such a thing. Ben Cohen explains in The 'Hot Hand' May Actually Be Real - evidently it's got something to do with coin flips. Regardless of how this works out, everyone should read (or reread) Gilovich's fantastic book, How We Know What Isn't So.

4. Less junk science → Better evidence → Better world The American Council on Science and Health has a mission to "provide an evidence-based counterpoint to the wave of anti-science claims". @ACSHorg presents its views with refreshingly snappy writing, covering a wide variety of topics including public policy, vaccination, fracking, chemicals, and nutrition.

5. Difference of differences → Misunderstanding → Bad evidence Ben Goldacre (@bengoldacre) of Bad Science fame writes in The Guardian that the same statistical errors – namely, ignoring the difference in differences – are appearing throughout the most prestigious journals in neuroscience.

Tuesday, 29 September 2015

Data blindness, measuring policy impact, and informing healthcare with baseball analytics.

This week's 5 links on evidence-based decision making.

Hear me talk October 14 on communicating messages clearly with data. Part of the HealthEconomics.com "Effective HEOR Writing" webinar series: Register here.

1. Creative statistics → Valuable insights → Reinvented baseball business Exciting baseball geek news: Bill James and Billy Beane appeared together for the first time. Interviewed in the Wall Street Journal at a Netsuite conference on business model disruption, Beane said new opportunities include predicting/avoiding player injuries - so there's an interesting overlap with healthcare analytics. (Good example from Baseball Prospectus: "no one really has any idea whether letting [a pitcher] pitch so much after coming back from Tommy John surgery has any effect on his health going forward.")

2. Crowdsourcing → Machine learning → Micro, macro policy evidence Premise uses a clever combination of machine learning and street-level human intelligence; their economic data helps organizations measure the impact of policy decisions at a micro and macro level. @premisedata recently closed a $50M US funding round.

3. Data blindness → Unfocused analytics → Poor decisions Data blindness prevents us from seeing what the numbers are trying to tell us. In a Read/Write guest post, OnCorps CEO (@OnCorpsHQ) Bob Suh recommends focusing on the decisions that need to be made, rather than on big data and analytics technology. OnCorps offers an intriguing app called Sales Sabermetrics.

4. Purpose and focus → Overcome analytics barriers → Create business value David Meer of PWC's Strategy& (@strategyand) talks about why companies continue to struggle with big data [video].

5. Health analytics → Evidence in the cloud → Collaboration & learning Evidera announces Evalytica, a SaaS platform promising fast, transparent analysis of healthcare data. This cloud-based engine from @evideraglobal supports analyses of real-world evidence sources, including claims, EMR, and registry data.

Tuesday, 22 September 2015

Writing skills series, encyclopedia of slide layouts, and fantasy sports decision-making

This week's 5 links on evidence-based decision making.

1. Well-crafted writing → Evidence explained → Uptake of ideas On October 14, I'll be talking about Communicating Messages Clearly with Data. My presentation will include techniques for writing about data, and telling a simple story about complex science.

2. Use the Slide Chooser → Tell your story → Inspire action The Extreme Presentation method is a step-by-step approach for designing presentations of complex or controversial information. It's based on empirical research, and has been tested with big companies. The companion book is the Encyclopedia of Slide Layouts by Paul Radich and Andrew Abela.

3. Fear of losses → Baseball decisions → Daily fantasty sports results In baseball daily fantasy sports, as in life, we are more motivated to minimize losses than to maximize gains. Dr. Renee Miller explains how cognitive biases expedite decision-making and influence outcomes.

4. Rethink strategy → Lower blood pressure targets → Reduce death risk The U.S. National Institutes of Health released early findings from a big study of blood pressure management in people over 50. The Sprint trial seems to tell us that keeping systolic pressure below 120 can reduce cardiovascular disease and risk of death by as much as one third.

5. Large data sets → LASSO method → Valid predictions Today's large data sets create fantastic opportunities to make useful predictions - but traditional methods of variable selection are unwieldy and unreliable. Daniel Samarov writes on the Experfy blog about LASSO, a modern way to select variables for predictive models. This sparse regression, using Least Absolute Shrinkage and Selection Operator, is becoming a mainstay for analyzing data with lots of variables.

Tuesday, 08 September 2015

'What Works' toolkit, the insight-driven organization, and peer-review identity fraud.

This week's 5 links on evidence-based decision making.

1. Abundant evidence → Clever synthesis → Informed crime-prevention decisions The What Works Crime Toolkit beautifully synthesizes - on a single screen - the evidence on crime-prevention techniques. This project by the UK's @CollegeofPolice provides quick answers to what works (the car breathalyzer) and what doesn't (the infamous "Scared Straight" programs). Includes easy-to-use filters for evidence quality and type of crime. Just outstanding.

2. Insights → Strategic reuse → Data-driven decision making Tom Davenport explains why simply generating a bunch of insights is insufficient: "Perhaps the overarching challenge is that very few organizations think about insights as a process; they have been idiosyncratic and personal." A truly insight-driven organization must carefully frame, create, market, consume, and store insights for reuse. Via @DeloitteBA.

3. Sloppy science → Weak replication → Psychology myths Of 100 studies published in top-ranking journals in 2008, 75% of social psychology experiments and half of cognitive studies failed the replication test. @iansample delivers grim news in The Guardian: The psych research/publication process is seriously flawed. Thanks to @Rob_Briner.

4. Flawed policy → Ozone overreach → Burden on business Tony Cox writes in the Wall Street Journal that the U.S. EPA lacks causal evidence to support restrictions on ground-level ozone. The agency is connecting this pollutant to higher incidence of asthma, but Cox says new rules won't improve health outcomes, and will create substantial economic burden on business.

5. Opaque process → Peer-review fraud → Bad evidence More grim news for science publishing. Springer has retracted 64 papers from 10 journals after discovering the peer reviews were linked to fake email addresses. The Washington Post story explains that only nine months ago, BioMed Central - a Springer imprint - retracted 43 studies. @RetractionWatch says this wasn't even a thing before 2012.

Wednesday, 19 August 2015

Science of criminal sentencing, pharma formulary decisions, and real astrology?

This week's 5 links on evidence-based decision making.

1. Criminal patterns → Risk assessment → Science of sentencing The Marshall Project describes the new science of sentencing, where courts use statistically derived risk assessments to inform their decisions about which prisoners should be released on parole, and how bail should be set. (Thanks to Gregory Piatetsky, @kdnuggets.)

2. Clinical & cost effectiveness → Evidence base → Pharma formulary Express Scripts, a large U.S. pharmacy benefits manager, has released its 2016 formulary outlining which drugs will be covered, and which will not. @BioPharmaDive explains the decision process: An independent group of physicians reviews the evidence on clinical and cost effectiveness of each candidate. "Me-too" products aren't making the cut.

3. March birthday → Atrial fibrillation → Real astrology? The Journal of the American Medical Informatics Association has findings from a retrospective population study that systematically explored (with a phenome-wide method) the connection between birth month and disease risk for 1,688 conditions. Authors claim that for 55 diseases, "seasonally dependent early developmental mechanisms may play a role in increasing lifetime risk."

4. Data-driven → Fewer middle managers → Nimble decision processes Data-driven management processes need careful driving, says Ed Burns. Benefits include transparent and objective decisions, and more nimble ones when analytics can eliminate middle managers. However, some efforts have backfired. More in this podcast by @EdBurnsTT, What are your tips for putting in place data-driven management strategies?

5. Aggregated economic data → Positive trends → Data-driven optimism Economist Max Roser is an optimist. Jeff Rothfeder writes in @stratandbiz about Roser's analysis of disparate data covering "everything from African development to violent death rates", and his conclusions that evidence unambiguously shows a world evolving for the better.

Tuesday, 11 August 2015

Book of Bad Arguments, evidence-based social services, and a fresh hell of confusing numbers.

This week's 5 links on evidence-based decision making.

1. Bad logic → Bad arguments → Bad decisions The Book of Bad Arguments is a simple explanation of common logical flaws / barriers to successful, evidence-based decisions. This beautifully illustrated work by Ali Almossawi (@AliAlmossawi) should be on everyone's bookshelf. Now available in several languages.

2. Home visits for children → Lifelong benefits → Better society Susan Urahn, EVP of Pew Charitable Trusts, says evidence-based policymaking can guide better use of taxpayer dollars. Her @GOVERNING piece explains how measuring effectiveness will transform delivery of social services. Since research confirms that early childhood affects lifelong behavior and health, states are investing in family support and coaching programs. [Note: Want to participate in an online collaboration on the evidence base for home visits? Let us know: tracy@uglyresearch.com.]

3. Words+Numbers → Statistics → Fresh Hell Hilda Bastian (@hildabast) reminds us to keep our sense of humor: "The comedic possibilities of clinical epidemiology are known to be limitless." Her wonderful blog is Statistically Funny. "OMG that spider is HUGE!" "Where? What - that little thing?" Words mean different things to different people, but numbers do also. Her latest post explains the evidence on why we struggle to get our heads around - and describe - big numbers and relative risks: We do okay with "18 out of 20" but not so much with "18,000 out of 20,000".

4. Data → Analytics → Decision skills You've got the data and the analytics. Now what? One finance executive found a way to improve his team's data literacy with data-driven decision making bootcamps.

5. Climate change evidence → Energy crystal ball → Economic predictions Kudos to @McKinsey for taking a hard look at the energy predictions they made 8 years ago. Their 2007 energy crystal ball was spot-on in several areas, but missed the mark in a few.

Tuesday, 28 July 2015

10 Years After Ioannidis, speedy decision habits, and the peril of whether or not.

1. Much has happened in the 10 years since Why Most Published Research Findings Are False, the much-discussed PLOS essay by John P. A. Ioannidis offering evidence that "false findings may be the majority or even the vast majority of published research claims...." Why are so many findings never replicated? Ioannidis listed study power and bias, the number of studies, and the ratio of true to no relationships among those probed in that scientific field. Also, "the convenient, yet ill-founded strategy of claiming conclusive research findings solely on... formal statistical significance, typically for a p-value less than 0.05."
Now numerous initiatives address the false-findings problem with innovative publishing models, prohibition of p-values, or study design standards. Ioannidis followed up with 2014's How to Make More Published Research True, noting improvements in credibility and efficiency in specific fields via "large-scale collaborative research; replication culture; registration; sharing; reproducibility practices; better statistical methods;... reporting and dissemination of research, and training of the scientific workforce."

2. Speedy decision habits -> Fastest in market -> Winning. Dave Girouard, CEO of personal finance startup Upstart & ex-Google apps head, believes speedy decision-making is essential to competing: For product dev, and other organizational functions. He explains how people can develop speed as a healthy habit. Relatively little is "written about how to develop the institutional and employee muscle necessary to make speed a serious competitive advantage." Key tip: Deciding *when* a decision will be made from the start is a profound, powerful change that speeds everything up.

3. Busy, a new book by Tony Crabbe (@tonycrabbe), considers why people feel overwhelmed and dissatisfied - and suggests steps for improving their personal & work lives. Psychological and business research are translated into practical tools and skills. The book covers a range of perspectives; one worth noting is "The Perils of Whether or Not" (page 31): Crabbe cites classic decision research demonstrating the benefits of choosing from multiple options, vs. continuously (and busily) grinding through one alternative at a time. BUSY: How to Thrive in a World of Too Much, Grand Central Publishing, $28.

4. Better lucky than smart? Eric McNulty reminds us of a costly, and all-too-common, decision making flaw: Outcome bias, when we evaluate the quality of a decision based on its final result. His strategy+business article explains we should be objectively assessing whether an outcome was achieved by chance or through a sound process - but it's easy to fall into the trap of positively judging only those efforts with happy endings (@stratandbiz).

5. Fish vs. Frog: It's about values, not just data. Great reminder from Denis Cuff @DenisCuff of @insidebayarea that the data won't always tell you where to place value. One SF Bay Area environmental effort to save a fish might be endangering a frog species.

Monday, 20 July 2015

The Cardinal Sin of data science, Evidence for Action $, and your biases in 5 easy steps.

My 5 weekly links on evidence-based decision making.

1. Confusing correlation with causation is not the Cardinal Sin of data science, say Gregory Piatetsky (@kdnuggets) and Anmol Rajpurohit (@hey_anmol): It's overfitting. Oftentimes, researchers "test numerous hypotheses without proper statistical control, until they happen to find something interesting and report it. Not surprisingly, next time the effect, which was (at least partly) due to chance, will be much smaller or absent." This explains why it's often difficult to replicate prior findings. "Overfitting is not the same as another major data science mistake - confusing correlation and causation. The difference is that overfitting finds something where there is nothing. In case of correlation and causation, researchers can find a genuine novel correlation and only discover a cause much later."

2. July 22, RWJF (@RWJF) will host a webinar explaining its Evidence for Action program, granting $2.2M USD annually for Investigator-Initiated Research to Build a Culture of Health. "The program aims to provide individuals, organizations, communities, policymakers, and researchers with the empirical evidence needed to address the key determinants of health encompassed in the Culture of Health Action Framework. In addition, Evidence for Action will also support efforts to assess outcomes and set priorities for action. It will do this by encouraging and supporting creative, rigorous research on the impact of innovative programs, policies and partnerships on health and well-being, and on novel approaches to measuring health determinants and outcomes."

3. Your biases, in 5 tidy categories. We've heard it before, but this bears repeating: Our biases (confirmation, sunk cost, etc.) prevent us from making more equitable, efficient, and successful decisions. In strategy+business, Heidi Grant Halvorson and David Rock (@stratandbiz) present the SEEDS™ model, grouping the "150 or so known common biases into five categories, based on their underlying cognitive nature: similarity, expedience, experience, distance, and safety". Unfortunately, most established remedies and training don't overcome bias. But organizations/groups can apply correctional strategies more reliably than we can as individuals.

4. PricewaterhouseCoopers (@PwC_LLP) explains how four key stakeholders are pressuring pharma in 21st Century Pharmaceutical Collaboration: The Value Convergence. These four: government agencies, emboldened insurers, patient advocates, and new entrants bringing new evidence, are substantially shifting how medicine is developed and delivered. "Consumers are ready to abandon traditional modes of care for new ones, suggesting billions in healthcare revenue are up for grabs now. New entrants are bringing biosensor technology and digital tools to healthcare to help biopharmaceutical companies better understand the lives of patients, and how they change in response to drug intervention." These include home diagnostic kits to algorithms that check symptoms and recommend treatments."

5. Remember 'Emotional Intelligence'? A 20-year retrospective study, funded by the Robert Wood Johnson Foundation (@RWJF) and appearing in July's American Journal of Public Health, suggests that "kindergarten students who are more inclined to exhibit “social competence” traits —such sharing, cooperating, or helping other kids— may be more likely to attain higher education and well-paying jobs. In contrast, students who exhibit weaker social competency skills may be more likely to drop out of high school, abuse drugs and alcohol, and need government assistance."

Tuesday, 14 July 2015

Data-driven organizations, machine learning for C-Suite, and healthcare success story.

1. Great stuff on data-driven decision making in a new O'Reilly book by Carl Anderson (@LeapingLlamas), Creating the Data-Driven Organization. Very impressive overview of the many things that need to happen, and best practices for making them happen. Runs the gamut from getting & analyzing the data, to creating the right culture, to the psychology of decision-making. Ugly Research is delighted to be referenced (pages 187-188 and Figure 9-7).

2. Healthcare success story. "Data-driven decision making has improved patient outcomes in Intermountain's cardiovascular medicine, endocrinology, surgery, obstetrics and care processes — while saving millions of dollars in procurement and in its the supply chain."

3. 1) description, 2) prediction, 3) prescription. What the C-Suite needs to understand about applied machine learning. McKinsey's executive guide to machine learning 1.0, 2.0, and 3.0.

4. Place = Opportunity. Where kids grow up has a big impact on what they earn as adults; new evidence on patterns of upward mobility. Recap by @UrbanInstitute's Margery Austin Turner (@maturner).

5. Open innovation improves the odds of biotech product survival. Analysis by Deloitte's Ralph Marcello shows the value of working together, sharing R&D data.

Tuesday, 07 July 2015

Randomistas fight poverty, nurses fight child abuse, and decision support systems struggle.

1. Jason Zweig tells the story of randomistas, who use randomized, controlled trials to pinpoint what helps people become self-sufficient around the globe. The Anti-Poverty Experiment describes several successful, data-driven programs, ranging from financial counseling to grants of livestock.

2. Can an early childhood program prevent child abuse and neglect? Yes, says the Nurse-Family Partnership, which introduces vulnerable first-time parents to maternal and child-health nurses. NFP (@NFP_nursefamily) refines its methodology with randomized, controlled trial evidence satisfying the Coalition for Evidence-Based Policy’s “Top Tier”, and producing a positive return on investment.

3. Do recommendations from decision support technology improve the appropriateness of a physician's imaging orders? Not necessarily. JAMA provides evidence of the limitations of algorithmic medicine. An observational study shows it's difficult to attribute improvements to clinical decision support.

4. Is the "data-driven decision" a fallacy? Yes, says Stefan Conrady, arguing that the good alliteration is a bad motto. He explains on the BayesiaLab blog that the concept doesn't adequately encompass casual models, necessary for anticipating "the consequences of actions we have not yet taken". Good point.

5. A BMJ analysis says the knowledge system underpinning healthcare is not fit for purpose and must change. Ian Roberts says poor-quality, published studies are damaging systematic reviews, and that the Cochrane system needs improvement. Richard Lehman and others will soon respond on BMJ.