Evidence Soup
How to find, use, and explain evidence.

Tuesday, 24 November 2015

Masters of self-deception, rapid systematic reviews, and Gauss v. Legendre.

This week's 5 links on evidence-based decision making.

1. Human fallibility → Debiasing techniques → Better science
Don't miss Regina Nuzzo's fantastic analysis in Nature: How scientists trick themselves, and how they can stop. @ReginaNuzzo explains why people are masters of self-deception, and how cognitive biases interfere with rigorous findings. Making things worse are a flawed science publishing process and "performance enhancing" statistical tools. Nuzzo describes promising ways to overcome these challenges, including blind data analysis.

2. Slow systematic reviews → New evidence methods → Controversy
Systematic reviews are important for evidence-based medicine, but some say they're unreliable and slow. Two groups attempting to improve this - not without controversy - are Trip (@TripDatabase) and Rapid Reviews.

3. Campus competitions → Real-world analytics → Attracting talent
Tech firms are finding ways to attract students to the analytics field. David Weldon writes in Information Management about the Adobe Analytics Challenge, where thousands of US university students compete using data from companies such as Condé Nast and Comcast to solve real-world business problems.

4. Discover regression → Solve important problem → Rock the world
Great read on how Gauss discovered statistical regression, but thinking his solution was trivial, didn't share. Legendre published the method later, sparking one of the bigger disputes in the history of science. The Discovery of Statistical Regression - Gauss v. Legendre on Priceonomics.

5. Technical insights → Presentation skill → Advance your ideas
Explaining insights to your audience is as crucial as getting the technical details right. Present! is a new book with speaking tips for technology types unfamiliar with the spotlight. By Poornima Vijayashanker (@poornima) and Karen Catlin.

Tuesday, 17 November 2015

ROI from evidence-based government, milking data for cows, and flu shot benefits diminishing.

This week's 5 links on evidence-based decision making.

1. Evidence standards → Knowing what works → Pay for success
Susan Urahn says we've reached a Tipping Point on Evidence-Based Policymaking. She explains in @Governing that 24 US governments have directed $152M to programs with an estimated $521M ROI: "an innovative and rigorous approach to policymaking: Create an inventory of currently funded programs; review which ones work based on research; use a customized benefit-cost model to compare programs based on their return on investment; and use the results to inform budget and policy decisions."

2. Sensors → Analytics → Farming profits
Precision dairy farming uses RFID tags, sensors, and analytics to track the health of cows. Brian T. Horowitz (@bthorowitz) writes on TechCrunch about how farmers are milking big data for insight. Literally. Thanks to @ShellySwanback.

3. Public acceptance → Annual flu shots → Weaker response?
Yikes. Now that flu shot programs are gaining acceptance, there's preliminary evidence suggesting that repeated annual shots can gradually reduce their effectiveness under some circumstances. Scientists at the Marshfield Clinic Research Foundation recently reported that "children who had been vaccinated annually over a number of years were more likely to contract the flu than kids who were only vaccinated in the season in which they were studied." Helen Branswell explains on STAT.

4. PCSK9 → Cholesterol control → Premium increases
Ezekiel J. Emanuel says in a New York Times Op-Ed I Am Paying for Your Expensive Medicine. PCSK9 inihibitors newly approved by US FDA can effectively lower bad cholesterol, though data aren't definitive whether this actually reduces heart attacks, strokes, and deaths from heart disease. This new drug category comes at a high cost. Based on projected usage levels, soem analysts predict insurance premiums could rise >$100 for everyone in that plan.

5. Opportunistic experiments → Efficient evidence → Informed family policy
New guidance details how researchers and program administrators can recognize opportunities for experiments and carry them out. This allows people to discover effects of planned initiatives, as opposed to analyzing interventions being developed specifically for research studies. Advancing Evidence-Based Decision Making: A Toolkit on Recognizing and Conducting Opportunistic Experiments in the Family Self-Sufficiency and Stability Policy Area.

Tuesday, 10 November 2015

Working with quantitative people, evidence-based management, and NFL ref bias.

This week's 5 links on evidence-based decision making.

1. Understand quantitative people → See what's possible → Succeed with analytics Tom Davenport outlines an excellent list of 5 Essential Principles for Understanding Analytics. He explains in the Harvard Business Review that an essential ingredient for effective data use is managers’ understanding of what is possible. To counter that, it’s really important that they establish a close working relationship with quantitative people.

2. Systematic review → Leverage research → Reduce waste This sounds bad: One study found that published reports of trials cited fewer than 25% of previous similar trials. @PaulGlasziou and @iainchalmersTTi explain on @bmj_latest how systematic reviews can reduce waste in research. Thanks to @CebmOxford.

3. Organizational context → Fit for decision maker → Evidence-based management A British Journal of Management article explores the role of ‘fit’ between the decision-maker and the organizational context in enabling an evidence-based process and develops insights for EBM theory and practice. Evidence-based Management in Practice: Opening up the Decision Process, Decision-maker and Context by April Wright et al. Thanks to @Rob_Briner.

4. Historical data → Statistical model → Prescriptive analytics Prescriptive analytics finally going mainstream for inventories, equipment status, trades. Jose Morey explains on the Experfy blog that the key advance has been the use of statistical models with historical data.

5. Sports data → Study of bias → NFL evidence Are NFL officials biased with their ball placement? Joey Faulkner at Gutterstats got his hands on a spreadsheet containing every NFL play run 2000-2014 (500,000 in all). Thanks to @TreyCausey.

Bonus! In The Scientific Reason Why Bullets Are Bad for Presentations, Leslie Belknap recaps a 2014 study concluding that "Subjects who were exposed to a graphic representation of the strategy paid significantly more attention to, agreed more with, and better recalled the strategy than did subjects who saw a (textually identical) bulleted list version."

Tuesday, 03 November 2015

Watson isn't thinking, business skills for data scientists, and zombie clickbait.

This week's 5 links on evidence-based decision making.

1. Evidence scoring → Cognitive computing → Thinking?
Fantastic article comparing Sherlock Holmes to Dr. Watson - and smart analysis to cognitive computing. This must-read by Paul Levy asks if scoring evidence and ranking hypotheses are the same as thinking.

2. Data science understanding → Business relevance → Career success
In HBR, Michael Li describes three crucial abilities for data scientists: 1) Articulate the business value of their work (defining success with metrics such as attrition); 2) Give the right level of technical detail (effectively telling the story behind the data); 3) Get visualizations right (tell a clean story with diagrams).

3. Long clinical trials → Patient expectations → Big placebo effect
The placebo effect is wreaking havoc in painkiller trials. Nature News explains that "responses to [placebo] treatments have become stronger over time, making it harder to prove a drug’s advantage." The trend is US-specific, possibly because big, expensive trials "may be enhancing participants’ expectations of their effectiveness".

4. Find patterns → Design feature set → Automate predictions
Ahem. MIT researchers aim to take the human element out of big-data analysis, with a system that searches for patterns *and* designs the feature set. In testing, it outperformed 615 of 906 human teams. Thanks to @kdnuggets.

5. Recurrent neural nets → Autogenerated clickbait → Unemployed Buzzfeed writers?
A clickbait website has been built entirely by recurrent neural nets. Click-o-Tron has the latest and greatest stories on the web, as hallucinated by an algorithm. Thanks to @leapingllamas.

Bonus! Sitting studies debunked? Corey Doctorow explains it's not the sitting that will kill you - it's the lack of exercise.

Wednesday, 21 October 2015

5 practical ways to build an evidence-based social program.

Notes from my recent presentation on how social programs can become evidence-based - in our lifetime. Get the slides: How Can Social Programs Become Evidence-Based? 5 Practical Steps. #data4good

Highlights: Recent developments in evidence-based decision making in the nonprofit/social sector. Practical ways to discover and exchange evidence-based insights. References, resources, and links to organizations with innovative programs.

Social Innovation Fund Evidence Evaluation

Data-Driven is No Longer Optional

Whether you're the funder or the funded, data-driven management is now mandatory. Evaluations and decisions must incorporate rigorous methods, and evidence review is becoming standardized. Many current concepts are modeled after evidence-based medicine, where research-based findings are slotted into categories depending on their quality and generalizibility.

SIF: Simple or Bewildering? The Social Innovation Fund (US) recognizes three levels of evidence: preliminary, moderate, and strong. Efforts are being made to standardize evaluation, but they're recognizing 72 evaluation designs (!).

What is an evidence-based decision? There's a long answer and a short answer. The short answer is it's a decision reflecting current, best evidence: Internal and external sources for findings; high-quality methods of data collection and analysis; and a feedback loop to bring in new evidence.

On one end of the spectrum, evidence-based decisions bring needed rigor to processes and programs with questionable outcomes. At the other end, we risk creating a cookie-cutter, rubber-stamp approach that sustains bureaucracy and sacrifices innovation.

What's a 'good' decision? A 'good' decision should follow a 'good' process: Transparent and repeatable. This doesn't necessarily guarantee a good result - one must judge the quality of a decision process separately from its outcomes. That said, when a decision process continues to deliver suboptimal results, adjustments are needed.

Where does the evidence come from? Many organizations have relied on gathering their own evidence, but are now overwhelmed by requirements to support decision processes with data. Marketplaces for evidence are emerging, as the Social Innovation Research Center's Patrick Lester recently explained. There's a supply and a demand for rigorous evidence on the performance of social programs. PepperSlice is a marketplace where nonprofits can share, buy, and sell evidence-based insights using a standard format.

Avoid the GPOC (Giant PDF of Crap). Standardized evidence is already happening, but standardized dissemination of findings - communcating results - is still mostly a free-for-all. Traditional reports, articles, and papers, combined with PowerPoints and other free-form presentations, make it difficult to exchange evidence systematically and quickly.

Practical ways to get there. So how can a nonprofit or publicly financed social program compete?

  1. Focus on what deciders need. Before launching efforts to gather evidence, examine how decisions are being made. What evidence do they want? Social Impact Bonds, a/k/a Pay for Success Bonds, are a perfect example because they specify desired outcomes and explicit success measures.
  2. Use insider vocabulary. Recognize and follow the terminology for desired categories of evidence. Be explicit about how data were collected (randomized trial, quasi-experimental design, etc.) and how analyzed (statistics, complex modeling, ...).
  3. Live better through OPE. Whenever possible, use Other People's Evidence. Get research findings from peer organizations, academia, NGOs, and government agencies. Translate their evidence to your program and avoid rolling your own.
  4. Manage and exchange. Once valuable insights are discovered, be sure to manage and reuse them. Trade/exchange them with other organizations.
  5. Share systematically. Follow a method for exchanging insights, reflecting key evidence categories. Use a common vocabulary and a common format.

 Resources and References

Don’t end the Social Innovation Fund (yet). Angela Rachidi, American Enterprise Institute (@AngelaRachidi).

Why Evidence-Based Policymaking Is Just the Beginning. Susan Urahn, Pew Charitable Trusts.

Alliance for Useful Evidence (UK). How do charities use research evidence? Seeking case studies (@A4UEvidence). http://www.surveygizmo.com/s3/2226076/bab129060657

Social Innovation Fund: Early Results Are Promising. Patrick Lester, Social Innovation Research Center, 30-June-2015. "One of its primary missions is to build evidence of what works in three areas: economic opportunity, health, and youth development." Also, SIF "could nurture a supply/demand evidence marketplace when grantees need to demonstrate success" (page 27).

What Works Cities supports US cities that are using evidence to improve results for their residents (@WhatWorksCities).

Urban Institute Pay for Succes Initiative (@UrbanInstitute). "Once strategic planning is complete, jurisdictions should follow a five step process that uses cost-benefit analysis to price the transaction and a randomized control trial to evaluate impact." Ultimately, evidence will support standardized pricing and defined program models.

Results 4 America works to drive resources to results-driven solutions that improve lives of young people & their families (@Results4America).

How to Evaluate Evidence: Evaluation Guidance for Social Innovation Fund.

Evidence Exchange within the US federal network. Some formats are still traditional papers, free-form, big pdf's.

Social Innovation Fund evidence categories: Preliminary, moderate, strong. "This framework is very similar to those used by other federal evidence-based programs such as the Investing in Innovation (i3) program at the Department of Education. Preliminary evidence means the model has evidence based on a reasonable hypothesis and supported by credible research findings. Examples of research that meet the standards include: 1) outcome studies that track participants through a program and measure participants’ responses at the end of the program.... Moderate evidence means... designs of which can support causal conclusions (i.e., studies with high internal validity)... or studies that only support moderate causal conclusions but have broad general applicability.... Strong evidence means... designs of which can support causal conclusions (i.e., studies with high internal validity)" and generalizability (i.e., studies with high external validity).

Tuesday, 20 October 2015

Evidence handbook for nonprofits, telling a value story, and Twitter makes you better.

This week's 5 links on evidence-based decision making.

1. Useful evidence → Nonprofit impact → Social good
For their upcoming handbook, the UK's Alliance for Useful Evidence (@A4UEvidence) is seeking "case studies of when, why, and how charities have used research evidence and what the impact was for them." Share your stories here.

2. Data story → Value story → Engaged audience
On Evidence Soup, Tracy Altman explains the importance of telling a value story, not a data story - and shares five steps to communicating a powerful message with data.

3. Sports analytics → Baseball preparedness → #Winning
Excellent performance Thursday night by baseball's big data-pitcher: Zach Greinke. (But there's also this: Cubs vs. Mets!)

4. Diverse network → More exposure → New ideas
"New research suggests that employees with a diverse Twitter network — one that exposes them to people and ideas they don’t already know — tend to generate better ideas." Parise et al. describe their analysis of social networks in the MIT Sloan Management magazine. (Thanks to @mluebbecke, who shared this with a reminder that 'correlation is not causation'. Amen.)

5. War on drugs → Less tax revenue → Cost to society
The Democratic debate was a reminder that the U.S. War on Drugs was a very unfortunate waste - and that many prison sentences for nonviolent drug crimes impose unacceptable costs on the convict and society. Consider this evidence from the Cato Institute (@CatoInstitute).

Wednesday, 14 October 2015

5 ways to tell a value story with data.


Source: Wikipedia. Anscombe's quartet.

[Notes from my recent talk on writing about health data.] Always remember it's not a data story you're telling, it's a value story. To make that happen, you must demonstrate clarity and establish credibility.

First put together this checklist and review it several times: What is the message? Why is this valuable, or at least interesting, to your audience? Where did the data come from? Why are the data believable?

Follow these 5 tips to get to clarity and credibility:

1. Bold opening statement or question. Begin with a crisp, clear message. If a reader's time is cut short, what key point should they remember? When opening with a question, be sure to answer it explicitly in closing summaries/conclusions (sounds simple, but oftentimes it's missed, draining impact from the content).

2. Inverted pyramid. Follow your opening statement with a summary of the key points: What, who, when, where, why. Use the journalism approach of giving away the ending, and then filling in background. Apply the inverted pyramid concept to both writing and data; so for example, present important charts or tables first, and raw data or other supporting data later.

3. Data visualization. Give them some 'Ooh, shiny', but not too much (I'm growing weary of the hero worship of artistic data viz creators). Visuals can tell a story that writing cannot: Reference the classic Anscombe's Quartet graphic above. Anscombe illustrated beautifully how four distinct data sets can have the same mean x, mean y, sample variance, etc. - and that only through visuals do we see their notable differences. A simple presentation of the statistics would not tell the whole story.

4. Explain the source. Writing must tell the rest of the value story: Where did the data come from? Why were they analyzed this way? Why is this a valid and useful finding? After providing clarity, now you're establishing credibility.

5. Engage the skeptics. Essential to establishing credibility. Identify potential challenges and tough questions expected from the audience. When possible, discuss the limitations and acknowledge the gaps in your findings. What questions remain? What further research is needed? By addressing these directly, you can spark a conversation with the audience.

Examples & Sources

Writing about data:
Excellent journalist - Jason Zweig
Health economics analytics - Context Matters
Health consultants - Evidera
Business and trade groups
American Medical Writers Association
ISPOR (International Society For Pharmacoeconomics and Outcomes Research)

Presenting data / Data visualization:
Stephen Few
Flowing Data - Nathan Yau
Business intelligence tech vendors, such as Tableau; Great article - Why the beautiful, time-tested science of data visualization is so powerful
Edward Tufte's book - Beautiful Evidence

Tuesday, 13 October 2015

Decision science, NFL prediction, and recycling numbers don't add up.

This week's 5 links on evidence-based decision making.

Hear me talk October 14 on communicating messages clearly with data. Part of the HEOR Writing webinar series: Register here.

1. Data science → Decision science → Institutionalize data-driven decisions
Deepinder Dhingra at @MuSigmaInc explains why data science misses half the equation, and that companies instead need decision science to achieve a balanced creation, translation, and consumption of insights. Requisite decision science skills include "quantitative and intellectual horsepower; the right curiosity quotient; ability to think from first principles; and business synthesis."

2. Statistical model → Machine learning → Good prediction
Microsoft is quite good at predicting American Idol winners - and football scores. Tim Stenovec writes about the Bing Predicts project's impressive record of correctly forecasting World Cup, NFL, reality TV, and election outcomes. The @Bing team begins with a traditional statistical model and supplements it with query data, text analytics, and machine learning.

3. Environmental concern → Good feelings → Bad recycling ROI
From a data-driven perspective, it's difficult to justify the high costs of US recycling programs. John Tierney explains in the New York Times that people's good motives and concerns about environmental damage have driven us to the point of recovering every slip of paper, half-eaten pizza, water bottle, and aluminum can - when the majority of value is derived from those cans and other metals.

4. Prescriptive analytics → Prescribe actions → Grow the business
Business intelligence provides tools for describing and visualizing what's happening in the company right now, but BI's value for identifying opportunities is often questioned. More sophisticated predictive analytics can forecast the future. But Nick Swanson of River Logic says the path forward will be through prescriptive analytics: Using methods such as stochastic optimization, analysts can prescribe specific actions for decision makers.

5. Graph data → Data lineage → Confidence & trust
Understanding the provenance of a data set is essential, but often tricky: Who collected it, and whose hands has it passed through? Jean Villedieu of @Linkurious explains how a graph database - rather than a traditional data store - can facilitate the tracking of data lineage.

Tuesday, 06 October 2015

Superforecasting, hot hand redux, and junk science.

This week's 5 links on evidence-based decision making.

Hear me talk on communicating messages clearly with data. Webinar October 14: Register here.

1. Good judgment → Accurate forecasts → Better decisions Jason Zweig (@jasonzweigwsj) believes Superforecasting: The Art and Science of Prediction is the "most important book on decision-making since Daniel Kahneman's Thinking Fast and Slow." Kahneman is equally enthusiastic, saying "This book shows that under the right conditions regular people are capable of improving their judgment enough to beat the professionals at their own game." The author, Philip Tetlock, leads the Good Judgment Project, where amateurs and experts compete to make forecasts - and the amateurs routinely win. Tetlock notes that particularly good forecasters regard their views as hypotheses to be tested, not treasures to be guarded. The project emphasizes transparency, urging people to explain why they believe what they do. Are you a Superforecaster? Find out by joining the project at GJOpen.com.

2. Better evidence → Better access → Better health CADTH (@CADTH_ACMTS), a non-profit that provides evidence to Canada's healthcare decision makers, is accepting abstract proposals for its 2016 Symposium, Evidence for Everyone.

3. Coin flip study → Surprising results → Hot hand debate The hot hand is making a comeback. After a noteworthy smackdown by Tom Gilovich, some evidence suggests there is such a thing. Ben Cohen explains in The 'Hot Hand' May Actually Be Real - evidently it's got something to do with coin flips. Regardless of how this works out, everyone should read (or reread) Gilovich's fantastic book, How We Know What Isn't So.

4. Less junk science → Better evidence → Better world The American Council on Science and Health has a mission to "provide an evidence-based counterpoint to the wave of anti-science claims". @ACSHorg presents its views with refreshingly snappy writing, covering a wide variety of topics including public policy, vaccination, fracking, chemicals, and nutrition.

5. Difference of differences → Misunderstanding → Bad evidence Ben Goldacre (@bengoldacre) of Bad Science fame writes in The Guardian that the same statistical errors – namely, ignoring the difference in differences – are appearing throughout the most prestigious journals in neuroscience.

Tuesday, 29 September 2015

Data blindness, measuring policy impact, and informing healthcare with baseball analytics.

This week's 5 links on evidence-based decision making.

Hear me talk October 14 on communicating messages clearly with data. Part of the HealthEconomics.com "Effective HEOR Writing" webinar series: Register here.

1. Creative statistics → Valuable insights → Reinvented baseball business Exciting baseball geek news: Bill James and Billy Beane appeared together for the first time. Interviewed in the Wall Street Journal at a Netsuite conference on business model disruption, Beane said new opportunities include predicting/avoiding player injuries - so there's an interesting overlap with healthcare analytics. (Good example from Baseball Prospectus: "no one really has any idea whether letting [a pitcher] pitch so much after coming back from Tommy John surgery has any effect on his health going forward.")

2. Crowdsourcing → Machine learning → Micro, macro policy evidence Premise uses a clever combination of machine learning and street-level human intelligence; their economic data helps organizations measure the impact of policy decisions at a micro and macro level. @premisedata recently closed a $50M US funding round.

3. Data blindness → Unfocused analytics → Poor decisions Data blindness prevents us from seeing what the numbers are trying to tell us. In a Read/Write guest post, OnCorps CEO (@OnCorpsHQ) Bob Suh recommends focusing on the decisions that need to be made, rather than on big data and analytics technology. OnCorps offers an intriguing app called Sales Sabermetrics.

4. Purpose and focus → Overcome analytics barriers → Create business value David Meer of PWC's Strategy& (@strategyand) talks about why companies continue to struggle with big data [video].

5. Health analytics → Evidence in the cloud → Collaboration & learning Evidera announces Evalytica, a SaaS platform promising fast, transparent analysis of healthcare data. This cloud-based engine from @evideraglobal supports analyses of real-world evidence sources, including claims, EMR, and registry data.