Evidence Soup
How to find, use, and explain evidence.

Tuesday, 17 May 2016

Magical thinking about ev-gen, your TA is a bot, and Foursquare predicts stuff really well.

Dreaming about Ev-Gen

1. Magical thinking about ev-gen.
Rachel E. Sherman, M.D., M.P.H., and Robert M. Califf, M.D. of the US FDA have described what is needed to develop an evidence generation system - and must be playing a really long game. "The result? Researchers will be able to distill the data into actionable evidence that can ultimately guide clinical, regulatory, and personal decision-making about health and health care." Recent posts are Part I: Laying the Foundation and Part II: Building Out a National System. Sherman and Califf say "There must be a common approach to how data is presented, reported and analyzed and strict methods for ensuring patient privacy and data security. Rules of engagement must be transparent and developed through a process that builds consensus across the relevant ecosystem and its stakeholders." Examples of projects reflecting these concepts include: Sentinel Initiative (querying claims data to identify safety issues), PCORNet (leveraging EHR data in support of pragmatic clinical research), and NDES (the National Device Evaluation System).

2. It pays to play the long game with data.
Michael Carney shares great examples in So you want to build a data business? Play the long game. These include "Foursquare demonstrating, once again, that it’s capable of predicting public company earnings with an incredible degree of accuracy based on real world foot traffic data.... On April 12, two weeks in advance of the beleaguered restaurant chain’s quarterly earnings report, Foursquare CEO Jeff Glueck published a detailed blog post outlining a decline in foot traffic to Chipotle’s stores and predicting Q1 sales would be 'Down Nearly 30%.' Yesterday, the burrito brand reported a 29.7% decline in quarter over quarter earnings.... Kudos to the company for persisting in the face of public scrutiny and realizing the true potential of its location-based behavioral graph."

3. Meet Jill Watson, AI TA.
Turns out, college students often submit 10,000 questions to their teaching assistants. Per class, per semester. So a Georgia Tech prof experimented with using IBM's Watson Analytics AI engine to pretend to be a live TA - and pulled it off. Cool stories from The Verge and Wall Street Journal.

4. Burst of unsettling healthcare news.
- So now that we know more about the cost of our healthcare, evidence suggests price transparency doesn't seem to cut our outpatient spending. Healthcare reform is hard.

- Recent findings indicate patient-centered medical homes aren't cutting Medicare costs. Buzzkill via THCB.

- Ever been told to have surgery where they do the most procedures? Some data show high-volume surgeries aren't so closely linked to better patient outcomes. Ouch.

Wednesday, 11 May 2016

How Integrative Propositional Analysis shapes evidence into a graph.

  IPA beers

I admire any effort to create a simple presentation of complex evidence. Having developed some models of my own, I know I’m on the right track when someone’s initial response is “That’s too simplistic; it’s much more complicated.” I believe you’ll really struggle if you don’t begin with a top-down perspective.

Now we can choose from useful frameworks for synthesizing and rating the quality and relevance of evidence: GRADE for medical evidence, and the U.S. Dept. of Education's evidence guidelines are just two examples. Integrative Propositional Analysis (IPA) is a method of integrating and analyzing the propositions (theories) stated in a study, strategic plan, or other document.

Bernadette Wright and Steven Wallis write in Sage Open that IPA structures relationships and quantitatively measures the inter-connectedness among concepts found within theories. I think this is a promising idea. IPA is briefly introduced in Three Ways of Getting to Policy-Based Evidence: Why researchers and practitioners are shifting away from expensive new studies toward the effective synthesis of existing research (Stanford Social Innovation Review). The ‘three ways’ are Randomista (requiring randomized experiments to generate evidence), Explainista (requiring strong data with synthesized explanation), and Mapista (preferring a  holistic knowledge map of a policy, program, or issue).

Graphista? IPA falls under Mapista, but we might instead say Graphista, since it constructs a graph and applies straightforward analytics. These six steps are involved:

  1. Find the logical statements/propositions in a theory (found in a publication).
  2. Diagram the propositions (a box for each concept/ term, an arrow for each causal link).
  3. Combine those smaller diagrams where they overlap to create a larger diagram.
  4. Count the number of concepts with two or more causes (“concatenated” concepts).
  5. Count the total number of concepts in the theory (“Complexity”).
  6. Divide concatenated concepts by total concepts to assess “Systemicity.”

Quantifying complexity and systemicity. Wright and Wallis explain that “The systemicity score computed in the final step is a key measure of causal inter-relatedness in IPA. The greater the proportion of concepts in a theory that are concatenated, the more the theory’s concepts are causally interrelated (Wallis, 2013). In previous studies across diverse fields in the physical and social sciences, paradigm-changing scientific theories have shown greater systemicity (inter-connectedness among concepts) than earlier, less successful scientific theories (Wallis 2010a).” Returning to the graph comparison, this brings to mind graph connectivity.

Integrative Propositional Analysis map

Talking this week with Bernadette Wright, she added that “Policy research is an applied science. It has the same problem of all applied sciences. It’s not enough to make new discoveries. We also need to apply existing knowledge to real-world problems. Integrative Propositional Analysis and related mapping techniques provide a rigorous way to connect existing studies into a larger pattern. This lets managers quickly see what’s known on a topic. So they can use that information to make a bigger difference for the people they serve.”

I have some questions, such as: Do different people construct the same IPA maps for the same theories? The authors also raise the question of inter-rater reliability in their discussion (page 7).

Cool idea. Wright and Wallis have developed a gamified version of IPA, where people co-create knowledge maps for experiental learning. For more on that and some other insights, go to  AEA365 - A Tip-a-Day for Evaluators.

Tuesday, 03 May 2016

Bitcoin for learning, helping youth with evidence, and everyday health evidence.

College diploma

1. Bitcoin tech records people's learning.
Ten years from now, what if you could evaluate a job candidate by reviewing their learning ledger, a blockchain-administered record of their learning transactions - from courses they took, books they read, or work projects they completed? And what if you could see their work product (papers etc.) rather than just their transcript and grades? Would that be more relevant and useful than knowing what college degree they had?

This is the idea behind Learning is Earning 2026, a future system that would reward any kind of learning. The EduBlocks Ledger would use the same blockchain technology that runs Bitcoin. Anyone could award these blocks to anyone else. As explained by Marketplace Morning Report, the Institute for the Future is developing the EduBlocks concept.


Market share MIT-Sloan

2. Is market share a valuable metric?
Only in certain cases is market share an important metric for figuring out how to make more profits. Neil T. Bendle and Charan K. Bagga explain in the MIT Sloan Management Review that Popular marketing metrics, including market share, are regularly misunderstood and misused.

Well-known research in the 1970s suggested a link between market share and ROI. But now most evidence shows it's a correlational relationship, not causal.


Adolescent crime

3. Evidence-based ways to close gaps in crime, poverty, education.
The Laura and John Arnold Foundation launched a $15 million Moving the Needle Competition, which will fund state and local governments and nonprofits implementing highly effective ways to address poverty, education, and crime. The competition is recognized as a key evidence-based initiative in White House communications about My Brother’s Keeper, a federal effort to address persistent opportunity gaps.

Around 250 communities have responded to the My Brother’s Keeper Community Challenge with $600+ million in private sector and philanthropic grants, plus $1 billion in low-interest financing. Efforts include registering 90% of Detroit's 4-year-olds in preschool, private-sector “MBK STEM + Entrepreneurship” commitments, and a Summit on Preventing Youth Violence.

Here's hoping these initiatives are evaluated rigorously, and the ones demonstrating evidence of good or promising outcomes are continued.


Eddie Izzard

4. Everyday health evidence.
Evidence for Everyday Health Choices is a new series by @UKCochraneCentr, offering quick rundowns of the systematic reviews on a pertinent topic. @SarahChapman30 leads the effort. Nice recent example inspired by Eddie Izzard: Evidence on stretching and other techniques to improve marathon performance and recovery: Running marathons Izzard enough: what can help? [Photo credit: Evidence for Everyday Health Choices.]

5. Short Science = Understandable Science.
Short Science allows people to publish summaries of research papers; they're voted on and ranked until the best/most accessible summary has been identified. The goal is to make seminal ideas in science accessible to the people who want to understand them. Anyone can write a summary of any paper in the Short Science database. Thanks to Carl Anderson (@LeapingLlamas).

Tuesday, 26 April 2016

Baseball decisions, actuaries, and streaming analytics.

Cutters from Breaking Away movie

1. SPOTLIGHT: How are innovations in baseball analytics like data science?
Last week, I spoke at Nerd Nite SF about recent developments in baseball analytics. Highlights from my talk:

- Data science and baseball analytics are following similar trajectories. There's more and more data, but people struggle to find predictive value. Oftentimes, executives are less familiar with technical details, so analysts must communicate findings and recommendations so they're palatable to decision makers. The role of analysts, and  challenges they face, are described beautifully by Adam Guttridge and David Ogren of NEIFI.

- 'Inside baseball' is full of outsiders with fresh ideas. Bill James is the obvious/glorious example - and Billy Beane (Moneyball) applied great outsider thinking. Analytics experts joining front offices today are also outsiders, but valued because they understand prediction;  the same goes for anyone seeking to transform a corporate culture to evidence-based decision making.

Tracy Altman @ Nerd Nite SF
- Defensive shifts may number 30,000 this season, up from 2,300 five years ago (John Dewan prediction). On-the-spot decisions are powered by popup iPad spray charts with shift recommendations for each opposing batter. And defensive stats are finally becoming a reality.

- Statcast creates fantastic descriptive stats for TV viewers; potential value for team management is TBD. Fielder fly-ball stats are new to baseball and sort of irresistible, especially the 'route efficiency' calculation.

- Graph databases, relatively new to the field, lend themselves well to analyzing relationships - and supplement what's available from a conventional row/column database. Learn more at FanGraphs.com. And topological maps (Ayasdi and Baseball Prospectus) are a powerful way to understand player similarity. Highly dimensional data are grouped into nodes, which are connected when they share a common data point - this produces a topo map grouping players with high similarity.

2. Will AI replace insurance actuaries?
10+ years ago, a friend of Ugly Research joined a startup offering technology to assist actuaries making insurance policy decisions. It didn't go all that well - those were early days, and it was difficult for people to trust an 'assistant' who was essentially a black box model. Skip ahead to today, when #fintech competes in a world ready to accept AI solutions, whether they augment or replace highly paid human beings. In Could #InsurTech AI machines replace Insurance Actuaries?, the excellent @DailyFintech blog handicaps several tech startups leading this effort, including Atidot, Quantemplate, Analyze Re, FitSense, and Wunelli.

3. The blind leading the blind in risk communication.
On the BMJ blog, Glyn Elwyn contemplates the difficulty of shared health decision-making, given people's inadequacy at understanding and communicating risk. Thanks to BMJ_ClinicalEvidence (@BMJ_CE).

4. You may know more than you think.
Maybe it's okay to hear voices. Evidence suggests the crowd in your head can improve your decisions. Thanks to Andrew Munro (@AndrewPMunro).

5. 'True' streaming analytics apps.
Mike Gualtieri of Forrester (@mgualtieri) put together a nice list of apps that stream real-time analytics. Thanks to Mark van Rijmenam (@VanRijmenam).

Wednesday, 20 April 2016

How to lead people through evidence-based decisions.

Decision Quality book

There's no shortage of books on strategy and decision-making - and many of them can seem out of touch. This one is worthwhile reading: Decision Quality: Value Creation from Better Business Decisions by Carl Spetzler, Hannah Winter, and Jennifer Meyer (Wiley 2016).

The authors are decision analysis experts with the well-known, Palo Alto-based Strategic Decisions Group. Instead of presenting schemes or templates for making decisions, they get to the heart of the matter: Decision quality, when making big decisions or smaller choices. How will you decide? How will you teach your team to make high-quality decisions? And how will you define 'high quality'?

For example, for a healthcare formulary decision, outline in advance what findings will be considered. Cost-effectiveness modeling? Real-world evidence? How will evidence be weighted - possibly using multi-criteria decision analysis? How will uncertainty be factored in?


“If you want to change the culture of an organization, change the way people make decisions.” -Vincent Barabba

Key takeaways from this book:

- You can lead a meaningful change by encouraging people to fully understand why it's the decision process, not the outcome, that is under their control.

- Teach your team to make high-quality decisions. Build organizational capability so people use similar language and methods to assess evidence and analyze decisions.

- Get more buy-in with a better process, from initial concept to execution. Judge the quality of a decision as you go along.


Tuesday, 12 April 2016

Better evidence for patients, and geeking out on baseball.

Health tech wearables

1. SPOTLIGHT: Redefining how patients get health evidence.

How can people truly understand evidence and the tradeoffs associated with health treatments? How can the medical community lead them through decision-making that's shared - but also evidence-based?

Hoping for cures, patients and their families anxiously Google medical research. Meanwhile, the quantified selves are gathering data at breakneck speed. These won't solve the problem. However, this month's entire Health Affairs issue (April 2016) focuses on consumer uses of evidence and highlights promising ideas.

  • Translating medical evidence. Lots of synthesis and many guidelines are targeted at healthcare professionals, not civilians. Knowledge translation has become an essential piece, although it doesn't always involve patients at early stages. The Boot Camp Translation process is changing that. The method enables leaders to engage patients and develop healthcare language that is accessible and understandable. Topics include colon cancer, asthma, and blood pressure management.
  • Truly patient-centered medicine. Patient engagement is a buzzword, but capturing patient-reported outcomes in the clinical environment is a real thing that might make a big difference. Danielle Lavallee led an investigation into how patients and providers can find more common ground for communicating.
  • Meaningful insight from wearables. These are early days, so it's probably not fair to take shots at the gizmos out there. It will be a beautiful thing when sensors and other devices can deliver more than alerts and reports - and make valuable recommendations in a consumable way. And of course these wearables can play a role in routine collection of patient-reported outcomes.


2. Roll your own analytics for fantasy baseball.
For some of us, it's that special time of year when we come to the realization that our favorite baseball team is likely going home early again this season. There's always fantasy baseball, and it's getting easier to geek out with analytics to improve your results.

3. AI engine emerges after 30 years.
No one ever said machine learning was easy. Cyc is an AI engine that reflects 30 years of building a knowledge base. Now its creator, Doug Lenat, says it's ready for prime time. Lucid is commercializing the technology. Personal assistants and healthcare applications are in the works.

Photo credit: fitbit one by Tatsuo Yamashita on Flickr.

Tuesday, 05 April 2016

$15 minimum wage, evidence-based HR, and manmade earthquakes.


Photo by Fightfor15.org

1. SPOTLIGHT: Will $15 wages destroy California jobs?
California is moving toward a $15/hour minimum wage (slowly, stepping up through 2023). Will employers be forced to eliminate jobs under the added financial pressure? As with all things economic, it depends who you ask. Lots of numbers have been thrown around during the recent push for higher pay. Fightfor15.org says 6.5 million workers are getting raises in California, and that 2/3 of New Yorkers support a similar increase. But small businesses, restaurants in particular, are concerned they'll have to trim menus and staff - they can charge only so much for a sandwich.

Moody's Analytics economist Adam Ozimek says it's not just about food service or home healthcare. Writing on The Dismal Scientist Blog, "[I]n past work I showed that California has 600,000 manufacturing workers who currently make $15 an hour or less. The massive job losses in manufacturing over the last few decades has shown that it is an intensely globally competitive industry where uncompetitive wages are not sustainable." 

It's not all so grim. Ozimek shows that early reports of steep job losses after Seattle's minimum-wage hike have been revised strongly upward. However, finding "the right comparison group is getting complicated."

Yellow Map Chance of Earthquake

2. Manmade events sharply increase earthquake risk.
Holy smokes. New USGS maps show north-central Oklahoma at high earthquake risk. The United States Geological Survey now includes potential ground-shaking hazards from both 'human-induced' and natural earthquakes, substantially changing their risk assessment for several areas. Oklahoma recorded 907 earthquakes last year at magnitude 3 or higher. Disposal of industrial wastewater has emerged as a substantial factor.

3. Evidence-based HR redefines leadership roles.
Applying evidence-based principles to talent management can boost strategic impact, but requires a different approach to leadership. The book Transformative HR: How Great Companies Use Evidence-Based Change for Sustainable Advantage (Jossey-Bass) describes practical uses of evidence to improve people management. John Boudreau and Ravin Jesuthasan suggest principles for evidence-based change, including logic-driven analytics. For instance, establishing appropriate metrics for each sphere of your business, rather than blanket adoption of measures like employee engagement and turnover.

4. Why we're not better at investing.
Gary Belsky does a great job of explaining why we think we're better investors than we are. By now our decision biases have been well-documented by behavioral economists. Plus we really hate to lose - yet we're overconfident, somehow thinking we can compete with Warren Buffet.

Tuesday, 29 March 2016

Rapid is the new black, how to ask for money, and should research articles be free?


1. #rapidisthenewblack

The need for speed is paramount, so it's crucial that we test ideas and synthesize evidence quickly without losing necessary rigor. Examples of people working hard to get it right:

  • The Digital Health Breakthrough Network is a very cool idea, supported by an A-list team. They (@AskDHBN) seek New York City-based startups who want to test technology in rigorous pilot studies. The goal is rapid validation of early-stage startups with real end users. Apply here.
  • The UK's fantastic Alliance for Useful Evidence (@A4UEvidence) asks Rapid Evidence Assessments: A bright idea or a false dawn? "Research synthesis will be at the heart of the government’s new What Works centres" - equally true in the US. The idea is "seductive: the rigour of a systematic review, but one that is cheaper and quicker to complete." Much depends on whether the review maps easily onto an existing field of study.
  • Jon Brassey of the Trip database is exploring methods for rapid reviews of health evidence. See Rapid-Reviews.info or @rapidreviews_i.
  • Miles McNall and Pennie G. Foster-Fishman of Michigan State (ouch, still can't get over that bracket-busting March Madness loss) present methods and case studies for rapid evaluations and assessments. In the American Journal of Evaluation, they caution that the central issue is balancing speed and trustworthiness.

2. The science of asking for donations: Unit asking method.
How much would you give to help one person in need? How much would you give to help 20 people? This is the concept behind the unit asking method, a way to make philanthropic fund-raising more successful.

3. Should all research papers be free? 
Good stuff from the New York Times on the conflict between scholarly journal paywalls and Sci-Hub.

4. Now your spreadsheet can tell you what's going on.
Savvy generates a narrative for business intelligence charts in Qlik or Excel.

Tuesday, 22 March 2016

Who should prepare evidence for human consumption? A possible fix to the hiring dilemma.

Pasta being sliced

Health economists, financial analysts, policy advisers, along with newly minted data scientists, face diverse challenges: Besides data-gathering and modeling, they analyze findings, demonstrate value, and advocate to others. No wonder these positions are so difficult to fill.

At some point, new evidence - economics, health, marketing - will have to be explained for human consumption. Who should do the explaining?

Writing in Tech Republic, Matt Asay (@mjasay) reminds us data science falls into two categories, depending on whether it's intended for human or machine consumption - the same can be said for all sorts of analytical activities. But eventually, most people who develop complex models, or create algorithms, or report findings will need to describe their work to someone making budget and strategy decisions.

That's some skill set. Delivering evidence for human consumption requires talent above and beyond complex technical or scientific expertise; communication skills are essential. In a recent Harvard Business Review, Michael Li explains three key presentation capabilities:

  1. Articulating business value (defining success with metrics).
  2. Giving the right level of technical detail (the story behind the data).
  3. Getting visualizations right (telling a clean story with diagrams).

How many hats do I have to wear? Many organizations want their strong technical and scientific talent to be skillful presenters, capable of articulating insights to decision makers. Some are also expected to function as influencers, and champion analytical methodologies to business units or partners across the organization. But not everyone aspires - or is able - to wear all these hats.

Consider a recent job posting for an Audience Insights Manager at a consumer-facing tech firm. The company is seeking someone with a “proven understanding of consumers and an ability to distill data into compelling, actionable insights”. But that's not all: “The ideal candidate will be a subject matter expert on analytics and have a strong track record of building and managing high-performing teams. You will be required to articulate the vision of the team to sales and marketing leadership and be a strong advocate for insights. You must be a data expert, but also know how to articulate data into insights.” (Is this realistic?)

Why not pair up? Would it make more sense to pair people, say for instance partnering an analytics expert with an internal influencer whose talent is synthesizing evidence and articulating value to executive decision makers? Or a pharma investigator with an 'insight interpreter'? We've all seen settings where a subject matter expert handed work product to a writer who created deliverables. I'm thinking of an integrated team of peers, so each individual can contribute as much value as possible.

I welcome your thoughts.

Photo credit: Pasta Being Sliced by Dldrlks on Flickr.

Wednesday, 16 March 2016

Rethinking Abstracts & Bibliographies: Three Examples.

In healthcare, communication is long overdue for innovation. Here's three approaches that change how research evidence and health economics are presented to decision makers and other stakeholders. 

1. From Taylor & Francis, cartoon abstracts are innovative introductions to journal articles. T&F says cartoons have already generated 11,000+ extra downloads for papers in science, technology, and math. "With the authors represented through characters in the cartoon strip, they’re also a useful networking tool amongst peers."

Taylor & Francis cartoon abstract


2. PepperSlice is an evidence synthesis format and technology I created with my team at Ugly Research. Key conclusions are presented in a brief narrative, and each supporting citation appears in a graphical format that highlights the data and how it was evaluated. (To get your name on the PepperSlice beta list, email me at tracy@uglyresearch.com.)

PepperSlice evidence synthesis format


3. Many online Elsevier journals supplement the print articles with interactive graphics. This effort began as the Article of the Future project.

Elsevier article of the future


Let me know what innovations you're developing, or would like to see.