Evidence Soup
How to find, use, and explain evidence.

72 posts categorized "fun-with-evidence friday"

Monday, 14 December 2015

'Evidence-based' is a thing. It was a very good year.

2015 was kind to the 'evidence-based' movement. Leaders in important sectors - ranging from healthcare to education policy - are adopting standardized, rigorous methods for data gathering, analytics, and decision making. Evaluation of interventions will never be the same.

With so much data available, it's a non-stop effort to pinpoint which sources possess the validity, value, and power to identify, describe, or predict transformational changes to important outcomes. But this is the only path to sustaining executives' confidence in evidence-based methods.

Here's a few examples of evidence-based game-changers, followed by a brief summary of challenges for 2016.

What works: What Works Cities is using data and evidence to improve results for city residents. The Laura and John Arnold Foundation is expanding funding for low-cost, randomized controlled trials (RCTs) - part of its effort to expand the evidence base for “what works” in U.S. social spending.

Evidence-based HR: KPMG consulting practice leaders say "HR isn’t soft science, it’s about hard numbers, big data, evidence."

Comparative effectiveness research: Evidence-based medicine continues to thrive. Despite some challenges with over-generalizing the patient populations, CER provides great examples of systematic evidence synthesis. This AHRQ reportillustrates a process for transparently identifying research questions and reviewing findings, supported by panels of experts.

Youth mentoring: Evidence-based programs are connecting research findings with practices and standards for mentoring distinct youth populations (such as children with incarcerated parents). Nothing could be more important. #MentoringSummit2016

Nonprofit management: The UK-based Alliance for Useful Evidence (@A4UEvidence) is sponsoring The Science of Using Science Evidence: A systematic review, policy report, and conference to explore what approaches best enable research use in decision-making for policy and practice. 

Education: The U.S. House passed the Every Student Succeeds Act, outlining provisions for evidence collection, analysis, and use in education policy. Intended to improve outcomes by shifting $2 billion in annual funding toward evidence-based solutions.

Issues for 2016.

Red tape. Explicitly recognizing tiers of acceptable evidence, and how they're collected, is an essential part of evidence-based decision making. But with standardizing also comes bureacracy, particularly for government programs. The U.S. Social Innovation Fund raises awareness for rigorous social program evidence - but runs the risk of slowing progress with exhaustive recognition of various sanctioned study designs (we're at 72 and counting).

Meta-evidence. We'll need lots more evidence about the evidence, to answer questions like: Which forms of evidence are most valuable, useful, and reliable - and which ones are actually applied to important decisions? When should we standardize decision making, and when should we allow a more fluid process?

Monday, 17 November 2014

Cognitive analytics, satisficing, sabremetrics, and other cool ways of deciding.

My weekly five links on evidence & decision-making. 

1. I was delighted when a friend sent a link to 10 Things I Believe About Baseball Without Evidence. Ken Arneson (@kenarneson) looks at sabremetrics with things like the linguistic relativity principle and the science of memory-based prediction. Warning to Oakland A's fans: Includes yet more reminders of the epic 2014 collapse.

2. The concept of satisficing came up on one of my current projects, so I reread some overviews. Herbert Simon coined this term to explain decision making under circumstances where an optimal solution can't be determined. (Simon sure has staying power: During grad school I wanted a copy of Administrative Behavior (1957), and there it was in paperback on the shelf at the Tattered Cover in Denver.)

3. Eric Topol (Medscape) interviewed Margaret Hamburg, US FDA commissioner, on weighing risks/benefits in complex agency decision-making. She talked specifically about the newly restrictive policy on opiate painkiller prescriptions (see page 3). I'd like to see FDA's analysis of unintended side effects, such as the concurrent rise of U.S. heroin use/overdoses.

4. Organizations are challenged with finding the pony in all their data, and marketing-spin hijinks have ensued. Seth Grimes (@sethgrimes) has a great discussion of efforts to go beyond the fundamental volume, velocity, etc. in Avoid Wanna-V Confusion.

5. Steve Ardire (@SArdire) authored a Dataversity paper on cognitive computing, including survey results and some choice comments, both pro and con, about the prospects for this emerging area. For creating business value, 'business intelligence/cognitive analytics' looks promising. Cognitive Computing: An Emerging Hub in IT Ecosystems.

Curated by Tracy Altman of Ugly Research


Friday, 18 October 2013

The Illustrated Book of Bad Arguments.

It's a glorious Fun-with-Evidence Friday. Because I've discovered The Illustrated Book of Bad Arguments. The author is Ali Almossawi (@alialmossawi), a metrics engineer in San Francisco. It's fantastic. Available online now, and soon in hardback.

Illustrated Book of Bad Arguments

Besides the fun illustrations, you'll find serious explanations of logical fallacies, plus definitions of key terms:

"Soundness: A deductive argument is sound if it is valid and its premisses are true. If either of those conditions does not hold, then the argument is unsound. Truth is determined by looking at whether the argument's premisses and conclusions are in accordance with facts in the real world." (BTW, I did not know premise is also spelled premiss.)

Almossawi says "I have selected a small set of common errors in reasoning and visualized them using memorable illustrations that are supplemented with lots of examples. The hope is that the reader will learn from these pages some of the most common pitfalls in arguments and be able to identify and avoid them in practice."

Happy weekend, everyone. 


Thursday, 22 August 2013

Major study debunks belief that people in the distance are really tiny.

Evidence shows our eyes have been playing tricks on us. As reported by The Onion, a "five-year study, conducted by researchers at Princeton University, has shattered traditionally accepted theories that people standing some distance away from you are very small, and people close-by are very big."

Wishing you a happy Fun-with-Evidence Friday. On a serious note, though, it seems to be getting tougher to tell the faux research from the silly, funded-with-other-people's-money stuff we often read about. For more on this, take a look at the Annals of Improbable Research.

Friday, 14 June 2013

Fun-with-Evidence Friday: Science works, b!t3^&s!

Sciencetee  Spotted this great t-shirt in Berkeley, California. Evidently the creator made this to summarize his doctoral research: Science: It Works, B!t3^&s.  Sciencetee2







What does "clinically studied" mean? (A cynic might interpret that as "research funds were spent").  I happened to see GNC's Mega Men Sport multivitamin, labeled as a "Clinically Studied Multivitamin^". Labels are small, so detailed findings can't appear on the front of a consumer package. But I wondered about the meaning in this context. Here's what the product page says.

Clinicallystudied"Lutemax 2020™
^In a randomized, double-blind, placebo-controlled study of 112 healthy volunteers, subjects taking the GNC vitamin and mineral blend in this product for six weeks experienced statistically significant improvements in markers of B vitamin and antioxidant status, as well as improvements in SF-36 Vitality and Mental Health scores compared to those taking a placebo."

And there you have it: A little evidence of something or other.

Happy Fun-with-Evidence Friday, everybody.

Friday, 26 April 2013

We've got to stop drooling at shiny data visualizations and keep searching for non-obvious evidence.

Yes, we want people to provide hard evidence supporting their claims. But aren't some things just too obvious to bother with? Shouldn't we use our talents to discover things we don't already know?

Fbfriends_wolfram2I like a shiny data visualization as much as the next guy, but recent details about Facebook analysis made me wonder why some great minds (with great resources) are focusing on some pretty unimportant stuff.

Here's what happened. Stephen Wolfram - who is certainly much smarter than me - released findings from his Wolfram|Alpha Personal Analytics for Facebook project. Among the revelations is that we have a peak number of friends around age 20, with that number slowly dwindling as we reach our 70s.

Sort of like the actual world, wouldn't you say? Doesn't this just confirm what we already know, only with a cool, crowd-sourced, data-donation methodology and attractive data visualization?

Doesn't pass the "six smart people" test. This reminds me of something Rita Gunther McGrath has talked about. She, too, is critical of obvious / we-already-know-that research, saying that if "six smart people in a room" can immediately see it, then it isn't something that needs formalized research. (Rita's a smart one, and I suggest following her at @rgmgrath).


The Daily Mail covered this with the headline "Growing old on Facebook: Search data reveals we talk more about the weather and  politics as we age." (Sadly, reading their story, it's tough to tell if they're being sarcastic or not.) Wolfram found we accumulate clusters of friends as we grow older - with the average 35-year-old having four clusters (again, no kidding: neighbors, work friends, friends from back home, etc.).

Wolfram has done some great stuff, such as A New Kind of Science. I know social network analysis can be important, and reveal useful things about how new knowledge develops. Let's see more of that.

Happy Fun-with-Evidence Friday. Have a great weekend, everybody.

Friday, 08 March 2013

Skeptical about the evidence in business books? The evidence on sales volumes is iffy, too.

I quit business books a long time ago, and I'm not the only one. Yet we hear endlessly about things like cheese moving and power posing.

Business book sales evidenceTurns out, we should question not only the evidence being claimed by the authors, but also some of their claims about popularity.

A recent Wall Street Journal investigation reveals evidence that some 'bestsellers' don't belong on those lists at all. In The Mystery of the Book Sales Spike [How Are Some Authors Landing On Best-Seller Lists? They're Buying Their Way], Jeffery Trachtenberg explains how authors can hire a marketing firm to purchase books ahead of their publication date, creating artificial - and short-lived - sales spikes.

"Last August, a book titled 'Leapfrogging' hit The Wall Street Journal's list of best-selling business titles upon its debut. The following week, sales of the book, written by first-time author Soren Kaplan, plunged 99% and it fell off the list.

Something similar happened when the hardcover edition of 'Networking is Dead,' was published in mid-December. A week after selling enough copies to make it onto the Journal's business best-seller list, more hardcover copies of the book were returned than sold, says book-sales tracker Nielsen BookScan."

Buyer beware. Even if the evidence doesn't actually confirm that an author is "best-selling", his/her speaking and consulting fees will likely get a nice bump. (Creating more opportunities for hapless folks to pay for bad evidence.)

Of course, the keepers of the bestseller lists are nervous about this type of thing, and have controls in place to prevent such trickery. Yet some books are getting through anyway.

Happy Fun-with-Evidence Friday.


Friday, 01 February 2013

Fun-with-Evidence Friday: Do you suffer from evidence euphoria? Or research rapture?

Well, hell. Another malady to worry about. Sean Pidgeon described research rapture in a recent New York Times column. Here I am, searching for evidence to support my every move, only to learn it's an addiction. I'll call it evidence euphoria.

I thought I was just being thorough. As Pidgeon explains: "If I may paraphrase various informal definitions that are to be found online, research rapture is something like this: A state of enthusiasm or exaltation arising from the exhaustive study of a topic or period of history; the delightful but dangerous condition of becoming repeatedly sidetracked in following intriguing threads of information, or constantly searching for one more elusive fact."

Maybe this is why my dissertation took so long. Happy Fun-with-Evidence Friday, everybody.

Friday, 18 May 2012

Fun-with-Evidence Friday: Limericks about wine swirling and scams about shoe toning.

Happy Fun-with-Evidence Friday. I have two things for you.

First, from the always-delightful Annals of Improbable Research. Scientists are commended for contributing valuable evidence about wine swirling. Poets chose co-winners in the Wine Swirling Limerick Competition, which honored the study 'Oenodynamic': Hydrodynamic of Wine Swirling, by Martino Reclari et al. (Video here.)

Winning limericks came from investigator Patrick McKeon:

In a lab during end of year crunch,
Thought a researcher needing a hunch:
 'This wine I do cherish,
  So (publish or perish!)
I'll send in a film of my lunch.'

And from limerick laureate Matrin Eiger:

They use several metrics to show
The circular motion and flow
  Of wine as it swirls,
  As it whirls and it twirls,
Although why, I am too drunk to know.

Shame on you, Joe Montana. And of course, there's the settlement of charges against Skechers for false claims about their so-called toning shoes. The U.S. Federal Trade Commission announced the Skechers shoe company will pay $40 million for deceiving consumers with cherry-picked evidence (provided by celebrity spokespeople). Among other things, the FTC alleged unsupported claims that Shape-ups would provide more weight loss, and more muscle toning and strengthening than regular fitness shoes.

Gotta love their can-do attitude, though: According to the San Francisco Chronicle, Skechers CFO David Weinberg denied the allegations, saying the company only settled to avoid a lengthy legal battle. "While we believe we could have prevailed in each of these cases, to do so would have imposed an unreasonable burden on the company, regardless of the outcome."

Oops. As The Consumerist points out, one chiropractor who testified in support of the shoes said "After performing a six week clinical trial testing the benefits of SKECHERS Shape-ups, I am confident in recommending them to patients to increase their low back endurance and improve gluteal strength. Patients also benefited from weight loss and improved body composition.” But turns out he is married to a Skechers marketing exec.

Have a great weekend, everybody.

Friday, 13 April 2012

Hard evidence teaches hard lessons: Rapture billboard in the rearview mirror.

Happy Fun-with-Evidence Friday. I've been traveling for family stuff. Driving I-80 through Wyoming, I spotted this orphaned billboard. Remember that Rapture many were predicting for May 21 last year (according to Harold Camping's biblical numerology)? I got this quick snapshot from the car (somewhere between Laramie and Little America). The yellow circle to the left reads "The Bible Guarantees It!".

This is why you can't get a straight answer out of an attorney or a statistician. I've got to admire people who show such courage in their convictions. After a Judgment-free May 21 came and went, Camping said he'd had a "really tough weekend." But you can't fake reality. Past performance is not a reliable indicator of future anything, etc etc. And while some made pretty funny jokes about punking the Rapture, others were seriously hurt by their beliefs, which isn't so funny.

Now folks are predicting Doomsday on December 21 - all that Mayan calendar, 12/21/12 stuff. Dan Piraro, creator of the Bizarro comic (and most awesome member of my high school graduating class), offers the best explanation I've seen: That's all the room they had on their ancient calendar.

Happy weekend, everybody. I look forward to dishing up more Evidence Soup as we approach the End of Days.