Evidence Soup

Does the evidence show that Dunbar's number really matters? Or are weak ties just as good as strong ones?

Contributors to the Smart Data Collective forum frequently offer useful insights about analytics and other topics. I liked a recent piece by Jacob Morgan about Dunbar's number: According to Wikipedia, this is a "theoretical cognitive limit to the number of people with whom one can maintain stable social relationships." Dunbar's is typically believed to be about 150. Makes sense to me - especially when I think about the circle of people I stay in regular contact with, invite to parties, interact with on Facebook, etc.

I'm intrigued by social network research: The structural holes stuff, etc. Lots of folks are exploring how people make connections, and what difference these connections (or lack thereof) can make. Of course, I'm most interested in understanding the role of social networks in spreading useful evidence as quickly as possible.

Strong vs. weak. Dunbar's number describes what many folks call "strong ties". Is a strong connection required to get people to consider new evidence, especially when that evidence challenges their current beliefs? Or are weak ties enough?

Morten T. Hansen, in his book Collaboration: How Leaders Avoid the Traps, Create Unity, and Reap Big Results, talks about the importance of weak ties. "[R]esearch shows that weak ties can prove much more helpful in networking, because they form bridges to worlds we do not walk within. Strong ties, on the other hand, tend to be worlds we already know; a good friends often knows many of the same people and things we know." And he offers a pragmatic explanation of why your weak-tie number can go much higher than 150: "It’s less time consuming to talk to someone once a month (weak tie) than twice a week (a strong tie). People can keep up quite a few weak ties without them being a burden."

Authority matters, too. When we consider new evidence, the authority of the source is important - this is a key component of making something persuasive. Technology has made it much easier to learn about new evidence and locate new authorities, even when we have weak - or possibly nonexistent - ties to those sources. (Consider Twitter. Evidence often comes from people you're following, or people who are re-Tweeted, even if you don't know them.)

Beating the number. For those who do want to be strongly connected to as many people as possible, Chris Brogan writes about "beating Dunbar's number" - for instance, by establishing yourself in multiple groups based on specific geography, a specific industry vertical, etc. Some of his points are a different way of saying "it's not who you know, it's who knows you" - and I agree with him on that. Brogan writes: "[M]ake it work in reverse. By that, I mean this: if people think of you as part of their 150, then they come to you, and they seek you out. This comes from having some value to offer. If, for instance, you’re helpful in finding others business, those people will seek you out the next time they have an opportunity, or the next time they need your help. In either case, you’re in the network."

The bottom line? I think Dunbar's number is useful for evaluating a certain type of connection: The ones characterized by strong ties. But for spreading evidence to a wider audience, it's not important. You can successfully communicate evidence to folks with whom you have only weak ties.

By the way, Dunbar has a book called Grooming, Gossip, and the Evolution of Language. I've not read it, but the reviews are quite positive, and I've heard it's a worthwhile read.

The Lancet retracts evidence about autisim and vaccines, saying 1998 study was flawed.

Update: Thanks to Peter West (@WestPeter) for my favorite new phrase: Evidence-(de)based.

Better to own up to your mistakes than pretend they didn't happen, hoping they'll go away. So kudos to The Lancet for retracting a paper they published in 1998. Dr. Andrew Wakefield had claimed his research findings substantiated his belief that the measles-mumps-rubella (MMR) vaccine causes autism in children.

As reported in the LA Times, Britain's medical regulator, the General Medical Council, concluded recently that "The British doctor whose suggestion of a link between the MMR shot and autism helped cause vaccination rates to plunge conducted his now-discredited research in a dishonest and irresponsible manner.... His study, however, was based on just 12 children. Lancet later declared that it should not have published the report, and further studies have not been able to replicate Wakefield's results."

A separate LATimes story quoted Dr. Jeffrey Boscamp of the Hackensack University Medical Center in New Jersey as saying that the original report "was outrageous," and that "Most of the authors asked for their names to be removed from the study. It's unfortunate that it undermined confidence in vaccines when in fact it wasn't true at all." In its comment today, The Lancet said that the "claims in the original paper that children were 'consecutively referred' and that investigations were 'approved' by the local ethics committee have been proven to be false."

Which professions provide the most evidence-free predictions?

Happy Fun-with-Evidence Friday. The folks at Cracked.com are at it again with a hilarious, and impressively evidence-based, post: This one is The 6 Most Statistically Full of Shit Professions. Sure, it would be easy to come up with more than six, but they've limited their picks to these: Stock market analysts, wine tasters, art critics, criminal profilers, weather forecasters, and sportswriters. [BTW, I haven't posted in awhile (and some Evidence Soup readers even noticed!). I was sick for a few weeks - many thanks to those who wrote in with well wishes.]

Christina H. at Cracked.com doesn't just take shots at people who make dubious predictions and pronouncements for a living: She provides evidence to back up her claims, and does it with tremendous humor. For example, in a discussion of wine tasting, she says:

"Sure, Joe Consumer actually likes cheaper wines better, but that's because Joe Consumer is a stupid Philistine. The experts can tell the difference between a 2006 and 2007 Stag's Leap Cabernet Sauvignon in their sleep because everyone knows 2006 was a pedestrian year for Napa Valley reds. Hell, they are so good they can tell the difference between two bottles of the same wine. In one experiment, wine experts were given two bottles of the same wine, only one was labeled a 'vin de table' (France's version of 'Night Train') and one was labeled a 'grand cru' (top-rated vineyard since 1855)."

As you've probably guessed, the experts rated the grand cru much more favorably. "Not only were their tasting skills put to shame, it didn't even occur to them that nobody buys a $40-plus bottle of wine for a university experiment." The story references hands-on evidence provided in the book The Wine Trials: 100 Everyday Wines Under $15 that Beat $50 to $150 Wines in Brown-Bag Blind Tastings. And to her credit, Christina H also links to authoritative evidence and research findings, in this case results cited in the New Yorker:

"Studies suggest that the experience of smelling and tasting wine is extremely susceptible to interference from the cognitive parts of the brain. Several years ago, Frédéric Brochet, a Ph.D. student in oenology at the University of Bordeaux, did a study in which he served fifty-seven participants a midrange red Bordeaux from a bottle with a label indicating that it was a modest vin de table. A week later, he served the same wine to the same subjects but this time poured from a bottle indicating that the wine was a grand cru. Whereas the tasters found the wine from the first bottle 'simple,' 'unbalanced,' and 'weak,' they found the wine from the second 'complex,' 'balanced,' and 'full.' Brochet argues that our 'perceptive expectation' arising from the label often governs our experience of a wine, overriding our actual sensory response to whatever is in the bottle."

On that happy note, I'm going to enjoy a glass of the $12 Spanish vino tinto I bought the other day.

Evidence-based trash talk.

Happy Fun-with-Evidence Friday. Apparently it's not that easy being a good trash talker on the football field, or anywhere else for that matter. It takes research, preparation, and focus. The New York Times recently profiled Bart Scott of the NFL's New York Jets about his lifelong dedication to creative, high-impact trash talk. Yes, it's humorous, but his advice could benefit anyone presenting evidence to an ornery or unconvinced audience.

As the story goes, he "uses a three-step trash-talk template. He starts with research. He scours ESPN, Google and scouting reports, which include pictures. He wants to understand the opponents he will talk to, understand what angers them, what makes them tick. He looks for police incidents, problems with wives or girlfriends, expanding stomachs, funny faces. To aid in his delivery, Scott watches wrestling videos, or famous trash talkers like Muhammad Ali."

Scott says "I keep ammo on everybody... Because I will never be caught off-guard. No one will outtalk me. Ever." His guide to becoming an expert trash talker includes "Do your research." And "Know your audience. Save your best material for last, and be conscious of whom you are speaking to and what reaction you can get."

A guide for evidence-informed health policymaking. SUPPORT provides that, and other great resources.

SUPPORT is an excellent resource providing a variety of information about evidence-based medicine and health policy (for example, a section on making judgments about the quality of evidence, and resources for conducting trials in low-income countries). The home page is here. Also included are tools for evidence-informed health policymaking. These are presented in a new supplement to the open-access journal Health Research Policy & Systems. The series is "for people responsible for making decisions about health policies and programmes and for those who support these decision makers."

The editorial Evidence-informed health policy: are we beginning to get there at last? [pdf] says policy informed by evidence "has long been the hope, and indeed sometimes the expectation, of those reforming health research systems. Now though, there are grounds for believing that the hopes are increasingly beginning to be turned into realities. This optimism is based on factors including... capacity to conduct systematic reviews of that evidence; and the growing attention being given to the policymaking structures necessary to absorb and use research evidence."

This takes me back to methods I studied in graduate school (PhD in Public Policy Analysis), reading classics such as Stokey & Zeckhauser. Always good stuff to know, giving people a methodical way to look at a complex situation. SUPPORT is a respectable summary of how to do this, encouraging people to systematically ask questions such as:

1. What's the problem?
2. How did the problem come to attention, and has this influenced the prospect of it being addressed?
3. What indicators can be used to establish the magnitude of the problem, and measure progress in addressing it?
4. How can a problem be framed (or described) in a way that will motivate different groups? 

Evidence-informed says it better. I think 'evidence-informed' is a better description of what we are trying to achieve, but it's used far less frequently than 'evidence-based', and that seems unlikely to change. The policy-making information is divided into 18 sections, with topics such as:

• 4: Using research evidence to clarify a problem.
• 8: Deciding how much confidence to place in a systematic review.
• 11: Finding and using evidence about local conditions.

The guide says policymakers "find themselves in situations that spur them on to work out how best to define a problem. These may range from being asked an awkward or challenging question in the legislature, to finding a problem highlighted on the front page of a newspaper. The motivations for policymakers wanting to clarify a problem are diverse. These may range from deciding whether to pay serious attention to a particular problem that others claim is important, to wondering how to convince others to agree that a problem is important. Debates and struggles over how to define a problem are a critically important part of the policymaking process."

Available in other languages? Some of the policy-making documents say "Links to Spanish, Portuguese, French and Chinese translations of this series can be found on the SUPPORT website http://www.support-collaboration.org/" - but I was unable to locate those links.

More graphics needed. There's lots of good advice here, although some of it could be improved. For example, the following table describes how to search for a research article or systematic review -- screen shots of conducting a search would have been preferable.

Additional resources & tools. Links to useful web sites are provided, such as the WHO Evidence-Informed Policy Network and the Candian Cochrane Centre. ResearchtoPolicy.ca talks about a policy decision-making program.

New evidence raises doubts about teaching traditions.

New evidence is sparking debate about teaching: Do we need to accommodate learning styles, and should we introduce math to toddlers? First, let's talk about the sacred cow we know as learning styles: The Chronicle of Higher Education discusses controversial new research - anyone who has studied to be an educator has probably been told they should match teaching techniques to student styles. "For example, kinesthetic learners - students who learn best through hands-on activities - are said to do better in classes that feature plenty of experiments, while verbal learners are said to do worse. Now four psychologists argue that... [t]here is no strong scientific evidence to support the 'matching' idea.... And there is absolutely no reason for professors to adopt it in the classroom.

"'We were startled to find that there is so much research published on learning styles, but that so little of the research used experimental designs that had the potential to provide decisive evidence,' says Harold E. Pashler, a professor of psychology at the University of California at San Diego.... 'Lots of people are selling tests and programs for customizing education that completely lack the kind of experimental evidence that you would expect for a drug,' Pashler says. 'Now maybe the FDA model isn't always appropriate for education — but that's a conversation we need to have.'"

"Advocates of learning styles respond that Mr. Pashler is the one who lacks evidence. Robert J. Sternberg, dean of arts and sciences at Tufts University and a psychologist who has done a lot of work on learning styles, says in an e-mail message to The Chronicle that the researchers did not fully survey the scholarly literature, and thus 'come across looking either biased about or largely ignorant of the field.'" Okay, so they didn't include a comprehensive literature survey - but where are the rebuttals to their specific findings, such as those charted above?

My two cents. I always figured it was unrealistic to expect busy educators to routinely tailor lesson plans to individual student styles -- I've done my share of teaching, and it seems the best thing is to employ a variety of methods, to cover the range of learning styles and hopefully keep the students more engaged. As Brain Rules puts it, we don't pay attention to things that are boring.

Show me the evidence. Kudos to David Glenn at the Chronicle of Higher Education for identifying not only the source of the published research paper (Psychological Science in the Public Interest), and its name (Learning Styles: Concepts and Evidence [pdf]), but for also linking to it. The research abstract says "For more than three decades, the idea that instructional methods should match a student’s particular learning style has been a powerful influence in education.... In a new assessment of the available evidence, authors Harold Pashler, Mark McDaniel, Doug Rohrer, and Robert Bjork conclude that the learning-styles hypothesis has little, if any, empirical grounding. A large number of studies have purported to show that different kinds of learners (such as 'auditory learners' and 'visual learners') learn best when taught in their preferred modality; but the majority of such studies have not used the type of randomized research designs (e.g., classify learners into categories, then randomly assign the learners to use one of several different learning methods and assess effectiveness of the learning methods with a test given to all participants) that would make their findings credible. What psychological evidence does show is that people are inclined to hold false beliefs about how they learn and that they tend to learn and teach others in nonoptimal ways. Among other things, the report has significant implications for instructional approaches, and underscores the need to ensure that teaching methods are informed by sound scientific research, not fad educational theories or intuition."

Math for four-year-olds? A recent New York Times story talks about educators who are questioning conventional ideas about teaching math: Studying Young Minds, and How to Teach Them says educators and scientists "have believed that children could not learn math at all before the age of five, that their brains simply were not ready.But recent research has turned that assumption on its head - that, and a host of other conventional wisdom about geometry, reading, language and self-control in class. The findings, mostly from a branch of research called cognitive neuroscience, are helping to clarify when young brains are best able to grasp fundamental concepts."

Great information. But true to NYT tradition, the article doesn't tell us where to look if we want to see the evidence for ourselves. For example, it says "In one recent study, for instance, researchers found that most entering preschoolers could perform rudimentary division, by distributing candies among two or three play animals." (Which study?) And "This relationship is new and still awkward, experts say, and there is more hyperbole than evidence surrounding many 'brain-based' commercial products on the market. But there are others, like an early math program taught in Buffalo schools, that have a track record." Can we get the name of those experts? The article does quote some educators, but doesn't link them specifically to this statement.

Doctor's Orders: The challenges of evidence-based medicine. And the difficulty of measuring it.

Doctor's Orders is a collaboration between MedPageToday and ABC. In a recent feature, Practicing Evidence-Based Medicine Is a Challenge, physicians discussed some of the hurdles. They also discuss the substantial difficulties of really knowing how many diagnoses aren't in keeping with current evidence. "With the amount of research being published in medical journals and presented at meetings, it should not be surprising when a new finding slips by a busy physician. Nor should it be surprising, then, that some decisions about patient care might be made without benefit of the most recent evidence.

"Although experts interviewed by MedPage Today agreed that keeping up with the most current information is challenging, it's unclear exactly how widespread the phenomenon of the outdated doctor is. 'To some degree or another, I think it's very widespread,' said Richard Deyo, MD, MPH, a professor of evidence-based family medicine at Oregon Health & Science University in Portland. But he added that it's not a black-and-white issue, because physicians can be up to date in one area and lagging behind in another. 'I think we all are sort of somewhere along a continuum,' he said."

"Lori Heim, MD, president of the American Academy of Family Physicians, agreed that it's difficult to put a solid number on how many doctors are practicing outdated medicine. She said a good place to start would be with the numerous studies that have found that many patients do not receive recommended care for various conditions. One such study, released in the New England Journal of Medicine in 2003, reviewed the care received by surveyed adults in the two years preceding a telephone interview. A review of their medical records found that only 54.9% of the time did they get the care recommended for their condition.

How do you really know? "However, Heim said, one can't conclude from that data that the other 45.1% of the care was delivered by doctors who were not up to date on the most recent evidence. Perhaps, for instance, a diabetic patient was scheduled to come in for hemoglobin A1c screening but missed the appointment. That might have been listed as a failure to get the recommended care, said Heim, a hospitalist at Scotland Memorial Hospital in Laurinburg, N.C."

When evidence slaps you in the face: Bad predictions.

Happy Fun-with-Evidence Friday. I often wonder why people make bold, public predictions. Aren't they just asking for folks to gather up some evidence and prove them wrong? The sites WrongTomorrow.com and BusinessWeek.com are doing some of that taunting.

Wrong Tomorrow is a clever site that tracks how well predictions survive the test of time. They monitor when a particular prediction will be testable and then label it right or wrong. Let this be a lesson for all of us next time we start to open our mouths.

Business Week has a good piece, The Worst Predictions About 2009. One that might have seemed safe until very recently is: "Tiger never does anything that would make him look ridiculous," appearing, sadly, in the brand-new January 2010 Golf Digest.

Only because it's Friday. You know celebrity gossip isn't my thing. But while we're on the subject of Golf Digest and TigerGate, I'll mention the new Wall Street Journal story "How Tiger Protected His Image" which claims the golfer agreed to an interview and cover photo in Men's Fitness magazine -- in exchange for the National Enquirer tabloid squelching a story about infidelity. This wasn't recently: It was in 2007. Even though Golf Digest "spent as much as $1 million annually on donations to the Tiger Woods Foundation, printing the charity's annual report and sponsoring many of Mr. Woods's preferred tournaments.... In return, Mr. Woods agreed to contribute monthly articles on golf techniques and limit his appearances in competing publications." (The story appears on page W6 of today's Wall Street Journal print edition.) Adding insult to injury, Woods appears on the cover of the new Golf Digest. Oy.

Health News Review looks at how evidence is presented in the media.

HealthNewsReview.org provides independent, expert reviews of news stories. Their mission it to "hold health and medical journalism accountable". The project is led by Gary Schwitzer (a journalism prof) and funded by the Foundation for Informed Medical Decision Making.

They clearly identify their 10 review criteria -- based on what they believe consumers need from stories about medical tests, procedures, and products. Kudos to them: "Criterion #4: Does the story seem to grasp the quality of the evidence? ...the relevant standard is that the journalist is expected to present information about the quality of the clinical evidence on which claims are based. In the case of a major claim about the efficacy of a new treatment this should be based on the results of a randomized trial with relevant clinical endpoints.

"Sometimes, the first story about a new treatment is in the form of promising results.... nothing intrinsically wrong with writing about this, but the story should make clear the limitations of the evidence, and contain a caution about interpretation of uncontrolled data. For example, a story about a non-randomized cohort or observational study should explain that researchers were not able to adjust for all factors that might account for an observed difference. The hierarchy of evidence is an important factor for journalists to consider, and to explain to readers and viewers...."

I'm especially impressed by Health News Review's apparent lack of anti-industry bias. They remind us that "Just because a study was funded by industry, it doesn't mean the findings lack integrity."

Other points they make about reporting the evidence: "We all should realize that not all studies are equal. Not all evidence is bullet-proof. There is not a certainty or a finality to everything that appears in the New England Journal of Medicine or any other journal.... But often, just because it's a study - or just because it's published somewhere - seems to suggest that the proof is in. The story is over. And it almost never is. That's not the way science works.... Before you jump to conclusions about research results after hearing some publicity, ask yourself:

-Is there any information on the limitations of the evidence?

-Was the study done in only a few people?

-Did the study report on an outcome that you really care about - like illness or death? Or did it only report on test results, markers, or scores?

-Did the information simply present anecdotes as evidence of a treatment's harms or benefits - rather than real numbers from the entire study group?"

They also provide tips on understanding studies to help journalists decipher things like absolute-vs.-relative risk and number needed to treat (NNT).

How to get evidence that search-engine optimization (SEO) is working.

Search engine optimization (SEO) is a way to improve traffic to a web site through organic (unpaid) search results. You can spend a lot or a little on SEO: Buying links to your site, tweaking content, etc., -- some methods are legitimate, and some might actually be harmful. So when you hire an SEO consultant, how do you know you're getting your money's worth?

The nature of online marketing makes it easy to collect volumes of data, count clicks, etc. The challenge is to look at the *right* evidence. Eric Enge of Stone Temple Consulting explains in Measuring SEO Results (on Search Engine Watch) that people look at several different metrics to assess SEO success: These include:

• Ranking reports: "[t]his data is deceptive. These... exercises always focus on high volume terms, the ones that make up 10 percent or less of total search volume."
• Number of links added: "[W]e remind clients that one link can be worth 100,000 times more than another. It's certainly interesting to track the links added, but only if you include a measurement of link quality...."
• Non-branded search traffic: A great metric. "Focus on daily unique visitors and chart that over time. [F]actor out branded search traffic because few Web sites need SEO help to rank for their brand name."
• Visitor engagement: "bounce rate, time on site, and page views per visitor are all useful...."
• Conversion: "This is the most important thing to measure." Useful evidence includes increases in product sales, 'contact us' requests, completed forms, and whitepaper or widget downloads.

Wisely, Enge reminds us to identify what our goals are before we attempt to measure whether we achieved them.