Evidence Soup

Does the evidence show that Dunbar's number really matters? Or are weak ties just as good as strong ones?

Contributors to the Smart Data Collective forum frequently offer useful insights about analytics and other topics. I liked a recent piece by Jacob Morgan about Dunbar's number: According to Wikipedia, this is a "theoretical cognitive limit to the number of people with whom one can maintain stable social relationships." Dunbar's is typically believed to be about 150. Makes sense to me - especially when I think about the circle of people I stay in regular contact with, invite to parties, interact with on Facebook, etc.

I'm intrigued by social network research: The structural holes stuff, etc. Lots of folks are exploring how people make connections, and what difference these connections (or lack thereof) can make. Of course, I'm most interested in understanding the role of social networks in spreading useful evidence as quickly as possible.

Strong vs. weak. Dunbar's number describes what many folks call "strong ties". Is a strong connection required to get people to consider new evidence, especially when that evidence challenges their current beliefs? Or are weak ties enough?

Morten T. Hansen, in his book Collaboration: How Leaders Avoid the Traps, Create Unity, and Reap Big Results, talks about the importance of weak ties. "[R]esearch shows that weak ties can prove much more helpful in networking, because they form bridges to worlds we do not walk within. Strong ties, on the other hand, tend to be worlds we already know; a good friends often knows many of the same people and things we know." And he offers a pragmatic explanation of why your weak-tie number can go much higher than 150: "It’s less time consuming to talk to someone once a month (weak tie) than twice a week (a strong tie). People can keep up quite a few weak ties without them being a burden."

Authority matters, too. When we consider new evidence, the authority of the source is important - this is a key component of making something persuasive. Technology has made it much easier to learn about new evidence and locate new authorities, even when we have weak - or possibly nonexistent - ties to those sources. (Consider Twitter. Evidence often comes from people you're following, or people who are re-Tweeted, even if you don't know them.)

Beating the number. For those who do want to be strongly connected to as many people as possible, Chris Brogan writes about "beating Dunbar's number" - for instance, by establishing yourself in multiple groups based on specific geography, a specific industry vertical, etc. Some of his points are a different way of saying "it's not who you know, it's who knows you" - and I agree with him on that. Brogan writes: "[M]ake it work in reverse. By that, I mean this: if people think of you as part of their 150, then they come to you, and they seek you out. This comes from having some value to offer. If, for instance, you’re helpful in finding others business, those people will seek you out the next time they have an opportunity, or the next time they need your help. In either case, you’re in the network."

The bottom line? I think Dunbar's number is useful for evaluating a certain type of connection: The ones characterized by strong ties. But for spreading evidence to a wider audience, it's not important. You can successfully communicate evidence to folks with whom you have only weak ties.

By the way, Dunbar has a book called Grooming, Gossip, and the Evolution of Language. I've not read it, but the reviews are quite positive, and I've heard it's a worthwhile read.

How to get evidence that search-engine optimization (SEO) is working.

Search engine optimization (SEO) is a way to improve traffic to a web site through organic (unpaid) search results. You can spend a lot or a little on SEO: Buying links to your site, tweaking content, etc., -- some methods are legitimate, and some might actually be harmful. So when you hire an SEO consultant, how do you know you're getting your money's worth?

The nature of online marketing makes it easy to collect volumes of data, count clicks, etc. The challenge is to look at the *right* evidence. Eric Enge of Stone Temple Consulting explains in Measuring SEO Results (on Search Engine Watch) that people look at several different metrics to assess SEO success: These include:

• Ranking reports: "[t]his data is deceptive. These... exercises always focus on high volume terms, the ones that make up 10 percent or less of total search volume."
• Number of links added: "[W]e remind clients that one link can be worth 100,000 times more than another. It's certainly interesting to track the links added, but only if you include a measurement of link quality...."
• Non-branded search traffic: A great metric. "Focus on daily unique visitors and chart that over time. [F]actor out branded search traffic because few Web sites need SEO help to rank for their brand name."
• Visitor engagement: "bounce rate, time on site, and page views per visitor are all useful...."
• Conversion: "This is the most important thing to measure." Useful evidence includes increases in product sales, 'contact us' requests, completed forms, and whitepaper or widget downloads.

Wisely, Enge reminds us to identify what our goals are before we attempt to measure whether we achieved them.

PR consultancy announces “evidence-based communications” methodology: A scientific approach to delivering client messages.

Burson-Marsteller, a global consultancy, has adopted a research-focused approach they call 'Evidenced-Based Communications' (complete announcement here). This year, the firm has rolled out a methodology for developing and measuring programs. “The media and communications landscape is changing and so have the needs of our clients,” said Burson-Marsteller CEO Mark Penn. “In a world of citizen journalism, social media, and instant information about events happening around the globe, we are investing in a more diligent and scientific approach to developing and delivering key messages.”

It's great to see marketing and media moving in this direction. Here's how the firm describes the benefits (there is, um, a bit of spin applied). "Evidence-based communications ends the guess work. All strategies are based on evidence, not speculation." (Okay. But trying something creative and new involves unknowns. Don't let evidence crowd out big ideas and experimentation.) "It is cost-effective... ensuring that each client’s communications dollars are spent on tactics and messages that will deliver results. It is measurable. By... benchmarking at the beginning of a program and measuring effectiveness at the end, clients will be able to demonstrate a positive communications return-on-investment."

They continue, saying theirs is a holistic approach: "While it is common to use some basic research to drive a communications message or to assess the reach of a program at the end, the Evidence-Based approach is a complete methodology. The approach ensures a thorough use of data and tools designed specifically to insert science into the process where appropriate. It provides proof of PR value to the organization’s C-Suite. By using an Evidence-Based approach, communications professionals can demonstrate the value that PR brings to their organization at large."

I wish them well with their evidence-based methodology.

Center for Health Design wants to accredit people in evidence-based healthcare building design.

Those of us who are advocates of evidence-based ____ understand the difficulty of establishing what's evidence-based and what's not. This happens even in medicine, where evidence-based management is several years ahead of such methods in many other fields. It takes a substantial body of evidence to eventually iron out conflicts in research findings. And by the time that happens, there's new evidence to consider. But regardless how complex the evidence is, it's a good idea to develop guidelines and resources that help people figure out for themselves for what really works and what doesn't: So they can avoid strategies that simply won't move things in the right direction, and be strong advocates for evidence-based management.

Along those lines, there's now an accreditation program for evidence-based design in the healthcare field, which means using evidence to protect patient safety, improve the level of care, and provide good working environments for caregivers. Brought to you by the Center for Health Design, "Evidence-based design (EBD) is the process of basing decisions about the built environment on credible research to achieve the best possible outcomes." The Center is a non-profit organization, so it should be able to function effectively as a clearinghouse for evidence on healthcare building design.

The Evidence-Based Design Accreditation & Certification (EDAC) program "educates and assesses individuals on their understanding of how to base design decisions on available, credible evidence." These professionals "will demonstrate a clear understanding of these components and use an evidence-based design process to meet and/or exceed the recommended minimum requirements. The program's goal is to test on the proper process to identify, hypothesize, implement, gather, and report the data associated with their project. Once an individual is EDAC accredited, he or she will have an ethical obligation to employ an evidence-based design process in his or her work."

Covering the gamut. This had me wondering if they are emphasizing the how of finding and interpreting evidence, or the what contained in the body of evidence (or both). I took a look at their study guides and the content outline describing what's on the certification exam (8-page pdf here). The examination has five sections: evidence-based design for healthcare, research, predesign, design, and construction and occupancy. Here's what the study guides cover:

• Study Guide 1:  An Introduction to Evidence-Based Design: Exploring Healthcare and Design. It "explores the history and evolution of evidence-based design through its present defintion and defines the key steps of the EBD process. Once the context is set, the guide will explore the components of the healthcare delivery system, the trends affecting it and the various settings in which delivery occurs."
• Study Guide 2: Building the Evidence Base: Understanding Research in Healthcare Design. "Guide two explores the role of research in evidence-based design. The types of research and evidence will be discussed as well as methodologies for doing that research. The guide covers the value of using credible evidence in healthcare projects, and how through research design solutions are empirically evaluated and scientific evidence is generated."
• Study Guide 3: Integrating Evidence-Based Design: Practicing the Healthcare Design Process. "Guide three pulls together the EBD process to walk you through the key steps of the design process in detail from pre-design, design, construction and occupancy. Practical examples demonstrate key areas and show how EBD is practically applied."

The content outline covers a wide range of stuff: There's everything from "how evidence-based design can contribute to attributes of a therapeutic environment" to "describe the role of the Board of Directors in understanding and championing evidence-based design processes" -- and lots in-between. Overall, I'd say this definitely demonstrates an understanding of the importance of following the evidence.

Teaching them how to fish? There's no one way to design a medical facility, and the body of evidence will always be changing -- so testing on specific design elements would be tricky. EDAC seems to be focused on the process stuff, so that's good: Their study materials include a section on critical evaluation, saying "Determine the relevance of evidence to the project based on one or a variety of
factors (e.g., sources, author qualifications and / or experience, appropriateness of research methodology, replication, composition of sample, reliability, validity, generalizability)". Excellent.

ECAD is discussed in Evidence-Based Design, a nice article appearing in a publication called NEXT (Spring 2009). (Alas, it's viewable in a particularly un-reader-friendly online magazine format.) The author, C. Richard Hall, says "As awareness increases and the body of evidence grows, more and more people are seeking to build hospitals with the guidance of those skilled in the practice. But up until now, there was no standard definition of evidence-based design." He explains that the EBD discipline has emerged over the past 15 years.

EDAC's stated mission is to "develop a community of accredited industry professionals through education and assessment of an evidence-based design process." And the vision statement says they want to see "a world where all healthcare environments are created using an evidence-based design process."

Getting the word out. Besides this certification process, the Center does a nice job of explaining to healthcare providers why they should pay attention to evidence-based design. Their web site has a section for healthcare leadership, presenting "evidence-based design resources for healthcare executives." This includes a white paper on the role of the CEO in evidence-based design, and guidance on establishing the business case for EBD.

The 'anti-Malcolm Gladwell' speaks: Lack of verifiable evidence tells us the influencer theory is bogus.

Lots of people have questioned whether there is hard evidence to support the influencer theory described in Malcolm Gladwell's The Tipping Point. I wrote about it here and here. One of the best-known critics is Duncan Watts, who was profiled in Fast Company

Brand Week's Todd Wasserman has a new article, Scientist: Influencer Theory is Bogus, covering an interview with Watts, who "first came to the marketing world's attention in 2003, when a New York Times article positioned the Columbia University professor as the anti-Malcolm Gladwell. While Gladwell's 2000 book The Tipping Point laid out an entertaining theory that fads like Hush Puppy shoes ...spread because of groups of people called influencers, Watts has argued that that model is deeply flawed. Watts, now a research scientist with Yahoo, charges that influencers have been poorly defined and that scientific data show you're just as likely to spread a message or product by targeting a random group of consumers as you would by going after so-called influencers. Mostly, Watts says marketers should be much more scientific in their approach, especially as social media grows in importance."

"Brandweek: What's wrong with the influencer model? Duncan Watts: ...when you scrutinize [the claims] carefully, they turn out to not really be very meaningful. Or to put it another way, everyone thinks they know what an influencer is and everyone thinks they know why they matter, but everybody thinks something different. Is an influencer the hipsters in the East Village or Oprah Winfrey? What makes Oprah influential is very different from what makes the hipster in the East Village influential. And so by failing to differentiate carefully between all these different types of influencers you really undermine the ability of the theory to say anything predictive.... Somebody asked the publisher of the surprise bestseller Eats, Shoots & Leaves why that book was so successful and he said it was successful because lots of people bought it. ...Hits are highly unpredictable. It's very difficult to even retrospectively go back and show that there are certain kinds of consistent attributes that result in being popular.

"[Brandweek]: So what would your advice be to a marketer trying to learn from things like that? [Watts]: My first advice would be stop fooling yourself.... Marketers have been chasing influencers for a decade and they haven't found them. And the reason is not that influence doesn't matter. It may very well matter.... The reason is that history is a very poor guide to the future.

"...[T]he whole influencer theory is actually more of a rhetorical device than a theory. It's not like people actually have an explicit theory of who is influential and they go out there and they use that theory to decide who to target. It's more that they do whatever they do. They throw parties or they give away free samples or they advertise in particular publications that brand themselves as reaching an influential audience and run some campaign in their normal manner. And if it works, they say, 'We reached the influencers.' ...In that sense, it's just a rhetorical device to help you explain the randomness that you actually experience in the world.... So it's one of these impossible-to-falsify theories because who you identify as an influencer is always after the fact."

Ski resort wants Colorado tourism officials to show them the evidence.

Evidence-based marketing (or any type of evidence-based management, for that matter) shouldn't happen only in the private sector. So I like what Vail officials are saying about the management of Colorado tourism dollars. According to today's Rocky Mountain News, they want to "hold the state's tourism office more accountable for the marketing money it spends by setting up an industry panel that would advise and oversee the government agency. The state budgets almost $20 million a year for tourism marketing. In a five-page proposal obtained by the Rocky Mountain News, the Broomfield-based company told the Colorado Tourism Office it should be held to 'the same strict return on investment requirements that corporate marketing groups are held to.'"

Good idea. President Obama is pushing for more government transparency. This sounds like a good place to make that happen.

Evidence from functional MRI shows cigarette warning labels make smokers want to light up.

This is one of the findings reported in Martin Lindstrom's new (and brilliantly titled) book, Buyology: Truth and Lies About Why We Buy. The book recaps how functional MRI (fMRI) was used to conduct a multi-year, multi-million dollar "neuromarketing" study. This is a very different type of evidence than I'm familiar with.

Evidence about warning labels. From the book: "In short, the fMRI results showed that cigarette warning labels not only failed to deter smoking, but by activating the nucleus accumbens, it appeared they actually encouraged smokers to light up. We couldn't help but conclude that... [labels] had become a killer marketing tool for the tobacco industry" (page 15). Assuming the evidence is valid, this is one tragically funny public policy lesson.

Evidence re: product placement. The book talks about advertising and product placement on the American Idol show. Coca Cola and Ford (automotive) were two key companies evaluated, and Coke was evidently the winner. Lindstrom claims that people actually remembered *less* about Ford after the study -- saying that the "Coke-saturated show actually suppressed" memories about Ford (page 49). He explains these results by saying that a product needs to play an integral part in the narrative -- you can't just plunk random products on the screen. I like his example about the movie E.T.: "Elliott didn't just pop those Reese's Pieces into his mouth during a thoughtless bike ride; they were an essential part of the storyline..." (page 51). If you're planning to blizzard people with brand after brand, he says you might as well set fire to those advertising dollars.

Is this real? Lindstrom was interviewed on NPR this week. He presents his findings enthusiastically, and I enjoyed listening to him. Typically, when I see someone touting a book about how companies market to the consumer's subconscious, I keep on clicking. But this is interesting. I'll confess, I know next to nothing about fMRI, so don't know if the Buyology conclusions are valid. But it certainly seems plausible.

(Still) searching for evidence of that elusive influencer.

It's doubtful there will ever be consensus on whether the mythical online "influencer" is real. Like many others, I've stated my skepticism about claims made in the Tipping Point book and elsewhere. But I'm becoming convinced there's one important type of influencer, based on a new research report by Rubicon Consulting, which found "Online comments and reviews posted by the enthusiasts are second only to word of mouth as a purchase driver for all web users. Those personal reviews are far more influential than official reviews posted by a website or magazine, or information posted online by a manufacturer."

Does Contributor = Influencer? Evidence in the report supports the common belief that 10% of the people contribute most web content (however, instead of the 90:10 rule I often hear repeated, this research shows 80% of content is produced by 9% of the community). So the question is, are the MFCs (most frequent contributors) also the most important influencers? And how would you go about demonstrating that?

Read/Write Web covered Rubicon's findings here. The report claims that "Small groups of enthusiasts dominate most online conversations, but that doesn’t mean online communities matter only to a narrow segment of people." This data seems to support that.

From the report: "The Rubicon survey documented how the influencer process actually works: The MFCs write most of the online reviews and comments, and other web users rely heavily on those comments when making purchasing decisions. Online comments are the new word of mouth, and MFCs do the most commenting."

Am I under the influence? I don't always look for recommendations -- but at our house, we rarely buy electronics or household appliances without first searching products and checking reviews at amazon. Small, but telling, example: This year we needed a sturdy cover for our new stainless-steel grill. An amazon seller offered one that seemed right -- until I read a negative review, with accompanying photos, showing mesh vents that let the weather in. I found one reviewers liked better, and now we're ready for the snow to fly. (Kudos to amazon for allowing reviewers to include photos and videos.) Shoe reviews at Zappos can also be useful (especially because they *structure* reviews around info like whether a shoe fits true-to-size) -- but amazon has more critical mass to make magic happen. (Zappos is a fantastic company, by the way.)

Apparently I'm somewhat typical: Rubicon's findings show that people are most heavily influenced by online information when buying electronics or cars; planning vacations; looking for jobs; and choosing movies and restaurants. And Rubicon's evidence draws a picture of who is doing the influencing: MFCs are younger, more ethnically diverse, and more technically skilled than "typical" web users.

Recapping Rubicon's claims about its findings: "This means the old idea of 'influencers' is confirmed and explained. The most frequent contributors are the influencers, and they have a strong influence on purchase decisions because they write most of the online recommendations and reviews."

Evidence of political ad effectiveness still hard to find.

Over the last few months in the U.S., some clever people have creatively gathered evidence indicating who was or wasn't ahead in the political races. Nate Silver of FiveThirtyEight.com has done some particularly impressive stuff (GQ has a nice writeup on Nate).

But beyond the polling and statistical analysis, evidence-based management isn't making much headway in the world of politics. A recent On the Media story looked at something I've often wondered about: What evidence is there showing that political advertising really makes a difference? They say "Despite the billions spent on elections, there's little research on how effective commercials, phone calls, or door-to-door efforts are at producing votes. Professor Donald Green, who's conducted hundreds of studies over the past decade, found that advertising and robo-calls do very little but that a novel, if slightly disturbing*, approach actually does increase turnout."

In the radio story, Brooke Gladstone concluded that "Donald Green, director of the Institution for Social and Policy Studies at Yale, has spent years assessing the power of all these methods to both mobilize and persuade voters. Green found that campaigns generally aren't getting much bang from their bucks."

Green said researchers have "studied robo-calls by celebrities, by governors, by local officials, by local clergy, partisan, non-partisan. We just don't find any effect on whether people vote. That we're pretty sure about, that their effects on mobilization are very meager.... Only one randomized study, of which I know, has looked at the effects of TV advertising on vote choice. We did find a momentary effect. The effect lasted a few days, but a week later it was all gone."

*Some would say that nominating Sarah Palin for VP was novel and slightly disturbing. (Sorry, couldn't help myself.)

Which is better evidence: Traditional TV viewer ratings, or measures of online engagement with the silent majority?

Now that people view, critique, and chat about TV shows online, traditional Nielsen viewer numbers probably don't provide enough evidence to satisfy advertisers. Networked Insights is collecting evidence differently -- they call it measuring the social, which includes evaluating the activities of a so-called online silent majority.

According to the company, "We set out to measure the social for ratings of network TV shows in comparison to Nielsen’s age-old weekly top 10 TV ratings.... ratings have largely defined where and how much money advertisers spend, but as more and more ad dollars continue to shift online, these measurements are becoming irrelevant...." They're attempting to measure "the entire online audience, including those who post and those who interact by reading, rating, sharing, linking and inviting. Equally important, we also take into account the influence of each interaction — if Tiger woods shares a golf video with some friends, that interaction carries far more weight than you or me doing the same."

Based on this approach, some popular shows (including Desperate Housewives and The Office) appear in Nielsen's Top 10, but not in Networked Insights' Top 10. (However, I interpret the appearance of 'Two and a Half Men' on *both* lists as hard evidence that the world will soon be coming to an end. But I digress.) More info at SocialSights.com. TechCrunch covered this also.