Evidence Soup

Which is better evidence: Traditional TV viewer ratings, or measures of online engagement with the silent majority?

Now that people view, critique, and chat about TV shows online, traditional Nielsen viewer numbers probably don't provide enough evidence to satisfy advertisers. Networked Insights is collecting evidence differently -- they call it measuring the social, which includes evaluating the activities of a so-called online silent majority.

According to the company, "We set out to measure the social for ratings of network TV shows in comparison to Nielsen’s age-old weekly top 10 TV ratings.... ratings have largely defined where and how much money advertisers spend, but as more and more ad dollars continue to shift online, these measurements are becoming irrelevant...." They're attempting to measure "the entire online audience, including those who post and those who interact by reading, rating, sharing, linking and inviting. Equally important, we also take into account the influence of each interaction — if Tiger woods shares a golf video with some friends, that interaction carries far more weight than you or me doing the same."

Based on this approach, some popular shows (including Desperate Housewives and The Office) appear in Nielsen's Top 10, but not in Networked Insights' Top 10. (However, I interpret the appearance of 'Two and a Half Men' on *both* lists as hard evidence that the world will soon be coming to an end. But I digress.) More info at SocialSights.com. TechCrunch covered this also.

Chunk It! offers another way to search for evidence.

As I've been writing about recently, I believe a plain-vanilla list of search results is inadequate for professionals. Chunk It! is a web browser plugin offering something different. It doesn't look like it's going to be the answer to all my prayers, but I think it's a step in the right direction.

Chunk It! says it "searches the contents hidden behind the links on any Web page to extract the valuable 'chunks' of information that really matter." In other words, they present not only a results page listing relevant links -- they also present your search terms in the context of the relevant site, and show you "chunks" of content quoted from those sites. It saves you the hassle of going to sites individually to see what's actually there.

A Read Write Web reviewer found that "Having ChunkIt display all the instances of a keyword in their context does indeed safe [sic] you from having to click through to all the top search results. Often, if you are just looking for a specific fact, the 'chunks' in the sidebar will already give you all the information you need without having to visit any other site."

I tried Chunk It! (it requires a downloaded browser toolbar). As you can see below, when I searched on "employee satisfaction," the left side of my screen showed relevant links along with paragraphs quoted  from the linked sites. You can hide or show the quoted chunks from each linked site. (The right side of my screen showed standard Google search results for "employee satisfaction".)

Use analytics to achieve data-driven, evidence-based marketing.

Eric Enge looks at some of the challenges of using analytics in his article Building a Data-Driven Organization. Even though he's talking specifically about search marketing, many of his observations apply to any effort to support evidence-based management with analytics.

A sophisticated analytics program can get complicated, take a lot of thought, and force people to change established processes. But if done right, it's worth it. Enge sums up the requirements this way:

1. "Having a voracious appetite.
2. "Putting in place the systems and technology to collect the data on an ongoing basis.
3. "Hiring the people who can decide what metrics are important, gathering the data, and then analyzing it.
4. "Being prepared to act on what you learn, even if it means changes in organizational policies or structure."

To avoid becoming overwhelmed by all the data, Enge reminds us to focus on things that are meaningful and actionable. He also points out the importance (and difficulty) of connecting the dots, rather than looking at individual numbers. Here's a good example from a search engine marketing campaign:

"An increasing number of people who buy online do multiple searches before buying. They may come to your site on a generic term, such as 'digital cameras,' but only make the purchase at a later time after searching on a very specific term, such as 'Canon PowerShot A590 IS'. Your evaluation of... 'digital cameras' in your [pay-per-click] PPC campaign may show that it's losing money. So you may lower the bid, but end up lowering your profits at the same time. Why? Because the search on Canon PowerShot A590 IS would never have taken place if the search on digital cameras had not taken place first. If you weren't there on the term digital cameras you wouldn't have gotten the sale."

What evidence suggests that performance reviews are worth keeping?

I was intrigued by Monday's Wall Street Journal article Get Rid of the Performance Review! (behind paywall).

Samuel A. Culbert says "To my way of thinking, a one-side-accountable, boss-administered review is little more than a dysfunctional pretense. It's a negative to corporate performance, an obstacle to straight-talk relationships, and a prime cause of low morale at work. Even the mere knowledge that such an event will take place damages daily communications and teamwork."

I agree. I've worked for some well-managed companies where I felt appreciated, challenged, part of a team -- all that good stuff. But when review time came around, it seemed demeaning: Even good managers talked about small stuff that happened months before. Although we did discuss how we might make more good things happen, those conversations were not part of the review process. Whether it's positive reinforcement or not-so-much, why not address things when they happen, without making a big ceremony out of a formal review? The best time to intervene and improve someone's performance is when the performance is (or isn't) happening.

Culbert offers seven reasons why reviews are harmful, including:

• Performance Doesn't Determine Pay: Pretending an individual's review has a lot to do with their compensation is ridiculous.
• Objectivity is Subjective: He says "The absurdity is even more obvious when bosses -- as they so often do -- base their reviews on anonymous feedback received from others. This illogic is highlighted in the contemporary performance-reviewing fad called '360-degree feedback.' Hate mail, I suppose, is similarly 'objective.'"

What does Culbert recommend? "Performance previews instead of reviews. In contrast to one-side-accountable reviews, performance previews are reciprocally accountable discussions about how boss and employee are going to work together even more effectively than they did in the past. Previews weld fates together. The boss's skin is now in the game."

Survey says: If the comments on Monday's article are representative of people's experiences, then the performance review should be dead and gone: The only commenter supporting reviews said they "make sure the employee knows what is important to his development and advancement in the organization". Well, of course -- but that could be done in a more collaborative setting. (The lone supporter also called Culbert an idiot... just the kind of person I'd want doing my performance review.)

As I said in my post yesterday, if HR execs are going to run with the big dogs, they need to demonstrate that they can, and do, practice evidence-based management. There's substantial evidence that performance reviews are painful at best -- and that superior alternatives for communication and performance management are available.

Does the evidence show that Swoop-and-Poop managers cause heart disease?

You know the type: Stays uninvolved while others do the heavy lifting, then swoops in at the last minute to deposit some "management input". This leads to chaos and resentment -- and possibly heart disease, according to the Talent Management article Bad Bosses Do More Than Kill Productivity -They Can Kill Employees. Travis Bradberry recommends some ways to avoid disruptions caused by so-called seagull managers: "Each manager must discover when and where he or she succumbs to seagull tendencies. Every single one of us is a seagull manager sometimes, in some situations, with some people." I think that's a great point.

Show us the evidence (or link to it, at least). To underscore the hazards of Swoop-and-Poop management, Bradberry talks about "researchers from the Finnish Institute of Occupational Health" who "followed British civil servants for 15 years to see if the type of boss one works for has any impact on long-term, physical health. ...The team from Helsinki found seagull-type managerial behaviors lead to a much higher incidence of employee coronary heart disease. In fact, employees working for a seagull manager were 30 percent more likely to develop coronary heart disease than those who do not. What's more, the incidence of coronary heart disease — the No. 1 killer in Western societies — was measured after the researchers had removed the influence of typical risk factors, such as age, ethnicity, marital status, educational attainment, socioeconomic position, cholesterol level, obesity, hypertension, smoking, alcohol consumption and physical activity."

Seriously? But Bradberry doesn't reference the specific research, or provide a link to follow. He interprets the findings for us, and that's where I have a problem with the article. We're told there's evidence that "seagull-type managerial behaviors" cause coronary disease -- but I'd be really, really surprised if "seagull manager" was one of the variables identified in the study. I would like to see for myself how the research was conducted.

Run with the big dogs. Human resources executives often admit they don't always have a seat at the executive table, and want to be more involved in strategic decision-making. They could advance their cause by demonstrating that they can, and do, practice evidence-based management. Alas, Bradberry isn't setting a very good example (though he could have without making his article look like a thesis).

Is this it? I looked for the source of this evidence using Infovell, the research site I recently recommended. I found one potential match, the 2007 BMJ article Social inequalities in self reported health in early old age: follow-up of prospective cohort study. It looks at civil servants over 19 years, but focuses on health differences among pay grades -- so maybe it's not the research Bradberry talked about. [FYI: The authors reported "RESULTS: Physical health deteriorated more rapidly with age among men and women from the lower occupational grades." Authors: Chandola, Ferrie, Sacker, & Marmot. Citation: BMJ (Clinical research ed.) , Volume 334 (7601) :( 990 ).]

Multiple-Frequency-Technology watchmaker says "The best evidence is you."

Happy Fun-with-Evidence Friday. In anticipation of the holiday shopping season, lately I'm seeing lots of advertising for high-priced watches. This week I received a catalog from Neiman Marcus saying "Discover how people worldwide improve their 'everyday' by experiencing natural frequency-based technology exclusively available in Philip Stein timepieces.... The best evidence is you."

Looking at the Philip Stein site, here's what I learned: The company "intertwines well-being and beauty with integrated technologies that help balance the body’s energy flow, also referred to as 'Chi'....  Philip Stein® wearers report better sleep, less stress, better concentration, increased performance and energy level, quicker recovery from workouts and less effects from jet lag. Philip Stein® frequency-based technology helps your body return to its most natural and interference-free state.... Every Philip Stein timepiece has integrated active TESLAR technology designed to reinforce the human body's electromagnetic field with a natural earth signal associated with calm, meditation, relaxation, and enhanced performance."

Who knows? Maybe they're onto something -- and if all else fails, there's always the placebo effect. The company was covered in the Miami Herald article Chichi Watches to Balance Your Chi.

An aside. The past few years, I've noticed luxury retailers referring to watches as "Timepieces". I wonder if that's very search-engine-optimization friendly. I see that some of them are back to calling them watches.

X-Ray glasses can detect a blogger's sources of evidence.

Well, not exactly. But I still like what they're doing with a project called memeorandum colors: They are evaluating bloggers' linking activity to detect what sources are being referenced. In this case, they're evaluating U.S.-related political blogs. I'd like to see this approach applied to other web pages to evaluate trends in linking activity -- this could be an interesting indicator when people are offering evidence.

Natch, they highlight liberal references in blue and conservative in red, which means a liberal (blue) blog that tends to reference conservative sources will appear red. The magic happens using a GreaseMonkey script -- I installed it, and got the above results at the Memeorandum site.

The script evaluates blogs that are aggregated on the Memeorandum political site. Joshua Schachter, one of the project developers, previously created delicious, one of my all-time favorite web applications (and something I can't imagine living without). Read/Write Web wrote about Memeorandum Colors here.

Happy Fun-with-Evidence Friday, everybody!

Open letter to the Information Overload Research Group: Here's how to remove some barriers to evidence-based management.

I'm thinking about joining the Information Overload Research Group. BigCo's founded it (Microsoft, IBM, Google, Intel), but now individuals and smaller companies are also members. IORG describes itself as "a group of industry practitioners, academic researchers, and consultants dedicated to reducing information overload, a problem which diminishes the productivity and quality of life of knowledge workers worldwide." The group held a conference in July.

IORG's stated mission: "build awareness of the world's greatest challenge to productivity, conduct research, help define best practices, contribute to the creation of solutions, share information and resources, offer guidance and facilitation, and help make the business case for fighting information overload." Information Overload is significant enough to have a Wikipedia page -- it's reportedly a multi-billion dollar problem (though exactly how you quantify its effects, I'm not sure).

Productivity is part of it. How many times have you been forced to take someone's word for it, because you didn't have time to dive deep into the evidence yourself? So I've got nothing against productivity movements like the Getting Things Done phenomenon or Stephen Covey's time-management seminars. But frankly, they're a little light when you're trying to get people to practice evidence-based management.

Information re-design is needed. Besides reducing information overload generally (by filtering communications, for instance), I believe we need to do the following:

• Improve the likelihood that valid, relevant evidence will get to people when they need it. And present it in a way that helps people absorb and interpret it in a reasonable period of time.
• Formally recognize evidence as a specific sub-category of information, and develop ways to identify it so search engines and publishing technology can recognize evidence. For instance, it  could be tagged in prescribed ways so people would know what type of evidence was being presented -- think of it as an Evidence-Based Markup Language. This effort could dovetail with current development of applications for the semantic web. We've seen some progress along these lines with respect to medical evidence -- but more needs to be done, both to support evidence-based medicine and also evidence-based management generally.

Matt Richtel has covered IORG in the New York Times. The group has also been covered by Jonathan Richards in the Times Online.

Why didn't more people act on this evidence?

If you showed your executive management team a chart like this, I'm guessing they might think it was evidence worthy of their attention, possibly representing an opportunity or a threat. And I'd imagine if your CEO showed numbers trending from 2 trillion to over 12 trillion in 17+ years, you might have noticed.

What are these numbers? U.S. Residential Mortgage Debt Outstanding, 1990-2008 Q2. You heard right: Outstanding residential mortgages exploded from $2.13 to $12.91 trillion.

Who put these numbers together? The U.S. Office of Federal Housing Enterprise Oversight. OFHEO's stated mission is to "promote housing and a strong national housing finance system by ensuring the safety and soundness of Fannie Mae (Federal National Mortgage Association) and Freddie Mac (Federal Home Loan Mortgage Corporation). OFHEO works to ensure the capital adequacy and financial safety and soundness of two housing government-sponsored enterprises - Fannie Mae and Freddie Mac." Unless you've been unplugged the last few weeks, you know the rest.

Could evidence-based management have helped avoid our mortgage crisis? Hindsight is 20/20, so it's easy to lay blame now. But even accounting for inflation, was it reasonable that residential mortgage debt grew so rapidly after 1990? Given this evidence, should more people have been sounding an alarm? Or is the lending system so complex that the evidence wasn't enough to inspire people to find  a proactive solution?

Try Infovell when digging deep for evidence.

When you're looking for basic info, standard web search results are fine. But searching the web at work, we (researchers, writers, managers, and other professionals) deserve better, both in terms of what information is searched, and how the results are presented. I've been experimenting with Infovell, a site with a unique take on searching scholarly information (they currently offer free trials). Although Infovell now emphasizes life science and medicine, I believe their approach could include other relevant information on the so-called deep web. This screen shot is an example of how they're different: As described below, I've done some testing on my own.

 

Worth a look. I searched Infovell for findings on 'blood pressure fish oil' -- and was quite impressed. As these screen shots show, rather than return an ordinary list of links, their site:

1. Allows users to create folders and save searches,
2. Provides a visual indicator showing the source of the information (such as the journal name), and
3. Groups results by data source (such as "Clinical Trials").

What are others saying? According to EContent, "Infovell’s new research engine is based on its KeyPhrase algorithm, which indexes words and phrases. The technology works in any language and can handle queries up to 25,000 characters. Infovell also comes with filtering and visualization features." Read/Write Web described Infovell this way: "The engine scours through open-access repositories of information like PubMed Central and the U.S. Patent and Trademark Office Claims, but it also allows access to scholarly journals such as those from Oxford University Press, SAGE, Taylor & Francis, Annual Reviews, Mary Ann Liebert Publications, and more."

PhysOrg.com offers a good explanation of the basic deep-web problem: "While many popular Web sites are specifically designed to be search-engine friendly, a lot of deep Web content is unstructured, making it difficult for keyword-based search engines to index. Further, the deep Web does not receive much traffic, meaning these pages don´t have many incoming links and therefore aren´t ranked highly by systems such as Google´s PageRank. And for private sites, barriers such as registration and subscription requirements also make it difficult for search engines to access them."

Why hasn't Google done this? Google Scholar offers relevant results, but the presentation isn't very good -- no visual indicators, and just a traditional list of results without sophisticated management such as grouping. Here's what I got for 'blood pressure fish oil':

Compares favorably to the status quo. Infovell saves time by searching lots of publishers at once -- and its display of results is better than what you'll get from a traditional journal. Here's my fish oil* search on the Sage site (which offers free access during October, as I wrote about recently). Note that this result also appeared in my Infovell search, because Sage journals are included.

 

Should information be free? Dean Giustini on the Open Medicine blog expressed enthusiasm for improved search results, but was not thrilled about Infovell's subscription fee. I have no doubt that access to academic research and other peer-reviewed pubs will eventually be free -- it's just a matter of how the business models will change, and when (not sure I'll live long enough to see it!). In the mean time, I expect many people will find that Infovell's subscription fee is worth it.

*Brief tangent: Fish oil offers many benefits (lowers triglycerides, etc.). If you take it, I recommend GNC's Triple Strength Fish Oil because it's highly purified and concentrated.