Evidence Soup

Why do we pay attention to business consultants/gurus?

Why do we hire famous management consultants to drop by and tell us how to run our business? And buy their books by the millions? Some recent articles annoyed me a little, though I'm not sure why. I've wondered before if my annoyance with management gurus results from either crankiness or envy, particularly in the case of Malcolm Gladwell (though I'm not alone, since I received plenty of emails from Evidence Soup readers skeptical of his claims after I covered some researchers who are producing evidence showing that the Tipping Point probably doesn't exist).

Yesterday's Wall Street Journal identified the most influential U.S. business thinkers/gurus (ranked based on Google search results, media mentions in LexisNexis, and academic citations -- read about their ranking methodology here). The Top 5 are Gary Hamel, Thomas Friedman, Malcolm Gladwell, Bill Gates, and Howard Gardner. (Robert Reich was No. 6 -- one of my friends in grad school called him a "policy hustler". Heh.) WSJ published two articles: New Breed of Business Gurus Rises and also Quest for Innovation, Motivation Inspires the Gurus. At least they didn't imply that these guys are above criticism, describing one of them this way: "Gary Hamel, 53 years old, is a prolific writer and speaker who has kept his popularity despite falling victim to a peril of gurudom: His 2000 book, "Leading the Revolution," lionized Enron Corp. (The Enron case study was removed from later editions.)" Now some are chipping away at Friedman's world-is-flat theory, claiming that nationalism is interfering: "Governments, as the story goes, are reasserting their role in the lives of individuals and businesses, causing barriers to rise. If true, that trend counters a premise of New York Times columnist Thomas Friedman's popular book, The World Is Flat, that the Internet and other types of IT are wiping away national borders."

So. What is it that bugs me? I've read too many business books and sat through too many keynotes to critique them individually here. Generally speaking, though, I've found that the advice of many guru types exhibits these flaws:

• They're not specific enough, or comprehensive enough. A few well-phrased anecdotes don't cut it when providing prescriptive, actionable advice.
• I always suspect they chose their examples conveniently, preferring the ones that support their "frameworks" (a problematic bugaboo of qualitative research).
• Rarely do they expose what types of evidence they used, tell us what variables were considered, describe how they arrived at their conclusions, and let us critique their methodology.
• Seldom do they have accomplishments of their own to examine and draw from (Bill Gates and Carly Fiorina being notable exceptions).

How do they command those big bucks? Sometimes an event organizer wants a well-known keynote speaker; nothing wrong with that. Another explanation for the popularity of some of these gurus is that "People have a natural desire to look for some kind of framework or a way of explaining what's going on." Well, no sh*t. I'm thinking people would be better off if they enrolled in some philosophy, business, liberal arts, and public policy courses and came up with a framework for themselves. But I suppose it's tempting to simply pay for a spoon-fed framework that can be cited to colleagues at key moments in meetings... I can almost see them rolling their eyes from here. Woops, there I go again, being cranky.

How can you encourage people to look at the evidence?

Those of us who are already believers can talk endlessly about the virtues of evidence-based management, decision analysis, data mining, etc. But how do we shake the trees and encourage more people to look rigorously at causes and effects, paying close attention to research findings and other available evidence? One way is to bring folks together for a seminar: By engaging an entire group, you can establish new ground rules for how people should assess problems and defend their recommended solutions -- without pointing a finger at the habits (or analytical shortcomings) of any particular individual.

HeadScratchers is a firm that helps people apply critical thinking techniques to business problem-solving, and I was pleased to see they are now offering open-enrollment workshops. I know Mike Kallett, the guiding force behind HeadScratchers, and can vouch for him as a helpful, thoughtful guy who has survived some tough business and product development challenges. Here are some of the workshop topics covered by HeadScratchers:

• HeadScratchers Critical Thinking Model.
• Elements and Tools of thinking, problem solving and decision making; with exercises.
• Critical Thinking Techniques and Templates with interactive exercises.
• Specific examples and exercises using real business issues from the attendees.

Oops, new evidence says maybe we've been teaching math wrong.

A train is traveling west at 65 mph, while another travels east at... are you as bored as I am? So-called "story" problems are often a painful experience for the math-phobic. I happen to enjoy math, and enjoy teaching math -- but most folks don't. (I can sympathize: My eye-hand coordination is pathetic, to the point where I was terrified to play even right field in elementary-school softball.)

Now, new evidence shows that our efforts to demystify math by providing real-world examples may be counterproductive -- and this may have implications for how we teach finance, etc. to MBA students and to business professionals in the workplace. The current issue of Science reports on a recent study: The Advantage of Abstract Examples in Learning Math presents some surprising findings, concluding that "Undergraduate students may benefit more from learning mathematics through a single abstract, symbolic representation than from learning multiple concrete examples." The authors conducted a randomized, controlled study -- something you don't often see in educational research.

The New York Times discussed the findings in Study Suggests Math Teachers Scrap Balls and Slices: "Entranced, perhaps, by those infamous hypothetical trains, many educators in recent years have incorporated more and more examples from the real world to teach abstract concepts. The idea is that making math more relevant makes it easier to learn. That idea may be wrong, if researchers at Ohio State University are correct. An experiment by the researchers suggests that it might be better to let the apples, oranges and locomotives stay in the real world and, in the classroom, to focus on abstract equations, in this case 40 (t + 1) = 400 - 50t, where t is the travel time in hours of the second train."

The Times explained that college students "learned a simple but unfamiliar mathematical system, essentially a set of rules. Some learned the system through purely abstract symbols, and others learned it through concrete examples like combining liquids in measuring cups and tennis balls in a container. Then the students were tested on a different situation — what they were told was a children’s game — that used the same math. 'We told students you can use the knowledge you just acquired to figure out these rules of the game,' Dr. Kaminski said. The students who learned the math abstractly did well with figuring out the rules of the game. Those who had learned through examples using measuring cups or tennis balls performed little better than might be expected if they were simply guessing."

What a relief! I've studied lots of math, and methodologies for teaching math, and am certified as a secondary mathematics teacher. Despite my scorn for the dreaded "two trains" algebra problem, being who I am, with lots of business experience, I am interested in real-world examples, especially ones that make something abstract more visual -- but students connect with examples from their daily lives, not sterile textbook problems. I've personally witnessed students struggling with algebra, expressing frustration over "Why are we talking about trains? And now pizza slices?" As one of the study authors pointed out, it's a belief, not an established principle, that concrete examples are more beneficial. Now they may be leading educators (including me) down a new path of evidence-based enlightenment.

Should we be worrying about Peak Water? Or which water to pair with the tilapia tonight?

A couple of years ago I attended the 4th World Water Forum in Mexico City, and it was eye-opening. Since today is Earth Day, I've once again been thinking about water -- and yikes, there are lots of things to think about. Here's a rundown on a few of them:

Bottled water: Today a story on The World (my favorite radio program, BTW) said we now drink more bottled water than milk or beer. But lots of energy, plastic, and other resources are required to bottle and transport all that stuff -- even though most people in the U.S. don't need it. As The World reported, "Drink Fiji water in Los Angeles, and the bottle travels around 9,000 miles. Drink it in New York, and it's well over 11,000 miles." So Fiji Inc. is attempting to make their product "carbon negative" by planting trees, intending to negate at least 120% of the carbon that the company contributes to the atmosphere. (Luckily, I can be like the Tap Water Girl in their story, because my city has delicious water right from the tap -- though I'll confess to keeping a six-pack of Arrowhead in the car, 'cause I gotta have water with me anywhere I go.)

Fine water: I've got nothing against luxury products. But now "fine" water is all the rage, and some fancy schmancy restaurants actually have water sommeliers. Yes, I said water sommeliers. You can even buy a book about it, such as Fine Waters: A Connoisseur's Guide to the World's Most Distinctive Bottled Waters. "The key to a good water sommelier isn't knowing the chemical detail of every water across the world.  Not a science, but an art, being a water sommelier is purely based on pairing water with food. And although it may be new to most, these water-passionate men and women know that when the right water is paired with the right cuisine, the entire dining experience can be more enjoyable." Oh, please.  If we must drink bottled water (and apparently we do), I suppose the environmental lesson for today is to drink one that's locally produced.

Infrastructure: In Nor Any Drop to Drink, Lee Bruno reports at Always On that 25-30% of drinking water is lost to leaks. 

Evidence: As explained in the May issue of Wired, "One barrier to better management of water resources is simply lack of data — where the water is, where it's going, how much is being used and for what purposes, how much might be saved by doing things differently. In this way, the water problem is largely an information problem. The information we can assemble has a huge bearing on how we cope with a world at peak water. [But] That data already shows the era of easy water is ending. Even economically advanced regions face unavoidable pressures — on their industrial output, the quality of life in their cities, their food supply."

Stationarity: The latest Newsweek writes about an impending water crisis, citing the new book Common Wealth: Economics for a Crowded Planet by Jeffery Sachs. "Economists and geologists have identified one culprit in the water-management problem, a mind-set they call "stationarity"—the belief that natural systems fluctuate within a narrow, predictable range, even over long periods. 'Stationarity is dead,' says Chris Milly, author of a recent Science paper on the issue—done in by population growth, climate change and economic development. But the effect of the stationarity fallacy has been to leave water policy in the hands of relatively shortsighted municipal and state authorities, while the federal government has been looking the other way." But besides long-term solutions, Sachs says there are straightforward things we can do now to improve our water supply -- for instance, store rainwater in ponds for irrigation during dry spells, recycle wastewater, and manually replenish natural underground aquifers with treated wastewater or storm runoff.

Peak water: A Wired article on so-called Peak Water (analagous to Peak Oil) says the problems are real: "Call it peak water, the point at which the renewable supply is forever outstripped by unquenchable demand. This is not to say the world is running out of water. The same amount exists on Earth today as millions of years ago — roughly 360 quintillion gallons. ...But 97 percent of it is in the oceans, where it's useless unless the salt can be removed — a process that consumes enormous quantities of energy. [...And] Like oil, water is not equitably distributed or respectful of political boundaries; about 50 percent of the world's freshwater lies in a half-dozen lucky countries."

Commonality: Now that Mother Earth is getting more attention, I hope it's a reminder to all of us that we do have something -- something incredibly important -- in common.

What are the costs and benefits of cost-benefit analysis?

In the early '90s, I was helping some Fortune 500 companies with U.S. regulatory compliance, and I wanted to understand the rationale, motivation, information flow, and review process associated with various federal rules. I knew that major new regulations had to be supported by a cost-benefit analysis -- but I didn't know how, if, or when it really factored into final rulemaking decisions. So I included this in the scope of my dissertation research: I was looking at policy discourse, specifically how decision-makers find, transfer, and use expert knowledge that is fragmented and often contradictory.

Huh? Why this unscheduled trip through the wayback machine, you ask? Because I stumbled upon The Myth of Cost-Benefit Analysis in Booz Allen Hamilton's journal, strategy+business. Denise Caruso outlines some problems with cost-benefit analysis and suggests alternative processes. (Caruso heads the Hybrid Vigor Institute in San Francisco, which supports interdisciplinary collaboration on science, technology, and social issues. Amen to that.) The article is worthwhile reading, touching on ideas relevant to business managers, even if they're not involved in public deliberations.

[benefits] - [costs] = ? It sounds tempting: Systematically weighing the costs of doing something against its potential benefits. If you're gung ho for evidence-based management, what's not to like? When Reagan signed Executive Order #12291 in 1981, it was supposed to help depoliticize an inherently political process by requiring decision-makers to use objective, rational analysis: A cost-benefit study was required for any regulation with a potential economic impact of $100+ million. (President Clinton authorized a similar requirement in 1993, as Executive Order 12866.)

It doesn't count if it can't be counted. Cost-benefit analysis is often institutionalized because it lends itself to widespread use more easily than many other decision-making tools: It requires that everything -- even value judgments -- somehow be expressed in economic terms. That can be difficult and time-consuming to do, and the results can be misleading. I believe one of the best reasons to do this type of study is to force people to be more disciplined about identifying specific impacts and costs -- typically, they'll discover unanticipated new knowledge as they grind through the details. But among the challenges are:

• Deciding what evidence counts: Caruso's introduction provides an example from US EPA regulations limiting mercury emissions from coal-burning power plants. One study predicted a health benefit of $50 million/year, while another estimated annual public health benefits of $5 billion. Why the difference? One group considered only fresh-water fish, while the other believed it was necessary to factor in potential mercury effects on salt-water fish, a much larger food source.
• Deciding who counts: What do we mean when we say "the public"? Do some people count more than others (e.g., children)? How are we supposed to consider impacts on our nearby neighbors (Canadians, etc.), or people around the globe (such as with global warming)?
• Deciding how to count: Since we're expressing everything in economic terms, do we have to assign a dollar value to human or animal lives that may be lost (or the quality of those lives)?
• Deciding which side effects to consider: There always will be side effects -- good and bad, some expected and some not -- that don't get counted. (My favorite book on this topic is Why Things Bite Back: Technology and the Revenge of Unintended Consequences by Edward Tenner.)
• Deciding how to weigh risks: It's difficult enough to do this when the outcomes of a recommended action are easily predicted. But oftentimes they're not, and we find ourselves applying all kinds of fudge factors to account for uncertainty.

What does this have to do with evidence-based management? This has everything to do with it because the analytical challenges are similar. For example, imagine you're defending a promotional marketing program, attempting to show it's worth more than it's costing: You have to convince people what evidence counts, and figure out how to count some fuzzy, qualitative stuff (even if you don't have to worry about how much mercury is in the tuna salad). Managers have a choice: They can insist on strict adherence to rigid, somewhat simplistic tools like cost-benefit and ROI. Or they can choose to be more entrepreneurial, collaborative, and innovative by embracing a variety of evidence-gathering techniques, without insisting that everything be reduced to dollars. I believe methodologies like cost-benefit analysis can be worthwhile exercises, as long as the findings are viewed as just one piece of the puzzle.

Are you ready for creative readiness?

The Conference Board has a new research report out, Ready to Innovate: Are Educators and Executives Aligned on the Creative Readiness of the U.S. Workforce? ($75 US). They say that "Innovation is crucial to competition, and creativity is integral to innovation. In November 2007, The Conference Board and Americans for the Arts, in partnership with the American Association of School Administrators, surveyed public school superintendents and American business executives (employers) to identify and compare their views surrounding creativity. Overwhelmingly, both the superintendents who educate future workers and the employers who hire them agree that creativity is increasingly important in U.S. workplaces, yet there is a gap between understanding this truth and putting it into meaningful practice."

I have not yet read this report, but it sounds relevant to my Tuesday post Many companies want their hiring to be more evidence-based. But is it even possible?. One of the key findings is that "Employers concerned with hiring creative people rarely use profile tests to assess the creative skills of potential employees. Instead, they rely on face-to-face interviews." The Board's concept of "creative readiness" is intriguing -- is this a leading indicator for innovation and strategic inventiveness? All too often, I've seen smart engineers and other professionals stuck in an environment that smothers creativity. And I'll admit to a distaste for the traditional University engineering degree programs, which in my experience are too rigid, too narrowly focused, and too comfortable graduating risk-averse, deductive thinkers rather than inductive thinkers with budding leadership abilities.

 

At least some in the music industry are trying evidence-based marketing.

If they awarded a trophy to the industry making the most awkward adjustment to an Internet-savvy business model, surely the music recording business would be the winner. I'm opposed to music theft under any circumstances. But the evidence shows that it's possible to include free listening as part of a sustainable business model.

Last.fm, a service I very much admire, is making it work using innovative ideas, commitment to a good user experience, and cutting-edge tagging* technology. As explained on Read/Write Web, "Last.fm's on-demand service, which lets users play any particular song, only allows a user to stream a song in full three times. After which, they're prompted to purchase the track through one of the affiliate services. Not only has this on-demand service been good for Last.fm ... [it's] good for partners, too, like Amazon, whose overall CD and download sales through Last.fm increased by 119%. And since the service launched, Last.fm users are purchasing 66% more albums than before."

To a point, I sympathize with the traditional music publishers -- few of us would voluntarily cannibalize our own established revenue streams. But too frequently they've used fear-based tactics, when they could have more enthusiastically embraced innovation, backed up by evidence-based sales and marketing programs.

*Last.fm is so slick. I stream music frequently while I'm working. Among other things, they use tags to categorize songs, so you can easily say what type of music you want to hear, and they'll do the rest. I use the Listen / Tag option (as opposed to specifying a particular artist) -- for example, I enter the tag Acid Lounge, and they queue up lots of Acid Lounge Tag Radio.

Many companies want their hiring to be more evidence-based. But is it even possible?

A friend of mine runs a marketing firm. I know from personal experience that hiring someone into a middle-management role at a small services firm is a Really. Big. Deal. So last year, when my friend wanted to fill a new position, she agonized: Her team administered tests, did lots of reference checking, etc. Eventually they identified a strong candidate ...but after only three weeks on the job, they knew it was not only a mistake, but potentially a disaster for the client relationships they'd worked so hard to develop. Ouch.

Lately one of my favorite topics is the effort to make hiring (and post-hiring "human capital" management) more evidence-based. I was delighted to see the article Dilbert the Inquisitor in last week's Thursday Styles section in the New York Times. Lisa Belkin wrote about the explosion of assessment techniques being adopted by corporate hiring managers. The use of profiling and evaluation instruments is growing rapidly, according to a recent survey of human resources managers. And there's a wide range of tools available: Naturally, there's the well-known Myers-Briggs Type Indicator, but there are also assessments requiring only about 5 minutes to complete, and some that involve a full day of role-playing. For example, DDI offers in-depth simulations of potential work situations. Belkin went through an exercise during which she pretended to be a new vice president at a foundering robotics company in 2025: "I held a 'meeting' with an actor pretending to be a passive-aggressive employee who was sabotaging a new sales plan with his vocal disapproval. I had a 'conference call' with an actress who was convincingly frosty as she refused to share key research and manpower that her department had and mine needed." This approach is intriguing -- at the very least, you would have an opportunity to see a candidate in quasi-real action.

Can you be 30% evidence-based? I'm confident that lots of research, data-collection, and testing has gone into the development of many hiring-related assessment tools. So does that mean they are evidence-based? I would say so. The problem is they can assess just so much -- several rather fuzzy factors will weigh into a hiring decision, and into how a person performs once they've joined a company: Despite your best efforts, your sparkling new hire can end up in the ditch in a hurry. An assessment tool can't “'be the be all and end all,' said Robert Blankstein, the vice president of client services at Caliper, an international management consulting company, based in Princeton, N.J. 'We tell client companies ‘Don’t take it as more than it is. Twenty to 30 percent of your decision at the most should be based on an assessment.’ ' ”

Too vague? Another potential problem is how to interpret results. It doesn't help -- and it could hurt -- to summarize the results of an assessment in terms that are too general to really distinguish candidates from one another, or identify who's best suited to a particular role. In the New York Times mini-experiment, one evaluation indicated that Belkin is "detail-oriented," while another said she is not (wisely, she wonders "Perhaps it depends on how engaging or useful I find any given set of details?"). I would hate to think a hiring manager would rule someone out because a simple test said he or she was not detail-oriented, without getting into the specific context of the work involved.

I'm an ENTJ. You? Before I die, I'd love for someone to show me hard evidence that the money they spent administering Myers-Briggs tests actually made a difference to their company. (All I know is that my husband and I are both ENTJs, as are many of our colleagues. What that says about us I'm not sure.) Hiring is one of the most important things a company does, yet few managers can devote much time to the interview/evaluation process. I think busy people should be taking advantage of the more sophisticated assessment tools -- as long as they remind themselves not to place too much emphasis on the results.

More skepticism about the Tipping Point.

There's been more online discussion about the Tipping Point concept, and skepticism about  how often one really happens. I wrote recently about Fast Company coverage of a Yahoo! researcher who is debunking the idea, based on hard evidence derived from email patterns (see Tripping Over the Tipping Point). Now a market research firm has substantiated the belief that Gladwell's supposed influencers don't influence nearly as many as he suggested. In Influencers Possess Less Clout, Pollara has found that "popularity doesn't always equate to credibility."

Read/Write Web says "The new data from Pollara does say that people use online social networks to make buying decisions, but they trust the advice of their friends and family on those networks far more than they do high-profile bloggers."

In an unsurprising development, some active bloggers want to debunk the debunkers, claiming that the Tipping Point is alive and well. The Yahoo! research was of course, intended to help marketers understand how best to sell on the web. And influencing an idea -- such as kicking off a public policy debate -- is certainly a different phenomenon from influencing a consumer buying decision. Read/Write Web makes a good point that "The study is complicated by the fact that it focused on buying things. The biggest blogs on the web aren't places readers go to find ways to spend money. There's an almost rabid rejection in blog reading communities of anything that costs money, in fact."

Are CIOs to blame for IT's talent shortage?

Allan Alter of CIO Insight says yes. In Blame CIOs for the IT Skills Shortage, he claims that employers aren't making the investments they need to develop their people, are refusing candidates who lack narrowly defined skills, and then complaining about their IT staffs. I've been thinking about this a lot lately (see We can't hire enough "talent." Oh, and we're eliminating 3,000+ jobs.), and I agree with his conclusion that "Many IT executives gripe that universities are not producing a stream of IT graduates who are prepared to function in the business world. Some worry about the unflattering image of technical professionals as socially awkward. But no one is more to blame for the skills shortage than CIOs, especially those at large companies. The reality is that IT executives are creating the skills shortage they grumble about." He quotes training investment statistics to support his claims.

Does this touch a nerve? Alter's view seems to resonate with lots of folks. This is the first time I recall seeing a CIO Insight article, or any IT-related article for that matter, so highly rated (as of this afternoon, it had received five stars out of five from dozens of reviewers). Alter suggests a way around the IT talent problem: "There’s a three-part solution to the skill shortage: One, no matter how the economy affects your firm, increase training for employees. Two, when recruiting from outside your company, be willing to interview capable IT professionals, even if their skills aren’t a perfect match for the job. Three, be willing to provide new hires with technical training." Or, as he puts it, "So stop griping about the skill shortage, broaden your searches, and start creating the skilled workforce your company needs."

Anecdotal evidence. Here's an example from one of my colleagues. He works in an in-demand engineering field, and recently told me his previous company claimed to value people, but was completely clueless about developing or retaining experienced talent. For instance, the executives allowed a not-very-people-savvy director to keep employees assigned to the same group essentially forever (or until they quit, whichever came first). Folks who asked to work in a different area within the company were refused, and given virtually no opportunities to expand their skill set. So my colleague left, and more hard-to-replace senior people continue to walk out the door.

What do you think? Is your company missing the opportunity to help people grow along with it? What can be done about it?