Evidence Soup

Wal-Mart claims to be saving U.S. families $260B this year. Does the evidence support this claim?

This week I visited my folks, who happen to live in Bentonville, Arkansas* -- better known as the home of Wal-Mart. The company's Home Office sign makes a bold claim: "So far this year, we've saved families $260,553,508,784." (It's actually more, since the displayed total steadily increases as you drive by, and I took this photo on Wednesday.)

Yes, that's $260 *billion*. Pretty bold statement, even for a company with such massive revenue (on Nov. 13, Wal-Mart announced quarterly net sales of $97.6B, with $61B coming from Wal-Mart U.S.). But they offer evidence to substantiate their claim: The mega-retailer has commissioned Global Insight to study its economic impact in the United States. You can download the 2007 research update here (Global Insight 2007 Price Impact Study).

Producing evidence. Wal-Mart's announcement describes how it's tallying these savings. Basically, they did what any other self-respecting corporate behemoth would do: Sponsored research. "In 2005, Global Insight, a leader in economic and financial analysis, analyzed Wal-Mart’s national and local impacts in terms of jobs, wages, prices, consumer buying power and GDP.... [T]he continued reduction in prices due to the presence of Wal-Mart and the growth in consumer expenditures over the 2004 to 2006 period translates directly into savings for consumers amounting to $287 billion in 2006. This corresponds to savings of $957 per person or $2,500 per household.... 'From the family vacation, to a daughter’s wedding, the savings American families realize at Wal-Mart bring the good things in life a good deal closer.... The new advertising tells the same story we’ve told since day one, how we’re working hard to save people money so they can live better.' The retailer will literally track the amount of savings so far this year by installing a 'savings ticker' outside of Wal-Mart’s home office in Bentonville, Ark."

From Global Insight: "In 2005, Wal-Mart commissioned Global Insight to undertake an independent research effort to analyze this issue. The goal of that research was to independently and credibly document Wal-Mart's national and local impacts in terms of jobs, wages, prices, consumer buying power, productivity, and gross domestic product.

"That report found that the existence of Wal-Mart between 1985 and 2004 resulted in a 3.1% cumulative reduction in consumer prices by 2004.... Our new study, titled The Price Impact of Wal-Mart: An Update Through 2006, looks at Wal-Mart's cumulative price impact and total cost savings as of 2006. Our findings continue to support the claim that an economy without Wal-Mart would have meant higher prices for consumers. It also concludes that the reduction in the price level due to the presence of Wal-Mart translates directly into consumer savings amounting to $287 billion in 2006—$957 per person and $2,500 per household."

The evidence is clear. I don't doubt that an economy without Wal-Mart would mean higher prices for U.S. consumers. We can quibble about the $ amount, but they do get our attention with that provocative sign, which is the *only* flashy thing you'll see when driving along Walton Blvd in Bentonville. Come to think of it, Bentonville is probably the best example of what Wal-Mart has done for the U.S. generally, especially small-town America. I've visited northwest Arkansas many times over the past 25+ years. The Bentonville-Rogers area has been transformed from a relatively unsophisticated place to enjoy the outdoors (with beautiful natural features, such as rivers, lakes, and limestone caves) to a reasonably cosmopolitan place with a few outstanding restaurants, access to good shopping, and things like a W "aLoft" hotel -- in the span of only a few years. All because of... Wal-Mart.

Seven, a new active book, tries a different approach to providing evidence about software business models.

I received a review copy of the book Seven Software Business Models. It's got plenty of useful insights for anyone interested in how the software business is changing, and how people intend to continue making money in the industry. The author, Timothy Chou, draws from his experiences as President of Oracle on Demand (LinkedIn profile here). Seven provides an overview of the various business models companies are using these days (see below) -- from traditional enterprise licensing to advertising-supported or freemium software. 

But the emphasis is on the future, not the past: Software-as-a-service, social applications, etc. At SoftwareSeven.com, Chou says the book "is written with a chapter focused on every functional area: R&D, Operations, Sales, Marketing, Finance HR. In addition, you'll also find lots of campfire stories from people who did these jobs at leading companies such as ...Salesforce.com, Webex, Netsuite...."

Fittingly, the book is sold and printed on demand, available from lulu.com. I really admire Chou for shaking the trees.

And even better, it's wiki-ish: Chou describes Seven as "the world's first Active Book. An Active Book combines both the depth and discipline required to write a physical book with the power of the Internet to connect people and information. This is a beta version, wander around, learn and keep coming back - you'll see it changing dramatically as we add new information and people over the next several months." People can request to become a contributor to this active book. Very cool. This is how evidence should be collected and distributed. (Though I believe they need to come up with a way to structure and present evidence so it's easier to pinpoint.)

Evidence? Yes, there's evidence. The book presents real-world P&L information for various software companies during funding rounds. But in business, lots of evidence comes in the form of anecdotes from experts, and there are plenty of those offered here. No, an anecdote isn't the same as a peer-reviewed study... but in this context, it can be equally useful.

Will evidence prevail over ideology in the battle over U.S. auto makers?

Age is a very high price to pay for maturity.   --Tom Stoppard

As I get more, um, mature, the more I understand that we (people, companies, communities) often aren't sufficiently motivated to make big, lasting changes until we reach the crisis stage. Gotta hit rock bottom first, all that stuff.

Time to tear off the Band-Aid. Things don't always have to go that far before good changes happen, of course. But I believe that's what's needed for the U.S. auto makers: They need to fundamentally change -- and I don't see much evidence that a government bailout will accomplish that. I think they need to face the music, even if it means reorganizing through bankruptcy.

In the red corner: Ideology! There's plenty of Big-I ideology to go around on this one.

• The free-markets people say "let Detroit succeed or fail on its own" -- although if we non-economists have learned anything over the last few months, we've learned that the free-market stuff is an admirable philosophy, but not happening in reality.
• Meanwhile, American Flag, Mom, & Apple Pie, Inc. (AFMAP) acts as if our rusty, trusty, Detroit-centric auto industry is a God-given right, and deserves a $25 billion Mulligan/freebie/handout. (Cue Springsteen singing Youngstown.)

And in the blue corner: Evidence! Looking at the evidence, I don't believe the auto makers should get a bailout from the U.S. government. Without repeating every detail and recapping every editorial, here are some highlights to support my argument:

• They've drained too much money from our economy already. In Just Say No to Detroit, an essay in Saturday's Wall Street Journal (behind paywall), David Yermack argues that the "The implications of this story for Washington policy makers are obvious. Investing in the major auto companies today would be throwing good money after bad.... GM has invested $310 billion in its business between 1998 and 2007. The total depreciation of GM's physical plant during this period was $128 billion, meaning that a net $182 billion of society's capital has been pumped into GM over the past decade -- a waste of about $1.5 billion per month of national savings. The story at Ford has not been as adverse but is still disheartening, as Ford has invested $155 billion and consumed $8 billion net of depreciation since 1998." Yermack claims we'd be better off simply writing checks to each GM employee... I think he has a point.
• Look to the future, not the past. In the New York Times DealBook column (yes, I said the New York Times), Andrew Ross Sorkin writes in A Bridge Loan? U.S. Should Guide G.M. in a Chapter 11 that an orderly bankruptcy for GM would be preferable to a taxpayer-funded bailout. This would allow them to tear up their crippling labor contracts and have a fresh start.

There's lots of evidence on both sides of this issue. But I believe the weightier evidence says Detroit has mis-managed and under-innovated for too long already -- instead of taking bold steps, they chose instead to hire lobbyists. Rather than throw more pearls before swine, I think it's more appropriate to continue providing assistance for programs like developing vehicles that run on alternative energy. And let the rest of the stuff sort itself out.

Forrester offers evidence that customer satisfaction affects stock price directly.

I'll admit, I think a lot of analyst reports are a big yawn. A recent one from Forrester, Customer Service: A Keystone Of Your Corporate Revenue Strategy (pdf), exceeds those expectations. (Although they can't help but fall into the tidy-explanation-of-why-some-companies-fail trap.)*

The report by Natalie Petouhoff begins with "Despite the pressure to become a profit center, most organizations’ customer service and contact centers continue to miss the mark on becoming a strategic business partner and are not considered part of the corporate revenue strategy." She makes some strong points in identifying catalysts to make customer service an "executive-level duty," including:

"Evidence of a direct financial tie to revenue. Studies reveal that customer service and the contact center directly affect corporate revenue.... An Accenture study cites customer service and converting customer information into business intelligence as the business improvement initiatives that produce the highest impact on a company’s financial success. For example, a typical $1 billion business could add $40 million in profit by increasing its customer-facing capabilities by 10%. According to... the American Customer Satisfaction Index, a company’s stock price is related** to customer satisfaction. This means that executive decisions that erode customer service are eroding corporate profitability."

The report is primarily a survey of relevant research, so if you're digging for evidence on this topic, your job just got a whole lot easier. Among the references (pdf) are these:

Good point. Another catalyst is "Direct executive accountability for large expenditures. With the SEC now requiring extensive documentation for executive pay-for-performance shareholder reports, never before have executive compensation decisions been so tightly linked to the decisions they make.... And because results in customer service and contact centers are so easily measured, a well-executed (or poorly executed) strategy can provide excellent SEC report material."

Ahem. Here are my footnotes:

*Figure 3 of the Forrester report, "Continued Poor Executive Leadership Is Responsible For Low Corporate Success Rates," is where they fall into the analyst/consultant trap of identifying why one company failed when others did not. While there is plenty of good stuff in analyses such as this, and lessons to be learned, I think we all know it's not as simple as listing a laundry list of root causes: The evidence is just too fuzzy. I believe the best refutation of the "Let Me Tell You Why" impulse is The Halo Effect. If you haven't read it yet, you should.

**Saying share price is "related" to customer satisfaction doesn't tell us much. I've seen evidence showing that nothing but free cash flow (FCF) truly correlates to stock price -- but arrgh, I can't find the citation right now. I'll keep looking.

Want to know more? Forrester provides a download of their report here (pdf) at no cost.

Are physical therapists ignoring the evidence about stretching?

After some significant hamstring problems, I recently saw a physiatrist at a sports medicine clinic, followed by a physical therapist. The P.T. came highly recommended, and seemed knowledgeable, explaining the why, not just the what, of the prescribed exercises. Emphasis throughout my treatment was on stretching the hamstring -- 3 times a day, for up to 3 minutes at a time. I've always advocated stretching, so was happy to comply with that advice.

Turns out, my problems were related partly to the chair I was using -- evidently lots of people with desk jobs develop hamstring problems. My exercises and stretches (combined with a new chair) have helped tremendously, and my faith in stretching is stronger than ever.

Is dynamic stretching the new black? Then along comes the article Stretching: The Truth in the New York Times' excellent sports magazine, Play. Judging from what they said, I was doing it all wrong: "The old presumption that holding a stretch for 20 to 30 seconds — known as static stretching — primes muscles for a workout is dead wrong. It actually weakens them.... Stretching muscles while moving, on the other hand, a technique known as dynamic stretching or dynamic warm-ups, increases power, flexibility and range of motion."

They've got studies (but true to New York Times tradition, they're not going to tell you where they are). "In a recent study conducted at the University of Nevada, Las Vegas, athletes generated less force from their leg muscles after static stretching than they did after not stretching at all. Other studies have found that this stretching decreases muscle strength by as much as 30 percent."

What does *all* the evidence say? Does this mean my doctor and physical therapist are ignoring the current, best evidence about stretching? I don't think so, necessarily. The Play article is about warming up and strengthening muscles for a workout or athletic activity, and that's not why I went in for treatment. They advised me to stretch because the area where my hamstring connects to my joint was aggravated (it was rather angry, actually). So based on what I've read, I think this is an apples-vs-oranges thing. I'm gonna keep on stretching. Statically, not dynamically.

Art world perpetuates faulty evidence, and journalists let them get away with it.

Art auction houses routinely include their commissions when reporting sale prices, significantly inflating the numbers above and beyond what people actually pay for works of art. Said differently, they're misrepresenting evidence about the current art market. In today's In the Fray column (Wall Street Journal, behind paywall), Lee Rosenbaum explains how it happens, and calls for a more transparent process.

In Making Sales Look Stronger: The auction houses have an interesting way of reporting their auction results., Rosenbaum says even though recent auctions were reported to have dismal results, "Actually, the situation is worse than reported. Contrary to what you might expect, press accounts, relying on the information released by the auction houses, don't normally measure a sale's success by comparing an object's hammer price -- the last amount announced by the auctioneer -- with the presale estimate of hammer price. Instead, they almost invariably compare the estimate of hammer price to a figure arrived at by adding hammer price to the commission that the auction house charges the buyer. The result is an apples-to-oranges comparison that makes the sale results look better than they actually are, because they've been inflated by the commission.

$1 million here, $1 million there, pretty soon you're talking about real money. Rosenbaum says commissions are 25% for sales prices up to $50K, dropping on a sliding scale to about 12% for amounts over $1 million. So when an auction brings in $100 million in sales, those fees are hefty. He continues: "Despite the fact that hammer prices of individual lots are announced as they happen during the sale, journalists who ask for them (or for a sale's hammer total) afterward may need time and persistence to get the information -- resources in short supply on a tight deadline. At a time when confidence in markets has been seriously shaken, it's more important than ever for the auction houses to provide a true reflection of how their sales are performing. They can easily provide hammer price lists and totals, along with the figures that include the buyer's fee. They should then use these figures draw the correct comparisons -- hammer price to presale estimate -- in analyzing how well their offerings have fared in the saleroom."

Since today's issue of the Wall Street also included an article reporting recent auction results, I wonder -- did this reflect a similar overstatement of actual sales prices? 

The evidence is undeniable: When the door closes on one buzzword, another one opens.

Given today's challenging selling environment, I understand the temptation to introduce new buzzwords to the marketplace. This week I was invited to a webinar where they're claiming that old-school "knowledge management" is inadequate, and that we should be enabling genius management instead. Really.

I wish people wouldn't use vague phrases like "knowledge" or "genius" in conjunction with "management". They really don't belong together, and these "activities" are unobservable (let's face it, "evidence-based" is plenty fuzzy for most people.) And I believe vendors inadvertently diminish the value of useful technologies or management methods when they apply such fuzzy labels. Plenty of impressive Intranet software helps people collaborate, make discoveries, and exchange ideas -- and it doesn't have to be called knowledge management.

Below are excerpts from the webinar invitation I received. They're making some valid points (particularly about moving away from a document-centric world view), and describing some interesting stuff -- so why give it such a lame label?

"Traditional Knowledge Management was defined by the centralization of documents — the objective was to centralize materials with the hope that people across the organization knew where to look. The problem with this approach?

• It assumes that everyone has time to navigate through layers of architecture.
• It assumes documents are the only resources customer-facing people need.
• It focuses too little on the behaviors that we’re looking to reinforce via Knowledge Management.

"Tomorrow’s most competitive organizations are those that have already begun to recognize that Knowledge Management is really an act in Genius Management. Instead of focusing on documents, they are working to harness the right genius from across the organization and provide it to the people who need it most — front-line sales reps. Join [various companies] to learn why Genius Management is the best strategy ahead for 2009.

"Learn how top companies... are leading the charge in Genius Management and hear first-hand how they are driving top-line impact by:

• Arming every rep with the best information to drive the right sales conversations
• Replicating the best practices of top performers across the sales force
• Reducing rep prep time and increasing field selling time
• Allowing salespeople to contribute insights from the field—without administrative burdens
• Ramping new hires quickly to start closing deals faster."

Doesn't this list represent some good SFA (sales-force automation) ideas, combined with tight integration between marketing and sales operations? I wish they would just say that.

Happy Fun-with-Evidence Friday, everybody.

Evidence struggles to overcome popular opinion, even when it's only 1 week old.

No matter how strong your evidence, assumptions, myths, and flawed statistics often get in your way. Artfully overcoming mistaken beliefs is a key challenge for those of us pursuing evidence-based management, particularly when issues are complex and data are ambiguous. Here's a recent example from the world of politics, though there's a lesson here for everyone.

Following last week's U.S. elections, people noticed a result in California that seemed contradictory: Even though Barack Obama and the Democrats won the state in a landslide, the conservative Proposition 8 initiative also won. (Proposition 8 bans marriage between two individuals of the same gender.) In only a few days, people came up with all kinds of explanations for this apparent contradiction. (Maybe to fan the flames of political rhetoric just a bit longer?) They mistakenly framed the problem around race -- though as we'll see in a moment, it's probably more about people's age than their race:

• Dan Walters in the Sacramento Bee concluded that minority-voter turnout caused the Prop 8 win: "The pro-Obama surge among the [African American and Latino] groups gave Proposition 8 an extra 500,000-plus votes, slightly more than the measure's margin of victory. To put it another way, had Obama not been so popular and had voter turnout been more traditional – meaning the proportion of white voters had been higher – chances are fairly strong that Proposition 8 would have failed."

• Others argued that not enough blacks voted against the measure, even as they were voting for Obama. NPR had people talking about it in Why Black Voters Didn't Fight Prop. 8. In the LA Times, Jasmyne Cannick wrote: "I am black. I am a political activist who cares deeply about social justice issues. I am a lesbian.... But even I wasn't inspired to encourage black people to vote against the proposition.Gay marriage? Please. At a time when blacks are still more likely than whites to be pulled over for no reason, more likely to be unemployed than whites, more likely to live at or below the poverty line, I was too busy trying to get black people registered to vote, period."

But what about the evidence? But evidence genius Nate Silver of FiveThirtyEight.com fame said not so fast (he's worth listening to, having developed a reputation for reliable statistical methods that successfully call things like presidential primaries and elections). In disputing the various theories about the Prop 8 outcome, he avoids ranting about standard deviations and cross-tabs, instead offering a well-articulated, plain-English explanation. In Prop 8 Myths, he says "Now, it's true that if new voters had voted against Prop 8 at the same rates that they voted for Obama, the measure probably would have failed. But that does not mean that the new voters were harmful on balance -- they were helpful on balance. If California's electorate had been the same as it was in 2004, Prop 8 would have passed by a wider margin. Furthermore, it would be premature to say that new Latino and black voters were responsible for Prop 8's passage. Latinos aged 18-29 (not strictly the same as 'new' voters, but the closest available proxy) voted against Prop 8 by a 59-41 margin.... At the end of the day, Prop 8's passage was more a generational matter than a racial one."

On Wednesday's Countdown show, Keith Olbermann compared the dominant public commentary to Silver's use of solid evidence, saying Nate "relies on statistics, not what he hears in the hallway." As I mentioned last week, GQ has a nice writeup on Nate and how he got started.

Study shows significant "brain waste" in U.S. An opportunity for evidence-based public policy?

It's a touchy subject: What to do when physicians and other educated people immigrate to the U.S., but without credentials that automatically qualify them to pursue their professions? We don't want to arbitrarily override our established standards for practicing medicine, etc. But isn't it also true that a mind is a terrible thing to waste? And isn't the U.S. facing shortages of physicians, nurses, teachers, and other skilled people? I believe this is an opportunity for evidence-based public policy (which, I think, is synonymous with "informed compromise").

A recent report by the Migration Policy Institute says that "More than 1.3 million college-educated immigrants are unemployed or working in unskilled jobs such as dishwashers, security guards, and taxi drivers — representing one of every five highly skilled immigrants in the US labor force. Their work in these jobs constitutes a serious waste of human capital — one that can be addressed by both immigrant admission and immigrant integration policies." As shown here, immigrants from Africa are especially burdened by under-employment.

The report continues: "Though often overlooked amid controversies over the flow of unauthorized, largely lowskilled immigrants, legal immigration channels have produced a steady flow of newcomers with substantial levels of education. As of 2006 there were more than 6.1 million immigrants 25 or older with a bachelor’s or higher degree, representing 15.2 percent of all college educated persons in the US civilian labor force. We estimate that more than half (53.4 percent) of these highly skilled immigrants obtained their education prior to migration, so that the United States benefits from schooling they received and that was paid for elsewhere. The great majority of these immigrants eventually do well here. Yet many experience considerable difficulty securing well-paying positions that use their credentials. Some never achieve employment commensurate to their qualifications. Numerous studies have shown that highly skilled immigrants contribute to the economy through innovation and entrepreneurship. In addition, research shows they produce a surplus for public coffers by paying more in taxes than they take out in services.

"Thus, the brain waste documented in this report represents unrealized returns not only to these immigrants and their families but also to the nation as a whole. In an economic environment in which human capital drives productivity and development, strategies to maximize the available human capital deserve the close attention of federal, state, and local policymakers." Refreshingly, this report comes with footnotes to substantive evidence (go here to download the pdf).

The LA Times covered the report, and interviewed Luis Garcia, a Peruvian physician who is now a dog walker. "'I feel lost,' Garcia said. 'Sometimes I'm embarrassed to talk to my family back home and tell them I'm taking care of dogs. But I know someday I will be able to do my geriatrics practice, and I know there are people here who need my help.'" And the Times pointed out the urgency of the situation: "Michael Fix, senior vice president of the Migration Policy Institute, said the need to help immigrant professionals gain the requisite credentials and experience is particularly acute now that the nation faces the impending retirement of 77 million baby boomers, considered the most skilled workforce in history. In California, for instance, the fastest growing occupations are computer software engineer and registered nurse."

Infovell, now DeepDyve, releases free version.

Update: The DeepDyve site had trouble signing people in yesterday (probably because publicity attracted an onslaught of new users). It's up and running today. But many are asking whether a free search engine should require a sign-in. I'm not sure why DeepDyve is asking for personal information just to let people search. It's well-known that these requests often collect lousy data -- or, as a speaker at last week's Defrag conference said, such sites seem to attract a disproportionate population from Beverly Hills (zip code 90210) and Schenectady, New York (12345).

I wrote in October about the deep-web research engine Infovell. It offers a better way to look for, and keep track of, evidence -- something I believe is long overdue.

Infovell has been renamed as DeepDyve, and begun offering a free version. See Read/Write Web coverage here. DeepDyve's scope is still somewhat limited, so isn't ready to be the go-to research engine for everybody. RWW points out that "DeepDyve is slowly expanding into more search verticals, but for now, its focus is on life sciences, physical sciences, and patents, though it also indexes a few humanities journals. The service also indexes newspaper and lets you search for Wikipedia articles as well." Their index reportedly includes about 500 million pages.

Freemium model. I think it's great that they're offering both a free version and DeepDyve Pro. Here's how they explain the benefits of the $45/month Pro version:

• "Dynamic Grouping. No more weeding through lists of results. DeepDyve Pro organizes all of your results into easy-to-navigate folders using dynamic grouping technology. Dynamic grouping allows you to quickly skim the entire universe of related topics and focus on the most relevant results.
• "Visual Clustering and Venn Diagrams. Only DeepDyve Pro users receive access to advanced visualization features such as visual clustering and Venn diagrams. Search results are visually clustered by topic, illustrating their relationship at a glance. When clusters have topics in common, they are represented by overlapping Venn diagrams, making it fast and easy to identify all related results.
• "Advanced Search. Take advantage of advanced search capabilities, such as the ability to filter searches by journal, patents, medicines, and other data sources. DeepDyve Pro saves time and effort to make your searches as productive as possible."