Evidence Soup

U.S. agencies aren't supposed to allow political interference with scientific evidence. Good luck with that.

Evidence Soup is back in business. These past 3 months, I've been distracted by a number of things, including a move from Denver, Colorado to the San Francisco Bay Area. So, where were we?

Will "4.37 Degrees of Separation" play at the multiplex? Awhile back I wrote about recent research to test the Six Degrees of Separation theory. New evidence suggests that people are separated by an average of 4.74 degrees (only 4.37 in the U.S.). Doesn't really roll off the tongue, and I wouldn't expect Will Smith to star in a sequel. But this latest research applies only to people on Facebook; a New York Times piece reminds us "the cohort was a self-selected group, in this case people with online access who use a particular Web site".

Desperately seeking scientific integrity. Early on, President Obama launched an effort to ensure that the public can "trust the science and scientific process informing public policy decisions". His March 9, 2009 memorandum made agencies responsible for "the highest level of integrity in all aspects of the executive branch's involvement". Great idea to de-politicize science, though it's much easier said than done -- as evidenced by the nearly-two-year delay in providing "guidelines" for agencies issuing this policy. Really. Those guidelines were published December 17, 2010 [pdf here].

Oh, and meanwhile the Obama Administration was widely criticized for using sloppy science to support its moratorium on offshore drilling after the BP disaster on the Gulf Coast. It's never simple: In real life it requires weighing risks, factoring in economic impacts, and making political tradeoffs (consider this analysis of the economic impact of that ban - [pdf here]). A scientific integrity rule won't give you all the answers you need when making such complex decisions: It's surely not as simple as determining whether scientific findings were manipulated for political purposes.

As explained in a recent National Public Radio story, now we're seeing the first challenge to federal evidence-gathering under this new regime: It's directed at the Bureau of Land Management (a branch of the Department of the Interior). More about that in a moment.

Why is this so hard? It's been rough going for agencies issuing scientific integrity policies. The basics are straightforward enough: Preventing people from twisting or quashing scientific evidence. But here are some reasons why the process is so problematic.

1. Scope. There's no bright line identifying what "science" should be included in an assessment, and therefore subjected to integrity requirements. This complicates things enough when you're working with "objective" evidence. It's even trickier when you bring in fuzzier stuff, like the dismal science (economics), or risk assessment. How can we say for sure what should be considered? Our assumptions about what's important determine what evidence we recognize - whether consciously or subconsciously. Our choices are influenced by our values - but we may not be fully aware of our values, or we may not want to articulate them in a transparent way.
2. Dissemination. Each agency's integrity policy is supposed to provide for open communication, and guide how evidence is presented to the public (see the 2010 guidelines mentioned earlier). It wouldn't serve anyone to have a free-for-all; consistent, controlled dissemination can improve usefulness and understanding. But not everyone agrees on the rights and responsibilities of scientists who want to discuss their findings with the public.
3. Whistle blowing. Among the substantial hurdles is the handling of whistle blowers. (I suppose if such a policy is going to have teeth, the people who want to blow whistles need to feel they can do so without losing their heads.)
4. Transparency. Scientific groups - such as the Union of Concerned Scientists - say they still want to see external accountability under these policies. So far, investigations of misconduct are internal.

12291 all over again? Thirty years ago, President Reagan signed Executive Order 12291, requiring cost-benefit analysis for 'major' federal regulations (those expected to impact the U.S. economy by $100 million or more). Clinton issued a similar order in 1993. In theory, this should have de-politicized some agency decision-making processes. The results (or lack thereof) were the subject of my doctoral dissertation.

As with the new mandate for scientific integrity, a requirement to weigh regulatory costs against benefits leaves lots of room for interpretation, and requires value judgments. When EPA issues a rule under the Clean Air Act, it's difficult enough to estimate how many hospital visits or early deaths are caused by a particular type of airborne particulate matter. But figuring out the social costs is harder still: Requiring businesses & governments to reduce those emissions can lead to job cuts and economic loss, which themselves cause poverty and negative health impacts.

Challenging BLM's process. Citing the new scientific integrity policy, a group called Public Employees for Environmental Responsibility (PEER) has filed a complaint against the BLM, saying "The U.S. Bureau of Land Management is carrying out an ambitious plan to map ecological trends throughout the Western U.S. but has directed scientists to exclude livestock grazing as a possible factor in changing landscapes....  [O]ne of the biggest scientific studies ever undertaken by BLM was fatally skewed from its inception by political pressure.... As a result, the assessments do not consider massive grazing impacts even though trivial disturbance factors such as rock hounding are included, [and they] limit consideration of grazing-related information only when combined in an undifferentiated lump with other native and introduced ungulates (such as deer, elk, wild horses and feral donkeys)." I didn't know we had a feral ungulate problem. But I digress.

This is a good example of how choices about collecting evidence can strongly influence the results. NPR explains that the Dept. of Interior has a scientific integrity officer who is responsible for investigating allegations of political interference. I wish him Godspeed.

 

Big Ideas may not have supporting evidence, but they sell books by the boatload.

Haven't we had enough of authors pitching an oversimplified analysis of something important? It's great to boil things down into plain language, but when an entire book is based on A Big Idea, complicated things are glossed over, evidence is cherry-picked, we get bamboozled. (And books are sold. Maybe I'm just jealous.)

Writing in New York Magazine, Kathryn Schulz takes Tina Rosenberg to task for touting a "social cure", formerly known as peer pressure, "a panacea so powerful that it can help you quit smoking, lose weight, escape poverty, ace calculus, combat Islamic terrorism, overthrow a dictator". (Thanks to Elizabeth Lusk, @gestaltKT, for the link. Here's her Evidence Soup interview.)

Group Think praises Rosenberg's earlier work (she is a MacArthur Foundation grant recipient). But Schulz has a bone to pick with her latest book, Join the Club: How Peer Pressure Can Transform the World. [Photo courtesy of New York Magazine.] She explains:

Big Idea books have been around for a long time; see The Communist Manifesto. But the Big Idea Book Club (I mean “club” as Rosenberg defines it: an identifiable in-group with enough status to influence the behavior of others) is a recent phenomenon. Its accidental founder and president in apparent perpetuity is Malcolm Gladwell. Its membership, like the membership of most powerful groups, is largely male. Its combined sales are stratospheric; whatever these books are hawking, we can’t stop buying it.

As for the books themselves, I’ll generalize (as, often, do they). Big Idea tomes typically pull promiscuously from behavioral economics, cognitive science, and evolutionary psychology. They coin phrases the way Zimbabwe prints bills. They relish upending conventional wisdom: Not thinking becomes thinking, everything bad turns out to be good, and the world is—go figure—flat.

(With Gladwell’s Blink, this mania for the counterintuitive runs top-speed into a wall, crumples to the ground, and stares dizzily at the little birds circling overhead. This is, let me remind you, a best-selling book about the counterintuitive importance of thinking intuitively.) What troubles me about the Big Idea Book Club is the way ideas often slide toward ideologies - grand unifying theories of culture, cognition, happiness, talent, the Internet, the future, you name it.

Well said, Ms. Schulz.

Three people to follow. So you can follow the evidence.

Three people I recommend following: John D. Cook, Carnival of Math, and William M. Briggs. Each of these guys uses math/statistics to explore the evidence in helpful ways. Thankfully, they bring the smarts *and* the funny.

1. John D. Cook is into programming, research, math, and statistics. He tweets via an impressive list of accounts (entire list here), including @AnalysisFact and @StatFact.

Responding to the common criticism of evidence that's not grounded in classic RCT (randomized, controlled trial) methodology, he writes: "As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence-based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence-based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."

2. Carnival of Math (oh, how I love that name). By day he goes by Mike Croucher, making insightful and witty observations, such as this one about the 95% confidence interval: "Last year around 27,000 psychology papers used a 95% confidence interval. So we can say with 95% confidence that at least 1,292 of them were wrong."

Mr. Carnival also writes technical stuff at Walking Randomly "because it's more fun than getting there in a straight line."

3. William M. Briggs bills himself as the Statistician to the Stars! He consults on statistical problems, and is an adjunct professor of statistical science at Cornell University (Ithaca, New York, USA). Awhile back, Briggs wrote an excellent piece Classical statistics is like a criminal trial: "Some thing has happened, and a hypothesis is put forth which, based on probative background information, the investigator thinks likely caused the thing. Evidence -- mostly in the form of “data”, but non-quantitative information too -- for this hypothesis is put forth.... The jury consists of mathematical formulae whose duty is to report solely on the likelihood of the evidence given the hypothesis is false. And since the evidence nearly always appears rare assuming the hypothesis is false, most trials result in a conviction -- meaning, the investigator’s viewpoint is confirmed."

Briggs explains how he thinks we should look at things, using global warming as an example: "Trying to fit carbon dioxide into the frame as the culprit for a historically observed increase in temperature is vastly less interesting and useful than in being able to accurately predict what will happen. It is easy enough to find enough evidence to convict our poor gas classically. But if it were part of a gang of gases, the others would go free. Meanwhile, in celebrating our conviction, we would remain ignorant of what will happen, or will issue poor predictions because we were so intent on assigning blame."

New evidence shows that evidence-based management can boost productivity.

There's good news about the evidence on evidence-based management. In When There’s No Such Thing as Too Much Information, Steve Lohr (@stevelohr ) talks in the New York Times about recent research by Erik Brynjolfsson, a prof at MIT's Sloan School of Management. (A big thank you to @ariegoldshlager and @terrigriffith for sending me this link.)

Brynjolfsson (@erikbryn) and two co-authors just published Strength in Numbers: How Does Data-Driven Decisionmaking Affect Firm Performance? Their findings suggest that large companies who adopt data-driven decision-making can achieve 5-6% higher productivity -- beyond what can be explained by factors such as technology investment.

Lohr explains that in this research study "data-driven decision making was defined not only by collecting data, but also by how it is used — or not — in making crucial decisions, like whether to create a new product or service. The central distinction, according to Mr. Brynjolfsson, is between decisions based mainly on 'data and analysis' and on the traditional management arts of 'experience and intuition.'"  

Big data = Big dollars. Even without these findings, companies have already recognized the value of data analytics. Lohr says that the MISO vendors — Microsoft, IBM, SAP and Oracle — have collectively spent more than $25 billion buying up companies specializing in data and analytics: "IBM alone says it has spent $14 billion on 25 companies that focus on data analytics. That business now employs 8,000 consultants and 200 mathematicians. I.B.M. said last week that it expected its analytics business to grow to $16 billion by 2015."

From the research abstract: "We examine whether performance is higher in firms that emphasize decisionmaking based on data and business analytics (which we term a data-driven decisionmaking approach or DDD). Using detailed survey data on the business practices and information technology investments of 179 large publicly traded firms, we find that firms that adopt DDD have output and productivity that is 5-6% higher than what would be expected given their other investments and information technology usage. Using instrumental variables methods, we find evidence that these effects do not appear to be due to reverse causality. Furthermore, the relationship between DDD and performance also appears in other performance measures such as asset utilization, return on equity and market value. Our results provide some of the first large scale data on the direct connection between data-driven decisionmaking and firm performance."

Stay tuned. I downloaded a free pdf of this research from SSRN here. I'll be exploring the details of the methodology and findings later this week. I'm particularly interested in their measures.

Evidence shows random passwords not necessarily better than usable ones. Tell your IT guy.

I happened upon a nice analysis of password usability and security by Thomas Baekdal. Evidently, our typical approach -- combining upper- and lower-case letters with numerical digits -- isn't necessarily better than choosing two or three common words, such as "fluffy is puffy". (Thanks to @davewiner for tweeting "Why the password 'this is fun' is 10 times more secure than 'J4fS<2'.)

Some IT management isn't as evidence-based as it could be. As Baekdal explains, we can choose a password that's both usable and secure. He goes through detailed calculations to predict the security of selected passwords under various hacking scenarios. Here are excerpts from his findings:

As you can see, the password J4fS<2, chosen by a brute-force method, is "secure for life". But this is fun can be considered "secure forever".

I wouldn't have guessed that "this is fun" would be so much more secure than "woosaa". Good to know.

Evidence-based smackdown! Denning says Sutton's "good boss" characterization doesn't follow the evidence.

What makes a "good boss"?* On the Harvard Business Review site, Bob Sutton suggests a list of 12 things that good bosses believe: Things like productive/intelligent handling of mistakes. Okay.

But over at Forbes.com, Steve Denning finds a crucial flaw in Sutton's characterization: He asks why this definition of "good boss" doesn't include a focus on the customer. Denning says that:

The contrast to a dictatorial command-style manager no doubt helped make the [Sutton] post very popular. From another perspective, however, there is one shocking, even mind-boggling, omission from the list: There is no mention of the client or the customer (or the stakeholder).... That’s because Sutton’s “good boss” is the natural inhabitant of hierarchical bureaucracy. This is a world that operates from an inside-out perspective, pushing products at customers, parsing and manufacturing demand, tweaking the value chain to achieve ever greater efficiencies, and concerned about outputs more than outcomes.... It leads to a preoccupation with style over substance.

As the author of Hard Facts, Dangerous Half-Truths And Total Nonsense: Profiting From Evidence-Based Management, Sutton presents his picture of a "good boss” as “evidence-based” and “rooted in real proof of efficacy.”

Sutton’s picture of the “good boss” however overlooks one huge mass of evidence: the comprehensive study of the productivity of 20,000 US companies from 1965 compiled by Deloitte’s Center for the Edge under the title of the Shift Index. Its conclusions? Running organizations as hierarchical bureaucracies has proved to be an utter disaster.

Denning seems to be asking "What does the evidence say about organizational structure, incentives, and long-term performance?" A question they'll be answering long after all of us are gone. Read the rest of Denning's post here.

*For me, any list of good boss characteristics has to start with "never uses the word 'boss'".

Looking for evidence showing you can get rich *and* be nice to people? Exhibit #1 is Steve Ells.

I've been meaning to write about Steve Ells and Chipotle for some time. Last night's premiere of America's Next Great Restaurant prompted me to (more about that in a minute). I've been eating at Chipotle since the first location opened (in Denver, where I happened to live nearby*). The food is better than ever; I highly recommend the posole.

The evidence. Good people policies, forward-thinking environmental programs, and good business results can co-exist. (Yes, there are others, such as Zappos.)

Good business. As the company took off, McDonald's invested substantially, and in 2006 spun off Chipotle as a successful IPO. Here's a writeup about their solid performance, and a look at EVA (economic value added). (And McDonald's - no stranger to hard evidence - presented evidence-based management methods to the younger chain, such as video-recording lines outside during busy lunch hours, so they could calculate how many customers left due to excessive wait times.... which don't exist, because even a line of 15 people moves through rather quickly.)

Do one thing really well. Clearly, Chipotle follows Steve Jobs' advice to "say no to 1,000 things." I've always wished they would add Asian options to the menu (hey, they've already got the rice -- and yippee! they recently added brown rice, which evidence shows has a lower glycemic index than white, and is associated with lower incidence of diabetes). But of course they're too smart and strategic for that: Now they're developing a separate chain of Asian-inspired restaurants (more news here). The word on the street says it will be called ShopHouse Southeast Asian Kitchen. I dearly hope they offer tofu.

Good environmental programs.  Chipotle emphasizes Food with Integrity: The company's supply chain demonstrates respect for the animals, the environment, and the farmers (ahem, I thought people who raised animals were called ranchers, but I'm from Oklahoma... what do I know?). All this comes at a price: It's not the cheapest food in town. But people will gladly pay for differentiation.

Good people policies. At Chipotle, a general manager can earn in the six figures (yes, over $100,000/yr). They hire a diverse group of people (and not just to satisfy some bogus quota). Take a look at this picture on their careers page. Over the years, I've eaten many times at more than a dozen of Chipotle's stores (in several U.S. states), and I can assure you that The Man isn't running these places.

Chipotle will help any employee become fluent in English. A restaurant reviewer who I very much admire, Laura Shunk, waxes poetic about her experience working there -- and she's not the only foodie/culinary type who agrees the company has good people policies *and* good food.

Reality TV!! Ells is an investor/judge on the new NBC show America's Next Great Restaurant. (Bingo! His participation is great marketing for Chipotle.) What an incredible opportunity for the entrepreneur who wins: Ells will be a mentor as they open three locations in LA, Minneapolis, and New York. I can't think of a better way for a budding restauranteur to gather great evidence.

*The first Chipotle opened near the University of Denver (it's still there, at 1644 East Evans Avenue).

How Vegas has kept the odds in its favor: Modern casinos do an impressive job of following the evidence.

I've been reading James Ellroy's Blood's a Rover ('violent story-telling with hep-cat syncopation' is the best I can describe it). Fiction, but illustrative of early Vegas reliance on crony capitalism and personal relationships.
Flash forward to now, and the continued success of Las Vegas (brief history here). It's a great example of modern management practices and the need to evolve or die.

The Experimenter column in MIT's Technology Review has a piece about Gary Loveman. He's got some bio: PhD Economist, Federal Reserve Bank guy, Harvard prof, and CEO of Ceasars Entertainment. (Many thanks to Arie Goldshlager for sending me this link - I follow him on Twitter: @ariegoldshlager.)

I love this guy. Loveman says things like: "We need to overcome hunch and intuition with empirical evidence. We've set up a process and a discipline for evaluating our intuitions and improving our understanding of what our customers prefer. We can start with a hunch or strong belief, but we act on it through experiment. We want evidence. We've gone from the introduction of experimentation as a technique to a culture of experimentation as a business discipline. We hire people predisposed to do this by temperament and by background. Organizationally, we're committed - and I'm committed - to making sure we have the discipline to have the decisions we make informed by this evidence."

 A big casino operation is a lot like Yahoo! or Facebook - they're fortunate to have high customer traffic that supports lots of experiments. Loveman "has impressed the importance of data analysis on his employees, who are expected to quickly scale small tests into company-wide initiatives. For example, they might test which is likelier to get customers to spend more: a free meal or a free night in a hotel."

Crunch data or die. Asked about the biggest hurdles to adopting evidence-based methods, Loveman says "There's a romantic appreciation for instinct and, frankly, an absence of rigor for the application of more scientific approaches. What I found in our industry was that the institutionalization of instinct was a source of many of its problems."

There's no end state that is "evidence-based management". The goal is to get more people to consume valid evidence.

I ran across this: Assessing the Uptake of Evidence-Based Management: A Systems Approach by Georges Potworowski and Lee A. Green. The article appears in the March 2011 issue of Industrial and Organizational Psychology. (You can download the 3-page pdf here.)

Who consumes evidence? Who makes it available for consumption? I like how the authors talk about consumption of evidence, and the need to understand the mechanisms that influence how it moves from origination into practice. They're right to suggest that we need to better understand the whole system.

In complex organizations, there's no end state where everybody is cheerfully practicing "evidence-based management": There's never complete evidence, and there's never complete agreement on what the evidence means (or even what counts as evidence). But I think we can agree that evidence-informed action is a good thing, so we need to create systems that make it possible.

Evidence mobilization? What Potworowski and Green suggest isn't a new idea. For years, people have tried to figure out how to get good information into the hands of people who can apply it (and how to get them to use it once they have it). They've studied barriers to dissemination and to uptake - whether technical or organizational.

We can think of "evidence" as discussed in this article as explicit knowledge derived from research findings. As the authors point out, the flow of medical evidence (or lack thereof) provides some useful examples for folks who want to encourage consumption of evidence within organizations. Other relevant efforts include:

• Knowledge transfer.
• Knowledge mobilization (here's a brief KMb description by Gary Myers (@KMbeing) from the KMbeing blog).
• Implementation Science is a journal of "research relevant to the scientific study of methods to promote the uptake of research findings into routine healthcare in both clinical and policy contexts."

Call for papers: Conference on Evidence-Based Human Resources Management.

Announcing the 7th Biannual Conference of the Dutch Human Resources Management network: This year's topic is Evidence-Based HRM.

The meeting is scheduled for 10 & 11 November 2011. Included will be keynote presentations from Susan Jackson (president of the Academy of Management) and Sara Rynes (former editor of Academy of Management Journal).

If you'd like to present, proposals are due 31-March-2011.