Evidence Soup

Blaming bed lunch for weight gain? Blaming sugar when your kids are hyper? The evidence says think again.

I love the term bed lunch. While a friend of mine was growing up in North Dakota, this is what they called a piece of pie, or other snack, people enjoyed right before bedtime. Now there's good news about bed lunch -- it doesn't contribute disproportionately to weight gain.

Rachel C Vreeman and Aaron E Carroll are pediatrics researchers fighting the good fight to bring more evidence into family life. Previously they've busted myths about drinking 8 glasses of water/day, fingernails continuing to grow after death, and other common beliefs. Now they've published new evidence challenging conventional wisdom about typical wintertime activities:

"In the pursuit of scientific truth, even widely held medical beliefs require examination or re-examination. Both physicians and non-physicians sometimes believe things about our bodies that just are not true. As a reminder of the need to apply scientific investigation to conventional wisdom, we previously discussed the evidence disputing seven commonly held medical myths. The holiday season presents a further opportunity to probe medical beliefs recounted during this time of the year."

Sugar makes children hyperactive. "At least 12 double blind randomised controlled trials have examined how children react to diets containing different levels of sugar. None of these studies, not even studies looking specifically at children with attention-deficit/ hyperactivity disorder, could detect any differences in behaviour between the children who had sugar and those who did not. This includes sugar from sweets, chocolate, and natural sources.... Scientists have even studied how parents react to the sugar myth. When parents think their children have been given a drink containing sugar (even if it is really sugar-free), they rate their children’s behaviour as more hyperactive. The differences in the children’s behaviour were all in the parents’ minds."

Bed lunch makes you gain more weight. There is some correlation between overweight and late-night eating, but "just because obesity and eating more meals at night are associated, it does not mean that one causes the other." Research shows "taking in more calories makes you gain weight regardless of when calories are consumed."

Poinsettias are toxic. (I blogged about this in 2006, doing my part to salvage the reputation of the beleaguered flower in No, Virginia, there really aren't any poisonous poinsettias).

Hatless people lose 1/3 of their body heat through their heads. Actually, you lose heat from any exposed area of your body -- proportionally. This myth resulted from a flawed army field study that just won't die.

Suicides increase over the holidays. Nope: "The epidemiological evidence just does not support that the holidays are a time of increased risk."

Permit me an Ebenezer moment: Among many others, our local CBS News covered these medical myths. I'm pleased the mainstream media picked up the story, though I still cringe when they say things like "highly scientific". And I wish their web coverage included links to the journals publishing the research, or at least the names of the published articles.

Vreeman and Carroll's article, Christmas 2008 Seasonal Fayre: Festive medical myths, can be cited as BMJ 2008;337:a2769.

What can a horse teach you about evidence-based management?

When my husband began his Executive MBA program, the first assignment was to fly to San Diego and go sailing with the cohort (having just completed my doctorate, I wasn't convinced this was "important" enough). But one of the tasks -- sailing with a 4-person team without uttering a single word -- got everybody's attention. Now there's an executive education program out of New York claiming to improve corporate performance and enhance communication skills.

The Horse Institute promises "You get measurable results. We are constantly improving our measuring tools to show the lasting benefits of our workshops. It’s not enough to send the participants back to the home office with great memories… you want training programs with long-term benefits. Our propriety tools are available to you; or we’ll work with you to develop new tools to fit your needs."

The institute says "If you have ever heard of Outward Bound or ropes courses, you are familiar with experiential learning.... based on the belief that adults learn best when they are engaged and actively involved in solving problems. These tried and true principles... have been well documented by Malcolm Knowles, the father of adult education (1913-1997)." I'm a big fan of his approach to andragogy, and wrote about it before.

Horse sense, or horse s#!t? Harry Hurt III of the New York Times was impressed by what he observed there, and he wasn't alone: "I’d secretly suspected from the beginning that the value of all this horse play would be minimal. But to my surprise, the group debriefing with Marie-Claude and the Horse Institute staff at the end of the day yielded testimonials that ranged from the predictably mundane to the sacred. A nurse coordinator with more than two decades of hospital experience said she would literally start to look outside her box-shaped office for more creative problem-solving techniques. One of her longtime colleagues said she’d realized that 'trying to make everybody happy is an impossible task.'"

And NPR went to see what this was all about, watching as people from an insurance company went through one of the programs. "The women are told they have to make one of the horses jump over a bar, but they can't touch the horse.... The exercise is aimed at getting people to think about how they solve problems and not assume there's a right or wrong way to tackle a task."

Measurement, schmeasurement. I'd be interested in seeing the institute's evaluation tools. I don't doubt this could be a worthwhile experience -- but things like this are really hard to measure.

Developers of Implicit Association Test claim to provide evidence about hidden biases.

I've been looking at Project Implicit, which says it "blends basic research and educational outreach in a virtual laboratory at which visitors can examine their own hidden biases. Project Implicit is the product of research by three scientists whose work produced a new approach to understanding of attitudes, biases, and stereotypes." There's a demonstration site for the Implicit Association Test, where you can complete their Three Countries or U.S. Election tasks. I tried them both:

• For Three Countries, you complete "short questionnaires and tasks in which you will identify words associated with China, India, and Japan and group them into categories as quickly as possible." First they asked if I preferred one country over another. Then they presented a series of words -- if I saw a "good" word, or a word associated with the particular country being tested, I was supposed to press "P" on my keyboard; for all others, I pressed "Q." So for India, I pressed "P" for things like "friend" and "curry". And so on.
• During the U.S. Election timed task, they showed photos of Obama and McCain, and I quickly clicked either "E" or "I" on my keyboard to indicate which I saw. Then they showed "good" and "bad" words and I did the same thing -- finally, they associated those things together.

What does the evidence say? Here are some of the conclusions they've drawn: "Findings observed in seven years of operation of the Project Implicit web site:

• Implicit biases are pervasive. They appear as statistically "large" effects that are often shown by majorities of samples of Americans. Over 80% of web respondents show implicit negativity toward the elderly compared to the young...
• People are often unaware of their implicit biases. Ordinary people, including the researchers who direct this project, are found to harbor negative associations in relation to various social groups (i.e., implicit biases) even while honestly (the researchers believe) reporting that they regard themselves as lacking these biases.
• Implicit biases predict behavior. From simple acts of friendliness and inclusion to more consequential acts such as the evaluation of work quality, those who are higher in implicit bias have been shown to display greater discrimination."

Gaining ground? The IAT folks say "The published scientific evidence is rapidly accumulating. Over 200 published scientific investigations have made use of one or another version of the IAT." On the Media's Brooke Gladstone interviewed one of the IAT creators, Tony Greenwald, recently (listen to Hidden Persuaders) -- but some commenters were unimpressed: Matt in Arlington, VA says "The test does not reveal anything at all! Matching words and images has absolutely no external validity. OTM should have done their due diligence before elevating a parlor game to the prominence of coverage on NPR. That due diligence would include familiarizing yourself with the statistical and methodological theories underlying experimental designs that Tony Greenwald purports to operate within. He does not operate within the methodological mainstream because he ignores the external validity of IAT."

Transparent evidence? Sort of. Related research articles are easily accessible from Project Implicit. MEGO alert! But my eyes glazed over, so proceed with caution: These are typical academic research articles, filled with dense language and an apparent inability to just get to the point. (See example* at the end of this post.)

Is there a practical application? I'm skeptical. Is this a big "so what," or can this approach teach us something meaningful about ourselves? Greenwald et al.* have said that "The IAT’s property of producing a palpable and possibly unsettling reaction during performance may be its central asset." Project Implicit offers corporate services, workshops, etc. that might have some value: "We provide a compelling face to face demonstration of the pervasive role of unconscious mental processes in everyday behavior. We can develop segments drawn from organizational situations that elicit unintended biases." Now if they could develop a way to test for implicit biases against evidence -- and the pursuit of evidence-based management -- we'd really have something.

*Excerpt: "Any new psychological measure is under challenge to establish its validity. Validity may include theoretical value (construct validity), empirical value (predictive validity), or applied value (ecological and consequential validity) [Ed. note: So far, so good.] Blanton and Jaccard... offered a new validity label, arbitrariness: 'We define a metric as arbitrary when it is not known where a given score locates an individual on the underlying psychological dimension or how a one-unit change on the observed score reflects the magnitude of change on the underlying dimension'.... Like Blanton and Jaccard, we use meaningfulness as the opposite of arbitrariness, and we therefore take metric meaningfulness to be the appropriate label for their preferred alternative. Blanton and Jaccard’s (2006) conception of metric meaningfulness is similar to and is largely contained within the concept of consequential validity that was described in this journal by Messick (1995, also cited by Blanton & Jaccard). Messick defined consequential validity as the aspect of construct validity that 'appraises the value implications of score interpretation as a basis for action as well as the actual and potential consequences of test use, especially in regard to sources of invalidity related to issues of bias, fairness, and distributive justice'"

From Greenwald, A. G., Nosek, B. A., & Sriram, N. (2006). Consequential validity of the Implicit Association Test: Comment on the article by Blanton and Jaccard. American Psychologist, 61(1), 56-61.

Does the evidence support a lower drinking age?

A group of college presidents is asking lawmakers to lower the legal drinking age to 18. Their group, the Amethyst Initiative, is concerned about widespread binge drinking on U.S. campuses and believes the evidence shows this will help remedy the problem. They say that "Twenty-one is not working: Alcohol education that mandates abstinence as the only legal option has not resulted in significant constructive behavioral change among our students. Adults under 21 are deemed capable of voting, signing contracts, serving on juries and enlisting in the military, but are told they are not mature enough to have a beer."

Today NPR interviewed John McCardell, President Emeritus of Middlebury College. He makes an interesting argument. Let's look at the evidence. When I first viewed AmethystInitiative.org, I wasn't sure they were going to show us any evidence. The site appears to be targeting lawmakers and influencers. But on the page Why Sign? they list several documents under a heading Learn More About the Drinking Age Debate. Finally, we're getting somewhere. If you open the Choose Responsibility and Debating the Issues documents [pdf], you'll see page after page of analysis and data -- followed by dozens of references. So they get points for exposing the evidence they're using to make their case. However, their evidence should be easier to find and presented more clearly. And they could do a better job of using interactive tools that encourage people to blog about their organization, send info to a friend, etc.

But there's another side to the story. At least one interest group, MADD (Mothers Against Drunk Driving) is opposed to lowering the drinking age.

• MADD's site offers resources to people who want to lead  changes in their community. And they provide lots of research and statistics -- though their research page wouldn't really load properly for me (perhaps because I'm using Firefox?). In media interviews, the Amethyst group claims that alcohol-related deaths have *increased* since the legal age was raised to 21, but MADD disagrees. People on both sides seem are finding statistics to support their arguments: Some are focusing on data for traffic-related deaths, which is of course only one piece of the story.
• MADD has also established a site at Why21.org targeting young people. Why21 is more web-savvy, with an RSS feed, for example. But they don't include hard evidence or link to references on their Myths & Facts page, they mostly just present arguments.

So, who is right? Well, I am, of course -- isn't that the point of having my own blog? :-) Both sides make compelling arguments about this important (and emotional) issue -- they do a good job of encouraging people to sign pledges, take action, get involved, etc. If I had to decide based on what I've researched, I would side with the Amethyst folks and their comparison to Prohibition. But my opinion isn't based purely on evidence - I also believe it's silly to send young men to Iraq with guns in their hands if we don't believe they're responsible enough to buy beer. I'd like to see more consistency across our public policies.

Is your Greek mythology a bit rusty? (I probably knew this back in high school.) Here's where the Amethyst group got its name: "The word Amethyst is derived from the Ancient Greek words meaning 'not' (a-) and 'intoxicated' (methustos). According to mythology, Amethyst was a young girl who incurred the wrath of the God Dionysus after he became intoxicated with red wine. Amethyst cried to Goddess Diana for help. Diana immediately turned the girl into a white stone. Upon discovering what had happened Dionysus wept, and, as his tears fell into his goblet, the wine spilled over the white rock, turning it purple." I'll never look at my amethyst earrings the same way again.

Resolved: Open-minded teenagers can improve on established methods of tossing evidence back and forth at each other.

I often wonder which is most difficult: getting evidence, deciding what it means, or communicating it to someone who can use it. Generally speaking, I believe people make it more about theory and hard data when they're less experienced, and gain a greater appreciation for intangibles, context, and communication as they soak up more interdisciplinary and real-world knowledge.

Oops, wrong again. But a couple of teenagers recently reminded me how tricky interpretation is. I was on my high school debate team way back when -- so I looked forward to seeing the 2007 documentary Resolved, and wasn't disappointed (it's listed on Wikipedia, played on HBO recently, and is available on Netflix). Unlike olden times, debaters now talk really fast (like, hundreds of wpm fast) so they can argue as many points as possible -- it's gibberish to anyone not accustomed to listening to it (Barack Obama and Ronald Reagan, these kids are not). Lots of the debaters in Resolved were extremely impressive -- they could do intensive research and analyze alternatives far better than I could at that age. But I was most impressed by Richard Funches and Louis Blackwell -- who came from a Joe Schmoe public school rather than one full of privileged kids.

Evidence doesn't speak for itself. Not only did Funches and Blackwell win important competitions, such as the California State Championships, they did something unheard of: They began questioning the entire debate framework, arguing that simply speed-reading evidence to your opponent (and the debate judge) isn't necessarily the best way to discuss important issues, and that the process suffers from an inherent racial and class bias. (The two were inspired by the work of Paulo Freire.) They claimed the established approach left out considerations such as epistemology, and so they tried replacing the entrenched, supposedly objective debate style with arguments rooted in personal experience. Anyone who says teenage critical thinking skills are declining needs to see Resolved.

Looking back at my high school years, I suppose I liked it *because* it was all so logical and academic -- a nice, safe place to spill evidence. Little did I know how sheltered I was. I'm still debating this one. One of the policy questions I debated asked whether the electoral college violates the U.S. Constitution. Fast forward to 2000, when Bush took the presidency despite Gore's popular-vote victory: I can only hope debate teams are still arguing about that. (Recount, another film recently on HBO, offers a fascinating exploration of the tug-of-war over the 2000 election. I recommend it also.)

New evidence explodes stereotype that girls aren't good at math.

In the United States, people often lament that women don't go into the STEM (science, technology, engineering, mathematics) professions. Some speculate it's because boys are inherently better at math and science -- Lawrence Summers famously said as much back in '05 (see my post, Harvard's president declared full of 'white matter').

Others speculate that STEM careers simply don't appeal as much to women. In the past, there certainly was peer pressure on girls to choose 'suitable' careers. (Personal anecdote: Once, while walking to class my sophomore year as a Chemical Engineering major, a guy walking nearby struck up a conversation. After asking me what I was studying, he said "I don't think women should major in engineering." Sour grapes because he couldn't get through Algebra II? Maybe, but still appalling.) Things are getting better, though: The rise of technology and geek-chic can't hurt, especially when young women see stories about cool people like Marissa Mayer.

Do the math. Findings appearing in the July 25 Science article Diversity: Gender Similarities Characterize Math Performance show that "Standardized tests in the U.S. indicate that girls now score just as well as boys in math." The authors looked at lots of recent data and discovered that since the "1970s and 1980s, several crucial cultural shifts have occurred that merit a new analysis of gender and math performance. In previous decades, girls took fewer advanced math and science courses in high school than boys did, [which explained] ...superior male performance on standardized tests.... By 2000, high school girls were taking calculus at the same rate as boys.... Today, women earn 48% of the undergraduate degrees in mathematics." This is a big improvement over the days when Mattel thought it was a good idea for Teen Talk Barbie to say "Math class is tough!"

The Gender Similarities authors found that effect sizes for gender differences are trivial. "Of these effect sizes, 21 were positive, indicating better performance by males; 36 were negative, indicating better performance by females; and 9 were exactly 0.... In contrast to earlier findings, these very current data provide no evidence of a gender difference favoring males emerging in the high school years." ABC News ran a nice story on Charles Gibson's excellent nightly program, and ABC has another story here. Update: Good story about this also on SiliconValley.com, What math gender gap? I hope Lawrence Summers gets the message.

Don't put that pencil down just yet: Former SUNY provost says evidence shows we should keep the SAT.

[Thanks to Marta Mooney (Fordham University) for sending me this link.] Recently I wrote about some selective colleges who are dropping the SAT test as an application requirement: The scores are in, and it's evidence-based management: 1, standardized-test industry: 0. But Peter Salins, former Provost of the SUNY (State University of New York) system, offers up evidence that SATs are in fact good predictors of college success. At Minding the Campus, he argues passionately against dropping the SAT, saying recent decisions to do so are "empirically unfounded" and might place collegiate academic standards at risk.

In Abandoning The SAT - Fraud or Folly? Salins says "Let's begin with predictive validity. Among the countless studies done on this subject over the years, not a single one has failed to find a high correlation between SAT scores and academic performance in college, as measured by grades or persistence. ...I had my institutional research staff repeatedly review the relationship between SAT scores and academic success among our 33 baccalaureate campuses and their 200,000 + students, and found - as all the national research has confirmed - a near perfect correlation." He goes on to say "How about purported class and race biases? ...There is absolutely no evidence that such students, when admitted on other non-test criteria, have greater academic success in college than would have been predicted by their SATs."

Salins believes this trend could be the tip of an iceberg: "Frankly, if SAT defection were only limited to the rarified world of semi-elite liberal arts colleges, I would not care one way or the other. It's a free country; these are private institutions and if this is where political correctness du jour takes them, so be it. The greater potential harm in discounting SAT or comparable objective tests as admissions criteria will be visited on non-elite institutions, most of them public, that enroll the lion's share of American college students. They will drink the anti-SAT admissions Kool-Aid promoted by their richer, more 'selective' peers but won't have the resources to process and evaluate the kinds of subjective applicant characteristics that could serve as substitutes or surrogates for SATs. And, unlike their more affluent private peers, they certainly won't be able to compensate for the academic deficiencies of entering students with supplemental tutoring or counseling. I sincerely hope that SAT defection is only a fad, limited to small high-end institutions. The arguments advanced by the as yet tiny band of top colleges or universities that are abandoning the SAT are empirically unfounded, if not downright fraudulent. ...[T]he disparagement of the SAT as an admissions criterion by the top tier schools may set an unfortunate precedent for the rest, risking widespread debasement of national collegiate academic standards that are not all that high to begin with."

The scores are in, and it's evidence-based management: 1, standardized-test industry: 0.

(Note to email subscribers: Some of you recently received a Soup of the Day message containing several Evidence Soup posts from earlier this year. This mix-up was caused by Feedburner, a usually stellar blog feed service. My apologies for the inconvenience.)

Some educators and students have been questioning the value of standardized U.S. college entrance examinations, believing they do a poor job of evaluating performance and representing an applicant's potential to succeed in college. Over the years, several schools have eliminated the tests from their application processes. But yesterday Wake Forest University (Raleigh, North Carolina) announced that it will no longer require applicants to submit an ACT or SAT score -- this is big, because more selective schools are joining the movement.

Wake Forest is the first U.S. News & World Report "top-30" University to drop the requirement (students still may submit scores if they want to). In its news release, admissions officials said "The decision was made after a careful consideration of recent research done at various universities. 'While many top-tier universities are increasing their reliance on standardized testing in the admissions process, recent research suggests that standardized tests are not valuable predictors of college success,' said Wake Forest Provost Jill Tiefenthaler, the university’s chief academic officer whose office oversees admissions. Some studies indicate performance on the SAT is closely linked to family income and education level, while others suggest a possible testing bias against certain minority students."

Highly regarded Smith College in Massachusetts recently made a similar policy change, saying that "Evidence of correlations between race, household income and test performance, along with the recognition that SATs may not be the best predictor of academic potential, prompt the decision." Mount Holyoke and the University of Texas at Austin have also joined the list.

Those who oppose mandatory entrance exams often cite the experience of Bates College in Maine, where standardized test scores have not been required for 20+ years. A recent NPR story quoted Wylie Mitchell, the dean of admissions there, as saying that "academic performance has not changed as a result of that decision." NPR also quoted Robert Schaeffer of Fair Test, who is a critic of SAT exams: "What it measures is how well you take the SAT."

Why do we pay attention to business consultants/gurus?

Why do we hire famous management consultants to drop by and tell us how to run our business? And buy their books by the millions? Some recent articles annoyed me a little, though I'm not sure why. I've wondered before if my annoyance with management gurus results from either crankiness or envy, particularly in the case of Malcolm Gladwell (though I'm not alone, since I received plenty of emails from Evidence Soup readers skeptical of his claims after I covered some researchers who are producing evidence showing that the Tipping Point probably doesn't exist).

Yesterday's Wall Street Journal identified the most influential U.S. business thinkers/gurus (ranked based on Google search results, media mentions in LexisNexis, and academic citations -- read about their ranking methodology here). The Top 5 are Gary Hamel, Thomas Friedman, Malcolm Gladwell, Bill Gates, and Howard Gardner. (Robert Reich was No. 6 -- one of my friends in grad school called him a "policy hustler". Heh.) WSJ published two articles: New Breed of Business Gurus Rises and also Quest for Innovation, Motivation Inspires the Gurus. At least they didn't imply that these guys are above criticism, describing one of them this way: "Gary Hamel, 53 years old, is a prolific writer and speaker who has kept his popularity despite falling victim to a peril of gurudom: His 2000 book, "Leading the Revolution," lionized Enron Corp. (The Enron case study was removed from later editions.)" Now some are chipping away at Friedman's world-is-flat theory, claiming that nationalism is interfering: "Governments, as the story goes, are reasserting their role in the lives of individuals and businesses, causing barriers to rise. If true, that trend counters a premise of New York Times columnist Thomas Friedman's popular book, The World Is Flat, that the Internet and other types of IT are wiping away national borders."

So. What is it that bugs me? I've read too many business books and sat through too many keynotes to critique them individually here. Generally speaking, though, I've found that the advice of many guru types exhibits these flaws:

• They're not specific enough, or comprehensive enough. A few well-phrased anecdotes don't cut it when providing prescriptive, actionable advice.
• I always suspect they chose their examples conveniently, preferring the ones that support their "frameworks" (a problematic bugaboo of qualitative research).
• Rarely do they expose what types of evidence they used, tell us what variables were considered, describe how they arrived at their conclusions, and let us critique their methodology.
• Seldom do they have accomplishments of their own to examine and draw from (Bill Gates and Carly Fiorina being notable exceptions).

How do they command those big bucks? Sometimes an event organizer wants a well-known keynote speaker; nothing wrong with that. Another explanation for the popularity of some of these gurus is that "People have a natural desire to look for some kind of framework or a way of explaining what's going on." Well, no sh*t. I'm thinking people would be better off if they enrolled in some philosophy, business, liberal arts, and public policy courses and came up with a framework for themselves. But I suppose it's tempting to simply pay for a spoon-fed framework that can be cited to colleagues at key moments in meetings... I can almost see them rolling their eyes from here. Woops, there I go again, being cranky.

How can you encourage people to look at the evidence?

Those of us who are already believers can talk endlessly about the virtues of evidence-based management, decision analysis, data mining, etc. But how do we shake the trees and encourage more people to look rigorously at causes and effects, paying close attention to research findings and other available evidence? One way is to bring folks together for a seminar: By engaging an entire group, you can establish new ground rules for how people should assess problems and defend their recommended solutions -- without pointing a finger at the habits (or analytical shortcomings) of any particular individual.

HeadScratchers is a firm that helps people apply critical thinking techniques to business problem-solving, and I was pleased to see they are now offering open-enrollment workshops. I know Mike Kallett, the guiding force behind HeadScratchers, and can vouch for him as a helpful, thoughtful guy who has survived some tough business and product development challenges. Here are some of the workshop topics covered by HeadScratchers:

• HeadScratchers Critical Thinking Model.
• Elements and Tools of thinking, problem solving and decision making; with exercises.
• Critical Thinking Techniques and Templates with interactive exercises.
• Specific examples and exercises using real business issues from the attendees.