Baby Einstein debacle is a lesson for all of us: Publicity-hungry people use weak evidence to attack, draw attention to themselves.

Update: "The Walt Disney Co. said Thursday evening that it still wants the University of Washington to retract 'an inflammatory and misleading' press release concerning 'Baby Einstein' and other baby DVDs - a demand that was categorically rejected earlier Thursday by the school's president. The Disney Co. said it believes the press release was 'developed to gain media attention, and contradicts and distorts the study's own carefully limited and hedged findings.' On Thursday morning, University of Washington president Mark Emmert told Robert Iger, Disney's chief executive and president, that not only was the press release valid but so was the work of the three University of Washington professors whose study about baby DVDs and videos came under sharp attack by Disney."

It's every entrepreneur's dream: A Colorado startup created a fast-selling consumer product (Baby Einstein), and sold the company to a Fortune 100 firm (Disney). But I doubt they expected what happened this week: They're the focus of an ugly argument involving peer-reviewed evidence, a CEO attack on a University's reputation, a somewhat questionable press release, and a high-caliber PR machine.

Here's what happened. An August 7 press release -- titled "Baby DVDs, videos may hinder, not help, infants' language development" -- announced findings published in an article called Association Between Media Viewing and Language Development in Children Under 2 Years  [5-page pdf], saying in part that: "The scientists found that for every hour per day spent watching baby DVDs and videos, infants understood an average of six to eight fewer words than infants who did not watch them. Baby DVDs and videos had no positive or negative effect on the vocabularies on toddlers 17 to 24 months of age. The study was published today in the Journal of Pediatrics." Although the research wasn't limited to only Baby Einstein products, since they're marketed by Disney, they drew the most attention. The press release said the study suggests parents "probably should limit the amount of time they expose their children to DVDs and videos such as 'Baby Einstein' and 'Brainy Baby'."

Just because it's peer-reviewed doesn't mean it's even remotely interesting. Even if this is true, where is the evidence that 6-8 fewer infant words will matter in the long run? And couldn't the videos have other benefits not considered, such as time spent between baby and caregiver, focusing on learning-related activity in general? Another reason I doubt the long-term importance of the study: "Only 17 percent of 384 babies in the survey were put in front of videos for an hour or more each day. The average baby watched only about 9 minutes a day."

Disney said the press release was inflammatory, and came out with guns blazing to find fault with the study: "Our assessment, based on what we have been able to learn thus far, is that its methodology is doubtful, its data seem anomalous and the inferences it posits unreliable." The Baby Einstein story was covered rather widely in the media, such as NPR and the New York Times, and was discussed on Talk of the Nation. Many defended the videos. And some accused the researchers of going after Disney simply to draw publicity to themselves (if that was their intent, it does seem to have worked).

Fighting fire with fire. Disney's CEO demanded a retraction of the press statement in a letter to the president of the University of Washington. I like the way Disney went tit for tat, calling out specific weaknesses they saw. Their letter said in part: "While the press release highlights that the study is based on a survey of 1,008 parents of children aged 2 to 24 months, a closer examination shows that the study based its critical conclusions about the impact of baby videos on infants 8 to 16 months on a disturbingly smaller sample of just 384 children. Of this group, 44 percent watched no television of any kind, leaving a total of 215 infants with some television viewing -€” and with no indication whatsoever as to how many of this smaller number watch any baby videos at all. [my emphasis]  Whether your University is comfortable associating its name with analysis of this quality is, of course, your decision. And I would not be reaching out to you if all that was at stake was a poorly done academic study. But the actions of the University have caused much more to be at stake."

The University responded by expressing support for the research study. (At this point, did they have a choice? Anything less would undermine their reputation.) Much ado about nothing. I believe this incident is an unfortunate by-product of the publish-or-perish focus within academic communities: People must conjure up "significant" research to survive -- which *sounds* like a good thing -- but sometimes leads to publication of articles like the baby video piece, which offer little evidence, yet consume lots of people's resources.

How can you be ready with counter-evidence? What if your company -- big or small -- sells a product that comes under attack? Here are four common scenarios to prepare for:

1. Sell it, baby! If you have solid, defensible evidence, consider offering it ahead of time, possibly as a marketing tool. (I wrote about a recent example here: The Brain Fitness Program offers peer-reviewed, published research reporting results from a randomized, clinical trial.)
   
2. Emergency backup. If you have reasonable evidence, but it's not ready for prime time, keep it in your back pocket. Fight back with it if you must, being sure to explain it in clear terms that a general audience can understand -- but be sure to reference authoritative sources.
3. Launch an articulate defense. If you really don't have solid evidence of your own, do what Disney did: Find flaws in the other guy's evidence -- oftentimes, it's amazingly easy to poke holes in their argument. Be as specific, objective, and clear as possible about your criticisms.
4. Spin, spin, spin. If your own research produces evidence that is contrary to your value proposition, don't try to bury it: Learn from it, and then get busy improving your product. But be ready to spin it, in case that evidence gets into the hands of a competitor or other critic. (Of course, if the evidence shows your product is likely to fail or harm someone, all bets are off: You must distribute it to the appropriate channel.)
   

Comments

[Tracy Allison Altman]

Matthew, Thanks for your insightful comments.

[Matthew Sheldon]

As a business consultant who uses the statistical techniques employed in this research, I have to say that this study is a complete joke. It almosts borders on academic fraud.

The results were based on a very weak sample and the data was collected in a phone survey, which just does not pass the credibility test right out of the gate.

Disney rightly pointed out that the study only focuses on a narrow age group, and is at best an anomalous statistical result and at worst simply shoddy research.

I have reviewed the regression formula and it fails to control for the amount of interaction a parent has with the child. This is such a horrendous methodological gap as to be unconscionable. It is quite possible that parents who do not spend time interacting with the child park them in front of the TV. This I can believe. But under such circumstances is it the video that limits language development or the lack of parental attention? The fact that this study cannot answer that question, yet claims to in the press release smacks of a desire to publish anything that will get picked up in the mainstream media. They then offer caveat after caveat once the bold claim in the headline is out there.

The result effectively claims that placing the child in front of a plain white wall with no stimulus is better for development than seeing a baby video. The very suggestion is, on its face, absurd. Yet we are asked to trust this on the basis of weak methodology and poor statistical control variables.

The researchers have thus far refused to post their data, which tells you something. My guess is they ran several hundred regressions and left out the right variables to sieze on a statistical anomaly to get a cheap headline. If they released the data, this would be obvious, but they have not. Hmmmmm.

Shame, shame, shame. The bar for "research" is as low as it has ever been and this proves it.