« We can get good evidence from the so-called gray literature, too. | Main | Does the evidence support a different approach to web search? »

Friday, 25 July 2008

Comments

Tracy Allison Altman

Chris, I agree with what you said. Evidence-based management is really a goal, like evidence-based medicine... you're never finished. The evidence is always changing and there's always more to discover. I think the value of it comes from groups making a commitment to look at all the evidence available (within reason), objectively and consistently, so they can dig through it more quickly and apply it to their real-world activities.
By the way, I didn't think you were taking a shot at me :-)

Chris

Tracy,
it seems you have taken my statement out of context. From my previous comment:

"Which begs the question, did the author of this blog post actually read the Science article, or just the press coverage? I ask not to take a shot at the author, but because this raises a major concern I have with EB management: an incomplete understanding of the full body of evidence can lead to an incorrect conclusion, and consequently bad managerial decisions."

First, as I noted I did not bring this up to take a shot at you. Second, the only claim I made about you was that it appeared you didn't read the full article (I never said you should have read ALL the literature). If you did, then the disconnect is in your recollection of Summers' comments.

More importantly, and more relevant to the crux of this blog, is that it appears I was not clear in the previous post. The second part of the passage was not directly related to the issue of gender differences in math ability. I have no idea how a manager might use this information. My main point in that passage was about EBMgt in general. IN GENERAL, the practice of EBMgt could be derailed by managers who make wrong decisions based on erroneous summaries or interpretations of the existing evidence. For anyone who sees the value in an evidence-based approach to management, this is a hurdle we will have to be aware of and ready to take on.


Tracy Allison Altman

Chris, Thanks for your comments.
Yup, I ponied up $10 and read the entire Science article. And yes, as you said, "Summers' comments about the far right tail of math ability remains supported by the evidence." But is this interesting? You wrote that "an incomplete understanding of the full body of evidence can lead to an incorrect conclusion, and consequently bad managerial decisions." But what "managerial decisions" might be detrimentally affected if someone didn't know about this variance? I'd like to know what those might be.
Also, we already know that *lots* of things could explain the "underrepresentation of women at the highest levels in STEM careers" -- in fact, I mentioned one when I said "Others speculate that STEM careers simply don't appeal as much to women." So no fair saying I didn't look at the "full body of evidence," because no one does. Rather, I think you suspected me of not looking at the "full body" of statistics in one particular journal article... not the same thing. There's lots of other evidence the authors didn't address, some statistical and some otherwise: And neither did I, and neither did you.

Chris

The problem with evidence-based anything, is that there is still human error in reading and understanding the evidence. Case in point, this post. In reality, Larry Summers has (again) been supported by the evidence. Note the following passages from the Science article:

"Gender and variance. Another explanation for the underrepresentation of women at the highest levels in STEM careers has focused not on averages, but on variance, the extent to which scores of one gender or the other vary from the mean score....

"The variance ratio (VR), the ratio of the male variance to the female variance, assesses these differences. Greater male variance is indicated by VR > 1.0. All VRs, by state and grade, are >1.0 [range 1.11 to 1.21 (see top table on p. 494)]."

As an example, they highlight Minnesota:

"For whites, the ratios of boys:girls scoring above the 95th percentile and 99th percentile are 1.45 and 2.06, respectively, and are similar to predictions from theoretical models."

The media, of course, are running with the mean scores (which do show that there is no difference from the average male and the average female - something Sommers acknowledged). But Summers' comments about the far right tail of math ability remains supported by the evidence. (It also means that males overpopulate the far left tail - the really bad scores.)

Which begs the question, did the author of this blog post actually read the Science article, or just the press coverage? I ask not to take a shot at the author, but because this raises a major concern I have with EB management: an incomplete understanding of the full body of evidence can lead to an incorrect conclusion, and consequently bad managerial decisions.


ps. It should be noted that while the science article points to the differences between male and female test score variances, it doesn't explain WHY. Note the conclusion by the authors:

"There is evidence of slightly greater male variability in scores, although the causes remain unexplained. Gender differences in math performance, even among high scorers, are insufficient to explain lopsided gender patterns in participation in some STEM fields."

Answering the WHY - and producing solutions to undo the disparity - is where the real value lies.

The comments to this entry are closed.

Enter your email address:

Delivered by FeedBurner

Subscribe to the Soup feed

Evidence Soup is brought to you by Tracy Allison Altman.

Follow me on Twitter: @EvidenceSoup.
Read about my work at Ugly Research.

Why all the fuss about evidence-based ____?

Site search Web search

-->
Powered by TypePad