Update: Since I posted this, the essay's author, John Ioannidis, has engaged in a discussion with other research experts over the validity of some of the claims in his 2005 essay. Steven Goodman and Sander Greenland published a response on 24-April-2007, titled Why Most Published Research Findings Are False: Problems in the Analysis. On June 26, Ioannidis responded in Why Most Published Research Findings Are False: Author's Reply to Goodman and Greenland. Meanwhile, Ramal Moonesinghe, Muin J. Khoury, A. Cecile J. W. Janssens contributed the essay Most Published Research Findings Are False—But a Little Replication Goes a Long Way.
-TAA, 20-Aug-2007
[original post begins here] When I wrote yesterday about impact factors and how they can distort our perceptions about the real value of people's contributions, I started thinking about other consequences of our reward system.
One of the things they don't talk much about in graduate school is the "file cabinet problem" -- the very real possibility that much of the "truth" discovered by researchers is filed away and never published. Why would this be? Because many findings reveal no significant patterns or "interesting" relationships between variables. And therefore they aren't considered valuable for publication -- even though they may tell the truth about our world.
A related consequence is that researchers continually struggle to design studies and to choose variables that will yield something they can get published. And while their findings are typically based on reasonably solid work and statistical analysis, most research claims being published today are likely to be untrue. That is according to an essay by John P.A. Ioannidis in Why Most Published Research Findings Are False in the journal PLoS Medicine. He notes that "Research is not most appropriately represented and summarized by p-values, but, unfortunately, there is a widespread notion that medical research articles should be interpreted based only on p-values." The probability that a given claim is true depends on several things: Key factors include the number of studies done on the same research question, and the "ratio of true to no relationships among the relationships probed in each scientific field."
Citation: Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124.
Comments