Evidence Soup

HealthNewsReview.org provides independent, expert reviews of news stories. Their mission it to "hold health and medical journalism accountable". The project is led by Gary Schwitzer (a journalism prof) and funded by the Foundation for Informed Medical Decision Making.

They clearly identify their 10 review criteria -- based on what they believe consumers need from stories about medical tests, procedures, and products. Kudos to them: "Criterion #4: Does the story seem to grasp the quality of the evidence? ...the relevant standard is that the journalist is expected to present information about the quality of the clinical evidence on which claims are based. In the case of a major claim about the efficacy of a new treatment this should be based on the results of a randomized trial with relevant clinical endpoints.

"Sometimes, the first story about a new treatment is in the form of promising results.... nothing intrinsically wrong with writing about this, but the story should make clear the limitations of the evidence, and contain a caution about interpretation of uncontrolled data. For example, a story about a non-randomized cohort or observational study should explain that researchers were not able to adjust for all factors that might account for an observed difference. The hierarchy of evidence is an important factor for journalists to consider, and to explain to readers and viewers...."

I'm especially impressed by Health News Review's apparent lack of anti-industry bias. They remind us that "Just because a study was funded by industry, it doesn't mean the findings lack integrity."

Other points they make about reporting the evidence: "We all should realize that not all studies are equal. Not all evidence is bullet-proof. There is not a certainty or a finality to everything that appears in the New England Journal of Medicine or any other journal.... But often, just because it's a study - or just because it's published somewhere - seems to suggest that the proof is in. The story is over. And it almost never is. That's not the way science works.... Before you jump to conclusions about research results after hearing some publicity, ask yourself:

-Is there any information on the limitations of the evidence?

-Was the study done in only a few people?

-Did the study report on an outcome that you really care about - like illness or death? Or did it only report on test results, markers, or scores?

-Did the information simply present anecdotes as evidence of a treatment's harms or benefits - rather than real numbers from the entire study group?"

They also provide tips on understanding studies to help journalists decipher things like absolute-vs.-relative risk and number needed to treat (NNT).