Here at Evidence Soup world headquarters, we assign show-me-the-evidence "soup can" ratings on a scale of 1 to 5. Now seems like a good time to explain how the ratings are assigned.
When someone has something important to say, we look for information that shows "Why should we believe it?" "Why does it matter?" and "Will people be able to find and understand it, so this information can really make a difference?" Oops, there’s not enough good data to go around. This isn't as simple as handing out 5 soup cans to every mega-statistical, double-blind study that gets published, and assigning only 1 or 2 to isolated success stories. And it's not always about "right" vs. "wrong." It's about getting information from as many knowledgeable sources as possible, and then clearly showing what the evidence says.
Go for 3. As explained below, anything at the 3-soup-can level or higher is a contribution of evidence that, on the surface, appears to be valid. But people at Level 4 or 5 do more, going beyond specific cases to show useful ways to produce, compare, review, or keep track of evidence.
There’s no “there” there. At Level 1, someone is making a statement without showing supporting information, and without showing they have first-hand knowledge of the subject matter. Examples: Claiming that "All the high-fructose corn syrup is why kids aren't doing as well in school these days." Or "That's obviously a waste of money." (If we were talking about things like faith or art, it would be OK to ask us to simply believe: But that’s not the focus of Evidence Soup.)
Tell me more. At Level 2, we can see they have some first-hand knowledge of the subject matter -- and they may have valid evidence to back up what they are saying. But if they don’t give us anything specific to grab onto, we can't assign more than 2 soup cans. For instance: “We are finding that treatment ABC works better than XYZ.” Or “They tried that software system at the last place I worked, but I heard no one liked it.” Level 2 warning signs include (1) not giving specifics (such as not showing what type of evidence there is); and (2) not providing a source or online identity to validate. Comparisons and testimonials can be useful, but we need more detail to see for ourselves.
Now we’ve got something to work with. We hand out 3 soup cans when we can see specific evidence, who put it together, and how. It doesn't have to be a bunch of numbers - qualitative evidence counts, too. And it doesn't necessarily mean the information is earth-shattering, or even completely accurate: But it does appear to have integrity. Level 3 example: A published study associating the occurrence of Alzheimer's with certain activities, like doing crossword puzzles. The evidence doesn’t have to establish cause-and-effect, or any universal truths. It just needs to be transparent and verifiable so the rest of us can decide if we believe it.
Open up a big ol' can of evidence. When someone is at level 4, they aren’t necessarily more “right” than someone at level 3. But they get 4 soup cans because they are making an effort to put things in perspective, showing how to use evidence in real life. For instance, they might be showing where the weaknesses are in a specific type of evidence: Such as explaining that a new painkiller may have performed great in a study, but that it was only compared to a placebo to establish a statistically significant result --- when it really should have been compared to other pain killers already in use. Or a Level 4 analysis might show how to apply evidence to actual practice: For example, a case study about how to get better adoption of best-practice guidelines. At Level 4, details are appropriate for the intended audience: We would expect different types of evidence in a CFO’s financial analysis, a patient handout, and a published, peer-reviewed study.
Power to the people. When we award 5 soup cans, it means someone has gone to the trouble of presenting evidence so people can quickly see what it is, why it matters, and what others have to say about it. Typically they use plain language and visuals (graphics, charts, etc.), instead of page after page of words and footnotes. And they provide straightforward ways for people to comment and add value, rather than follow traditional, one-way communication channels. Level 5 example: A corporate wiki where people can contribute evidence to support a strategic plan, easily and transparently. Another example: An open-access, online research journal, where the peer community can comment on the works being published, and contributions of evidence are interconnected.
Comments