Evidence Soup

Struggling to explain your scientific evidence? What you need, my friend, is the perfect country & western song.*

It's always difficult to explain evidence to non-experts. Previously I've shared How to Explain Your Research at a Party. But the National Institute for Mathematical and Biological Synthesis (at the University of Tennessee) is taking it to a whole new level. NIMBioS has a Songwriter in Residence program to "encourage the creation and production of songs involving ideas of modern biology and the lives of scientists who pursue research in biology".

Each resident spends one month in the department, and is expected to write, compose, and produce several songs during that time. This is a serious effort. For example, this month's resident songwriter, Kay Stanton, aspires to become the Jane Goodall of music. (*According to the guidelines established by David Allan Coe, she'll need to work in some stuff about Mom, prison, and trucks.)

Move over, Bill Nye. This may sound silly to some, but I think otherwise: Cross-pollination and interdisciplinary learning are what we need more of. The process of explaining their research - and seeing it in a new light - can substantially benefit the scientists. And a layman's explanation of complicated evidence is always welcome.

NPR did a story this week on the NIMBioS program. Here's a video and interview with Jay Clark, one of the songwriters. His song Sexual Selection explains that "we all like to copulate". There may be a Top 40 hit in his future.

Health-evidence.ca supports evidence-informed decision making in public health.

Health-evidence.ca provides resources to support evidence-informed decision making in public health. The project is based at McMaster University (Ontario, Canada), intended to "promote an ongoing environment of collaboration between the research community and the decision-making and practice setting".

Get your evidence for free. The Twitter bio (@HealthEvidence) says "We search for, screen, and rate the latest systematic review evidence in public health and compile it in a free, searchable online registry." You can browse or search articles by topic - and filter specifically for narrative, meta-analysis,  or systematic review. All evidence provided is rated on a scale of 1 (weak) to 10 (strong). The site provides abstracts and links to the original published research (and pdf or full text of open-access information).

Workshop: Educating about evidence. I wanted to know more about health-evidence.ca, so requested information from Paula Robeson, RN, MScN, a Knowledge Broker with McMaster's School of Nursing (@paularobeson on Twitter). She told me about a workshop initiative for health professionals who work with evidence:

• Evidence-Informed Decision Making Workshop: One-week intensive course.
• McMaster University (Hamilton Ontario), May 1-6, 2011.
• Advance your skills in critical appraisal of research literature. Learn strategies for implementing evidence-informed decisions. Tutorial groups for health professionals in advanced practice, library services, oncology and public health, in addition to groups for faculty and other clinical nursing areas.

Evidence about the evidence. The folks behind the health-evidence project do their share of publishing, too. For instance, they investigated the role and effectiveness of knowledge brokering in A randomized controlled trial evaluating the impact of knowledge translation and exchange strategies.

Haynes' pyramid. The knowledge translation literature (including the article mentioned above) categorizes different forms of evidence useful to healthcare decision-making. These are sometimes placed on a pyramid, originally called Haynes 5S, now known as the 6S Pyramid of Pre-Processed Evidence:

1. Systems: Detailed decision-support services that match information from specific
   people (or populations) with the current best evidence.
2. Summaries: An integrated presentation of the evidence concerning options for a given health problem.
3. Synopses of syntheses: "Succinct descriptions of systematic reviews or meta-analyses" to inform an intervention.
4. Syntheses: A single set of findings that combines the results of multiple studies (including systematic reviews and meta-analyses).
5. Synopses of single studies: Brief descriptions of individual research studies.
6. Studies: Findings from individual research studies.

By developing their own rating system (scale of 1 to 10), health-evidence.ca is distinguishing between reviews based on strong evidence and others that provide weaker evidence.

You like insightful analysis of interesting evidence, yes? Then start Barking up the wrong tree.

I highly recommend the blog Barking up the wrong tree. Great stuff on psychology, decision-making, and such. Brought to you by Eric Barker - @bakadesuyo on Twitter. (Thanks to Douglas Heingartner for the heads up on this: I recently wrote about his Metastudies.Org.)

Just the interesting stuff. For each research topic he covers, Barker puts together a snappy summary that gets right to the point. Some examples of the things he writes about:

Do smart people perform worse than dumb people on some cognitive tasks? "[T]here are some cases where additional working memory has no benefit - or can even be a disadvantage. A great example of this can be found in Decaro et al's recent article in Cognition, who show that subjects with higher working memory capacity are actually worse at learning complex rules in a categorization task...

Can you tell if a politician is liberal or conservative just by looking at them? "[P]erceivers were more accurate when they rated politicians whose attitudes were opposite to their own position.... Finally, politicians who were rated accurately had higher chances of being reelected to the following parliamentary session."

Don't delay, start reading Barking up the wrong tree today. Barker provides citations to his sources. Also, I like how he includes links to related previous posts, and provides book recommendations. 

SAGE is the latest traditional publisher to open up some of the evidence: SAGE Open launches January 1.

I received an email from SAGE, a leading social science publisher, saying "Be the first to publish in SAGE Open." Here's their current landing page. This movement toward "open" journals is a step in the right direction. At least now they can accept and publish papers more quickly, unencumbered by a rigid editorial calendar. From their site:

"SAGE Open seeks to be the world's premier open access outlet for academic research. As such, unlike traditional journals, SAGE Open does not limit content due to page budgets or thematic significance. Rather, SAGE Open evaluates the scientific and research methods of each article for validity and accepts articles solely on the basis of the research.

"This approach allows readers greater access and gives them the power to determine the significance of each article through SAGE Open's interactive comments feature and article-level usage metrics. Likewise, by not restricting papers to a narrow discipline, SAGE Open facilitates the discovery of the connections between papers, whether within or between disciplines."

SAGE promises quick review and decision times, a continuous-publication online format, and global distribution via SAGE Journals Online. Now they are offering a $195 author acceptance fee (discounted from the regular price of $695.

 

MetaStudies.org features concise summaries of the latest peer-reviewed meta-analytical studies.

Metastudies.org features concise summaries of peer-reviewed meta-analytical studies - in categories such as economics, sociology, health, and education. I learned about it through Douglas Heingartner (@MetastudiesOrg on Twitter). It's a good-looking site, and I like what they're doing. They pose topics as research questions ("Do higher taxes on booze reduce alcohol-related death?").

What's a meta-study? They give an excellent description of what this is all about, explaining that a meta-study looks at "previous studies on the same topic. Say that one study suggests coffee can give you cancer, while another suggests coffee can help prevent cancer. Meta-analytic studies aim to clear up the discrepancies. Various definitions include:

• Any systematic procedure for statistically combining the results of many different studies (Wiktionary).
• The process or technique of synthesizing research results by using various statistical methods to retrieve, select, and combine results from previous separate but related studies (Answers.com).
• A meta-analysis combines the results of several studies that address a set of related research hypotheses (Wikipedia)."

Easily digestible recap. Metastudies says the site "doesn't add editorial insights, but instead summarizes key findings." For each review, they link to the source of the meta-analysis, and recap its purpose and results. (“Hybrid diesel buses were found to be the technology that has the lowest emissions for most of the studied pollutants.”)

For example, the recap of the alcohol tax study provides the "Source: Effects of Alcohol Tax and Price Policies on Morbidity and Mortality: A Systematic Review, by A.C. Wagenaar et al., in the American Journal of Public Health, September 23, 2010." and "Purpose of study: We systematically reviewed the effects of alcohol taxes and prices on alcohol-related morbidity and mortality to assess their public health impact.”

We still have some questions. The Metastudies site, on its own, can't fix the fact that research recaps are sometimes too vague. For instance, “Adherence to the Mediterranean diet was associated with a significant reduction of overall mortality… cardiovascular incidence… cancer incidence… and neurodegenerative diseases.” What is a "significant reduction"?

The girl interrupting my rant about passive writing makes some good points. But I respectfully disagree.

[Quick commercial announcement: I'm on Twitter @EvidenceSoup.]

Sylvia McLain, who writes the blog Girl, Interrupting (cool name), took issue with my recent post Hey you, the one with the scientific evidence. Enough with the passive writing already!. She argues that passive voice is actually good for science; in some respects I agree, but in others I don't.

Data, collect yourselves! I'd commented on the plain-language movement, and on a Harvard Health Blog post Medical journals: Stop being so passive. In her post It’s not about you, its about the data!, McLain points out that "Science is and should be about the data and it is damn nigh impossible to write about data in the active voice because data don’t collect themselves." True - and that's precisely why I believe active voice is needed: Data don't collect themselves, people do it.

GI (@girlinterruptin on Twitter) provides some good unscience-y examples, where passive voice seems right. She goes on to say: "in technical science journals you will see things like: ‘The data was collected’ (passive) which is about the data. Rather than ‘I collected the data’ (active) which is about YOU collecting the data in the first person...." And she continues: "As scientists we teach our students to write about the data – which are the object of any experiment. Why? Because the science is about the data and the data are INDEPENDENT of who did the experiment (well except if there is fraud). Science is about the physical world around us which is exhibited by the data and not the person doing the experiment."

But I beg to differ: The investigator isn't independent of their evidence, no matter how closely they follow good research practices. Data almost never speak for themselves, and I don't believe we can separate the science from the scientist. When we conduct research, we make important choices while setting priorities, selecting techniques, and presenting findings -- and we're influenced by our particular values, unrecognized biases, motivations, and experiences. (I'm not saying that no data can be viewed objectively. The degree to which outside factors influence an interpretation of the evidence varies from one field to another. Some forms of evidence are more transparent than others.)

They're not fraudulent - but many published research findings are still false. Besides taking an objective look at a set of data, we also need answers to questions like: 'Why did the researcher choose this set of variables to analyze?' and 'Why did they apply this statistical technique?' See my previous posts on Ioannidis' critique of published findings and on why good statistics can make bad evidence.

Color me proactive. I believe writing in an active voice can remind researchers to explain themselves: "I did this because ...." rather than simply saying "the data were collected over X # of days, blah blah blah". And active voice can encourage people to draw conclusions like "The evidence shows that ....", making it easier for the rest of us to understand the implications of their research.

 

Interview Wednesday: Rick Austin, health communicator.

Today for Interview Wednesday, we talk with Rick Austin, Senior Communication Specialist for the Research Into Action project at The University of Texas School of Public Health in Houston, Texas. (View Rick's blog here or follow him on Twitter @KTExchange.)

The Five Questions.
#1. What got you interested in evidence?
I would say that this job is what really got me focused, but I had always been interested, as a layman (which I still am!), in asking “how do we know this?” Related to this, my daughter, when she was a senior in high school, took a class entitled The Theory of Knowledge, which was a year-long conversation about “how do we know what we think we know?” I would’ve loved to sit in on that all year.

What types of evidence do you work with most often (medical, business research, statistics, social science, etc.)?
Since the Research Into Action (RIA) project is based in a school of public health, that’s where we generally start, with empirical public health research.

What is your involvement with evidence: applying it, advocating its use, researching/developing it, synthesizing/explaining/translating it, communicating it?
Not quite all of the above. RIA is a knowledge translation project, so that encompasses advocacy, synthesizing, translating, communicating.

Where do you go looking for evidence, and what types of sources do you prefer? (formally published stuff such as journals, or something less formalized?)
We start with peer-reviewed journal articles as a basis for the subject we’re translating, but then we’ll go wherever the search for support and amplification takes us. We’ve generally found that a single peer-reviewed research project may be a good starting point for the conversation, but the nature of empirical research is such that any useful conclusion is going to be too narrowly focused. So, we find ourselves looking for similar research that will help support our viewpoint.
For example, we did some knowledge translation work on a study that originated here at the UT School of Public Health, looking at physical activity and academic achievement among elementary school students. The PI found some reliable correlations between increased physical activity and improved academics. However, the single study wasn’t enough to hang our hat on: The sample was fairly small, and the improvements weren’t across the board. We went looking for, and found, a slew of related studies on physical activity and improved self-esteem, improved on-task behavior, and improved classroom management, to name a few. So, we broadened our focus slightly, and had a strong argument for increased physical activity.

#2. On a scale of 1 to 10, where 10= ‘It’s crystal clear.’ and 1=’We have no idea why things are happening.’, how would you describe the overall “state of the evidence” in your primary field?
That depends on what you define as “my field.” I think the field of public health research is very strong, inasmuch as it’s populated by not only PhD public health field practitioners, but many who are also MDs and/or laboratory-based researchers. So, probably an 8 or 9. The field of knowledge translation/dissemination/exchange/social marketing, is extremely fluid in the United States, and also siloed in many areas. For example, the NIH’s clinical translation work is heavily focused on bench-to-bedside, and, they’ve got all the money. They might find it very enlightening to talk a little more freely with the social marketers and health communicators at the CDC. So, from the standpoint of knowledge translation, probably a 2 or 3.

Which of these situations is most common in your field?
a) Much of the evidence we need doesn’t yet exist.
b) People don't know about the evidence that is available.
c) People don't understand the evidence well enough to apply it.
d) People don’t follow the evidence because it's not the expectation.
Hmm, this is a hard one because the umbrella I work under is so broad. From the knowledge translation standpoint, I’ll have to go with “much of the evidence we need doesn’t yet exist.”

#3. Imagine a world where people can get the evidence they need, and exchange it easily and transparently. What barriers do you believe are preventing that world from becoming a reality?
Looking at this from the perspective of the average citizen/consumer, the two things that immediately spring to mind are information overload and lack of critical thinking skills. You and I have both blogged about all of these energy field necklace/bracelet/patch scams on the market, and they’re a perfect illustration of the general inability/unwillingness to think critically. Just a few seconds of thought (“Hmm, it’s a little piece of plastic held against my wrist by an elastic band; that’s silly, what could it possibly do?”) would restore some sanity to the consumer health discussion.

Where do you see technology making things better?
The ability to connect worldwide, the speed of dissemination (a double-edged sword), and something that nobody has even imagined yet – look at all the experimentation going on in geo-location, mobile health, new hardware platforms, ubiquitous internet connectivity – that will come as a complete surprise.

#4. How do you prefer to share evidence with people, and explain it to them? Do you have a systematic way of doing it, or is there a format that you follow?
Depends entirely on the evidence and the audience. Actually, here at RIA we’re trying to hammer out a second draft of a model of knowledge translation right now. We hope to describe a lot of the knowledge translation scenarios and weave some systemization into what is currently a very ad hoc process. We’ll post it on the discussion board at KTExchange when we’ve got it ready, and let people throw darts at it.

What mistakes do you see people making when they explain evidence?
Several of our podcast interviewees have beaten researchers about the head and shoulders on this subject (see how I put the onus on the interviewees there? Twasn’t me!). One point they’ve made is that many lab researchers have neither the motivation nor the communication skills to explain what they do to laymen. Joanne Silberner, formerly a science reporter with NPR, talks about trying to encourage a laboratory pathologist by telling him to explain something the way he would at a cocktail party. He got huffy and said, “I don’t go to cocktail parties.” Another obstacle to explaining evidence is the researcher’s instinctive aversion to definitive statements. Their research is never definitive; there’s always room for more exploration. The next-door neighbor doesn’t understand that mindset at all.

#5. What do you want your legacy to be?
Right now, I don’t see any long-term legacies in my work. It’s great fun and very challenging, but it’s my job, not my life. I’ll have to go with a cliché instead: I’d like to be remembered for being a good father and a good friend, both of which are way harder than I thought they would be.

Thanks, Rick.

____________________________

Chime in. Would you like to be interviewed, or do you have someone to recommend? Drop me a note at tracy AT evidencesoup DOT com.

Shattering myths and communicating the evidence: National Drug Facts Week looks at the science behind drug abuse.

The National Institute on Drug Abuse (a U.S. government agency) sponsors programs to communicate evidence about drug abuse and addiction. This is National Drug Facts Week, a campaign to get information to young people.

The organization has a NIDA for Teens site, covering the science behind drug abuse, and a blog called Sara Bellum (cute). There's also a Drug Facts Week web site and a Facebook page. NIDA is on Twitter at @NIDANews (using hashtag #drugfacts2010 for this week).

November 9th is the annual Drug Facts Chat Day, when NIDA scientists have a Web chat with teens asking questions about drugs.

What's your IQ? Anyone can test their knowledge on drug abuse by taking the Drug IQ Challenge.

For instance, "What substance is most abused by high school seniors?" Marijuana, tobacco, alcohol, or ecstasy?

There's a contest (in conjunction with the Grammy Foundation) asking young musicians to submit original music and videos celebrating healthy living, or depicting the dark side of drug abuse. Who knows, maybe someone will write something on par with The Needle and the Damage Done, my personal favorite in that category.

Young people are the audience for the Drug Facts Week information. NIDA doesn't always list references for the evidence, such as in the Drug IQ quiz (they're trying to raise awareness, not write research papers). However, the Sara Bellum blog provides plenty of specifics about the science. As an example, the post First get drug smart, then think twice. presents information in plain-language terms and then links to scientific details.
Myth: "If I smoke cigarettes now and then, I won’t get addicted."
Think twice: "Each puff of a cigarette gives a smoker about 1 to 2 milligrams of nicotine.  Although that may not seem like much, it is enough to make someone addicted." bit.ly/cvgZvd
Following that link, the tobacco addiction page provides diagrams, scientific information, and references.

Hey you, the one with the scientific evidence. Enough with the passive writing already!

The plain language movement - which requires government agencies to use plain language when writing their publications - should be extended to researchers publishing in medical journals. This is a way to eliminate the dreaded passive language we find in medical writing. (The movement is growing - see the Plain 2011 conference in Stockholm or the Center for Plain Language.)

If you're interested in getting people to communicate evidence more effectively, don't miss this excellent piece on the Harvard Health Blog: In Medical journals: Stop being so passive, Harvard Heart Letter Editor P.J. Skerrett details the prevalence of passive language - and explains why it's harmful. [Thanks to Melissa Sweet (@croakeyblog) and @plain_language for this.]

Skerrett says "Although pundits occasionally rail against [passive] construction, no one has ever quantified its use in medical writing. Until now. Dr. Robert Amdur and colleagues from the University of Florida College of Medicine classified the voice of every sentence in 30 articles from each of three leading journals.... They did the same thing for 30 articles from the Wall Street Journal. The averages were telling. In the medical journals, 20% to 26% of the sentences were in the passive voice, compared to just 3% in the Wall Street Journal."

And this matters: "it gets in the way of effective communication, which should be the whole point of medical writing. It generates dreary, convoluted sentences that make medical research — much of it done with our tax dollars — even more inaccessible to most folks." Skerrett reminds us of Michael Crichton's NEJM essay taking medical writers to task, saying “Medical writing is weak.” (And Crichton would know.)

The research appears in AMWA Journal, Vol 25, No. 3, 2010: Use of the Passive Voice in Medical Journal Articles [pdf]. From the abstract:

"A common criticism of medical writing is excessive use of the passive voice, but there are no published data on its frequency in medical journal articles.... We studied the frequency of sentences with a passive voice construction in 3 types of articles from 3 medical journals: Opinion Papers, Review Articles, and Original Research Reports from the Journal of the American Medical Association, the New England Journal of Medicine, and The Lancet.... The median passive voice frequency was similar in all types of medical journal articles but much higher than in The Wall Street Journal articles (20%-26% vs 3%; P < .0001)."

I thought "we" would help. This finding surprised me: "There was no meaningful correlation between the frequency of passive voice and the use of first person pronouns (I or we) in the medical journal articles."

Calling for change. The researchers conclude that "Writing with a high passive voice frequency is a style of choice rather than a requirement for publication.... We recommend that medical journal editors make a passive voice frequency of ≤10% a publication requirement for all types of articles."

Amen to that.

Evidence Chart keeps people honest about the evidence.

I learned about a project called Evidence Chart through Peter Kraker on Twitter. Its purpose is to help "students to glean evidence from the research literature, articulate theories, and consider whether each piece of evidence supports or undermines each theory. Working scientists find the site useful for quickly creating a compact representation of the evidence for and against competing hypotheses."

Telling the whole truth. Evidence Chart isn't a fully fledged software product - but it's worth noting their simple approach to listing all relevant research/evidence, and identifying which items do, or do not, support a hypothesis that's being considered. Each piece of evidence is ranked (on a scale of -5 to 5) according to whether it opposes or confirms. The associated blog quotes Richard Feynman on scientific integrity, from his 1974 cargo-cult science commencement address:

"If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it.... In summary, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another."

Each Evidence Chart corresponds to a particular research question: For example, there's a chart about What is the role of sleep on hippocampus-dependent memory consolidation? (you'll need a free account to see this one - but you can view a simple example without signing in).

Competing hypotheses. The chart lists available evidence down the left-hand column, and across the top row lists competing hypotheses relevant to the research question. (In this example, the competing hypotheses are 'no role', 'passive role', 'permissive role', and 'active role'.) Also, you can view 'Dominant' descriptions, or 'Contrarian' descriptions if you want to hear what a devil's advocate would say about the evidence.

Of course, besides Evidence Chart there are many tools for ranking decision-making criteria. But few of them are about ranking evidence based on its support of a particular theory.