Evidence Soup

Evidence reveals least effective medical treatments: Field-tested do's and don'ts for primary care physicians.

The Good Stewardship Working Group has developed The "Top 5" Lists in Primary Care: Meeting the Responsibility of Professionalism. Their work was published May 23 in the Archives of Internal Medicine (paywall); free abstract here. Here's how they describe the project:

Physicians can adhere to the principles of professionalism by practicing high-quality, evidence-based care and advocating for just and cost-effective distribution of finite clinical resources. To promote these principles, the National Physicians Alliance (NPA) initiated a project titled "Promoting Good Stewardship in Clinical Practice" that aimed to develop a list of the top 5 activities in family medicine, internal medicine, and pediatrics where the quality of care could be improved.

NPA members sought to identify "ideal" medical activities, meaning they would be common in primary care practice, would be strongly supported by evidence, would lead to significant health benefits, and would reduce risks, harms, and costs. A number of primary care activites were field tested in family medicine, internal medicine, and pediatrics: Participants identified common clinical activities that could lead to higher quality care and better use of finite clinical resources. Based on the findings, the Good Stewardship group is recommending that its "Top 5" lists of activities be implemented in primary care practice across the United States.

Medscape recapped the findings in Most Wasteful Treatments, Procedures in Primary Care (free registration required). "A study commissioned by a reform-minded medical society has come up with specialty-specific lists of procedures in primary care that do little if anything to improve outcomes but excel at wasting limited healthcare dollars."

Medscape concludes that many of the Top 5 recommendations consist of "don'ts." For instance: "A prime example is routinely ordering diagnostic imaging for patients with low back pain — but with no warning flags, such as severe or progressive neurologic deficits — within the first 6 weeks, according to the study."

To put the findings into practice, NPA plans to "produce training videos to help physicians discuss the guidelines with patients and gain their understanding and trust, lest patients think they are being denied needed care (a common scenario when they ask for antibiotics to treat a viral infection)."

 

Interview Wednesday: Leart Shaka, crusader for good evidence about vaccination.

For today's Interview Wednesday, we talk with Leart Shaka, Editor of Vaccine Times. I first wrote about him last year after discovering his blog. As a self-described layman, he's created an impressive repository of information about vaccines - and is an example of what one passionately curious person can do. (His journey also illustrates the pathetic state of reliable, user-friendly evidence: In a perfect world, his efforts wouldn't be needed because the appropriate resources would have already been available.)

Leart tweets as @skepdude and also @VaccineTimes. Since I first wrote about his project, he's put together a team and launched Vaccine Times. He explains why he has put so much effort into the project: "The need became apparent to me as I was verifying a claim made by a Generation Rescue report. The claim was straightforward: the US mandates 36 vaccines by the age of 5, while the average of another 30 countries was only 18. The claim itself does not sound like much, count the doses and compare. Nevertheless, I soon found that after a couple of hours of work, finding information from reliable sources, compiling it, verifying the calculations etc., just weren’t enough. If it was that time-consuming to verify a simple claim like that, I wondered, how much more time-consuming is it to verify the other claims?" Amen to that!

#1. What got you interested in evidence?
I don’t think there was any one thing, or any one event, that got me interested in evidence. I think all people are interested in evidence, and I am not different from the rest of humanity in that regard. The differences, however, between people have to do with the kind of evidence they require, and how consistent they are at applying these requirements.

As a skeptic I subscribe to the “extraordinary claims require extraordinary evidence” line of thinking. Believers of woo-woo on the other hand set a very low bar for their side, while at the same time demanding an almost impossibly high bar from the other side. To them the only criteria for evidence is “does it support my belief?” instead of “is this evidence reliable?” That is why an anti-vaccine proponent will accept an anecdote that supports his belief that vaccines are dangerous, while at the same time dismissing a mountain of scientific evidence that goes against that belief.

What types of evidence do you work with most often (medical, business research, statistics, social science, etc.)?
Since I decided to focus my skeptical activities to countering the anti-vaccine misinformation, I have been dealing mostly with medical evidence, in the form of scientific studies about vaccines. However, the science behind vaccines is not the only focus of The Vaccine Times; we also deal with suffering caused by vaccine-preventable diseases, and raising the public’s awareness about the dangers of not vaccinating. As such we also rely on information from government sources such as the CDC, the World Health Organization website, articles from news websites, other pro-health websites, blogs etc.

What is your involvement with evidence: Applying it, advocating its use, researching/developing it, synthesizing/explaining/translating it, communicating it?
I am not a scientist, as such my involvement with evidence falls more in the communication aspect, limited of course by my non-expert understanding of it. The main goal of The Vaccine Times is to publicize, to advertise so to speak, the science-based evidence about vaccines to the laypublic. However, we try to do that in a way that is easily accessible. That is why we need to rely on other kinds of evidence, besides the scientific one, such as the ones I mentioned in the previous question (news items, other websites and blogs).

Where do you go looking for evidence, and what types of sources do you prefer? (formally published stuff such as journals, or something less formalized?)
That depends on what I am trying to communicate to the public. If for example, I am summarizing a new study about vaccines, I go straight to the source, the actual study. In those cases I prefer not to rely on a news item describing the study, or a press release, I want to go straight to the primary source. In general, questions about the science, should be settled by looking at the scientific studies.

If on the other hand, I simply want to communicate a recent development, say an outbreak of an infectious disease, then I rely on news stories from various sources on the web. If the information about the outbreak is available in a government website such as the CDC, or a state’s Health Department website, I prefer to get the information there. If that is not available, I rely on news items from the web, which I read carefully to try and find out what their primary source is, and try to follow their primary source to ensure that the reporting on the news item was correct. For example, if the article says : “According to the Minnesota Health Department”, that is a clue for me to go to the Minnesota Health Department web page and do a search for the topic in question.

Same goes for blog entries. I follow their links to try to get to the primary source. Many times a blog entry will link to a news article, which in turn I have to peruse for its source of information and so on. Determining if a source of information is reliable, if it is “good enough”, is as much of an art as it is a science. You get better the more you do it. The basic idea is that you always want to go to the primary source of the information, if you can. If it is not possible for whatever reason, then secondary sources will have to do, but our reliance on such sources has to decrease accordingly.

#2. On a scale of 1 to 10, where 10= ‘It’s crystal clear.’ and 1=’We have no idea why things are happening.’, how would you describe the overall “state of the evidence” in your primary field?
I have chosen to spend my time on the vaccination issue. The rating I guess depends on the kind of question being asked. If the question is “Do vaccines reduce incidence of the infectious diseases they are meant to protect against?”, I would say the rating would be a 9.999999. In that case the science is as crystal clear as it can be that yes, vaccines reduce incidence considerably. However, in science there’s always a margin of error; there is always a bit of uncertainty for every result; I don’t think you can ever get a 10 in matters of science.

On the other hand, I may not be the best person to answer this question. I am not an expert in vaccines, so any answer I give is limited to my layman’s understanding of the evidence.

#3. Imagine a world where people can get the evidence they need, and exchange it easily and transparently. What barriers do you believe are preventing that world from becoming a reality? (data incompatibility, lack of research skills, information overload, lack of standard ways of presenting evidence, lack of motivation to follow evidence-based practices, ...)
Information overload is the main problem I see. Scientific knowledge has exploded over the past couple hundred years, and it is almost impossible for a layperson to properly understand the science behind any issue, be it vaccines, global warming or what have you. There are literally thousands of studies done on vaccines. Some of them are good, some bad. First of all, one needs to be able to access all of them, read them all. Then, one has to be able to distinguish the reliable, from the less reliable, from the unreliable studies. Then one has to somehow summarize the information on the first two and figure out where that totality of evidence is pointing. This is a tremendous task.

This process requires large amounts of time and expertise. Most laypeople, including myself, don’t have either, so at best we get an educated “feel” for the verdict of all this evidence, but we have to be clear: our “feel” does not trump expertize. A layperson who has spent 20 years looking at an issue is not at par with an expert who has spent 20 years looking at an issue, and that is where most anti-vaccine proponents stumble. They think their Google University Education is just as, if not more, reliable than an expert’s education and experience.

Where do you see technology making things better?
I can see how technology can make information, scientific evidence, more accessible. In fact the Internet has already done so. With a few clicks I can find the abstract of any study I want, if not the full article. On the other hand we don’t yet have a technology that can distinguish between the reliable, less reliable, and unreliable sources of evidence. That is still up to the individual; and that is where we have a problem. There is nothing technology can do to make someone a better researcher. It’s a tool, and as all other tools if used properly it can be extremely useful, but in the hands of someone who does not understand how to use it, it becomes useless, and sometimes even dangerous.

#4. How do you prefer to share evidence with people, and explain it to them? Do you have a systematic way of doing it, or is there a format that you follow?
That depends on who you’re sharing with, what kind of evidence you’re sharing, and what your goal is. In general, I think, one has to tailor the format to the audience. There is no simple answer, no one format that can apply to all. You must communicate differently to a parent, than you do to a rabid anti-vaccine proponent, than you do to a science-minded individual, because all 3 care about different things. They are interested in different things, and if your goal is to make a difference, to persuade one person one way or another, one speech is not going to apply to all 3 scenarios.

In general, the idea is to understand the point the other person is making; if they are correct you want to reinforce that point by adding your evidence and if they are wrong you have to figure out how and where they went wrong, and then figure out which piece of evidence will address that error better, and then figure out what is the best way to communicate that piece of evidence.

Again, it’s an art and one gets better with practice.

What mistakes do you see people making when they explain evidence?
The mistakes are multiple. You get things such as cherry picking, i.e. choosing only the evidence that supports your point of view while disregarding contrary evidence; drawing conclusions beyond what the evidence supports; not applying the same criteria to your side as to the other side, in a few words: all the logic 101 fallacies and errors.

#5. What do you want your legacy to be?
I’d like to be remembered for being The Vaccine Times guy; the guy who figured out how to get the pro-health message to the parents in a way that made it easy for them to digest. I would like to be able to dilute the anti-vaccine propaganda as much as possible, to help parents find their way to science-based evidence about vaccines.

Vaccines are extremely important, millions of lives have been saved with the advent of vaccines and there are some today who are pushing people away from life-saving vaccines. We owe it to the next generation to try and fight this kind of misinformation to the best of our abilities. I guess if I had to put it in a few words I’d like to be remembered as the guy who tried to do the right thing for our children.

Thanks for sharing these insights, Leart. You are truly an inspiration.

____________________________

Chime in. Would you like to be interviewed, or do you have someone to recommend? Drop me a note at tracy AT evidencesoup DOT com.

Learning to hug it out: The competing logics of the physician-patient relationship and business-like, evidence-based health care.

Here's an interesting take on the struggle to shift toward evidence-based medicine: Managing the Rivalry of Competing Institutional Logics, by Trish Reay and C. R. (Bob) Hinings. This Organization Studies article is free from Sage Online through April.

Sorry, I don't follow your logic. The authors look at institutional logics, which provide our organizing principles: they're "the basis of taken-for-granted rules guiding behaviour of field-level actors, and they ‘refer to the belief systems and related practices that predominate’.... Logics are an important theoretical construct because they help to explain connections that create a sense of common purpose and unity". But there's often rivalry between an incumbent logic and a New World Order.

Oh, Canada! When the Alberta (Canada) provincial health care system introduced "new business-like health care", it presented "a direct challenge to the previously dominant logic of medical professionalism. [Originally] the physician – patient relationship guided all service provision. Physicians used their professional knowledge to determine appropriate care for their patients, and other professionals followed physician requests or orders. Patients were expected to follow medical direction. And it was the role of government to provide sufficient funds to meet the need determined by physicians. Thus the logics of medical professionalism and business-like health care were competing. Each was associated with different organizing principles, and each required a different set of behaviours from actors within the field."

Alberta adopted new governance structures to increase efficiency and do more with less. The principles "associated with a business-like logic were cost-effective treatment, lowest-cost provider and customer satisfaction. In particular, physicians were singled out as significant cost drivers and identified as a major source of current problems. Consequently, physicians were legislatively excluded from holding board positions in the newly created Regional Health Authorities (RHAs).... Physicians did not agree with the principles of business-like health care as set out by government. They did not believe that patient care should be provided based on government determinations of cost-effectiveness and patient satisfaction. They particularly did not believe that their behaviour should be determined by those principles."

Let's hug it out. The health system and the physicians found some ways to work together. My favorite? "Working together against the government" (p. 641), which provided common ground for collaborating. They also separated medical decisions from other RHA decisions, and collaborated on experimental sites.

The data. The authors used archival documents, interview data, and their own observations over time as they investigated organizational change. A sample finding: "Physicians were angered by government actions excluding them from positions on RHA boards and proposals for substituting other (less expensive) professionals for physicians when possible. They believed that their opinions and expertise were no longer considered valuable and they spoke out in letters to the editor and other public outlets."

The findings: Managed rivalry. From the research abstract: This study "identified four mechanisms for managing the rivalry of competing logics that facilitated and strengthened the separate identities of key actors, thus providing a way for competing logics to co-exist and separately guide the behaviour of different actors."

A must-read: Yudkowsky's gentle, brilliant explanation of Bayesian reasoning and evidence. And why we overestimate breast cancer occurrence 85% of the time.

Happy Fun-with-Evidence Friday. No videos today, just an important lesson told in an entertaining way. Eliezer Yudkowsky is a research fellow at the Singularity Institute for Artifical Intelligence. Boy howdy, can he explain stuff. He's written some great explanations of Bayes' theorem for non-practitioners. [Thanks to @SciData (Mike Will) for linking to this.]

How Bayesian reasoning relies on evidence. Bayes depends on something called 'priors': These are prior probabilities (think of them as 'original' probabilities before additional information becomes available). We use evidence to establish the value of these priors (e.g., the proportion of people with a particular disease or condition). Bayes' theorem is then used to determine revised, or posterior, probabilities given some additional information, such as a test result. (Where do people get priors? "There's a small, cluttered antique shop in a back alley of San Francisco's Chinatown. Don't ask about the bronze rat.")

Yudkowsky opens by saying "Your friends and colleagues are talking about something called 'Bayes' Theorem' or 'Bayes' Rule', or something called Bayesian reasoning. They sound really enthusiastic about it, too, so you google and find a webpage about Bayes' Theorem and... It's this equation. That's all. Just one equation. The page you found gives a definition of it, but it doesn't say what it is, or why it's useful, or why your friends would be interested in it. It looks like this random statistics thing." Then he walks through a simple example about calculating breast cancer risk for a woman with a positive mammography result. Very hands-on, including little calculators like this:

Risk, misunderstood 85% of the time? The scary part is this: "Next, suppose I told you that most doctors get the same wrong answer on this problem - usually, only around 15% of doctors get it right. ("Really?  15%?  Is that a real number, or an urban legend based on an Internet poll?" It's a real number. See Casscells, Schoenberger, and Grayboys 1978; Eddy 1982; Gigerenzer and Hoffrage 1995; and many other studies. It's a surprising result which is easy to replicate, so it's been extensively replicated.)"

Evidence slides probability up or down. I especially like Yudkowsky's description of how evidence 'slides' probability in one direction or another. For instance, in the breast cancer example, if a woman receives a positive mammography result, the revised probability of cancer slides from 1% to 7.8%, while a negative result slides the revised probability from 1% to 0.22%.

About priors. Yudkowsky reminds us that "priors are true or false just like the final answer - they reflect reality and can be judged by comparing them against reality. For example, if you think that 920 out of 10,000 women in a sample have breast cancer, and the actual number is 100 out of 10,000, then your priors are wrong. For our particular problem, the priors might have been established by three studies - a study on the case histories of women with breast cancer to see how many of them tested positive on a mammography, a study on women without breast cancer to see how many of them test positive on a mammography, and an epidemiological study on the prevalence of breast cancer in some specific demographic."

The Bayesian discussion references the classic Judgment under uncertainty: Heuristics and biases, edited by D. Kahneman, P. Slovic and A. Tversky. "If it seems to you like human thinking often isn't Bayesian... you're not wrong. This terrifying volume catalogues some of the blatant searing hideous gaping errors that pop up in human cognition."

You must read this. Yudkowky's Bayesian discussion continues with a more in-depth example, eventually leading to a technical explanation of technical explanation. I recommend that one, too.

Why do we frown on health guidelines that are based on expert knowledge? Sometimes that's the current, best evidence.

Is a medical guideline useless if it's not based on randomized, controlled trials? How much is expert knowledge (or opinion) worth? I'm asking myself these questions after reading Analysis of Overall Level of Evidence Behind Infectious Diseases Society of America Practice Guidelines, a new article in the Archives of Internal Medicine. Findings revealed that more than half of the associated recommendations are based on Level III evidence (expert opinion).

The question of how to weigh recommendations is relevant not only to healthcare professionals, but also to people in business, government, and the non-profit sector. I learned about these new findings via Gary Schwitzer's excellent Health News Review, which asked "New questions about medicine's 'best practice' guidelines: evidence or opinion?" (on Twitter: @HealthNewsRevu).

What's the fuss about? According to the abstract, the study analyzed 41 Infectious Diseases Society of America (IDSA) guidelines encompassing 4,218 individual recommendations. This analysis considered both the strength of recommendation (Levels A through C) and overall quality of evidence (Levels I through III). The researchers concluded that: "More than half of the current recommendations of the IDSA are based on level III evidence only. Until more data from well-designed controlled clinical trials become available, physicians should remain cautious when using current guidelines as the sole source guiding patient care decisions." (Ahem. Shouldn't they "remain cautious" even when recommendations are based on "better" evidence? Otherwise aren't they at risk of practicing cookie-cutter medicine?)

Sometimes expert knowledge *is* the current, best evidence. The basis of evidence-based management is making decisions based on current, best evidence: Randomized, controlled experiments are great. But they're not always available -- or in many fields, even possible. We can't disregard other forms of evidence: We're much better off systematically weighing the fuzzier evidence than we are disregarding everything that doesn't fit the narrow definition of 'Level I' evidence.

Here's a recap of the study findings:

• Quality of evidence. 14% of the recommendations were based on Level I evidence, 31% on Level II, and 55% on level III.
• Strength of recommendation. The investigators explain that among the Level A (strong) recommendations, 23% were based on Level I evidence (1 randomized controlled trial), while 37% were based on expert opinion only (Level III).

Maybe not apples & oranges, but different varieties of apples. The guideline story was widely covered in the medical media, including Fierce Health and Reuters. Reuters oversimplified the situation, saying "more than half the recommendations relied solely on expert opinion or anecdotal evidence." Okay, but that's what 'Level C' is for: To indicate a relatively weak recommendation. And since when is expert opinion not valuable?

Buyer beware. Here's what the Infectious Diseases Society says about its practice guidelines: They "are systematically developed statements to assist practitioners and patients in making decisions about appropriate health care for specific clinical circumstances [Institute of Medicine Committee to Advise the Public Health Service on Clinical Practice Guidelines, 1990]. Attributes of good guidelines include validity, reliability, reproducibility, clinical applicability, clinical flexibility, clarity, multidisciplinary process, review of evidence, and documentation. It is important to realize that guidelines cannot always account for individual variation among patients. They are not intended to supplant physician judgment with respect to particular patients or special clinical situations. IDSA considers adherence to the guidelines... to be voluntary, with the ultimate determination regarding their application to be made by the physician in the light of each patient’s individual circumstances."

 

Interview Wednesday: Robert W. Dubois, Chief Science Officer at the National Pharmaceutical Council.

For Interview Wednesday, I'm delighted to feature Robert W. Dubois, MD, PhD, the Chief Science Officer at the National Pharmaceutical Council. He demonstrates a deep understanding of the difference between imagining evidence-based medicine and actually making it happen: All of us should listen and learn.

At NPC, Dr. Dubois oversees research on policy issues related to the harmonization, dissemination, and communication of comparative effectiveness research, as well as on how health outcomes are valued. Previously, he was the Chief Medical Officer at Cerner LifeSciences, focused on comparative effectiveness and the use of an electronic health records infrastructure to implement clinical change. (Dubois is located in Washington, DC and Los Angeles, CA, USA.)

NPC (on Twitter: @npcnow) describes itself this way: The "overarching mission is to sponsor and conduct scientific analyses of the appropriate use of biopharmaceuticals and the clinical and economic value of innovation. The organization’s strategic focus is on comparative effectiveness research and evidence-based medicine for health care decision-making, to ensure that patients have access to high-quality care." Amen to that.

The Five Questions.
#1. What got you interested in evidence-based medicine? What has been your involvement with evidence: applying it, advocating its use, discovering it, synthesizing/explaining it, and/or communicating it?
If I had to sum up my career, it would be that I have tried to help define what is appropriate care as well as to develop approaches to ensure that appropriate care would be given to patients  (and inappropriate care would neither be given nor paid for).  Although this would not be a formal definition of evidence-based medicine, it reflects the intent of it.

After my training in internal medicine and my fellowship in health policy and research methods (Robert Wood Johnson Clinical Scholar at UCLA and RAND), I helped create an “expert system” software platform (at a company named Value Health Science which I helped co-found), used by insurance companies and HMOs in the late 1980s/early 1990s.  Built around nationally developed expert guidelines, this program matched a patient’s clinical presentation to those guidelines to determine whether proposed surgery would be appropriate. This prospective approach identified many patients where alternative less risky/costly therapies had not been attempted and where surgery might be avoided.   We were able to reach out to the surgeons and discuss alternative approaches. That experience taught me a substantial amount about the realities of evidence-based medicine and how it might be implemented.

In the mid-1990s, the disease management movement began and our group was very involved in designing some of the early approaches (at Protocare Sciences).  For the first time, the pharmaceutical industry got involved in not only the delivery of “pills” but also the wrap-around elements of care that might promote optimal patient outcomes (e.g., identifying patients most at risk, encouraging compliance with medication and lifestyle recommendations, close monitoring of clinical progress, and rigorous assessment of patient response). We were able to develop various tools to help educate patients as well as to assist providers in their patient care decisions.

Most recently, I worked at the Cerner Corporation, a very large electronic medical records company. Once again, my focus was on defining appropriateness, assessing cost effectiveness, and participating in the growing comparative effectiveness movement. I had the opportunity to work with many pharmaceutical and biotechnology companies and apply various evidence-based medicine methods to real-world problems. These approaches included:  analysis of patient care as documented in an electronic medical record, quantifying the burden of illnesses and identifying areas for improved care, and assessing the costs and cost effectiveness of various approaches to care. We also had an opportunity to build best practice guidelines into the electronic infrastructure which could alert providers when opportunities to help patients may not have been identified.

I joined the National Pharmaceutical Council about two months ago as their Chief Science Officer. My career has come full circle—beginning in the world of health policy and now being in Washington where comparative effectiveness and evidence-based medicine have become central issues.

During your career, where have you looked for evidence, and what types of sources have you found most helpful? (formally published journals, data mining, information services, etc.?)
Finding the “best” evidence presents various challenges. One typically begins with the published literature. However, key information may not yet have been published in peer reviewed articles. Adding information presented at conferences is also important, although those studies have not yet undergone the rigorous peer reviewed scrutiny that is needed.  Unfortunately, much “evidence” never gets published. This publication “bias” occurs because negative results (i.e., that a particular intervention did not work or was no better than alternatives) may not have been written up for publication or may not have been accepted by journals (who prefer positive results). Finding this “unpublished” information is important to get a complete view.

Complementing information from the above sources, I have always solicited input from expert clinicians. True evidence-based medicine needs to be a blend of what has been studied formally as well as the judgment of clinicians making day-to-day patient care decisions. Neither doctors’ opinions nor published literature by themselves can be sufficient.

#2. On a scale of 1 to 10, where 10=‘It’s crystal clear.’ and 1=’We have no idea why things are happening.’, how would you describe the “state of the evidence” in your primary field?

It is a mixed picture that I can best explain below.

Which of these situations is most common in your field?
a) Much of the evidence we need doesn’t yet exist.
b) People don't know about the evidence that is available.
c) People don't understand the evidence well enough to apply it.
d) People don’t follow the evidence because it's not the expectation.

Probably a, b, c, and d are all true. It has been estimated that at best, only one third to one-half of what we do as physicians has adequate evidence to support those decisions. The establishment and funding of the Patient Centered Outcomes Research Institute (PCORI) (as part of the Patient Protection and Affordable Care Act) will help over time to reduce some of these critical evidence gaps.  But, improving care is not just about creating more evidence. Perhaps only 50% of care that we know works is actually given to patients—so there is much room for improvement in the care that we provide.

#3. Imagine a world where people can get the evidence they need, and exchange it easily and transparently. What barriers do you believe are preventing that world from becoming a reality? (data incompatibility, lack of research skills, lack of standard ways of presenting evidence, lack of motivation to follow evidence-based practices, ...) And where do you see technology making things better?
Books have been written on this topic. I will highlight a few elements: more research is needed in determining not only what types of therapies are best for which types of patients, but also the manner that patients receive that care — should all diabetics be cared for by specialists, or by primary care physicians? In a group practice, or with individual physicians? Whomever they see needs access to full information about each of their patients. Until we have electronic medical record systems that can talk to each other (e.g., from the hospital, the specialist, the primary care doctor, the retail pharmacy), that goal will not be a reality.

And, it is more complex than just technologic integration. Each system must use the same patient ID, define key clinical terms in similar ways, and capture information in database fields rather than in just free text. Yet another key driver relates to the way we reimburse providers of care. In a fee-for-service environment, there will inevitably be silos of care. We will also reward volume of services provided rather than the quality of those services, the outcomes that patients achieve, or whether the care met evidence-based medicine principles.

#4. How do you prefer to share evidence with people, and explain it to them? Do you have a systematic way of doing it, or is there a format that you follow? What mistakes do you see people making when they explain evidence?
One size cannot fit all. Each stakeholder has different interests and among the many individuals, their level of domain expertise and understanding of research methods will also differ. To effectively convey information requires that the information be tailored to those individuals. Moreover, information will most likely be discarded or forgotten unless that information is provided at the point of decision making. I may read Consumer Reports each week and learn about the best car or washing machine. But, I will not really understand or remember the issues until I am actually shopping for that car or washing machine. In like fashion, we need to provide doctors and patients with the information that they need at the moments when they are making relevant decisions. I refer to these as “teachable” moments.

5. What do you want your legacy to be?
I’d like to be remembered for having a passion for understanding when care would be appropriate and helping to make it more likely to happen, and having done so in a scientifically rigorous way that is practical in the real world.

Many thanks, Dr. Dubois. Lots of food for thought here; I know the Evidence Soup community will be very interested in your perspective.

____________________________

Chime in. Would you like to be interviewed, or do you have someone to recommend? Drop me a note at tracy AT evidencesoup DOT com.

Health-evidence.ca supports evidence-informed decision making in public health.

Health-evidence.ca provides resources to support evidence-informed decision making in public health. The project is based at McMaster University (Ontario, Canada), intended to "promote an ongoing environment of collaboration between the research community and the decision-making and practice setting".

Get your evidence for free. The Twitter bio (@HealthEvidence) says "We search for, screen, and rate the latest systematic review evidence in public health and compile it in a free, searchable online registry." You can browse or search articles by topic - and filter specifically for narrative, meta-analysis,  or systematic review. All evidence provided is rated on a scale of 1 (weak) to 10 (strong). The site provides abstracts and links to the original published research (and pdf or full text of open-access information).

Workshop: Educating about evidence. I wanted to know more about health-evidence.ca, so requested information from Paula Robeson, RN, MScN, a Knowledge Broker with McMaster's School of Nursing (@paularobeson on Twitter). She told me about a workshop initiative for health professionals who work with evidence:

• Evidence-Informed Decision Making Workshop: One-week intensive course.
• McMaster University (Hamilton Ontario), May 1-6, 2011.
• Advance your skills in critical appraisal of research literature. Learn strategies for implementing evidence-informed decisions. Tutorial groups for health professionals in advanced practice, library services, oncology and public health, in addition to groups for faculty and other clinical nursing areas.

Evidence about the evidence. The folks behind the health-evidence project do their share of publishing, too. For instance, they investigated the role and effectiveness of knowledge brokering in A randomized controlled trial evaluating the impact of knowledge translation and exchange strategies.

Haynes' pyramid. The knowledge translation literature (including the article mentioned above) categorizes different forms of evidence useful to healthcare decision-making. These are sometimes placed on a pyramid, originally called Haynes 5S, now known as the 6S Pyramid of Pre-Processed Evidence:

1. Systems: Detailed decision-support services that match information from specific
   people (or populations) with the current best evidence.
2. Summaries: An integrated presentation of the evidence concerning options for a given health problem.
3. Synopses of syntheses: "Succinct descriptions of systematic reviews or meta-analyses" to inform an intervention.
4. Syntheses: A single set of findings that combines the results of multiple studies (including systematic reviews and meta-analyses).
5. Synopses of single studies: Brief descriptions of individual research studies.
6. Studies: Findings from individual research studies.

By developing their own rating system (scale of 1 to 10), health-evidence.ca is distinguishing between reviews based on strong evidence and others that provide weaker evidence.

What we all can learn from GRADE: How to translate evidence into recommendations. And how to improve people's understanding by presenting it differently.

It's tough enough to put together relevant, scientifically valid evidence. But it's equally difficult to synthesize it and make decisions based on it. Where medicine is concerned, the GRADE Working Group does a good job of translating evidence into actionable, plain-English recommendations. (GRADE= Grading of Recommendations Assessment, Development and Evaluation.)

Something we all can learn from. GRADE isn't just a way to rate pieces of evidence, but also to explain it and recommend what to do based on that evidence. The group describes the benefit this way: "A systematic approach to grading the strength of management recommendations can minimize bias and aid interpretation of expert-created medical guidelines."

Financial outcomes are evidence, too. I'm impressed that people involved in GRADE specifically recognize outcomes other than medical ones. It's not just about getting a particular health result - it's about recommending steps that will improve the overall well-being of the patient (and maybe everyone else involved). This overlaps the concept of comparative effectiveness. And it parallels the concepts of value-based medicine and patient-centered medicine, which I've written about before.

Summary of Findings Tables. The SoF table is a key piece of the GRADE methodology: It's where the evidence is summarized. SoF tables include several key items (displayed in columns), including:

• Outcomes: Important outcomes (desirable and undesirable).
• Assumed risk: Expected burden of the outcomes.
• Corresponding risk: The burden of the outcomes after the intervention has been applied.
• Relative magnitude of effect: The risk, odds, or hazard ratio.
• Number of participants/studies addressing these outcomes. 
• Rating of the quality of evidence for each outcome.

Presentation matters! As you can see in this example, SoF tables aren't pretty. But the format is a concise, organized presentation of lots of evidence - and it improves rapid retrieval.

There's evidence that the GRADE SoF approach is an effective way to present evidence. Based on small, randomized studies of people in an evidence-based practice workshop, "inclusion of an SoF table in a review improved understanding and rapid retrieval of key findings." (Results appeared in the June 2010 Journal of Clinical Epidemiology.)

Here's a recap of those results: Participants with the SoF table were more likely to “agree” or “strongly agree” that it was easy to find results for important outcomes than people without the SoF table (68% vs. 40%). And participants with the SoF table were more likely to correctly answer two questions regarding results than those without it: (93% vs. 44%, and 87% vs. 11%). Also, participants with the SoF table spent an average of 90 seconds to find key information, compared with 4 minutes for people without the SoF.

Tara Horvath has put together a good basic intro to SoF (PowerPoint). Also, you can download the GRADEpro software tool for developing a Summary of Findings table.

Translating evidence into recommendations. The recommendation is another key piece of the GRADE approach: "Judgments about evidence and recommendations in healthcare are complex. For example, those making recommendations must decide between recommending selective serotonin reuptake inhibitors (SSRI’s) and tricyclics for the treatment of moderate depression must agree on which outcomes to consider, which evidence to include for each outcome, how to assess the quality of that evidence.... Because resources are always limited... they may also need to decide whether any incremental health benefits are worth the additional costs."

Two flavors: Strong & weak. "Recommendations to administer, or not administer, an intervention, should be based on the tradeoffs between benefits on the one hand, and risks, burden and, potentially, costs on the other." Based on the available evidence, if clinicians are very certain that benefits outweigh risks, they make a strong recommendation. Here's an example of one: "Short-term aspirin reduces the relative risk of death after myocardial infarction by approximately 25%. Aspirin has minimal side effects and very low cost. People’s values and preferences are such that virtually all patients suffering a myocardial infarction would, if they understood the choice they were making, opt to receive aspirin. Clinicians can thus offer a strong recommendation for aspirin administration in this setting."

Resources. Here are some of the many valuable resources on the GRADE methodology (these pages link to free PDFs of the articles). Both were published in BMJ during 2008:

1. Going from evidence to recommendations.
2. Incorporating considerations of resources use into grading recommendations.

 

 

Vaccine Central is showing us the evidence on vaccination.

I ran across the blog Vaccine Central, making vaccine information readily available "so we may all make our decisions based on the facts not the hype". It's produced by @Skepdude on Twitter.

Taking a look at the evidence. On Vaccine Central there's a separate page for scientific evidence, linking to specific research articles and providing key quotes. There's also a page providing mortality rates for various diseases, as estimated by WHO (the World Health Organization).

The blog is intended "to make it easy for interested individuals, especially parents, to find good information on vaccines.... The idea for the Vaccine Central came to me as I was verifying a claim made by a Generation Rescue report.... If it was that time-consuming to verify a simple claim like that, I wondered, how much more time-consuming is it to verify the other claims?" Amen to that.

A good start. Compiling a comprehensive synthesis of all the evidence on vaccination would likely be an insurmountable task for most bloggers (and for most health professionals). I think this is a solid effort.

Skepdude asks for suggestions, so here are a couple: The site lists diseases and available vaccines. A side-by-side comparison in table format would be easier to read (e.g., are there diseases on the list without an available vaccine?). Also, I didn't quite follow the list of Categories - the organization is good, but the category names aren't links, so I wasn't sure what the purpose was.

Research into Action project (KTExchange.org) moves health-related evidence from the lab to the real world.

A new Research Into Action (RIA) network recently launched at KTExchange.org (KT = knowledge translation). The project comes from the Institute for Health Policy at The University of Texas*. It links to research, events, and funding sources, and provides a way to keep track of this stuff (by the way, they linked to an interesting site I'd missed, the Centre for Reviews and Dissemination, a "register of ongoing systematic reviews").

Knowledge translation is an effort to move research into the real world. (I'm all for that.) The institute has developed a model for disseminating findings: "RIA’s novel KT model places the widely recognized producer-push, user-pull model within a larger continuum of models and assumptions about the nature of knowledge and its production and use.... a flexible and adaptive mechanism for synthesizing and communicating findings..."

I registered at KTExchange.org and took a look at what they're doing. You can create a profile, sign up to receive notifications of new funding sources, search for literature citations or case studies, and learn about research-related events. You can also link to colleagues (sort of like connecting on LinkedIn).

I admire what KTExchange is trying to accomplish. These are early days (ver. 1.0) for the project, so I don't want to nitpick. But here are first impressions: Some of the site seems quite current, while some is a bit old-school. You can save links, such as citations, in a profile.

The site isn't particularly strong on interaction - even though you can "connect to colleagues". The social-network aspects are separated from the rest of the content. For example, when someone writes on the blog or discussion board, their name/photo aren't linked to their profile, so you can't easily learn more about these contributors - or connect to them as colleagues (e.g., this post by Shannon Rasp doesn't link to her profile). Likewise, when viewing someone's profile, the page doesn't include links to their contributions. 

*The Institute’s scope of work includes bridging and brokering, analysis, design and development, and education.

"Knowledge is the enemy of disease."
Tikki Pang, Muir Gray, Tim Evans - The Lancet (28-Jan-2006)