Members of the new Evidence-Based Management Collaborative
met recently at Carnegie Mellon University in Pittsburgh, Pennsylvania.
The meetup included discussions of things like "How can we
make it practical for practitioners to make evidence-informed
organizational decisions?"
What the group is about. "The Collaborative’s primary task is to design the architecture and support practices for on-line access to best evidence summarized in ways practitioners and educators can readily use." This effort may lead to a sort of mini-Cochrane Library offering research findings, systematic reviews, and other evidence about management practices. (At its next meeting, scheduled for January 10, the group intends to involve more practitioners in determining what forms of presentation are most useful.)
I've reviewed some of the presentations by researchers who spoke at the recent meeting. (Not surprisingly, many of the materials on evidence-based management were developed in the context of health care and evidence-based medicine, since that is still the most active area in this field.) David Denyer of Cranfield University presented a very nice synopsis of EBM: He talked about different forms of evidence useful to researchers and managers, and provided some definitions, including this one: "What is evidence? It is information or data that people select to help them answer questions... anything that can be used as evidence to answer questions. One person's evidence is another's information." (Knight 2004). And also this one: "What is evidence? The available body of facts or information indicating whether a belief or proposition is true or valid." (Solesbury 2004, citing the Oxford English Dictionary).
Denyer showed a hierarchy of evidence that was developed at the Sheffield School of Health and Related Research:
Rank (in declining order of "objectivity"):
- Systematic reviews and meta-analyses
- Randomized controlled trials
- Cohort studies
- Case-control studies
- Cross-sectional surveys
- Case reports
- Expert opinion
- Anecdotal
But he acknowledged that this hierarchy makes more sense when applied to physical or scientific evidence, not evidence for social science. Hammersley & Pawson put it this way: Are we supposed to "administer 20cc's of strategy" and wait to see what happens?
Denyer also offered insightful observations about the appropriate role of evidence, reminding us that evidence does not speak for itself. And he discussed various ways evidence can be synthesized - examples include:
Synthesizing Qualitative Evidence
Narrative summary
Grounded theory & Constant comparison
Meta-ethnography, Meta-synthesis, and Meta-study
Domain analysis
Discourse analysis
Synthesizing Quantitative Evidence
Content analysis
Case study/survey
Qualitative comparative analysis
Meta-analysis & Bayesian meta-analysis
Quasi-statistics & Frequency (or vote) counting
Although these are considered valid research methods within the academic community -- and yes, I learned many of these techniques in graduate school -- this list illustrates perfectly why there's still such a wide gap between managers who need evidence to help them make real-world decisions, and the type of information being offered by many researchers. With all due respect, I can't see myself recommending to a client that we do a "meta-ethnography" ...at the very least, I'd have to give it a different name. Otherwise, it would never make it past their B.S. filters.
Comments