Yesterday I wrote about a new study showing that researchers look online for evidence differently, compared to what they did back in the library-stack days (see New evidence about how people get evidence online.). But that included only peer-reviewed journals -- and lots of solid evidence comes from other sources, what academics somewhat smugly call the "gray" literature. This got me thinking about how we could study citation patterns in the real world to identify roadblocks that might limit the transfer of useful information.
Quick stroll down memory lane. I've always been interested in citation analysis, and did some myself as part of my doctoral research. I wanted to look at the extent of fragmentation of available evidence -- and see how this fragmentation might impact a decision-making process. The primary purpose of my dissertation was to see when, and how often, people reach across their particular institutional or educational boundaries for evidence when they're dealing with interdisciplinary problems. This is still a major concern for me -- because evidence-based management can't really succeed if people don't go outside the comfort zones they developed in school while studying in a specific field.
Show me the evidence -- gray or otherwise! I looked at a collection of documents pertaining to a series of complex public policy decisions -- this included research reports, books, journal articles, and other material intended to influence the outcome. (I used U.S. Clean Air Act regulatory policy as my test bed because it's highly contested and there's little consensus on available decision-making tools). I wanted to see who contributed to the discourse, what evidence they contributed, and what threats they invoked when making their arguments. I did an in-depth content analysis of 400 documents -- because many of them were not peer-reviewed, and instead were "gray" docs not covered by citation index services, each citation was manually entered into a database. Meanwhile, I identified what types of evidence were presented by each contributor. Selected highlights from my findings:
Citation analysis:
- One of the references I tracked was cited by 10% of the authors in my study. However, only 6% of the references cited were cited by even one of the other authors.
- Co-citation rose slightly from the 1970s to late 1990s, as the percentage of documents sharing at least one citation increased from 78% to 89%. Documents authored by University researchers were most likely to share at least one citation with another in the study (40% of the time -- perhaps this reflects a more cohesive knowledge base within the academic community).
- 72% of the study authors cited at least one reference document, though only 34% provided a formal list of references.
Content analysis:
- Economic data was provided 26% of the time. Roughly 30% of the authors provided hard science data.
- Cost-benefit analysis was addressed 21% of the time, and risk assessment was discussed by 10% of the authors in the study.
Since 94% of the references I tracked were not cited by any other author in the study, does this demonstrate the need for a common base of air quality knowledge? Or is this fragmentation an unavoidable consequence of the complex nature of interdisciplinary policy problems?
Are these findings consistent with recent research? The Evans study I wrote about yesterday showed that people are referencing fewer articles now, and reaching consensus more quickly -- that agrees with the increases in co-citation I discovered. But the newest documents in my study were published in 1998, when fewer things were available online, so it would be interesting to see if citation patterns changed between 1999-2008. However, since I looked at all kinds of policy-related documents, not just scholarly journals, it might not matter -- lots of the document types I analyzed (e.g., research reports, books) would not be available in full-text form even today.
Maybe it's because of Google. On the other hand, the explosion of email, blogs, and web indexing might be what's making the difference. Even when documents aren't available online, it's now much easier to find out what other people are reading, writing, or saying -- so perhaps the trends noted by Evans might hold true for the "gray" literature. And maybe a future citation analysis should look at other indicators, such as when publications were indexed on Google Scholar. And another possible indicator: A commenter on ars technica noted that "There has been a shift in many journals to limit the number of references allowed in an article, that would also create some of the patterns seen."
Comments