Toward Evidence-Based Criminology and Criminal Justice: The Campbell Collaboration
[This article was contributed by Anthony Petrosino (American Academy of Arts & Sciences and Harvard University), Robert F. Boruch (University of Pennsylvania), David Farrington (University of Cambridge), Lawrence Sherman (University of Pennsylvania) and David Weisburd (Hebrew University and University of Maryland). It is based on their manuscript under review at the Justice Policy Journal. For more information or to get involved, contact Anthony Petrosino at 617-576-5035 or e-mail apetrosino@amacad.org.]
An evidence-based approach to policy and practice requires that we learn from prior high-quality evaluation studies and integrate the results into decision-making. The Campbell Collaboration ( http://campbell.gse.upenn.edu) is a recently formed international organization created to help us learn what works and what does not from evaluation studies that test interventions relevant to social policy. This article briefly describes the organization and discusses some of the issues related to its primary goal, which is to facilitate the creation, updating, and rapid dissemination of high quality summaries of the scientific literature-known as systematic reviews-in criminal justice and other areas.
Why Are Systematic Reviews Needed?
When empirical evidence from evaluative studies are brought to bear on policy and practice decisions, traditional approaches are problematic. For instance, some rely heavily on a single experiment or "stand-alone" study to make relative judgements about what works. Sometimes we do have to act upon a single study because that is all we have. With each succeeding study, however, such an approach becomes more problematic. Rather than relying on only one or a few influential studies, reviewers have tried to conduct broad surveys of the literature to take account of many separate but similar evaluative studies in order to learn what works. Unfortunately some of these traditional and narrative reviews in criminal justice have been unsystematic, lacking explicit and transparent methods, preventing readers from understanding why studies were included or excluded, how they were critically appraised, and how the summary conclusions about what works were reached. Also, the criteria used by these reviewers to declare whether interventions were successful were questionable. Ironically, although reviews were often the most widely cited and influential papers, they were rarely subject to scrutiny in the same way as original research.
Emergence of a Science of Reviewing
Over the past hundred years, researchers who grappled with the problem of understanding how to interpret the results of separate but similar studies have created a body of methodological solutions we refer to as the "science of reviewing." Most of these developments occurred in the last 25 years. For example, in 1976, Gene Glass coined the term meta-analysis and developed one approach to quantitative reviewing of prior research. Among its strengths, meta-analysis provided reviewers with more precise ways to determine the "success" of interventions. The short history of meta-analysis has already proven valuable, as it has countered widely held assumptions about what works in all areas of social and medical science (Hunt, 1997).
Because a meta-analysis may not be explicit about methods, and not every rigorous review uses quantitative methods, the term systematic review (see box below) has become more popular to describe scientific syntheses.
Characteristics of Systematic Reviews
|
Problems with Current Systematic Reviews for Evidence-Based Policy
Although there is a growing consensus that systematic reviews can play an important role in evidence-based policy, there are some obstacles to overcome. For one, most systematic reviews are one-time efforts conducted only as long as grant money or investigator interest lasts. They are infrequently updated in light of new studies, new methodological developments, or cogent criticisms made by others.
Another obstacle is that a systematic review can be biased by a failure to gather all of the relevant evaluative studies. One reason is what is known as publication bias. Many investigators who conduct evaluations in criminal justice do not publish their results in peer-reviewed journals. Research in other fields has found that published articles more frequently report that interventions are effective than do unpublished documents like dissertations and technical reports (Smith, 1980). Failure to gather these unpublished or fugitive documents can potentially bias the review toward concluding that an intervention is effective.
Dissemination presents a major obstacle to evidence-based policy. Because reviews are made available in different outlets, there is a lack of uniformity in how they are reported. Policy and practice decision-makers also do not rely on academic journals-in which most reviews are published-for their research information (Weiss and Singer, 1988). Disseminating evidence to those who can use it seems haphazard at best. Print publications also can not devote space to updates and can take a year to publish findings. How can such obstacles be overcome?
The Campbell Collaboration
The Campbell Collaboration was officially established at an inaugural meeting held at the University of Pennsylvania in February 2000. It is named after Donald T. Campbell (1917-1996), an influential psychologist and evaluation theorist who drew our attention to the need for societies to assess more rigorously the effects of policies and practices. Its purpose is to facilitate the preparation, maintenance, and accessibility of systematic reviews of research on the effects of interventions. At the Inaugural, special coordinating review groups were created to oversee work in three major areas: Crime and justice, social welfare, and education. In addition, because decisions about the methods in original evaluative research and reviews ought to be evidence-based, a Campbell Methods Group was established to develop ways of doing this. The Communications and Dissemination Group was also formed at the meeting to help build bridges to potential users of systematic reviews. Work has begun on developing a register of evaluations to assist reviewers in finding studies and reduce potential for publication bias.
The precedent for the Campbell Collaboration is the Cochrane Collaboration in health care. Archie Cochrane was a noted epidemiologist with strong opinions about the lack of good evidence guiding many medical and health care practices and policies. He criticized the health care profession for ignoring scientific evidence in favor of medical school training, personal experience, and anecdotal successes and failures in making decisions. Such a choice, Cochrane argued, resulted in unnecessary death and suffering for patients. In 1993, the Cochrane Collaboration was formed to address his criticisms (www.cochrane.org). In short, the organization conducts systematic reviews of the evidence on what works in health care. Its major product is the Cochrane Library, an electronic publication available over the World Wide Web (or through CD-ROMs) that contains all the Collaboration's products. All Cochrane reviews are prepared electronically, structured with the same exact detail, and updated periodically to take into account new evidence. Already the Cochrane Library is widely regarded as the best single source of evidence on the effects of health care interventions, underscored by a recent decision by the National Institutes of Health to make it available to all 16,000 of its employees.
The Campbell Collaboration will build off the Cochane precedent but focus on what works in social policy. Using strict quality control guidelines, the Campbell Collaboration will produce scientifically rigorous and explicit reviews. The Campbell Collaboration has a multi-interest (not an exclusive research interest) to ensure that reviews are relevant and useful to decision-makers. Its multinational perspective is designed to overcome the usual English-only bias in conducting reviews and disseminating them. By using electronic publication via the World Wide Web, the Campbell effort hopes to overcome some dissemination problems posed by traditional journal outlets.
The Campbell Crime and Justice Group
This review group will coordinate the preparation, maintenance, and accessibility of systematic reviews of research on the effects of interventions relevant to crime and justice. These would include not only reviews of research on the effects of interventions designed to reduce crime or delinquency, but also reviews on improving the management and operations of the criminal justice system. A 13-member Steering Committee, chaired by Professor David Farrington of the University of Cambridge (U.K.) and representing 10 nations, guides the early development and activity of the Group. The UK Home Office supported, through a small seed grant to Professor Farrington, some early Group work. During year one, the Steering Committee solicited reviews in 25 areas of intervention, including boot camps, street lighting, restorative justice, child skills training and hot spots policing. The Australian Institute of Criminology hosts the Group's Web site (http://www.aic.gov.au/campbellcj).
Conclusion
The Campbell Collaboration intends to produce the best evidence on what works to inform decision-makers, researchers, and the general public. "Best evidence" means systematic reviews that are rigorous, updated in light of new studies and criticisms, relevant and accessible to end-users, cover studies published worldwide, and open to criticism and comment.
References