6: Measuring Outcomes and Evaluating Impact

About the Report The Report, Part I: Planning a Re-Entry Initiative:

chapters and policy statements:

About the Report of the Re-Entry Council

6: Measuring Outcomes and Evaluating Impact

Employ process and outcome evaluation methods to bring clarity to a program's mission, goals, and public value, as well as to assess and improve program implementation, efficiency, and effectiveness.

Overview

Full Overview »

Far too often, policymakers and practitioners implement re-entry initiatives without ever assessing the results of those initiatives, much less improving them once they have been implemented. Without such measures, several key questions will remain unanswered:

Is the program producing the desired results?

Is the program generating the greatest possible impact?

Is the program making the most efficient use of public funds?

Including an evaluation component in a re-entry initiative can enable program administrators and policymakers in a given jurisdiction to answer these questions, helping them to develop and prioritize goals, measure success in reaching those goals, and identify areas for future improvement. Strong outcomes or other program evaluation data can also provide a buffer against politically driven reactions to isolated incidents, such as when a released prisoner commits a particularly heinous crime while under supervision.

As further detailed in the recommendations below, a successful evaluation requires analysis of both a program's process and its outcomes. A process evaluation examines the theory underlying the program, how the program is administered, and ultimately whether the program is administered in accordance with its intended design. Such an evaluation also considers a program's operation in its entirety through performance measurements, which assess elements such as funding, staffing, and the number and type of clients served. While a process evaluation indicates whether an effort has been implemented and the extent to which its operation is consistent with its intended design, a rigorous outcome evaluation can identify the impact of a program, the reasons behind a program's success or failure, and whether the program is cost-effective. An outcome evaluation generally focuses on a program's effectiveness, examining whether the program achieved its intended goal of, for example, helping people who have been released from incarceration secure long-term employment or reducing public expenditures on re-entering individuals. Outcome evaluations often include intermediate measures to enable local policymakers and practitioners to monitor how agency staff and divisions are performing particular tasks and functions that are critical to producing a desired outcome as well as to make midcourse improvements to program design and implementation.

Evaluation is crucial to the ongoing success of a re-entry initiative. By generating a constant flow of detailed information about the program's operations, successes, and failures, process and outcome, evaluations help policymakers and program administrators make informed decisions about program design, resource distribution, and funding streams. The recommendations that appear below discuss the major evaluation methods that jurisdictions should undertake in order to understand and improve re-entry outcomes. This section does not, however, provide specific detail that can be applied to particular aspect of a re-entry initiative. Rather, it offers a broader discussion about various evaluation methods that should be considered by any community working to improve prisoner re-entry outcomes.

Before Embarking on Evaluation

One challenge for many jurisdictions is finding the resources, both financial and professional, to conduct evaluations using a methodology that meets the standards of the research community. To ensure that these standards are met--that, in effect, any analysis is meaningful and policies based on the research are accountable, state or local policymakers should designate a qualified person or agency to define, measure, and report on the success of a re-entry initiative. This designated person or agency must be given the authority and resources necessary to carry out any established evaluation goals.

Example: The Little Hoover Commission in California is a "bipartisan, independent state body that promotes efficiency and effectiveness in state programs." http://www.lhc.ca.gov/lhc.php

Example: The Washington State Institute for Public Policy carries out "practical, non-partisan research--at legislative direction--on issues of importance to Washington State."

http://www.wsipp.wa.gov/

A Note about Data : The success of any evaluation effort may ultimately rely upon the data available to researchers. While it is often desirable to collect original data on participants to assess the success of a particular program, such efforts are often costly. Thus, administrators should take into account both the shortcomings and assets of existing criminal justice data systems when identifying the data and methodology to be employed in the evaluation of a re-entry initiative. Indeed, much of the available data and information should have been collected and analyzed long before the program implementation phase, in the course of gathering information at the outset of a prisoner re-entry initiative. (See Policy Statement 2, Developing a Knowledge Base, for more on collecting jurisdiction-specific data on re-entry.) Importantly, such data must be collected and analyzed on an ongoing basis, rather than for a one-time evaluation. Further, given limitations on existing data and funding for research, policymakers and researchers should also consider collaboration and/or information-sharing with other agencies or data collection systems beyond those in the criminal justice field, provided that such sharing does not impinge on legally protected privacy interests. (See Policy Statement 8, Intake Procedure, for more on legislative and regulatory protection of privacy rights.)

Recommendations

Related Policy Statements

Our Publications

How and Why Medicaid Matters for People with Serious Mental Illness Released from Jail

Hundreds of thousands of people with mental illness are released from jail each year. Without continuity of care, they are likely to be reincarcerated. Enrollment in Medicaid increases access to treatment for people with mental illness released from jail, who typically lack other means to pay for those services.