NJJEC Assesses Evaluation Needs of OJJDP Grantees

Carrie Williamson, JRSA Research Associate

The National Juvenile Justice Evaluation Center (NJJEC), a JRSA project funded by the Office of Juvenile Justice and Delinquency Prevention (OJJDP), is designed to assist state, local, and tribal entities with the evaluation of juvenile justice programs and implementation of evidence-based initiatives. As part of this project, we recently completed a national assessment of the evaluation-related needs of OJJDP grantees. The purpose of the survey was to learn about the current activities of the grantees and their agencies or organizations that relate to evaluation and performance measurement, and to enable us to use NJJEC's training and technical assistance resources to meet their needs and help overcome their greatest challenges. A full report of the assessment findings, available on the NJJEC Web site, is being used to shape current and future project activities.

OJJDP grantees received the survey by email between June and August of 2011. Staff made an extensive effort to reach all direct recipients of OJJDP funds, as well as subgrantees from state, local, and tribal jurisdictions. The assessment included two survey instruments: one for state and local respondents, and a second for tribal respondents. Respondents were asked to designate their agency or organization as grant-making, providing direct services to youth, and/or operating one or more components of the juvenile justice system, or both. We hoped to determine differences in needs between the grant-making and service provider groups so that resources could be more narrowly targeted based on the function of the grantees' agencies or organizations. Further, we were able to ask respondents from grant-making agencies or organizations about their perception of the grantees' needs.

We received 962 responses from state and local grantees and 39 responses from tribal rantees. Because we allowed the survey to be forwarded to other individuals if the recipient felt he or she was not the most appropriate to respond to the questions, we were unable to ascertain an exact response rate or know how many responses were received from individuals working under the same grant or for the same program. Once data collection was complete, tabulated responses were examined to determine common evaluation and performance measurement activities, the level of need for training and technical assistance, and topics of interest related to evaluation and performance measurement. The survey also included open-ended questions to provide grantees with the opportunity to express needs and issues that might not be addressed in close-ended questions.

Survey Results
Funding and sustainability were two of the most frequently cited challenges of both service providers and grant-making agencies. Further, a substantial percentage of granting agencies stated that they want assistance using evaluation results, and need training and technical assistance related to cost-benefit analysis, a skill important for demonstrating program results and obtaining future funding. The focus on sustainability and related issues suggests that grantees may need help in making their performance measurement data and evaluation activities understandable to stakeholders to help them judge the level of a program's success.

Service providers said they need training and technical assistance on performance measurement development, developing program logic, and creating logic models. The majority of state and local respondents indicated that they collect additional performance measures beyond those required by OJJDP's performance measurement system, DCTAT. This suggests that emphasis on the development of appropriate measures may be needed, as well as discussion about the best sources of data and ways for program staff to collect data. Efficiency in data collection and analysis will be a key component of this discussion, as respondents consistently indicated a lack of time and staff to perform additional duties beyond service delivery.

Many responses from the tribal community reflected those received from state and local respondents. Unique challenges of tribal communities included data collection, information sharing, and cooperation among multiple aspects of the criminal justice system in responding to crimes committed in Indian country. Tribal grantees had difficulty with program participation and retention, and a lack of staff time and resources. Training and technical assistance created for tribal grantees, particularly in the area of sustainability, should address these issues.

Generally, self-rated level of need for training and technical assistance among all respondents was moderate; however, grant-making agencies rated their grantees' level of need substantially higher than their own level of need. Tribal grantees' self-rated need for training and technical assistance was higher than state and local respondents,' and nearly half indicated they had not received assistance related to evaluation or evidence-based practices in the past year. NJJEC activities should focus on generating new resources for tribal grantees, as well as promoting existing resources.

NJJEC staff will present information on the project, including information on the results of this survey, to the Federal Advisory Committee on Juvenile Justice (FACJJ) this summer and consider ways in which our activities can address the priorities of the Committee. The FACJJ advises the President, Congress, and OJJDP on issues related to juvenile justice. Two of its five priorities for 2012 are evidence-based youth justice practices and information sharing in youth justice. These priorities align with NJJEC activities focused on performance measurement and evaluation.