Monthly Archives: May 2012

About The NJJEC Tutorial

This project was supported by Grant No. 2010-JF-FX-0063 awarded by the Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice. Points of view in this document are those of the authors and do not … Continue reading

Posted in Learning | Leave a comment

Introduction

The NJJEC Tutorial comprises four modules. Important concepts related to performance measurement, evaluation, and evidence-based practices are conveyed through the story of Darcy Austen, a youth program director trying to combat rising juvenile crime. Module One contains an overview of … Continue reading

Posted in Learning | Leave a comment

Tutorial Objectives

HOW TO NAVIGATE THE TUTORIAL:  Navigate through the tutorial using the previous and next buttons provided at the bottom of the screen. The drop-down menu in the top right hand corner of your screen allows you to switch between modules.  … Continue reading

Posted in Learning | Leave a comment

Pre-Quiz

Continue to the Next Module →

Posted in Learning | Leave a comment

What Are Performance Measurement and Evaluation?

Welcome to Module Two! In this module, you will learn: What performance measurement and evaluation are. Seven key steps in the evaluation process. How to properly identify and show evidence for a problem in your community. How to select and … Continue reading

Posted in Learning2 | Leave a comment

What Are Performance Measurement and Evaluation?

Performance measurement and program evaluation are two ways juvenile justice program managers and staff members can assess what a program is trying to accomplish, how it is functioning, and what results are being achieved.   Performance measurement is a means … Continue reading

Posted in Learning2, sub | Leave a comment

Why Do We Evaluate Juvenile Justice Programs?

It is important that juvenile justice programs be evaluated for several reasons. Evaluation demonstrates whether a program is successful in meeting its goals and objectives. Evaluation takes performance measurement a step further by ruling out other explanations for program results. … Continue reading

Posted in Learning2 | Leave a comment

The Evaluation and Performance Measurement Process

There are seven key steps in the evaluation and performance measurement process.  In Module 1, we will focus on Steps One and Two: Define the problem. Implement evidence-based programming. Develop program logic. Identify measures. Collect and analyze data. Report findings. Reassess … Continue reading

Posted in Learning2 | Leave a comment

Step One: Define the Problem

In the juvenile justice field, programs and policies are developed to address the prevention or reduction of delinquent behavior. This tutorial presents program evaluation concepts through the use of a fictional city called Devonville. Devonville is a mid-sized, semi-urban metropolitan area with … Continue reading

Posted in Learning2 | Leave a comment

Step One: Define the Problem

CITY OF DEVONVILLE From the Office of Mayor Justice T. Smith Dear Ms. Austen, We regret to inform you that the Northwest Community Center is at risk for closure due to its failure to demonstrate the provision of effective, structured … Continue reading

Posted in Learning2, sub | Leave a comment

Step One: Define the Problem

Darcy has a lot of work to do.  First, she needs to demonstrate the magnitude and nature of the problem by providing data. To adequately define the problem, a program manager like Darcy needs to get a broad picture of all … Continue reading

Posted in Learning2, sub | Leave a comment

Step One: Define the Problem

As stated earlier, residents in the NW quadrant of Devonville have been complaining about unsupervised juveniles hanging out in the neighborhood and committing crime during the day. Darcy needs to provide solid evidence of an actual problem to support their … Continue reading

Posted in Learning2, sub | Leave a comment

Step One: Define the Problem

In addition to justifying a problem in the community, program managers should also be able to identify characteristics of the juveniles they are targeting for particular interventions– the program’s target population.  Because the effectiveness of many interventions depends on how … Continue reading

Posted in Learning2, sub | Leave a comment

Step One: Define the Problem

Darcy makes a trip down to the Devonville Police Department and meets with a crime analyst who pulls juvenile arrest statistics from their database. Darcy and and the crime analyst note that a large number of juvenile arrests in the … Continue reading

Posted in Learning2, sub | Leave a comment

Step Two: Implement Evidence-Based Programming

Once Darcy has properly described the magnitude and nature of the problem in her community, she is ready to move on to the next step: implementing evidence-based programming. The program Darcy chooses to implement should have previously demonstrated success in … Continue reading

Posted in Learning2 | Leave a comment

Step Two: Implement Evidence-Based Programming

The best question to ask when implementing evidence-based programming is: What is the linkage between the identified problem and the program you want to implement?  What evidence-based programs, practices, policies, or principles have addressed this type of problem successfully? Darcy … Continue reading

Posted in Learning2, sub | Leave a comment

Step Two: Implement Evidence-Based Programming

Not all programs and practices have the same amount of evidence.  Here are some frequently used terms to describe the level of support that exists for an evidence-based program or practice: Exemplary/model/best—clear evidence of effectiveness with multiple, rigorous evaluations; Promising—some … Continue reading

Posted in Learning2, sub | Leave a comment

Step Two: Implement Evidence-Based Programming

Darcy’s center is already well-attended by youth in the area, but she needs to focus the center’s activities to address the problems of truancy and dropout that seem to be contributing to the juvenile crime problem. From her search of … Continue reading

Posted in Learning2, sub | Leave a comment

Step Two: Implement Evidence-Based Programming

Darcy’s next step is to find programs that make use of these strategies to address delinquency. Darcy compiles available information on evidence-based programs that target juvenile delinquency and are well-suited for high school youth in NW Devonville. She uses online … Continue reading

Posted in Learning2, sub | Leave a comment

Step Two: Implement Evidence-Based Programming

Let’s take a look at some of the specific benefits of a mentoring program like Darcy has chosen: Bonding with positive role models Provision of structured and constructive activity during otherwise unsupervised hours Emphasis on the importance of staying in … Continue reading

Posted in Learning2, sub | Leave a comment

Creating a Logic Model

Welcome to Module Three! There are seven key steps in the evaluation and performance measurement process.  In Module 3, we will focus on Steps Three and Four: Define the problem. Implement evidence-based programming. Develop program logic. Identify measures. Collect and analyze … Continue reading

Posted in Learning3 | Leave a comment

What is a Logic Model?

Darcy has identified a problem in her community and selected an evidence-based mentoring program to implement.  Darcy’s next step is to define the elements of the program.  This visual depiction of a program plan is called a logic model. While … Continue reading

Posted in Learning3 | Leave a comment

What is a Logic Model?

A logic model provides: A format for identifying what the program expects to achieve. A basis for monitoring activities. A method to document what the program intends to do and what it is actually doing.   The logic model helps … Continue reading

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

To begin creating a logic model, Darcy needs to develop a GOAL statement for the mentoring program. A goal is a broad statement about what a program intends to accomplish.  A goal addresses the intended long-term outcome of a program. … Continue reading

Posted in Learning3 | Leave a comment

Step Three: Develop a Logic Model

Here are some possible goals for Darcy’s program:   To hire 50 mentors for the program within two months. The goal must be broad and tell what the program expects to achieve in terms of the problem. To hire 50 … Continue reading

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

Each GOAL has one or more OBJECTIVES associated with it. Each OBJECTIVE has a set of ACTIVITIES with which it is associated. If the activities are carried out successfully, they will lead to the accomplishment of the program’s objectives, which … Continue reading

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

Darcy’s program theory (also called program logic) should explain how the mentoring program will benefit Devonville and its youth. The logical relationships between the program’s goal, objectives, and activities can be expressed through a series of IF-THEN statements: IF high … Continue reading

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

This is Darcy’s theory of how a mentoring program will help the students of Devonville.  Her program logic spells out how the goal, objectives, and activities relate to each other.   Establishing these associations allows Darcy and any evaluator she … Continue reading

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

In addition to a logic model, Darcy should also provide the city with a program narrative. A program narrative includes a detailed description of the mentoring program, including how the program is organized, who the target population is, and where … Continue reading

Posted in Learning3, sub | Leave a comment

Program Narrative

The Northwest Community Center Mentoring Program The Northwest Community Center Mentoring program is for Devonville youth who are truant or may drop out of school.  This program is based on an evidence-based mentoring program that has repeatedly demonstrated effectiveness across … Continue reading

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

Now that Darcy has established her goal and written her narrative, she turns her focus to developing other components of the logic model: OBJECTIVES:  Expected achievements that are well defined, specific, measurable, and derived from the goal. ACTIVITIES:  The steps … Continue reading

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

Here are possible objectives, resources, and activities that could be associated with the goal of Darcy’s program:  To prevent juvenile delinquency in NW Devonville by strengthening students’ bonds to their school. Objective #1: Develop positive relationships between NW Devonville youth … Continue reading

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

Objective #2: Increase the grade point average for youth in Northwest Devonville participating in the program within 6 months of being matched to their mentor. Activities: For the duration of the program, mentors will meet with their assigned youth two … Continue reading

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

Now that Darcy has defined the problem, selected an evidence-based program, and developed the program logic, she can proceed to the next step in the evaluation process: identifying performance measures.  These measures should assess progress toward reaching the goals and … Continue reading

Posted in Learning3 | Leave a comment

Step Four: Identify Measures

PROCESS MEASURES Process measures are able to tell us whether the program is being implemented according to the original plan.  If substantial changes are made to an evidence-based program or practice, it may not work as effectively, so it is … Continue reading

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

OUTCOME MEASURES Outcome measures show the change (or lack of change) in the target population that are directly related to the goal(s) and objectives. There are three types of outcomes: initial, intermediate, and long-term. Initial outcomes are the immediate results … Continue reading

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

All performance measures should be: Well-defined Specific Measurable Derived from the goal Let’s begin by choosing process measures for Darcy’s mentoring program.

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

The main question to ask when thinking about process measures is this: What information should I collect to document what the program is doing?  Here are some of the ACTIVITIES Darcy would like to document: Each youth participating in the … Continue reading

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

Darcy selects her outcome measures by determining the change(s) she assumes will result from program activities.  Here are some of Darcy’s proposed objectives and outcome measures: OBJECTIVES: Increase the grade point average for 60% of youth participating in the program. … Continue reading

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

OUTCOMES AND PROGRAM LOGIC Each outcome that we expect the program to affect is directly related to the program’s theory or logic.  The program goal is the long-term outcome anticipated by the program.  Let’s look at Darcy’s theory about her … Continue reading

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

OUTCOMES AND PROGRAM LOGIC Here are two more IF-THEN statements that follow Darcy’s program logic: IF youths’ grades improve, attendance improves, and participation in extracurricular activities increases, THEN youths will feel stronger bonds or ties to their schools. In Darcy’s … Continue reading

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

OUTCOMES AND PROGRAM LOGIC Finally, here is Darcy’s last IF-THEN connection: IF youths adopt prosocial values and believe in the importance of academic achievement, THEN they are less likely to engage in delinquent behavior.  The decrease in the likelihood of … Continue reading

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

Darcy’s program operates within a larger system.  The last step in developing a logic model is identifying those factors external to the program that may affect whether the program will be able to achieve its goal and objectives.  It is … Continue reading

Posted in Learning3, sub | Leave a comment

Data Collection and Program Improvement

Welcome to Module Four! There are seven key steps in the evaluation and performance measurement process.  In Module 4, we will focus on Steps Five, Six, and Seven: Define the problem. Implement evidence-based programming. Develop program logic. Identify measures. Collect … Continue reading

Posted in Learning4 | Leave a comment

Data Collection and Program Improvement

Darcy has developed a strong set of process and outcome measures for her mentoring program. Even though she is not a professional evaluator, Darcy is able to demonstrate that she has a strong program design and will monitor program implementation … Continue reading

Posted in Learning4, sub | Leave a comment

Hiring an Outside Evaluator

Once her program has been fully implemented and has operated long enough to have an impact, Darcy may want to consider hiring a professional evaluator. If so, she should consult Hiring and Working With An Evaluator, a briefing paper that can … Continue reading

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

Darcy thinks about ways she can collect important program data. Data: Documented information or evidence of any kind. Darcy needs to measure the relationship that the youth and mentor develop during the program, the youths’ bonds to school, and youths’ prosocial … Continue reading

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

Anecdotal Evidence You might be tempted to show that a program is working by providing anecdotal evidence.  An individual might cite one or two instances of a certain result as “proof” that a program is working effectively without considering whether … Continue reading

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

Once data have been collected, they need to be analyzed. Data should clearly demonstrate whether or not a program met its objectives. Since most juvenile justice programs are trying to change attitudes and/or behaviors, data analysis usually focuses on: Whether … Continue reading

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

Data analysis will show that one of four possibilities has occurred: You realize that your program was implemented as designed and your objectives were achieved. You realize that your program was implemented as designed, but you did not reach your … Continue reading

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

Let’s look at the first possibility:  You realize that your program was implemented as designed and your objectives were achieved.  This scenario is ideal. How should Darcy respond? She might want to consider hiring an evaluator.  Because Darcy has chosen … Continue reading

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

The second possible result after data analysis is that you realize your program was implemented as designed, but you did not reach your intended objectives.  If this occurs, consider the following questions: Did something happen independent of the program that … Continue reading

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

The third possible result after data analysis is that you become aware that your program was not implemented correctly and your objectives were not achieved.  If this has occurred, focus on reasons why program implementation did not occur as it … Continue reading

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

Finally, let’s consider the fourth potential result of data analysis: You realize that your program was not implemented as designed, but your objectives were achieved.  While this is unusual, it is certainly possible. Let’s consider one of Darcy’s program activities: … Continue reading

Posted in Learning4, sub | Leave a comment

Step Six: Report Findings

It is important that Darcy include a plan for reporting data in her letter to the Mayor.  Remember that four requirements were listed as responsibilities of each applicant. These included:  Substantial information demonstrating the existence of the youth-related problem the … Continue reading

Posted in Learning4 | Leave a comment