About The NJJEC Tutorial

This project was supported by Grant No. 2010-JF-FX-0063 awarded by the Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice. Points of view in this document are those of the authors and do not necessarily represent the official position or policies of the U.S. Department of Justice.

Posted in Learning | Leave a comment

Introduction

The NJJEC Tutorial comprises four modules. Important concepts related to performance measurement, evaluation, and evidence-based practices are conveyed through the story of Darcy Austen, a youth program director trying to combat rising juvenile crime.

Module One contains an overview of the learning objectives and a pre-quiz.  Modules Two, Three, and Four provide information on program design, performance measurement, and evaluation.  A progress bar at the top of your screen will show you how much of the module you have completed and how much remains.  At the end of Module Four, a quiz will test your knowledge on the material.  Please click the SUBMIT button after you complete each quiz.  If you would like to have your responses and scores e-mailed to you, enter your name and e-mail address.  The four modules and two quizzes will take approximately one hour to complete.

 

Intro graphic
 

 

Posted in Learning | Leave a comment

Tutorial Objectives

HOW TO NAVIGATE THE TUTORIAL:  Navigate through the tutorial using the previous and next buttons provided at the bottom of the screen. The drop-down menu in the top right hand corner of your screen allows you to switch between modules.  Move quickly to different topics within each module using the topical menu provided on the left hand side of your screen.

Each module has a set of learning objectives that progress from the previous module:

Posted in Learning | Leave a comment

Pre-Quiz

Quiz for Module One

Welcome to the quiz for Module One! This quiz will be used to determine what you already know about performance measurement and evaluation. Please click the blue start button to take the quiz!

Start
Congratulations!  You have completed the pre-quiz.  You scored %%SCORE%% out of %%TOTAL%%, with %%PERCENTAGE%% correct.  Please click the SUBMIT button below. If you wish to receive the results of the quizzes, please provide us with your name and e-mail address. In order for NJJEC to gather information about who completes the tutorial, you may also provide us with your job title, number of years in position, and a rating of your knowledge of evaluation. %%FORM%%

Your Information

Your answers are highlighted below.
Return
Shaded items are complete.
12345
678910
1112131415
End
Return

Continue to the Next Module →

Posted in Learning | Leave a comment

What Are Performance Measurement and Evaluation?

Welcome to Module Two!

In this module, you will learn:

  • What performance measurement and evaluation are.
  • Seven key steps in the evaluation process.
  • How to properly identify and show evidence for a problem in your community.
  • How to select and implement evidence-based programming appropriate for the problem in your community.
Posted in Learning2 | Leave a comment

What Are Performance Measurement and Evaluation?

Performance measurement and program evaluation are two ways juvenile justice program managers and staff members can assess what a program is trying to accomplish, how it is functioning, and what results are being achieved.

 

Performance measurement is a means to ensure fidelity in implementation.  Fidelity means program activities are being carried out as planned.  While performance measurement tells us a lot about a program, evaluation is needed to show that the program’s activities caused any observed outcomes.

Posted in Learning2, sub | Leave a comment

Why Do We Evaluate Juvenile Justice Programs?

It is important that juvenile justice programs be evaluated for several reasons.

  1. Evaluation demonstrates whether a program is successful in meeting its goals and objectives.
  2. Evaluation takes performance measurement a step further by ruling out other explanations for program results.
  3. Evaluation reveals strengths and weaknesses of a program.
  4. Evaluation provides an objective assessment of program effectiveness.
  5. Programs using evaluation to demonstrate accomplishments are more likely to secure future funding.
Posted in Learning2 | Leave a comment

The Evaluation and Performance Measurement Process

There are seven key steps in the evaluation and performance measurement process.  In Module 1, we will focus on Steps One and Two:

  1. Define the problem.
  2. Implement evidence-based programming.
  3. Develop program logic.
  4. Identify measures.
  5. Collect and analyze data.
  6. Report findings.
  7. Reassess program logic.
Posted in Learning2 | Leave a comment

Step One: Define the Problem

In the juvenile justice field, programs and policies are developed to address the prevention or reduction of delinquent behavior. This tutorial presents program evaluation concepts through the use of a fictional city called Devonville.
Emeraldville
Devonville is a mid-sized, semi-urban metropolitan area with a population of approximately 700,000 people. The city is divided into quadrants (NW, NE, SW, and SE). Citizens in Devonville, specifically in the NW quadrant, have been complaining about juvenile crime during the day. They see more kids hanging out in and around their neighborhoods and say that they are using their idle time to commit crime.

Within the NW area, there is a city-funded community center that has existed for three years which primarily serves as an after school hang-out for teenagers. While the center holds numerous activities for youth, it has never had a formalized program designed to prevent or reduce juvenile delinquency. Darcy Austen, the Program Manager of the Center, recently received this letter:

Posted in Learning2 | Leave a comment

Step One: Define the Problem

CITY OF DEVONVILLE
From the Office of Mayor Justice T. Smith

Dear Ms. Austen,

We regret to inform you that the Northwest Community Center is at risk for closure due to its failure to demonstrate the provision of effective, structured programming for NW Devonville youth. Due to the dramatic change in juvenile crime in Devonville, the state government has established new programmatic guidelines and evaluation requirements for all programs. In order for the center to be awarded funds for the next grant period, the city requires that you demonstrate the Northwest Community Center is using grant funds for an evidence-based program. Please note in the current program application that we are requiring the following evaluation-related components to be included for consideration for future funding:

  1. Substantial information demonstrating the existence of the problem your program aims to address in the Northwest Community Center.
  2. A detailed work plan of the proposed evidence-based program in your center, including a logic model that provides program goals, objectives, and activities.
  3. The performance measures that will be used to track progress toward addressing the problem.

The Northwest Community Center has had a positive impact on various children’s issues in the past, but due to the increasing juvenile crime in the area, we can only support programs that clearly demonstrate a need and an evidence-based solution to the problem. We look forward to receiving your application for the next grant period. If you have any questions, please do not hesitate to call our office.

Sincerely,

Celia Hawkins

Celia Hawkins, Assistant Director
Office of Child Welfare and Social Services

Posted in Learning2, sub | Leave a comment

Step One: Define the Problem

Darcy has a lot of work to do.  First, she needs to demonstrate the magnitude and nature of the problem by providing data.

To adequately define the problem, a program manager like Darcy needs to get a broad picture of all the factors contributing to the current situation.  This is best done by stating the facts: Who? What? Where? When? Why?

Darcy comes up with the following list of individuals she can contact to find out more:

  • Local elected officials
  • Community leaders
  • Those working with juveniles in the community
  • Concerned parents and teachers
  • Media
  • Law enforcement

 

Posted in Learning2, sub | Leave a comment

Step One: Define the Problem

As stated earlier, residents in the NW quadrant of Devonville have been complaining about unsupervised juveniles hanging out in the neighborhood and committing crime during the day. Darcy needs to provide solid evidence of an actual problem to support their complaints.

Data are documented information or evidence of any kind.  There are a variety of sources where Darcy can obtain more information about her community to determine the extent of the juvenile crime problem, including:

  • Police records
  • School attendance logs
  • Conversations with members of the community
  • Census information
  • State criminal and juvenile statistics agencies

 

Here are some suggested resources for assessing a problem in your state or local community:

Posted in Learning2, sub | Leave a comment

Step One: Define the Problem

In addition to justifying a problem in the community, program managers should also be able to identify characteristics of the juveniles they are targeting for particular interventions– the program’s target population.  Because the effectiveness of many interventions depends on how suitable they are for the population being served, youth characteristics such as age, gender, language, and problem behaviors should be considered, in addition to characteristics of the youths’ schools, families, and communities.  This can be accomplished through quantifying and qualifying the problem.

Here are some questions to ask in order to best define the problem you are seeking to address:

  • Whom does it affect?
  • How serious is it?
  • What specifically is the problem?
  • How does it connect with other problems in the community?
  • What services are already in place to address the problem?
  • What are the gaps in services?
  • Why is there a problem?
Posted in Learning2, sub | Leave a comment

Step One: Define the Problem

Darcy makes a trip down to the Devonville Police Department and meets with a crime analyst who pulls juvenile arrest statistics from their database. Darcy and and the crime analyst note that a large number of juvenile arrests in the NW quadrant are taking place during school hours.  Also, there has been a 5% increase in juvenile arrests in the NW quadrant in the past two years, while there has been little change in the juvenile arrests in the rest of the city.

Darcy also speaks with the principals in charge of the three public high schools in the NW quadrant to see if they have any helpful information.  Darcy learns that these schools have had a 25% increase in truancy and 20% increase in dropout among students in the last two years.  The principals note that these truancy and dropout increases are much higher than schools in other parts of the city of Devonville.

Based on the information from the police and school system, Darcy defines the problem as follows:

The NW quadrant has experienced a 5% increase in juvenile crime in the past two years. There has been no significant increase in juvenile crime in the rest of the city. In addition, there has been a 25% increase in the truancy and a 20% increase in school dropout for the public high schools in the NW quadrant for each of the last two years.  The increases in truancy and dropout are also unique to the NW area of Devonville, and may be contributing to the juvenile crime problem.

Darcy has decided to review research on programs that have addressed these problems effectively with youth similar to those in NW Devonville.  After reviewing this research, she will select an evidence-based program that is appropriate for her target population.

Posted in Learning2, sub | Leave a comment

Step Two: Implement Evidence-Based Programming

Once Darcy has properly described the magnitude and nature of the problem in her community, she is ready to move on to the next step: implementing evidence-based programming.

The program Darcy chooses to implement should have previously demonstrated success in repeated, credible evaluations, and meet the specific needs of her target population.  Specifically, Darcy needs to choose a program that addresses juvenile delinquency by reducing truancy and dropout rates among high school students.

There are many resources available online to locate specific programs that have demonstrated results, including the Blueprints for Violence Prevention programs and OJJDP’s Model Programs Guide.

Posted in Learning2 | Leave a comment

Step Two: Implement Evidence-Based Programming

The best question to ask when implementing evidence-based programming is: What is the linkage between the identified problem and the program you want to implement?  What evidence-based programs, practices, policies, or principles have addressed this type of problem successfully?

Darcy must be able to demonstrate:

  1. Sufficient research exists to support the program plan as a possible solution for the problem Darcy wants to address.
  2. The program is well established as being effective for this target population.
  3. Results have been published and they are widely accepted.

 

Posted in Learning2, sub | Leave a comment

Step Two: Implement Evidence-Based Programming

Not all programs and practices have the same amount of evidence.  Here are some frequently used terms to describe the level of support that exists for an evidence-based program or practice:

  • Exemplary/model/best—clear evidence of effectiveness with multiple, rigorous evaluations;
  • Promising—some evidence of success, many questions remain; or
  • Innovative—derived from evidence-based programs, but has not yet been evaluated.

 

When Should I Use An Innovative Program?

An exact version of the program you envision does not have to exist in order for you to rely on evidence-based programming. Search the literature for programs which address issues similar to the problem you have identified. You may be able use an innovative program that is based in research evidence, but has not yet been sufficiently evaluated to be considered an evidence-based program.  You may also be able to combine strategies from more than one program.

Posted in Learning2, sub | Leave a comment

Step Two: Implement Evidence-Based Programming

Darcy’s center is already well-attended by youth in the area, but she needs to focus the center’s activities to address the problems of truancy and dropout that seem to be contributing to the juvenile crime problem.

From her search of online resources, Darcy learns that increased truancy and school dropout rates are risk factors for juvenile delinquency.  The data she has collected indicate these factors may contribute to the current juvenile crime problem in NW Devonville.  Darcy’s research indicates that the center can most effectively address the juvenile crime problem with the strategies listed below.

Given the problem with youths skipping or dropping out of school, Darcy needs to find a program to encourage high school youth to attend school.

Since most arrests involve groups of juveniles, Darcy assumes that some juveniles are being negatively influenced by the antisocial attitudes of their peers. She will therefore want to identify a program that will boost juveniles’ prosocial values, including valuing social institutions such as schools.

Posted in Learning2, sub | Leave a comment

Step Two: Implement Evidence-Based Programming

dorothyDarcy’s next step is to find programs that make use of these strategies to address delinquency.

Darcy compiles available information on evidence-based programs that target juvenile delinquency and are well-suited for high school youth in NW Devonville. She uses online resources to find programs that have demonstrated successes in multiple studies with strong evaluation designs.

After reviewing several programs, Darcy chooses a community-based mentoring program. One of the main reasons she chose this program is that it seeks to increase juveniles’ prosocial values, such as the importance of education, while at the same time directly addressing issues related to truancy and drop out, such as academic performance and participation in extracurricular activities.

Posted in Learning2, sub | Leave a comment

Step Two: Implement Evidence-Based Programming

Let’s take a look at some of the specific benefits of a mentoring program like Darcy has chosen:

  • Bonding with positive role models
  • Provision of structured and constructive activity during otherwise unsupervised hours
  • Emphasis on the importance of staying in school
  • Help with homework
  • Attention to delinquent behavior, including gang activity and drug use

 

Darcy believes these results will strengthen Devonville’s high school students’ bonds to their schools, which should increase their attendance in school and reduce dropout rates, resulting in reduced delinquent activity.

 

This is the end of Module Two!  Click “Continue” to move on to Module Three.

Continue to the Next Module →

Posted in Learning2, sub | Leave a comment

Creating a Logic Model

Welcome to Module Three!

There are seven key steps in the evaluation and performance measurement process.  In Module 3, we will focus on Steps Three and Four:

  1. Define the problem.
  2. Implement evidence-based programming.
  3. Develop program logic.
  4. Identify measures.
  5. Collect and analyze data.
  6. Report findings.
  7. Reassess program logic.
Posted in Learning3 | Leave a comment

What is a Logic Model?

Darcy has identified a problem in her community and selected an evidence-based mentoring program to implement.  Darcy’s next step is to define the elements of the program.  This visual depiction of a program plan is called a logic model.

While there are several ways to set up a logic model, typical components include:

  • GOAL(S)
  • OBJECTIVES
  • RESOURCES
  • ACTIVITIES
  • PROCESS MEASURES
  • OUTCOME MEASURES
  • EXTERNAL FACTOR

Click here for a blank logic model template you can print and fill in as Darcy develops her logic model.  Remember, you can set up your own logic model in the way that best presents your program’s main components– this is just a sample.

 

Posted in Learning3 | Leave a comment

What is a Logic Model?

A logic model provides:

  • A format for identifying what the program expects to achieve.
  • A basis for monitoring activities.
  • A method to document what the program intends to do and what it is actually doing.

 

The logic model helps maintain focus and program fidelity by linking project goals, objectives, activities, and performance measures. It clearly shows which activities should be carried out and for what purpose, as well as what data should be collected to keep track of program activities.

Darcy can include her logic model in her grant application to explain how her program will address rising juvenile crime in NW Devonville, and how she will demonstrate that the program is following this plan.

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

To begin creating a logic model, Darcy needs to develop a GOAL statement for the mentoring program.

A goal is a broad statement about what a program intends to accomplish.  A goal addresses the intended long-term outcome of a program.

 

Darcy comes up with three possible goal statements based on the above definition:

  • To hire 50 mentors within two months for the program.
  • To rid the Devonville community of all juvenile crime.
  • To prevent juvenile delinquency in NW Devonville by strengthening high school students’ bonds to their school.

 

Based on what you know about Devonville, which goal statement do you think is most appropriate for Darcy’s mentoring program? Remember, it should be broad and describe the long-term changes the mentoring program expects to achieve in Devonville.

Posted in Learning3 | Leave a comment

Step Three: Develop a Logic Model

Here are some possible goals for Darcy’s program:

 

To hire 50 mentors for the program within two months.

The goal must be broad and tell what the program expects to achieve in terms of the problem. To hire 50 mentors doesn’t say anything about what the program is expected to accomplish. Also, it is not a statement of what the program expects to achieve, it is a statement about a program activity.

 

To rid the Devonville community of all juvenile crime.

This goal statement is definitely broad, but it is unrealistic.

 

To prevent juvenile delinquency in NW Devonville by strengthening high school students’ bonds to their school.

This goal is broad, offers a solution to the crime problems in Devonville, and is realistic, assuming that everything Darcy has in mind for her program goes as planned. Great! Darcy has the program’s goal!

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

Each GOAL has one or more OBJECTIVES associated with it.
Each OBJECTIVE has a set of ACTIVITIES with which it is associated.

If the activities are carried out successfully, they will lead to the accomplishment of the program’s objectives, which will ultimately lead to the achievement of or progress towards the goal.

The set of relationships between a program’s goals, objectives, and activities are often referred to as program theory or program logic. These components specify the process by which the program will produce positive changes in the youth it serves.

Darcy uses the information she has collected about the evidence-based mentoring program to help her develop her program logic.

 

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

Darcy’s program theory (also called program logic) should explain how the mentoring program will benefit Devonville and its youth. The logical relationships between the program’s goal, objectives, and activities can be expressed through a series of IF-THEN statements:

if then graphic

IF high school youth develop positive relationships with mentors who help with homework and studying, monitor attendance at school, and encourage and facilitate participation in extracurricular activities,
THEN youth’s grades will improve, attendance will improve, and participation in extracurricular activities will increase.

IF high school youth’s grades improve, attendance improves, and participation in extracurricular activities increase,
THEN youths will feel stronger bonds or ties to their schools.

IF high school youth feel strong bonds or ties to their schools,
THEN they are more likely to adopt other prosocial values such as a belief in the importance of lawful behavior.

IF high school youth adopt other prosocial values,
THEN they are less likely to engage in delinquent behavior.

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

This is Darcy’s theory of how a mentoring program will help the students of Devonville.  Her program logic spells out how the goal, objectives, and activities relate to each other.

activities graphic
 

Establishing these associations allows Darcy and any evaluator she may work with to examine each individual connection, and to see where the connections succeed or fail. She can continually revisit the logic model to be sure her program is following its original plan. If adjustments are needed to improve the program or data collection, the program logic and logic model can be updated.

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

In addition to a logic model, Darcy should also provide the city with a program narrative. A program narrative includes a detailed description of the mentoring program, including how the program is organized, who the target population is, and where the program is located.

Putting ALL of this information on a logic model would cause it to be too cluttered. The logic model should be easy to read and understand very quickly, and the narrative provides all the extra (but important!) details about the program.

Darcy comes up with the following description. Note that she included a number of important features of the program, including the program plan’s evidence base, target population, program activities, location, and time needed to carry out the program plan.

Posted in Learning3, sub | Leave a comment

Program Narrative

The Northwest Community Center Mentoring Program

The Northwest Community Center Mentoring program is for Devonville youth who are truant or may drop out of school.  This program is based on an evidence-based mentoring program that has repeatedly demonstrated effectiveness across a range of settings. The program targets a number of the behavioral problems related to juvenile delinquency in NW Devonville, such as academic performance and attitudes towards school.

The program will be hosted at the Northwest Community Center. In order to be eligible for the program, youth must meet the following criteria:  Live in the Northwest area of the city; be a high school student between the ages of 14 and 18; have a GPA for the first quarter of school below a C average; have accrued three or more unexcused absences during the first quarter of school year; and agree to be mentored for at least one year.

Youth are referred to the program by a parent or guardian, a teacher, and/or by a school guidance counselor. Upon referral to the program, the program manager, Darcy Austen will schedule a meeting with the identified youth, his or her parent(s) or guardian (s), and at least one teacher within two weeks to develop a plan outlining all of the activities that the youth must partake in throughout the duration of the program. The plan is subject to change based on the youth’s progress.

Additionally, the youth will be matched with an adult mentor who will contact him or her within two weeks of the first meeting and the signing of the program consent form (which is considered the youth’s start date in the program).  For the duration of the program, mentors will meet with their assigned youth two times per week to go over homework and/or other academic issues.  All youth and their mentors will participate in a community-sponsored workshop on the requirements for being admitted to college.

For the duration of the program, mentors will contact their youths’ teachers weekly to discuss and address any problems related to truancy and/or any other academic issues. Mentors and their assigned youth will meet at least four additional times throughout the duration of the program to discuss youths’ interest and involvement in after school activities (sports, clubs).  Throughout the program, all youth will be assessed based on their academic performance, attendance in school, and effort they put forth with their mentors.

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

Now that Darcy has established her goal and written her narrative, she turns her focus to developing other components of the logic model:

OBJECTIVES:  Expected achievements that are well defined, specific, measurable, and derived from the goal.

ACTIVITIES:  The steps or tasks undertaken to meet the objectives of the program.

Darcy should also keep in mind the resources that are available to operate the program and to accomplish the program’s goals and objectives. Below are some examples of resources:

  • Building to house a program
  • Staff to manage and direct the program activities
  • Computers and staff to collect data and conduct analyses
  • Funding to support program objectives

There may be external factors such as the availability of afterschool activities that affect Darcy’s program.  If they seem significant, she might want to include these external factors in her logic model.

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

Here are possible objectives, resources, and activities that could be associated with the goal of Darcy’s program:  To prevent juvenile delinquency in NW Devonville by strengthening students’ bonds to their school.

Objective #1: Develop positive relationships between NW Devonville youth and their mentors by completion of the program.

 This is a good objective because it describes an initial expected effect of the program (developing relationships), it includes a direction (positive), a time frame (by program completion), and a target population (youth in NW Devonville).  Keep in mind that this is only one of several possible objectives for the program.

Activities: Each youth participating in the program will be matched with one adult mentor within two weeks of beginning the program.

To determine if this activity is appropriate, consider whether it appears to be a task that would contribute to meeting the objective.

Try to come up with your own objectives and activities for Darcy’s mentoring program. Click to the next slide for another objective example.

Posted in Learning3, sub | Leave a comment

Step Three: Develop a Logic Model

Objective #2: Increase the grade point average for youth in Northwest Devonville participating in the program within 6 months of being matched to their mentor.

Activities: For the duration of the program, mentors will meet with their assigned youth two times per week to go over homework and/or other academic issues.

 

Let’s think about some of the resources needed to carry out program activities.

Resources: Access to school records, text books, communication between school and mentor.

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

Now that Darcy has defined the problem, selected an evidence-based program, and developed the program logic, she can proceed to the next step in the evaluation process: identifying performance measures.  These measures should assess progress toward reaching the goals and objectives.

There are two types of performance measures: process measures and outcome measures.

PERFORMANCE MEASURES

PROCESS MEASURES: Data used to demonstrate the implementation of activities. These include products of activities and indicators of services provided.

OUTCOME MEASURES: Data used to measure achievement of objectives and goal(s).

 

Posted in Learning3 | Leave a comment

Step Four: Identify Measures

PROCESS MEASURES

Process measures are able to tell us whether the program is being implemented according to the original plan.  If substantial changes are made to an evidence-based program or practice, it may not work as effectively, so it is important to collect process measures on the execution of program activities.  Fidelity in implementation means that the program’s activities adhere to the program plan.

Darcy should collect at least one process measure for each activity, so that all program activities during the grant period are tracked. It is important to accurately track all activities to ensure the program was properly implemented.  These measures may also be used to determine why program results differ from what was expected, should that occur.

 

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

OUTCOME MEASURES

Outcome measures show the change (or lack of change) in the target population that are directly related to the goal(s) and objectives.

There are three types of outcomes: initial, intermediate, and long-term.

  • Initial outcomes are the immediate results of the program.
  • Intermediate outcomes are the results following the initial outcomes.
  • Long-term outcomes refer to the ultimate impact of the program and are related to the achievement of the program goal.

Note that a program may not have each of these types of outcome measures, depending on program design and ability to collect the data required for the measure.

To develop outcome measures, think about what information the program will collect to document what you expect to occur as a result of the activities.  What difference do you expect your program to make, for whom, and by when?

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

All performance measures should be:

  • Well-defined
  • Specific
  • Measurable
  • Derived from the goal

Let’s begin by choosing process measures for Darcy’s mentoring program.

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

The main question to ask when thinking about process measures is this: What information should I collect to document what the program is doing?  Here are some of the ACTIVITIES Darcy would like to document:

  1. Each youth participating in the program will be matched with one adult mentor within two weeks of beginning the program.
  2. For the duration of the program, mentors will meet with their assigned youth two times per week to go over homework and/or other academic issues, as well as discuss youths’ interest and involvement in after school activities (sports, clubs).
  3. All youth and their mentors will participate in a community sponsored workshop on the requirements for being admitted to college.
  4. For the duration of the program, mentors will contact their youths’ teachers weekly to discuss any problems related to truancy and/or any other academic issues.
  5. If problems are identified, the mentor will address issues with youth.

 

Darcy proposes these PROCESS MEASURES to assess the program’s activities:

  1. Number and percent of youth in the program matched with an adult mentor within two weeks of the beginning of the program.
  2. Number of meetings per week attended compared to total number of meetings possible (twice a week for the duration of the program).
  3. Number and percent of youth/mentor pairs who attended the college admissions requirements workshop compared to the number of youths participating in the program.
  4. Number of contacts made with youths’ teachers compared to total number of weeks in program.
  5. Number and percent of problems identified by teachers that were addressed in youth-mentor meetings.
Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

Darcy selects her outcome measures by determining the change(s) she assumes will result from program activities.  Here are some of Darcy’s proposed objectives and outcome measures:

OBJECTIVES:

  1. Increase the grade point average for 60% of youth participating in the program.
  2. Decrease the number of school absences by at least two days per quarter for 90% of participating youth by program completion.
  3. All program youth will begin to participate in at least one school sponsored during the course of the program.

 

Notice that Darcy has specified these objectives with the percent of youth she expects to show change.  She can adjust these percents later to further improve the program’s success.  For example, Darcy feels it is reasonable to expect 60% of youth to increase GPA by the end of the program, but the following year, she may increase that number to 75% if the program is a success.

The initial and intermediate outcome measures listed below are directly linked to program objectives, and can be measured during or at the end of the program.

OUTCOME MEASURES:

  1. Number and percent of youth with increased grade point average by completion of program.
  2. Number and percent of youth with at least two fewer absences in the final quarter of the program compared to the first quarter of the program.
  3. Number and percent of youth participating in at least one school-sponsored activity during the course of the program.

 

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

OUTCOMES AND PROGRAM LOGIC

Each outcome that we expect the program to affect is directly related to the program’s theory or logic.  The program goal is the long-term outcome anticipated by the program.  Let’s look at Darcy’s theory about her mentoring program to see how initial, intermediate and long-term outcomes are related to program logic.

Let’s take a look at Darcy’s first IF-THEN connection:

  • IF youth develop positive relationships with mentors who help with homework and studying, monitor attendance at school, and encourage and facilitate participation in extracurricular activities, THEN youths’ grades will improve, attendance will improve, and participation in extracurricular activities will increase.

 

All of the elements in the THEN statement (improvements in grades, attendance, and increased participation in extracurricular activities) are initial outcomes. These are the direct results of the mentors’ activities that we expect to see first.

 

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

OUTCOMES AND PROGRAM LOGIC

Here are two more IF-THEN statements that follow Darcy’s program logic:

  • IF youths’ grades improve, attendance improves, and participation in extracurricular activities increases, THEN youths will feel stronger bonds or ties to their schools.

In Darcy’s program theory, improvements in school performance result in increased bonding to school. Youth bonding to school is therefore an intermediate outcome. It is a result of the initial outcomes, and is itself necessary for the program to meet its goal.

  • IF youths feel stronger bonds or ties to their schools, THEN they are more likely to adopt other prosocial values, such as a belief in the importance of lawful behavior.

The adoption of prosocial values other than the importance of academic achievement is an intermediate outcome that is a necessary step in achieving Darcy’s program goal.

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

OUTCOMES AND PROGRAM LOGIC

Finally, here is Darcy’s last IF-THEN connection:

  • IF youths adopt prosocial values and believe in the importance of academic achievement, THEN they are less likely to engage in delinquent behavior. 

The decrease in the likelihood of delinquent behavior is a long-term outcome. This outcome is the ultimate result the program hopes to achieve; that is, the program’s GOAL.

Posted in Learning3, sub | Leave a comment

Step Four: Identify Measures

Darcy’s program operates within a larger system.  The last step in developing a logic model is identifying those factors external to the program that may affect whether the program will be able to achieve its goal and objectives.

 It is important to remember that a true evaluation, not just performance measurement, is needed to demonstrate that results have been caused by your program and not external factors.

Check out this logic model diagram with definitions for each of these components. 

 

 

This is the end of Module Three!  Click “Continue” to move on to Module Four.

Continue to the Next Module →

Posted in Learning3, sub | Leave a comment

Data Collection and Program Improvement

Welcome to Module Four!

There are seven key steps in the evaluation and performance measurement process.  In Module 4, we will focus on Steps Five, Six, and Seven:

  1. Define the problem.
  2. Implement evidence-based programming.
  3. Develop program logic.
  4. Identify measures.
  5. Collect and analyze data.
  6. Report findings.
  7. Reassess program logic.

Remember, you can always review a prior Module using the drop-down box above.

Posted in Learning4 | Leave a comment

Data Collection and Program Improvement

strong program designDarcy has developed a strong set of process and outcome measures for her mentoring program. Even though she is not a professional evaluator, Darcy is able to demonstrate that she has a strong program design and will monitor program implementation carefully to make sure each step is carried out as planned.  Even without the assistance of an outside evaluator, she is able to:

  • Demonstrate that her program is evidence-based.
  • Develop a logic model.
  • Develop measures to determine whether a program is meeting its goals and objectives.
  • Design data collection forms and procedures.
  • Build a database to record data collected.
  • Conduct simple data analyses.
  • Produce a written report about the successes and failure of her program.
Posted in Learning4, sub | Leave a comment

Hiring an Outside Evaluator

Outside InvestigatorOnce her program has been fully implemented and has operated long enough to have an impact, Darcy may want to consider hiring a professional evaluator. If so, she should consult Hiring and Working With An Evaluator, a briefing paper that can be accessed on the NJJEC website.

Resources to help with hiring an outside evaluator include:

 

Darcy will know a lot about how well the program is being implemented based on the performance measure data she is able to collect.  In addition, performance measure data will help Darcy determine when she is ready for an evaluation of her program.

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

Darcy thinks about ways she can collect important program data.

Data: Documented information or evidence of any kind.

Darcy needs to measure the relationship that the youth and mentor develop during the program, the youths’ bonds to school, and youths’ prosocial values.  To determine if there were changes, Darcy needs to assess these factors at two points: before the program begins (pre-test) and then again after the program is completed (post-test).

Pre- and post-tests are data collection tools administered at two points in time. A pre-test is administered before services or activities begin, while a post-test is administered at the conclusion of services or activities.

OUTCOMES MEASURED BY PRE- AND POST-TESTS

  • Youth and mentors indicate interest in and willingness to meet with each other: Mentor/Youth Relationship Scale (pre- and post-test).
  • Youths are bonded to school: School Bonding Scale (pre- and post-test).
  • Youth adopt pro-social values: Youth Values Questionnaire (pre- and post-test).

The scores from the scales above serve as the data that assess important components of the evidence-based program Darcy is implementing:  the relationship between youth and mentor, youths’ bonds to school, and youths’ prosocial values.

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

EmeraldvilleAnecdotal Evidence

You might be tempted to show that a program is working by providing anecdotal evidence.  An individual might cite one or two instances of a certain result as “proof” that a program is working effectively without considering whether the results are representative, or if the results were actually caused by the program instead of other factors.

For example, suppose a mother calls Darcy three months into her mentoring program to report that her son’s behavior at home is ten times better than it used to be. Her son is now taking out the trash without having to be asked; he’s using the phone much less; he’s obeying his curfew. These are signs that something is changing the youth’s life, but these changes may not be representative of the program outcomes.  Other youth in the program may not be showing the same improvements, or this youth’s behavior might be changing for some reason unrelated to his participation in the mentoring program.  Darcy cannot assume the program is a success overall based on the results experienced by this parent.

Systematic data collection—collecting the same data from all individuals participating in the program—gives better information about the impact of the program on its target group.

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

Once data have been collected, they need to be analyzed. Data should clearly demonstrate whether or not a program met its objectives. Since most juvenile justice programs are trying to change attitudes and/or behaviors, data analysis usually focuses on:

  • Whether a change occurred.
  • How much change occurred.
  • Whether or not a negative behavior was prevented.
  • Any changes to the program structure or operation itself.
  • Other unexpected changes that may affect the process or outcomes.
Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

Data analysis will show that one of four possibilities has occurred:

  • You realize that your program was implemented as designed and your objectives were achieved.
  • You realize that your program was implemented as designed, but you did not reach your intended objectives.
  • You realize that your program was not implemented as designed and your objectives were not achieved.
  • You realize that your program was not implemented as designed, but your objectives were achieved.

 

What should you do in each of these scenarios?

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

Let’s look at the first possibility:  You realize that your program was implemented as designed and your objectives were achieved.  This scenario is ideal.

How should Darcy respond? She might want to consider hiring an evaluator.  Because Darcy has chosen an evidence-based program and has performed quality performance measurement, it is reasonable to assume that her program has been a success.  However, in order to empirically (i.e. scientifically) demonstrate that the mentoring program actually caused the positive changes in participating youth, she will need to hire an evaluator and establish that program activities actually caused the outcomes.

Remember, evaluation and performance measurement both:

  • Focus on program performance
  • Can be used for monitoring and improvement
  • Generate evaluative information

 

… But evaluation takes performance measurement a step further, and determines whether the outcomes of a program can be attributed to the program or other factors.

 

Let’s take a look at the other scenarios.

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

The second possible result after data analysis is that you realize your program was implemented as designed, but you did not reach your intended objectives.  If this occurs, consider the following questions:

  • Did something happen independent of the program that affected your ability to achieve your objectives?
  • Were the performance measures specified correctly?
  • Have you expected to see results too soon?

 

Answering these questions will help you to determine why you have not achieved your program’s objectives.

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

The third possible result after data analysis is that you become aware that your program was not implemented correctly and your objectives were not achieved.  If this has occurred, focus on reasons why program implementation did not occur as it was intended, and consider ways to improve fidelity to the program model.

For example:

  • Are program participants from the correct target population?  If not, why not?  How might you select appropriate program participants in the future?
  • Were activities carried out in the manner and frequency they were intended?  If not, what resources might help you adhere more closely to the intended activities?
  • Have any external factors prevented you from properly implementing your program?  If so, how could you prevent this from occurring in the future?

 

Posted in Learning4, sub | Leave a comment

Step Five: Collect and Analyze Data

DorothyFinally, let’s consider the fourth potential result of data analysis: You realize that your program was not implemented as designed, but your objectives were achieved.  While this is unusual, it is certainly possible.

Let’s consider one of Darcy’s program activities:

For the duration of the program, mentors will meet with their assigned youth two times per week to go over homework and/or other academic issues.

Darcy’s analyses might reveal that in most situations, mentors met with their assigned youth once a week instead of two times per week, yet the objective to increase GPA was still achieved.  She may want to consider changing this activity since the program achieved objectives with less intensive services. On the other hand, she may want to retain the objective to see how different the outcome (GPA) would be if mentors do actually meet with youth twice per week.

Posted in Learning4, sub | Leave a comment

Step Six: Report Findings

It is important that Darcy include a plan for reporting data in her letter to the Mayor.  Remember that four requirements were listed as responsibilities of each applicant. These included:

  1.  Substantial information demonstrating the existence of the youth-related problem the program aims to address in the Northwest Community Center.
  2.  A detailed work plan of the proposed evidence-based program at the center, including a logic model that describes the program’s goals, objectives, and activities.
  3.  The development of measures that will be used to track progress toward reducing the problem.
  4.  A specific account of how the center will collect and analyze evidence of program success.

Once data have been collected and analyzed, the findings must be reported. This is Step Six of the evaluation process.

Reporting should consist of taking the results of the analysis and putting them into a user-friendly format appropriate for the audience. In the report stage, data should be both presented and interpreted so that the program information is useful for the community.

Posted in Learning4 | Leave a comment

Step Six: Report Findings

DorothyTo make her application stronger, Darcy should mention ways she will analyze and report her findings once she is able to collect data.  She should state how she will collect data related to her original goal, objectives, and activities, and discuss how the data will show that the program has been implemented according to the plan.  She may also explain that she will eventually be able to identify accomplishments and indicate areas where the mentoring program needs improvement. Finally, she should note when she will be able to produce a report.

With this information included in her application, Darcy has provided the Mayor with a good desciption of her mentoring program as requested.

Based on Darcy’s program design and data collection strategy, she should be able to produce a report on the implementation and preliminary outcomes of the newly designed program in approximately one year.

Posted in Learning4, sub | Leave a comment

Step Six: Report Findings

In addition to a report outline, Darcy submits a program timeline with her application. In planning her program, Darcy comes up with the following timeline that she will abide by:

Northwest Community Center Mentoring Program Timeline

Year 1

Months 1-3 Months 4-9 Months 10-12
Identify Mentors Provide 6 months of service to participants Data analysis and report writing

Year 2

Months 13-15 Months 16-21 Months 22-24
Identify Mentors Provide 6 months of service to participants Data analysis and report writing
Posted in Learning4, sub | Leave a comment

Step Seven: Reassess Program Logic

The final step in the evaluation process is to look at the program’s impact overall, and return to the logic model to see if it needs modification.  The data analyses lead to conclusions about how well the program has fulfilled its objectives. Reassessing program logic, design, and operation after the evaluation report is produced reveals the program’s strongest assets and also shows us where the program needs improvement.

At this point, ask:

Do we need to modify activities, develop new objectives, or reexamine program goals?

If you answer “yes,” changes should be made to improve the program.

Posted in Learning4 | Leave a comment

Step Seven: Reassess Program Logic

If you have implemented an evidence-based program correctly, measured consistently, and the problem still exists, you probably need an evaluation to determine why the program has failed to meet its goal(s).  A thorough evaluation will separate the program’s impact from external factors.  You should consider the following potential problems with a failed evidence-based program:

  • Did something happen in the community independent of the program that coincided with the program to cause an increase in juvenile crime?  Could program youth have been affected by these events?
  • Were program performance measures specified correctly?  For example, was the Mentor/Youth Relationship Scale used by Darcy a good measure to indicate whether program youth developed positive relationships with adult role models?
  • Have you expected to see results too soon?

Responses to these questions might be used to modify the program plan and improve program outcomes.

Posted in Learning4, sub | Leave a comment

Final Quiz

Quiz for Module Four

Welcome to the final quiz! This quiz will be used to determine what you have learned about data collection and program improvement. Please click the blue start button to begin!

Start
Congratulations!  You have completed the final quiz.  You scored %%SCORE%% out of %%TOTAL%%, with %%PERCENTAGE%% correct.  Please click the SUBMIT button below. If you wish to receive the results of the quizzes, please provide us with your name and e-mail address. In order for NJJEC to gather information about who completes the tutorial, you may also provide us with your job title, number of years in position, and a rating of your knowledge of evaluation. %%FORM%%








Your Information - Copy 2

Your answers are highlighted below.
Return
Shaded items are complete.
12345
678910
1112131415
End
Return
Posted in Learning4 | Leave a comment

Tutorial Completion

Great job!  You have completed the NJJEC tutorial.  Let’s review the seven steps of performance measurement and evaluation:

  • Define the problem.
  • Implement evidence-based programming.
  • Develop program logic.
  • Identify measures.
  • Collect and analyze data.
  • Report findings.
  • Reassess program logic.

 

Questions?  Comments?  E-mail njjec@jrsa.org

Subscribe to the NJJEC Bulletin for updates on trainings and conferences of interest, NJJEC project activities, and important research and evaluation issues in juvenile justice.

Posted in Learning4 | Leave a comment