Darcy has developed a strong set of process and outcome measures for her mentoring program. Even though she is not a professional evaluator, Darcy is able to demonstrate that she has a strong program design and will monitor program implementation carefully to make sure each step is carried out as planned.  Even without the assistance of an outside evaluator, she is able to:

• Demonstrate that her program is evidence-based.
• Develop a logic model.
• Develop measures to determine whether a program is meeting its goals and objectives.
• Design data collection forms and procedures.
• Build a database to record data collected.
• Conduct simple data analyses.
• Produce a written report about the successes and failure of her program.

Previous  Next

Once her program has been fully implemented and has operated long enough to have an impact, Darcy may want to consider hiring a professional evaluator. If so, she should consult Hiring and Working With An Evaluator, a briefing paper that can be accessed on the NJJEC website.

Resources to help with hiring an outside evaluator include:

• Office of Juvenile Justice and Delinquency Prevention
• American Evaluation Association
• American Society of Criminology
• American Psychological Association
• American Sociological Association

Darcy will know a lot about how well the program is being implemented based on the performance measure data she is able to collect.  In addition, performance measure data will help Darcy determine when she is ready for an evaluation of her program.

Previous  Next

Darcy thinks about ways she can collect important program data.

Data: Documented information or evidence of any kind.

Darcy needs to measure the relationship that the youth and mentor develop during the program, the youths’ bonds to school, and youths’ prosocial values.  To determine if there were changes, Darcy needs to assess these factors at two points: before the program begins (pre-test) and then again after the program is completed (post-test).

Pre- and post-tests are data collection tools administered at two points in time. A pre-test is administered before services or activities begin, while a post-test is administered at the conclusion of services or activities.

OUTCOMES MEASURED BY PRE- AND POST-TESTS

• Youth and mentors indicate interest in and willingness to meet with each other: Mentor/Youth Relationship Scale (pre- and post-test).
• Youths are bonded to school: School Bonding Scale (pre- and post-test).
• Youth adopt pro-social values: Youth Values Questionnaire (pre- and post-test).

The scores from the scales above serve as the data that assess important components of the evidence-based program Darcy is implementing:  the relationship between youth and mentor, youths’ bonds to school, and youths’ prosocial values.

Previous  Next

Anecdotal Evidence

You might be tempted to show that a program is working by providing anecdotal evidence.  An individual might cite one or two instances of a certain result as “proof” that a program is working effectively without considering whether the results are representative, or if the results were actually caused by the program instead of other factors.

For example, suppose a mother calls Darcy three months into her mentoring program to report that her son’s behavior at home is ten times better than it used to be. Her son is now taking out the trash without having to be asked; he’s using the phone much less; he’s obeying his curfew. These are signs that something is changing the youth’s life, but these changes may not be representative of the program outcomes.  Other youth in the program may not be showing the same improvements, or this youth’s behavior might be changing for some reason unrelated to his participation in the mentoring program.  Darcy cannot assume the program is a success overall based on the results experienced by this parent.

Systematic data collection—collecting the same data from all individuals participating in the program—gives better information about the impact of the program on its target group.

Previous  Next

Once data have been collected, they need to be analyzed. Data should clearly demonstrate whether or not a program met its objectives. Since most juvenile justice programs are trying to change attitudes and/or behaviors, data analysis usually focuses on:

• Whether a change occurred.
• How much change occurred.
• Whether or not a negative behavior was prevented.
• Any changes to the program structure or operation itself.
• Other unexpected changes that may affect the process or outcomes.

Previous  Next

Data analysis will show that one of four possibilities has occurred:

• You realize that your program was implemented as designed and your objectives were achieved.
• You realize that your program was implemented as designed, but you did not reach your intended objectives.
• You realize that your program was not implemented as designed and your objectives were not achieved.
• You realize that your program was not implemented as designed, but your objectives were achieved.

What should you do in each of these scenarios?

Previous  Next

Let’s look at the first possibility:  You realize that your program was implemented as designed and your objectives were achieved.  This scenario is ideal.

How should Darcy respond? She might want to consider hiring an evaluator.  Because Darcy has chosen an evidence-based program and has performed quality performance measurement, it is reasonable to assume that her program has been a success.  However, in order to empirically (i.e. scientifically) demonstrate that the mentoring program actually caused the positive changes in participating youth, she will need to hire an evaluator and establish that program activities actually caused the outcomes.

Remember, evaluation and performance measurement both:

• Focus on program performance
• Can be used for monitoring and improvement
• Generate evaluative information

… But evaluation takes performance measurement a step further, and determines whether the outcomes of a program can be attributed to the program or other factors.

Let’s take a look at the other scenarios.

Previous  Next

The second possible result after data analysis is that you realize your program was implemented as designed, but you did not reach your intended objectives.  If this occurs, consider the following questions:

• Did something happen independent of the program that affected your ability to achieve your objectives?
• Were the performance measures specified correctly?
• Have you expected to see results too soon?

Answering these questions will help you to determine why you have not achieved your program’s objectives.

Previous  Next

The third possible result after data analysis is that you become aware that your program was not implemented correctly and your objectives were not achieved.  If this has occurred, focus on reasons why program implementation did not occur as it was intended, and consider ways to improve fidelity to the program model.

For example:

• Are program participants from the correct target population?  If not, why not?  How might you select appropriate program participants in the future?
• Were activities carried out in the manner and frequency they were intended?  If not, what resources might help you adhere more closely to the intended activities?
• Have any external factors prevented you from properly implementing your program?  If so, how could you prevent this from occurring in the future?

Previous  Next

Finally, let’s consider the fourth potential result of data analysis: You realize that your program was not implemented as designed, but your objectives were achieved.  While this is unusual, it is certainly possible.

Let’s consider one of Darcy’s program activities:

For the duration of the program, mentors will meet with their assigned youth two times per week to go over homework and/or other academic issues.

Darcy’s analyses might reveal that in most situations, mentors met with their assigned youth once a week instead of two times per week, yet the objective to increase GPA was still achieved.  She may want to consider changing this activity since the program achieved objectives with less intensive services. On the other hand, she may want to retain the objective to see how different the outcome (GPA) would be if mentors do actually meet with youth twice per week.

Previous  Next

To make her application stronger, Darcy should mention ways she will analyze and report her findings once she is able to collect data.  She should state how she will collect data related to her original goal, objectives, and activities, and discuss how the data will show that the program has been implemented according to the plan.  She may also explain that she will eventually be able to identify accomplishments and indicate areas where the mentoring program needs improvement. Finally, she should note when she will be able to produce a report.

With this information included in her application, Darcy has provided the Mayor with a good desciption of her mentoring program as requested.

Based on Darcy’s program design and data collection strategy, she should be able to produce a report on the implementation and preliminary outcomes of the newly designed program in approximately one year.

Previous  Next

In addition to a report outline, Darcy submits a program timeline with her application. In planning her program, Darcy comes up with the following timeline that she will abide by:

Northwest Community Center Mentoring Program Timeline

Year 1

Year 2

Previous  Next

If you have implemented an evidence-based program correctly, measured consistently, and the problem still exists, you probably need an evaluation to determine why the program has failed to meet its goal(s).  A thorough evaluation will separate the program’s impact from external factors.  You should consider the following potential problems with a failed evidence-based program:

• Did something happen in the community independent of the program that coincided with the program to cause an increase in juvenile crime?  Could program youth have been affected by these events?
• Were program performance measures specified correctly?  For example, was the Mentor/Youth Relationship Scale used by Darcy a good measure to indicate whether program youth developed positive relationships with adult role models?
• Have you expected to see results too soon?

Responses to these questions might be used to modify the program plan and improve program outcomes.

Previous  Next