Juvenile Justice Evaluation Needs in the States:
Findings of the Formula Grants Program Evaluation Needs Assessment

INTRODUCTION

The Justice Research and Statistics Association (JRSA) is working with the Office of Juvenile Justice and Delinquency Prevention (OJJDP) on a project to develop and enhance juvenile justice evaluation capacity in the states. The goal of this project is to provide useful resources and practical technical assistance to local and state policymakers, program administrators and staff, and evaluators to strengthen their ability to implement and utilize evaluation studies and their findings. During the first phase of the project, JRSA conducted an assessment of the status of juvenile justice evaluation efforts in the states and territories by surveying key evaluation stakeholders, including juvenile justice specialists, State Advisory Group (SAG) chairs, and Statistical Analysis Center (SAC) directors. (See Appendix A for copies of the survey instruments.) This assessment focused on the evaluation practices related specifically to projects and initiatives funded by the Formula Grants Program, which was established under Title II, Part B, of the Juvenile Justice and Delinquency Prevention (JJDP) Act.

The first wave of surveys was sent to the SAC directors, JRSA's primary constituents, in February 1999. Section III of this report summarizes the findings from that survey. In addition, JRSA complied a list of all of the juvenile justice-related reports and publications completed by the SACs between 1994 and1999 (see Appendix B). In late May, a more comprehensive assessment survey was mailed to the juvenile justice specialists, who oversee the Formula Grants Program in each state. Findings from the juvenile justice specialist needs assessment are summarized in Section I of this report. Finally, to capture the perspective of the state policymakers appointed to review and establish programs and policies for the Formula Grants Program, a brief questionnaire was sent to the State Advisory Group (SAG) chairs in late May 1999. As provided by the JJDP Act, the SAG is an advisory board appointed by each state's governor. SAGs consist of 15 to 33 members who have training, experience, or special knowledge concerning the prevention and treatment of juvenile delinquency or the administration of juvenile justice. Findings from the SAG survey are summarized in Section II of this report.

Conclusions based on the findings from all three surveys are provided in Section IV of the report. The results of this assessment will be used to help determine the scope and content of the training and technical assistance JRSA develops to address the juvenile justice evaluation needs identified by these key groups. A national Advisory Group, with representatives from each of the juvenile justice stakeholder groups surveyed as well as two evaluation experts, will review the findings, discuss the implications for the states, and provide feedback regarding future project activities.
 
 

List of Tables

Table 1. State Evaluation Practices

Table 2. Juvenile Justice Specialists' Satisfaction with Evaluation Practices

Table 3. State Decisionmakers' Requests for Evaluation Information

Table 4. Evaluation Training and Technical Assistance Providers

Table 5. Training and Technical Assistance Needs Identified by the Juvenile Justice Specialists

Table 6. Juvenile Justice Stakeholder Groups that Need Training According to the Juvenile Justice Specialists

Table 7. Funding Availability for Evaluation Training

Table 8. Factors that Facilitate and Hinder Evaluation Efforts According to the Juvenile Justice Specialists

Table 9. State Advisory Group Chairs' Satisfaction with Evaluation Practices

Table 10. Training and Technical Assistance Needs Identified by the State Advisory Group Chairs

Table 11. Factors that Facilitate and Hinder Evaluation Efforts According to the State Advisory Group Chairs

Table 12. Statistical Analysis Centers' Juvenile Justice-Related Research and Activities

Table 13. Organizations Conducting Juvenile Justice Evaluations

Table 14. Factors that Facilitate and Hinder Evaluation Efforts According to the Statistical Analysis Center Directors
 
 

SECTION I

Results of Surveys of

Juvenile Justice Specialists

Surveys were sent to the state juvenile justice specialists to identify the evaluation practices and technical assistance needs in the states, especially as they related to the Title II, Part B Formula Grants Program. The survey was announced at the Coalition for Juvenile Justice annual meeting in early April 1999. During that meeting, JRSA held a focus group session with the regional juvenile justice specialists to gain their input about the content and format of the questionnaire.

Of the 56 states and territories, 48 replied to the survey (an 86% response rate). This section summarizes the findings from the juvenile justice specialist survey.
 

State Evaluation Practices

States assess the effectiveness of their juvenile justice projects/initiatives funded by the Formula Grants Program in a number of ways, making it difficult to create clear categories that capture the range of evaluation activities taking place. In the first survey question, the juvenile justice specialists were asked to select from a continuum of evaluation activities the approach(es) that best describe the evaluation practices in their states. The activities ranged from basic program monitoring to a state policy requiring all subgrantees to earmark funds for evaluation. The intent of the question was to begin to establish an overall picture of the nature and scope of the current state assessment activities, as well as the state policies that support those efforts. If none of the descriptions accurately characterized their state, the juvenile justice specialists were asked to write a brief description in an "Other"category. Table 1 summarizes the proportion of the states involved in specific strategies to evaluate projects funded by the Formula Grants Program. The numbers in the columns represent the number of juvenile justice specialists who listed that approach, and the percent is the proportion of the total 48 respondents.
 

Note. Percentages are based on the proportion of the 48 respondents. Multiple responses were permitted.
 

The most frequently-mentioned evaluation approach used by the states to assess juvenile justice programs is basic program monitoring. Almost all of the juvenile justice specialists (88%) indicated that they routinely collect standardized program information such as the number of clients served, the demographic characteristics of the program participants, and a general program description. More than half of the states (65%) require the subgrantees to include an evaluation plan with their grant application. However, only three states (6%) indicated that they require all subgrantees to allocate funds specifically for evaluation. In 23 states (48%), the state agency either conducts evaluations of programs or contracts with outside evaluators to assess specific programs. Four of these states reported that they set aside money specifically from the Formula Grant Program or Challenge Grant Programs to fund evaluations.

Specialists' responses to this question show that the majority of the states collect basic program monitoring data, which can serve as a foundation for a program evaluation. Fewer have policies requiring evaluation plans in subgrantee applications, and even fewer require subgrantees to specifically earmark funds to conduct an assessment.
 

Satisfaction With Existing Evaluation Practices

In the second survey question, the juvenile justice specialists were asked, "How satisfied are you with the evaluation practices in your state?" Table 2 shows their responses to this question.
 

Only five juvenile justice specialists (10%) indicated that they are satisfied with the existing evaluation practices in their state. As one specialist wrote, "Given the resources available, the existing system is sufficient." Almost half (46%) of the juvenile justice specialists reported that they are satisfied with parts of their state evaluation practices and not satisfied with others. Responses such as those were categorized as somewhat satisfied. For example, one specialist wrote, "The evaluation practices themselves are top-notch and rival those of any other state. However because the evaluator is several steps removed from any contracting or funding decisions, in practice, the evaluation tends to have minimal impact on programming."

A large proportion of the juvenile justice specialists (40%) responded that they are not satisfied with the evaluation practices in their state. Their reasons for dissatisfaction are listed below in the order of the most frequently mentioned reason to the least frequently mentioned:

1. evaluations are process rather than outcome oriented;

2. there is a lack of funding to conduct evaluations;

3. the quality of reporting and data collection by the subgrantees is inadequate;

4. it is difficult to compare across programs without common performance measures;

5. subgrantees and state staff do not have the expertise;

6. the large diversity of programs makes it difficult to develop standard evaluation outcome measures.

It is important to note that of the 41 juvenile justice specialists who were either somewhat or not satisfied with current evaluation practices, 16 reported that their states are actively involved in reviewing the evaluation process or have recently implemented policy changes to increase the quantity and quality of the assessments. Some of the evaluation policies recently implemented included: 1) requiring subgrantees to set aside evaluation funds; 2) requiring evaluation plans of subgrantees; 3) conducting evaluation training for all Formula Grant Program subgrantees; and 4) the state collecting and aggregating outcome data from individual programs.
 

State Advisory Group Use of Evaluation Information

Beyond use as a monitoring tool by the juvenile justice specialists responsible for overseeing the Formula Grants Program, evaluation data is also collected to assist decisionmakers charged with determining the merits of grant applicants for new and renewal grants. In every state, a State Advisory Group (SAG) is appointed by the governor to establish program policy and review state plans and Formula Grants Program applications. Because the SAG is the primary policymaking body overseeing the Formula Grants Program, the juvenile justice specialists were asked if they provide evaluation information to the SAG in their states, and if so, how the SAG uses the information.

Of the 48 respondents, 33 (69%) reported that they do provide evaluation information to their SAGs. According to the juvenile justice specialists, the most frequent uses of evaluation information by the SAGs include: 1) planning and program development; 2) determining continuation funding; 3) deciding if programs should be replicated; 4) informing the annual grant review process; and 5) reviewing contracts for independent evaluators.

Twelve of the juvenile justice specialists reported that they do not generally provide evaluation information to their SAGs. However, many of the juvenile justice specialists who do not currently provide information reported that they plan to in the future.
 

State Decisonmakers' Requests for Evaluation Information

The next survey question sought to determine which state decisionmakers other than the SAGs request evaluation information from the juvenile justice specialists. Table 3 shows the specialists' responses to this question.
 

Representatives from the State Legislature were the most likely policymaker group to ask for information about the assessment of juvenile justice programs and initiatives. Approximately one in three juvenile justice specialists reported that they have responded to requests from the State Executive Office. The fewest number of requests for evaluation information came from representatives from the State Budget Office who were mentioned by only nine (19%) of the juvenile justice specialists. Other agencies or individuals who have requested data include: 1) the state Auditor's Office; 2) representative from Congress; 3) university students; 4) the state Office of Contract Management; and 5) local officials. Fifteen of the juvenile justice specialists reported that they have not received a request for evaluation information from any of these policymaker groups.
 
 

Recent Evaluation Training and Technical Assistance in the States

Identifying the existing resources available to state and local juvenile justice program administrators is a first step in the process of building evaluation capacity in the states. The juvenile justice specialists were asked the following question: "List the evaluation-related training or technical assistance that the state Juvenile Justice staff or Formula Grants Program subgrantees received in the last two years." Because there is a fairly high turnover rate among the juvenile justice specialists in the states, it is likely that the number of evaluation training sessions was under-reported in the survey.

Twenty-seven states (56%) reported receiving some form of evaluation training or technical assistance in recent years, for a total of 51 sessions in all the states. It is important to note that of the 27 states who received some form of evaluation training, seven states were the recipients of three or more training sessions. In other words, seven states benefitted from almost half of the 51 training sessions reported.

The most frequently mentioned evaluation topics addressed during the sessions included: 1) introduction to program evaluation; 2) how to develop performance measures; 3) how to implement the Title V Community Self-Evaluation Workbook; and 4) how to improve the state's juvenile justice data/information system. The majority of the training/technical assistance identified by the specialists was held in their own states.

Juvenile justice specialists were also asked to identify who attended the 51 training sessions. Juvenile justice specialists or other state agency staff attended 75% of the sessions, prospective or current subgrantees attended 56% of the sessions, and other local sites or State Advisory Group members attended 10% and 4% of the sessions, respectively.

Table 4 describes the organizations that provided the evaluation training and technical assistance.
 

The distribution of providers conducting evaluation training and technical assistance sessions was fairly even. State and federal government agencies were responsible for 57% of the sessions, while research consultants/firms and universities conducted 29% and 16% of the sessions, respectively.
 

Future Evaluation Training or Technical Assistance Needs

Juvenile justice specialists were asked to assess the need for evaluation assistance in their states. More specifically, specialists were asked to describe: 1) the type of evaluation training or technical assistance that is needed in their states; 2) who should receive this training; and 3) whether funding is available for out-of-town training for key juvenile justice groups. Table 5 indicates the types of evaluation training needed in the states.
 
 
 
 
 

Note. Percentages are based on the proportion of the 48 respondents.
 

Almost half (48%) of all the juvenile justice specialists identified training needs in specific evaluation skills. The most frequently mentioned training topic in this category was identifying and developing performance measures, which was listed by one of every five juvenile justice specialists. Other frequently mentioned topics included: 1) outcome/impact evaluation training; 2) cost/benefit analysis; and 3) how to evaluate small programs.

Training topics falling under the broad category "basic program evaluation" were listed by 29% of the juvenile justice specialists. This category captures comments which identified the need for general and introductory evaluation training such as: 1) constructing a logic model; 2) performing a needs assessment; and 3) learning basic research methods.

A number of juvenile justice specialists (27%) requested training and technical assistance that is related to improving overall state evaluation practices. This category incorporates responses which are related to the development of a more systematic and standardized evaluation process including: 1) defining standard outcome measures for similar programs; 2) using standardized outcome measures for comparative purposes; and 3) developing a prototype evaluation model. Other training topics mentioned by the specialists that are related to the need to improve the state evaluation infrastructure include: 1) improving the grant application process; 2) creating more reliable reporting systems; 3) developing more comprehensive and valid information/data systems; 4) increasing the distribution and use of evaluation studies and findings; and 5) developing partnerships between state/local government and universities.
 

Table 6 lists the juvenile justice stakeholder groups that, in the opinion of the specialists, would benefit from evaluation training or technical assistance.
 

Note. Percentages are based on the proportion of the 48 respondents.

Prospective and current subgrantees and state juvenile justice agency staff are the groups that would benefit the most from evaluation training, according to the juvenile justice specialists. In addition, over half of the specialists (56%) listed the State Advisory Group members/state and local juvenile justice policymakers as juvenile justice evaluation stakeholder groups who should receive training.

Since limited funding is often a barrier to attending training sessions, the juvenile justice specialists were asked about the availability of funds for key groups to attend out-of-town training sessions. Table 7 summarizes the responses to this question.
 

In the majority of the states (88%), there is funding available for State Advisory Group members and juvenile justice specialists to attend an out-of-town evaluation training. On the other hand, only 33% of the juvenile justice specialists indicated that the Formula Grants Program subgrantees have funding available to attend non-local training sessions.
 

Factors That Facilitate and Hinder Evaluation Efforts

Juvenile Justice Specialists were asked to indicate those factors in the states that foster the juvenile justice evaluation efforts, and those that hinder those efforts. Table 8 summaries their responses to this question.
 

Note. Percentages are based on the proportion of the 48 respondents. Multiple Responses were permitted.
 

Staffing

Staffing was the most frequently mentioned factor that both facilitates and hinders juvenile justice evaluation efforts in the states. Juvenile justice specialists who reported that skilled people support their state's evaluation efforts described situations such as: 1) having knowledgeable juvenile justice staff or local program administrators who, as one specialist wrote, are "able to conduct evaluations or contract out for them, and work closely with juvenile justice programs to develop evaluation materials"; 2) partnering with local universities to engage in evaluation activities; and 3) having research and evaluation divisions within state government to support their efforts.

Unfortunately, staffing was even more frequently mentioned as an impediment to state evaluation efforts. The specialists who responded in this manner mentioned issues such as: 1) insufficient number of staff; 2) staff/program administrators lacking the knowledge of evaluation design/techniques; 3) not having a person in a leadership position to guide others through the process; and 4) high staff turnover. Many of the juvenile justice specialists commented that there is a great need for training for both state juvenile justice staff and local program administrators. One specialist simply stated, "They would appreciate any help provided."

Funding

After staffing, funding was the next most frequently mentioned factor that impacts juvenile justice evaluation efforts in the states. Three funding strategies were viewed as facilitating evaluation activities: 1) earmarking funds for evaluation in each subgrantee's award; 2) allocating specific state funds out of the Formula Grants Program award to conduct evaluation activities; and 3) funding evaluation efforts with Federal Drug Control and System Improvement funds.

More specialists viewed funding as hindering rather than facilitating evaluation in their states. Difficulties with funding included: 1) small project awards resulting in limited administrative budgets to fund an evaluation; 2) a federal pass-through requirement (67%) that limits monies available for evaluation; and 3) the perception of subgrantees that evaluation will leave no funds for the program itself. Although many specialists indicated that insufficient funding was the issue, others implied that the short-term nature of grant funding was the hindrance. For example, one specialist wrote, "It is an unrealistic expectation to have small, short-term grants track long-term outcomes given the cost and time that it would take." Another reported that, "The short duration of funding makes outcome measures difficult to track until after grant funds have been expended, and the obligation to evaluate then is impossible to enforce."
 
 
 

State Infrastructure

Specialists identified state infrastructure issues, such as evaluation-related policies and data systems, as other factors that affect evaluations. Policies that facilitate evaluation include: 1) requiring an evaluation plan as part of the grant application; 2) requiring subgrantees to incorporate an outside evaluation component; and 3) creating a standard evaluation system to collect data for comparative purposes. The lack of effective policies, such as those mentioned above, was identified as a hindrance to evaluation by many of the states.

Insufficient state juvenile justice data systems also present barriers to program assessment. These barriers include:1) not having baseline data collected and maintained by criminal justice agencies to determine the impact of programs; 2) no uniform measures collected among agencies; 3) difficulties obtaining data such as neighborhood crime trends from the police; and 4) administrative procedures developed for direct-service projects that are not appropriate for evaluation projects. The tension between the competing goals of standardizing state evaluation policies and practices and developing a system that accounts for the diversity of projects is illustrated by this account from one of the juvenile justice specialists:
 
 
 

Every grant-funded projects is different, therefore, evaluation measures will vary. That is ok, but there is constant demand for statewide results of whole funding programs rather than hundreds of separate sets of results for individual projects. We hope to develop common measures that are valid. It is unrealistic to expect relatively small federal grant programs to produce statewide impacts (e.g., a 3% reduction in recidivism). What is realistic is a sound evaluation of individual projects, and aggregating those results to the extent possible
 

Evaluation Environment

The category "evaluation environment" captured a range of comments that referred to the overall acceptance in the state that evaluation is critical in the development of effective strategies that improve programs and initiatives within the juvenile justice system. According to the juvenile justice specialists, a supportive evaluation environment is one in which key juvenile justice evaluation stakeholders possess the following: 1) a desire to know what works; 2) a shared belief that evaluation is necessary to determine program effectiveness and impact; 3) leaders who advocate outcome evaluations instead of just process evaluations; and 4) the desire to use evaluation findings to enhance state and local planning.

Conversely, an environment that is not supportive of evaluation is described as one in which: 1) subgrantees do not recognize the value of evaluations and therefore do not make the effort; and 2) ideological support for rigorous evaluation practices is lacking.

Leadership

Juvenile justice specialists indicated that the commitment of organizations and key individuals in leadership positions influences evaluation practices. The organizations and individuals most frequently mentioned included: 1) State Executive Office; 2) legislative branch of the state government; 3) local government officials; 4) and juvenile justice administrators in state and local agencies. Examples of supportive leadership practices that have improved evaluation efforts are provided in the following comments from the specialists:

"Legislative and executive branches interested in outcome measurements and recidivism when allocating funds for juvenile justice, delinquency prevention and reduction;"

"Leadership among the Governor, Lieutenant Governor, and the SAG to formulate state policy and fund programs that are effective based on empirical research with measurable objectives;"

"The SAG and the related agency structure must be supportive of evaluation efforts or they will not occur. Without this structural support, the resources necessary (to conduct evaluations) would not be available."

Five juvenile justice specialists also identified the lack of leadership effort to make evaluation a priority as a hindrance to evaluation, which is illustrated by the following statements:

"I'd like to see more leadership at the federal level on this subject, especially in providing a structure and funding to implement a good and consistent system;"
 
 
 

"[Their efforts are hindered by the] lack of concern for overall program effectiveness by the SAG and the fact that it is not a priority by the agency."
 
 

Summary

The purpose of surveying the juvenile justice specialists was to gain the perspective of the state staff responsible for overseeing the Formula Grants Program. According to this needs assessment, 88% of the states collect basic program monitoring data, and 65% require an evaluation plan from subgrantees. The majority of the juvenile justice specialists are not completely satisfied with the existing evaluation practices.

One of the main reasons for allocating resources to evaluation is to enable policymakers to make empirically based decisions about funding and replicating programs. But not all of the state Formula Grants Programs are under the same level of public scrutiny. The findings indicate many of the SAGs receive evaluation information on the programs being funding by the Formula Grants Program to assist them with decisionmaking and state planning. However, 25% of the juvenile justice specialists do not routinely provide information to their State Advisory Group, either because it is not available or not requested. Additionally, although 50% of the juvenile justice specialists reported that they had requests from at least one key policymaking group for evaluation information, 31% of the specialists reported that they had never received a request for information from representatives of the State Legislature, State Executive Office or the State Budget Office.

The evaluation training and technical assistance provided to the states should be targeted at improving both the individual evaluation capacity of state and local administrators and the state evaluation infrastructure, according to the juvenile justice specialists. Individually focused training topics identified included an introduction to basic program evaluation and developing specific skills, such as how to develop measurable outcome indicators. The technical assistance topics identified were related to improving the overall state evaluation process, included creating standardized evaluation models, improving the data systems, and developing partnerships between the state and the academic community. As Table 4 indicates, universities conducted only 8 of the 51 training sessions, which may indicate an underutilized resource. It is also important to note that the group least likely to having funding available to travel, the subgrantees, was identified by the specialists as the stakeholder group most in need of evaluation training.

Finally, the factors most frequently mentioned by the specialists that both facilitate and hinder current evaluation efforts were staffing and funding. Beyond training the existing state and local staff, the juvenile justice specialists indicated that their states would benefit from having additional staff responsible for implementing and monitoring evaluation activities. Although some juvenile justice specialists described successful funding strategies being used in their states to facilitate activities, insufficient funding was reported to be a problem. Specialists are concerned with the unrealistic expectation that very small programs have the resources necessary to conduct long-term outcome evaluations.
 
 

SECTION II

Results of Surveys of

State Advisory Group Chairs

This section summarizes the findings of the survey of State Advisory Group (SAG) Chairs. To obtain the perspective of these juvenile justice policymakers on the Formula Grants Program, a brief survey was developed which asked the SAG chairs to assess the evaluation policies and practices in their states. The survey was announced at the Coalition for Juvenile Justice annual meeting in early April. During that meeting, JRSA held a focus group session with the regional SAG chairs to acquire their input about the content and format of the questionnaire. The SAG chairs were mailed the survey in late May.

JRSA received 23 responses in total, but the responses represent only 20 of the states and territories. In one state, both the juvenile justice specialist and the SAG chair replied to the survey, and in another both the chair and an additional SAG committee member responded. The SAG chair of one state that recently became eligible for Formula Grants Program funding returned the survey, but did not have any experience upon which to make recommendations. Therefore, the findings presented are based on 22 responses from 20 states.
 

Satisfaction with Existing Evaluation Practices

The first question in the survey assessed the SAG chairs' overall satisfaction with the existing evaluation practices in their states (Table 9). Of the 22 responses to the SAG survey,
 
 
 
 
 
 
 

3 chairs are satisfied with the evaluation practices, 7 are somewhat satisfied, and 12 are not satisfied. Those who reported being satisfied with the existing evaluation practices are in the process of, or have recently implemented, changes in their evaluation system with the assistance of universities or consulting firms.

The vast majority of the SAG chairs who responded were less than completely satisfied with evaluation in their states. A concern with the overall quality of the assessments was the most frequently mentioned issue. More specifically, many respondents expressed frustration with the limited numbers of outcome evaluations being conducted. For example, one chair wrote, "Too often, grantees submit little impact evaluation data, and are not held accountable."

Other issues mentioned regarding the quality of evaluations included: 1) the need for on-sight visits to verify monitoring reports; 2) not enough cross-site and long-term evaluations; 3) poorly defined goals and objectives; and 4) the limitations of using self-assessment methods as analytical tools. Additional reasons given for not being satisfied were insufficient staffing and limited funding.

Evaluation Training and Technical Assistance Needs in the States

In Question 2, the SAG chairs were asked to identify the types of evaluation training or technical assistance needed in their state. Overall, 15 of the SAG chairs indicated that they did have evaluation needs, 2 reported that they did not need assistance, and 5 were unsure.

States with Training Needs

Of the 15 chairs who reported a need for evaluation training and technical assistance, the types of training requested fell into three categories: 1) basic program evaluation training; 2) specific evaluation skills; and 3) the state evaluation practices. Table 10 shows the number of the SAG chairs who identified needs in the three categories.
 

Seven of the SAG chairs indicated that their state would benefit from basic program evaluation training for both the SAG committee members and local program administrators. One chair wrote, "[We need] training on evaluation in general - what it is, how to do it, why it is important, and how it relates to the SAG." In addition to general evaluation training, other chairs commented that states could improve their evaluation capacity by identifying and using existing resources, such as survey instruments, that have already been developed.

Four of the chairs indicated that states would benefit from training in specific evaluation skills such as facilitating outcome-based planning and conducting outcome evaluations. Finally, four of the SAG chairs indicated that their states need technical assistance to standardize and simplify the existing state evaluation practices. According to one chair, "There is a need for more consistency in the process, [it is] too random, ad hoc, and inconsistent to be of much value. This is contributed to by state systems which are fragmented and under multiple agencies." Other examples of comments associated with standardizing the evaluation process include requests for: 1) a simple evaluation instrument; 2) a step-by-step format that standardizes the process; and 3) common terminology for evaluations and outcome measures. Another chair stated, "The critical need is in developing appropriate evaluation systems that address each of our program/service delivery systems."

States With No Training Needs/Unsure of Training Needs

Of the 22 respondents, 2 indicated that they did not need evaluation training or technical assistance because they have contracts with a research firm or a university. As one chair reported: "We use competent consultants for our evaluation needs." The remaining five chairs were either unsure of their training needs or unsure that training would be beneficial. For example, one SAG chair wrote, "[There is] no need to train if [there are] no funds for evaluation." A second chair indicated that training would only be necessary if OJJDP's expectations related to program assessment change. More specifically, the chair wrote, "I don't feel we need any TA unless the OJJDP is requesting evaluation of programs to measure causal relationships between program activities and outcome objectives. If OJJDP requires this, then I would recommend the funding be increased to insure providers can not only develop a more comprehensive evaluation plan, but also implement it."
 

Factors that Hinder and Facilitate Evaluation Efforts

Questions 3 and 4 asked the SAG chairs, in their professional opinion, what factors in their states foster the juvenile justice evaluation efforts, and what factors hinder those efforts. The aggregated responses merged into the following categories: 1) staffing; 2) policies; 3) funding; 4) leadership; and 5) evaluation environment. Table 11 describes the number of SAG chairs who mentioned each of the above factors.
 
 
 
 
 
 
 

Staffing

Staffing was the most frequently mentioned factor that both facilitates and hinders evaluation efforts in the states. The category staffing includes state juvenile justice staff, SAG committee members and local program administrators. Approximately half of the chairs indicated that they have knowledgeable staff in their states who are informed about evaluation.

In contrast, 15 chairs listed staffing-related issues as barriers to their states' evaluation practices. These issues included: 1) limited knowledge of evaluation; and 2) insufficient number of staff. One SAG chair reported that their state's ability to conduct evaluations is impeded by the "limited availability of knowledgeable individuals or organizations that can provide guidance throughout the evaluation process." In fact, another chair recommended: "We need a position within the JJPA agency, funded from administrative budget to coordinate the on-going process evaluation activities and develop and implement long-term program evaluations."

State Infrastructure

Existing juvenile justice evaluation policies and practices, or the state infrastructure related to the Formula Grants Program, was the second most frequently mentioned factor influencing the states' evaluation efforts. Policies and practices described by the chairs that facilitate evaluation efforts included: 1) requirements by the granting source that evaluation be done; 2) standardized format for the guidelines and criteria for grant recipients and evaluators; 3) the requirement that an evaluation plan be part of the grant proposal; and 4) the requirement that funding be set aside for evaluations.

Specific policy-related issues identified as hindrances to evaluation included: 1) policies that are inconsistent and not clearly understood by grant recipients; 2) the lack of a policy that links evaluation results to funding decisions; and 3) policies that prohibit access to data from other youth service agencies such as school systems. A comment from one SAG chair illustrates the frustration that some of the chairs have with the existing evaluation policies: "[We have] inconsistent policy statewide as to what evaluation means, what we really want, and whether we are willing to fund it."
 

Funding

Of the ten SAG chairs who listed funding as a factor, eight believe it is a barrier to current state evaluation efforts. The primary concern was with allocating limited Formula Grants Program funds to evaluation, rather than additional programming. This concern is illustrated by the following comment from one of the SAG chairs: "Funding is just enough to conduct programs, and evaluations would take away from those funds." In light of the fact that many programs receive small grant awards, it remains unclear to many of the chairs how to determine or justify the proportion of dollars that should be spent on evaluation.

Leadership

Five of the chairs indicated that strong leadership from the SAG, other policymakers or juvenile justice administrators is a key component of effective evaluation practices. Several chairs indicated that the SAG plays an important role in advocating evaluation and monitoring.

In contrast, the "lack of cooperation of the administrative officials with the SAG" was listed as an example of leadership presenting a barrier to program evaluation. "Increasing and improved federal monitoring of practices and procedures" was identified by one chair as a leadership strategy that would enhance evaluation practices.

Evaluation Environment

This category incorporates comments which refer to the overall acceptance in the state of the idea that evaluation is critical in the development of effective strategies to improve programs and initiatives. A supportive evaluation environment was described by the respondents as one in which key juvenile justice stakeholders have the "desire to know what works." However, to know what works, evaluations must be theoretically and methodologically rigorous, according to three of the chairs.

The lack of a systematic approach to assessing programs and initiatives, distributing the results, and incorporating the findings was identified by three of the respondents as elements of an unsupportive evaluation environment. For example, one chair reported, "The justice system in general, but especially the juvenile justice system, does not yet have a system focus that would promote the development and use of effective evaluation tools." Another chair indicated, "We need to address the question of what we do with the results or findings of an evaluation? If the results are negative, what form of technical assistance do we provide to improve programs."
 

SUMMARY

The findings from this needs assessment indicate that most SAG chairs believe that the current evaluation practices related to projects/initiatives funded by the Formula Grants Program are not completely effective. As stated earlier, the majority of the responding chairs were less than completely satisfied with the existing evaluation practices in their state.

Staffing was the most frequently mentioned factor that hinders evaluation efforts. Basic program evaluation training and training in specific evaluation skills are needed to enhance evaluation capacity in the states. The second most frequently mentioned factor is a systemic issue: evaluation policies. SAG chairs noted that technical assistance is needed to support the states in the development
 

of more standardized evaluation practices, ranging from improving the granting process to the creating consistent outcome measures.

SAG chairs who indicated that they are satisfied with their existing evaluation practices are those whose states, with the help of consultants and universities, are already in the process of increasing their evaluation capacity. The strategies they are using - improving the system through policy changes and improving state and local knowledge through training - are similar to the needs identified by the remaining SAG chairs.
 
 

SECTION III

Results of the Surveys of

Statistical Analysis Center Directors

The first wave of the assessment surveys was sent to the Statistical Analysis Center (SAC) directors, JRSA's primary constituents. The directors received the survey in early February. Of the 52 SACs in the states and territories, 42 replied to the request for information, which is an 81% response rate. This section summarizes the findings of the from the SAC director survey.
 

Juvenile Justice-Related Research and Evaluation Activities

Table 12 summarizes the data analysis, research, and evaluation activities that the SACs have conducted in the last five years to support their state juvenile justice system. This information was derived from the directors' responses to the survey, with additional information taken from the JRSA annual directory, Criminal Justice Issues in the States, which catalogs all the SAC reports and research activities. As Table 12 shows, the SAC activities could be classified as addressing three major areas: the juvenile justice system; community-based initiatives; and school-based initiatives. Within the category juvenile justice system, research was conducted on topics such as juvenile day reporting centers, court programs, programs for special offender populations, and data analysis for the Formula Grants Program three-year plan. Gang studies and assessments of local delinquency prevention programs were categorized as community-based initiatives. Research and activities that were included in the category "school-based initiatives" were evaluations of programs such as DARE, school resource officer projects, and student risk assessment surveys.
 
 
 
 
 

As Table 12 s hows, the level of involvement of the SACs with juvenile justice evaluation efforts varies. The findings of this survey showed that eight SACs are participating in four or more juvenile justice research activities: Delaware, Illinois, North Carolina, Ohio, Pennsylvania, Texas, Virginia, and West Virginia. It is also important to note that seven SACs are responsible for at least one of the activities (data analysis, monitoring, assessment and implementation) related to the OJJDP Title II, Part B, Formula Grants Program. Those states are Alaska, Indiana, Iowa, Massachusetts, Missouri, Ohio, and Rhode Island.

The increased involvement of the SACs in juvenile justice research and planning in the last few years can partly be attributed to the introduction of the State Justice Statistics (SJS) Program for SACs in 1997. This program supplanted the State-Level SAC and Information Network Program, which was funded exclusively by the Bureau of Justice Statistics (BJS) and provided partial support to state governments to establish and operate SACs to collect, analyze and report statistics on crime and justice. The new SJS program provides funds from BJS, as well as other Office of Justice Program (OJP) agencies, for SACs to conduct research on specific issues. Each year BJS, JRSA, and selected SAC directors determine the SJS program themes. Other OJP agencies provide supplemental funds for SAC projects under specific themes.

In 1998, the juvenile theme/topic requested was research, analysis and evaluation related to the provisions of the Juvenile Accountability Incentive Block Grants (JAIBG). SACs could submit applications for up to $50,000 in additional funding to conduct one or more studies related to states' certification for JAIBG grants or topics related to the 12 JAIBG program purpose areas. According to JRSA's assessment, 18 SACs played a role in the implementation of JAIBG in their states, which included activities ranging from data collection and analysis to program development and evaluation. The juvenile theme was unchanged in the SJS Application Guidelines for FY 1999, and supplemental funding was again available for JAIBG-related projects. Thus through the new SJS program, the SACs have received support to conduct research that examines critical juvenile justice issues. (See Appendix B for a list of recent SAC juvenile-justice related reports and publications).
 

The majority of SACs participate in at least one of the following activities, which support juvenile justice evaluation efforts in their states:

1) Respond to requests from juvenile justice advisory councils, legislators, state agencies, and other commissions to provide descriptive data and reports that assist decisionmaking processes;

2) Produce an annual report that provides statistical data and analysis on juvenile and adult crime trends in their state;

3) Analyze and report on population forecasts of juvenile offenders and institutional populations;

4) Participate on advisory boards/committees that assess, develop, or improve criminal and juvenile justice data collection and evaluation activities;

5) Provide funding or contract with other organizations, such as universities or research centers, to conduct evaluations of juvenile justice programs.
 
 

Other Organizations Conducting Juvenile Justice Evaluations

Question 3 in the survey asked the SAC directors to list other organizations involved in juvenile justice evaluation in their states. Table 13 shows the organizations listed by the SAC directors.Research institutes and universities were cited by 40% of the directors as other organizations conducting juvenile justice program evaluations in their states. Other state organizations conducting juvenile justice evaluations included: 1) state family or youth services; 2) state criminal justice planning; 3) state juvenile justice departments; and 4) governor-appointed commissions.
 
 
 
 
 
 
 

Factors that Facilitate and Hinder Evaluation Efforts

Questions 4 and 5 in the survey asked the SAC directors, in their professional opinion, what factors in their states foster juvenile justice evaluation efforts, and what factors hamper those efforts. Although their responses were varied and the examples were often specific to their state infrastructure, the comments converged into six themes: 1) funding; 2) staffing; 3) evaluation environment; 4) data systems; 5) state infrastructure; and 6) legislative interest.

Table 14 shows the number of SAC directors who listed factors in these areas. The numbers in the columns represent the number of directors who listed that factor, and the percentage is the proportion of the total 42 respondents. These factors are listed in the order of the most frequently mentioned to the least frequently mentioned.
 
 
 
 
 

Note. Percentages are based on the proportion of the 42 respondents.
 

Funding

Funding was the most frequently mentioned factor that both facilitates and hinders program evaluations of juvenile justice activities. It was named by 38% of the directors. According to the responses, two strategies that states are using to fund evaluation efforts are:

1) earmarking funds in state agency budgets; and 2) lobbying their state legislatures to set aside resources specifically for juvenile justice research and planning. In addition, one director wrote "The availability of funds from federal sources such as OJJDP makes (evaluation) more feasible."

Although SAC directors identified successful funding strategies, insufficient financial resources remain a concern. Legislators interested in reducing the rates of juvenile delinquency are enacting legislation for specific programs, but often do not appropriate the additional funding for an evaluation. As one director pointed out, "There is little state general revenue to support implementation and evaluation." The point was also made that "Evaluations that may not meet academic standards also provide good information, and the reality is that resources do not exist to always do formal academic studies. Therefore, we need to find a less costly means of getting reliable data that are good enough."
 

Staffing

After funding, staffing was the most frequently mentioned impediment to evaluation efforts. Staffing issues included: 1) not having enough staff; 2) staff not having adequate knowledge or time; 3) not having cooperative relationships with other agencies; and 4) not having a senior staff person responsible for evaluation activities. Nevertheless, many SAC directors wrote that the knowledgeable staff in their offices, as well as other state agencies, produce sound research. For example, one SAC director reported, "What evaluation is done is facilitated by the existence of the Juvenile Justice Advisory Board and a relatively well-staffed Department of Juvenile Justice Research Unit."

Evaluation Environment

This category captures a range of comments that refer to the SAC directors' perception of the states' "receptivity" to program evaluation for juvenile justice efforts. According to the SAC directors, a supportive evaluation environment is one in which one or more of the key evaluation stakeholders (grant administrators, practitioners, state agency heads, advisory group chairs, and legislators) share a sense of responsibility to respond to the demand for accountability, a desire to know what works, and a willingness to allocate resources to understand how policy changes affect the system as well as the lives of youth.

On the other hand, an environment that is not supportive of evaluation is described as one in which there is a lack of commitment to good research, practitioners and policymakers are impatient for findings, and evaluation is not thought about during the program planning stage.

Frustration was also apparent in comments such as, "Evaluations, even when conducted, are not used in practice to make decisions; agencies such as public safety receive funding regardless of evaluation results." And, "Evaluations will only be conducted if they are required and failure to do so results in a penalty." Another SAC director commented that "Evaluation findings need to be presented in a way that is more understandable and useful for policymakers and practitioners. Researchers need to find better ways of communicating findings to people making decisions."

Data Systems

Approximately 26% of the SAC directors indicated that data systems are a hindrance to juvenile justice evaluation efforts. The overriding concern is that data from the existing systems are unreliable and insufficient. In part, this is the result of data originating from different agencies (courts, law enforcement, and corrections) that do not record the same information. One director wrote, "There is a lack of data or databases at statewide or local levels which store timely, integrated information on juvenile processing offenses and supervision. There is resistance by service providers to collect evaluation-related data, and there is no accountability for grant-funded service providers to provide data and facilitate efforts." Another commented that there is "no comprehensive juvenile arrest data, and very summative court statistics with no breakdown by offense or offender type."

It is important to note that some of the data system limitations are purposeful. Unlike adult criminal history records, juvenile records are considered private, even among state agencies. Therefore the system never intended to provide easy access to youth records. As a result, the information that can be used as outcome measures for evaluation is limited. Recent legislation in states such as Pennsylvania and Illinois requires that youthful offenders be fingerprinted for felonies and the creation of juvenile criminal history records. Both data sources will increase the capacity of analysts in the states to conduct longitudinal studies, specifically those addressing recidivism.

A few states listed their data system and the receptivity to data-informed policy as positive factors. One director commented, "Support for unbiased factual analysis of problems is evident in our legislature and administration."

State Infrastructure

The SAC directors mentioned four common themes within the overarching category labeled "state infrastructure":

1) Collaborative Partnerships: Many of the directors indicated that having strong, collaborative relationships with local and state agencies, universities, and other research organizations has resulted in the SAC being more involved in juvenile justice evaluation efforts in their states.

2) State Granting Process: Although strong partnerships are valuable, they also present problems in state administrations. A director wrote, "The types of grants we are writing, especially the partnerships, do not fit into the existing government fiscal grant administration and contractual processes. This has resulted in it taking more time to implement program evaluations once conceptualized and approved than anticipated."

3) Decentralized State Juvenile Justice Systems: States that are large with geographically dispersed populations often operate highly decentralized juvenile justice systems. As a result of the decentralized systems, data collection procedures and reporting formats can be very inconsistent. This makes it difficult to conduct evaluations that require multijurisdictional involvement. A few of the SAC directors commented that this issue could be addressed by implementing a more standardized evaluation system. One SAC director indicated, "There is a need for either the OJJDP or a state agency to take the lead and develop clear expectations and standards for juvenile justice evaluation." For example, a state could create an evaluation plan that identifies different levels of evaluation (monitoring, process, impact) that are required for programs at certain points in the program lifecycle.

4) Oversight Committees: It is important that the committees or advisory groups overseeing the juvenile justice systems in the states are committed, both ideologically and financially, to evaluation.

Legislative Interest

The public concern with increasing juvenile violence in recent years has prompted state legislators to focus more closely on juvenile justice issues. As discussed above, the SAC directors indicated that legislation has been passed to implement new juvenile justice programming, and in some cases, money is being earmarked for evaluation. The Minnesota SAC successfully lobbied the state legislature to fund Teen Courts and to set aside resources to conduct an evaluation.

Legislative interest in juvenile delinquency is also being met with apprehension, in that the implementation and evaluation of specific programs is being legislatively mandated, but the mandates are not being funded. Unfunded mandates force state agencies to use monies not specifically earmarked for evaluation. And finally, the SAC directors acknowledged that even evaluations conducted under legislative mandate sometimes fail to affect policy. For example, one director wrote, "A large-scale assessment of the impact of race on the [state's] juvenile
 

justice system did not lead to any significant changes in [the legislature's] policies or practices."
 

Summary

The findings from this survey indicate that the majority of the state SACs are involved in a number of juvenile justice-related activities, which range from data collection and descriptive statistical analysis to conducting comprehensive program evaluations. The two juvenile justice research areas in which the largest number of SACs are involved are studies related to disproportionate minority confinement and efforts to document and improve existing juvenile justice information systems.

Comments from the SAC directors also reveal the need for additional training and technical assistance to support evaluation efforts. According to the SAC directors, funding is the main factor that either hinders or fosters evaluation efforts in the states. If the money is not specifically set aside for evaluation activities, they are much less likely to occur. One out of three SAC directors indicated that there is a need to either increase the number of staff who have evaluation expertise, or to provide training for existing personnel. Approximately 25% of the SAC directors stated that the evaluation environment in their state is a hindrance to the juvenile justice evaluation efforts. Both of these factors present barriers which can be addressed through training and technical assistance designed to increase the evaluation knowledge and awareness of staff responsible for conducting evaluations as well as juvenile justice planners and policymakers who use evaluation results.

Finally, an important goal of the assessment phase of this project is to identify the existing juvenile justice evaluation resources in the states. One juvenile justice area to which SACs bring expertise and experience is the development of juvenile data systems. One in five SAC directors indicated that they are involved in juvenile justice system data improvement projects. Quality program evaluation efforts require accurate and reliable data. Since the SACs are responsible for criminal justice data collection efforts in the states, they are in a position to improve the juvenile justice evaluation capacity by playing a significant role in the identification and development of credible juvenile justice data systems.
 
 

SECTION IV

Conclusions

Although some of the states surveyed are satisfied with their evaluation practices, it is clear from the assessment that the majority of the juvenile justice specialists and the SAG chairs are not completely satisfied with the status of current efforts. When the findings from the three surveys are summarized, what emerges is the need to enhance the juvenile justice evaluation capacity on three levels: 1) the individual level; 2) the state level; and 3) the relational level.

Individual Level

The assessment findings consistently indicate that maintaining a knowledgeable staff is essential to evaluation efforts. All of the juvenile justice stakeholder groups surveyed were asked to identify the factors that facilitate and hinder the evaluation efforts in their states. For all three groups, staffing was one of the three most frequently mentioned factors. Additionally, the SAG chairs and the juvenile justice specialists, when asked to identify the training and technical assistance needs in their states, reported that subgrantees, SAG members, state juvenile justice administrators and policymakers would all benefit from training designed to improve their ability to conduct and use evaluation studies and their findings. The individual training topics most frequently reported included specific evaluation skills such as developing outcome measures or conducting a cost/benefit analysis, and basic program evaluation training.

These findings suggest the need for training and evaluation technical assistance materials that are accessible to a broad base of juvenile justice stakeholders, with diverse backgrounds and skills, who have the need to improve their understanding of introductory program evaluation terminology, methodology and techniques. In order for the materials to be meaningful to a wide audience of juvenile justice stakeholders, evaluation terms and program examples should be specific to the field of juvenile justice.

State Level

The second level of assistance, according to the needs assessment findings, should be directed at improving the state juvenile justice evaluation system as a whole. Enhancing the evaluation capacity of individuals does not address the need, identified by all three juvenile justice stakeholder groups, to develop more standardized, consistent and useful state evaluation policies and practices. Currently, basic program monitoring is the most common assessment activity taking place in the states. Few states and territories require subgrantees to submit an evaluation plan with their grant application or conduct evaluations of specific programs. Only three of the states have a policy that requires subgrantees to earmark funds to conduct evaluation activities.

Another training and technical assistance area that was identified by both the SAG chairs and the specialists is how to improve state evaluation practices. The significance of a systematic approach to building evaluation capacity was supported by the finding that state infrastructure, which includes evaluation policies and practices, was cited as a factor that facilitates and hinders evaluation efforts in the states by all three stakeholder groups. The limitations of making changes without regard to the entire evaluation process are illustrated by the stakeholders' insights regarding funding as a factor that facilitates and hinders evaluation efforts. Although insufficient funding was reported to be a significant barrier to state evaluation efforts, the respondents also identified the need for more consistent evaluation funding policies and practices. In other words, without a state strategy to prioritize the programs to be evaluated and a policy which allocates specific funds for evaluation activities, just providing increased financial resources to the states may not eliminate funding-related barriers to a more effective evaluation system.

The assessment also revealed the perception that the juvenile justice leaders at the federal and state level are not setting evaluation priorities and clarifying expectations - OJJDP's expectations of the state juvenile justice agencies, and state juvenile justice agencies' expectations of local governments and individual programs.

All of these related findings suggest that technical assistance is needed in the states to facilitate the development of more standardized and consistent state evaluation policies and practices. Implementing a more systematic strategy to assess the impact of juvenile justice programs may clarify the expectations of state and local administrators, as well as provide more valid and reliable information for policymakers.

Relational Level

Finally, the assessment disclosed the need for technical assistance to develop state and local evaluation partnerships with state agencies or private organizations with evaluation capabilities. The states that reported being satisfied with their existing evaluation practices have utilized and fostered relationships with existing organizations that have evaluation expertise. Universities, research firms, Statistical Analysis Centers (SACs), and other state government departments were identified as organizations with the capacity to facilitate the creation of more effective state evaluation strategies. According to the SAC director survey, the majority of the SACs are involved in juvenile justice research and planning activities such as responding to data requests for the Formula Grants Program Three-Year Plan and conducting independent evaluation studies of specific programs.

These findings suggest the need for training and technical assistance that helps create sustainable state and local evaluation capacity and promotes evaluation partnerships that support the development of effective and useful state-specific juvenile justice evaluation efforts. A few states either have, or are in the process of developing, a state juvenile justice evaluation strategy, but they remain in the minority. The lessons those states have learned, as well as the practices and procedures they have implemented, may be useful for other states and territories committed to improving their current evaluation efforts.
 
 

APPENDIX A:

Survey Instruments
 
 
 
 

APPENDIX B:

SAC Juvenile Justice-Related Reports and Publications