VTC Evaluation 101 · The results you hope your VTC will achieve ... Problem •Headache Activities...

All Courts are Family Courts -

when participants

have children and other family

members.

3

Why Focus on Children? The Numbers

• 50% of OIF/OEF veterans are

parents

• Over a million children have

parents who served since

9/11

• 30-35% of deployed parents

have trauma and substance

abuse issues

You can’t coordinate what you can’t count

• The task is not diverting funds from VTCs; it is accessing

funds already available for children's services

• But to mobilize and link to new resources from other

agencies that already serve children

VTCs Don’t Have Do It Alone

• Maternal and child

health,

• Mental health

• Child development,

• Youth services

• Special education

• Delinquency prevention

That’s what collaborative means

• How many children of veterans are in your VTC?

• What services and supports do these children need?

• How can your VTC build lasting effective networks or

partnerships to respond to the needs of these

children?

3 Ns – Numbers, Needs, Networks

LEARNING OBJECTIVES

•Find out how to make evaluation work for you to improve your program, identify your successes and plan for sustainability

•Learn how and where to start your evaluation

•Understand how to choose an evaluation plan and an evaluator that works for you

WHAT IS EVALUATION?“Program evaluation is the

systematic collection of information about the activities, characteristics, and outcomes of

programs to make judgments about the program, improve

program effectiveness, and/or to inform decisions about future

programming” -PATTON, 1997

THE EVALUATION PROCESS

Evaluation refers to a RESEARCH PURPOSE

rather than to a specific research

method.

Many data collection methods are used,

such as surveys, focus groups, observations,

case management notes, etc.).

WHY EVALUATE?

• Improve Your VTC

• Identify Your Successes

• Plan For Sustainability

IMPROVE YOUR PROGRAM

•Which elements of your VTC are working, which

are not, and why?

• Assess program implementation

• Clearly articulate program design

• Assess program coverage

• What are your admission policies?

• Are you reaching the veterans that most need a VTC?

• Are veterans’ children and family members considered?

• Is there a point in your VTC program at which you’re losing

participants? (Conduct a drop-off analysis)

IDENTIFY SUCCESSES

•What are your …

•OUTCOME OBJECTIVES

• The results you hope your VTC will achieve

• Examples: Sobriety, Employment, Reduced Recidivism

•PROCESS OR IMPLEMENTATION OBJECTIVES

• Describe the accomplishment of VTC activities

• Examples: Numbers assessed, number treated, court

attendance, peer mentor/participant interactions

SUSTAINABILITY

Stakeholder involvement and

evaluation data

are the best assets to sustain your VTC.

SO, WHERE DO I START?Learn how and where to start your evaluation

LOGIC MODEL“A systematic and visual way

to present and share your understanding of the relationships

among the resources you have to operate your program,

the activities you plan to do,

and the changes or results you hope to achieve”

-WK KELLOGG FOUNDATION, 2001

CONNECT ACTIVITIES TO OUTCOMES

“I think you should be more explicit here in Step Two”

LOGIC MODEL

LOGIC MODEL: A UNIVERSAL EXAMPLE

Problem

• Headache

Activities

• Take aspirin

Outputs

• N of aspirin provided

Short/ Intermediate Term Outcomes

• Headache reduced

Long Term Outcomes

• More productive, less grouchy

Feedback

Aspirin helps with a headache

VTC EXAMPLE

Problem

• Substance Use in Response to PTSD

Activities

• Trauma-specific substance abuse treatment

Outputs

• N of Txsessions attended

Short/ Intermediate Term Outcomes

• Substance Use Reduction/ Abstinence

Long Term Outcomes

• Improved Family Relationships, Job Readiness

Feedback

Need to change activities to improve outcomes?

SMART OBJECTIVES

S Specific

M Measureable

A Action Oriented

R Realistic

T Time Specific

HOW DO I MEASURE MY VTC’S OBJECTIVES?

• How do we measure successful reintegration?

• How?

•Who?

•Where?

•When?

HOW DO I CHOOSE?• How to choose:

• An evaluation plan

• An evaluator that works for you

CONNECTING YOUR LOGIC MODEL TO THE EVALUATION FRAMEWORK

Problem

• Program Goals and Objectives

• What do you want to accomplish?

Activities

• Process Evaluation

• What did you do?

Outputs

• Process Evaluation

• Who did you serve?

Outcomes

• Outcome Evaluation

• Did you achieve your objectives?

TYPES OF EVALUATION PLANS

BASIC• Establishes accountability. Focuses on evaluation readiness and program processes—5% to

10% of Budget

• Adherence to Ten Key Components of VTCs

• Prepare your VTC for process evaluation

• Identify measureable objectives (GPRA/BJA)

• Examine program implementation

• Compare originally proposed VTC plan to Program in operation

• Compliance with key components of veterans treatment courts

• Integrates evidence-based standards

• Identify challenges and successes

• Document outputs (Units of service)

• Counts veterans’ children and family members

INTERMEDIATE• Basic evaluation PLUS—10 % to 20% of Budget

• More intensive process evaluation

• Detailed review of program coverage, establish client attributes at intake and conduct drop-off analysis to identify points of client attrition

• Analysis of implementation fidelity

• Performance measurement

• Prepare your program for outcome evaluation

• Identify means of measurement (instruments and methods)

• Data

• Identify data sources

• Database development and implementation

• Performance measurement and analysis

• Short term and intermediate outcomes

INTENSIVE• Intermediate evaluation PLUS—Minimum of 20% of the budget

• Impact (long term effects) of the program

• Measurement and analysis of long term outcomes

• Typically experimental or quasi-experimental designs

• Examine the “value added” by the program

• Cost studies

SELECTING AN EVALUATOR• Start ASAP!

• Internal or external evaluation?

• Cost

• Complexity of the evaluation

• How will the evaluation be perceived?

• Choosing an internal evaluator

• Basic skill set and experience

• Choosing an external evaluator

• Personal and professional orientation

• Ability to communicate with a broad

range of stakeholders

• Cultural sensitivity

• Credentials and past performance

Q & A

CHILDREN AND FAMILY FUTURES

25371 Commercentre Drive, Suite 140

Lake Forest, CA 92630

(714) 505-3525

www.cffutures.org

www.ncsacw.samhsa.gov

CONTACT INFORMATION:

Colleen M. Killian, PhDResearch Associate

Gregory Robinson, PhDSenior Research Associate

Holly Child, PhD ABDResearch Associate