Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
All Courts are Family Courts -
when participants
have children and other family
members.
3
Why Focus on Children? The Numbers
• 50% of OIF/OEF veterans are
parents
• Over a million children have
parents who served since
9/11
• 30-35% of deployed parents
have trauma and substance
abuse issues
You can’t coordinate what you can’t count
• The task is not diverting funds from VTCs; it is accessing
funds already available for children's services
• But to mobilize and link to new resources from other
agencies that already serve children
VTCs Don’t Have Do It Alone
• Maternal and child
health,
• Mental health
• Child development,
• Youth services
• Special education
• Delinquency prevention
That’s what collaborative means
• How many children of veterans are in your VTC?
• What services and supports do these children need?
• How can your VTC build lasting effective networks or
partnerships to respond to the needs of these
children?
3 Ns – Numbers, Needs, Networks
LEARNING OBJECTIVES
•Find out how to make evaluation work for you to improve your program, identify your successes and plan for sustainability
•Learn how and where to start your evaluation
•Understand how to choose an evaluation plan and an evaluator that works for you
WHAT IS EVALUATION?“Program evaluation is the
systematic collection of information about the activities, characteristics, and outcomes of
programs to make judgments about the program, improve
program effectiveness, and/or to inform decisions about future
programming” -PATTON, 1997
THE EVALUATION PROCESS
Evaluation refers to a RESEARCH PURPOSE
rather than to a specific research
method.
Many data collection methods are used,
such as surveys, focus groups, observations,
case management notes, etc.).
WHY EVALUATE?
• Improve Your VTC
• Identify Your Successes
• Plan For Sustainability
IMPROVE YOUR PROGRAM
•Which elements of your VTC are working, which
are not, and why?
• Assess program implementation
• Clearly articulate program design
• Assess program coverage
• What are your admission policies?
• Are you reaching the veterans that most need a VTC?
• Are veterans’ children and family members considered?
• Is there a point in your VTC program at which you’re losing
participants? (Conduct a drop-off analysis)
IDENTIFY SUCCESSES
•What are your …
•OUTCOME OBJECTIVES
• The results you hope your VTC will achieve
• Examples: Sobriety, Employment, Reduced Recidivism
•PROCESS OR IMPLEMENTATION OBJECTIVES
• Describe the accomplishment of VTC activities
• Examples: Numbers assessed, number treated, court
attendance, peer mentor/participant interactions
SUSTAINABILITY
Stakeholder involvement and
evaluation data
are the best assets to sustain your VTC.
SO, WHERE DO I START?Learn how and where to start your evaluation
LOGIC MODEL“A systematic and visual way
to present and share your understanding of the relationships
among the resources you have to operate your program,
the activities you plan to do,
and the changes or results you hope to achieve”
-WK KELLOGG FOUNDATION, 2001
CONNECT ACTIVITIES TO OUTCOMES
“I think you should be more explicit here in Step Two”
LOGIC MODEL
LOGIC MODEL: A UNIVERSAL EXAMPLE
Problem
• Headache
Activities
• Take aspirin
Outputs
• N of aspirin provided
Short/ Intermediate Term Outcomes
• Headache reduced
Long Term Outcomes
• More productive, less grouchy
Feedback
Aspirin helps with a headache
VTC EXAMPLE
Problem
• Substance Use in Response to PTSD
Activities
• Trauma-specific substance abuse treatment
Outputs
• N of Txsessions attended
Short/ Intermediate Term Outcomes
• Substance Use Reduction/ Abstinence
Long Term Outcomes
• Improved Family Relationships, Job Readiness
Feedback
Need to change activities to improve outcomes?
SMART OBJECTIVES
S Specific
M Measureable
A Action Oriented
R Realistic
T Time Specific
HOW DO I MEASURE MY VTC’S OBJECTIVES?
• How do we measure successful reintegration?
• How?
•Who?
•Where?
•When?
HOW DO I CHOOSE?• How to choose:
• An evaluation plan
• An evaluator that works for you
CONNECTING YOUR LOGIC MODEL TO THE EVALUATION FRAMEWORK
Problem
• Program Goals and Objectives
• What do you want to accomplish?
Activities
• Process Evaluation
• What did you do?
Outputs
• Process Evaluation
• Who did you serve?
Outcomes
• Outcome Evaluation
• Did you achieve your objectives?
TYPES OF EVALUATION PLANS
BASIC• Establishes accountability. Focuses on evaluation readiness and program processes—5% to
10% of Budget
• Adherence to Ten Key Components of VTCs
• Prepare your VTC for process evaluation
• Identify measureable objectives (GPRA/BJA)
• Examine program implementation
• Compare originally proposed VTC plan to Program in operation
• Compliance with key components of veterans treatment courts
• Integrates evidence-based standards
• Identify challenges and successes
• Document outputs (Units of service)
• Counts veterans’ children and family members
INTERMEDIATE• Basic evaluation PLUS—10 % to 20% of Budget
• More intensive process evaluation
• Detailed review of program coverage, establish client attributes at intake and conduct drop-off analysis to identify points of client attrition
• Analysis of implementation fidelity
• Performance measurement
• Prepare your program for outcome evaluation
• Identify means of measurement (instruments and methods)
• Data
• Identify data sources
• Database development and implementation
• Performance measurement and analysis
• Short term and intermediate outcomes
INTENSIVE• Intermediate evaluation PLUS—Minimum of 20% of the budget
• Impact (long term effects) of the program
• Measurement and analysis of long term outcomes
• Typically experimental or quasi-experimental designs
• Examine the “value added” by the program
• Cost studies
SELECTING AN EVALUATOR• Start ASAP!
• Internal or external evaluation?
• Cost
• Complexity of the evaluation
• How will the evaluation be perceived?
• Choosing an internal evaluator
• Basic skill set and experience
• Choosing an external evaluator
• Personal and professional orientation
• Ability to communicate with a broad
range of stakeholders
• Cultural sensitivity
• Credentials and past performance
Q & A
CHILDREN AND FAMILY FUTURES
25371 Commercentre Drive, Suite 140
Lake Forest, CA 92630
(714) 505-3525
www.cffutures.org
www.ncsacw.samhsa.gov
CONTACT INFORMATION:
Colleen M. Killian, PhDResearch Associate
Gregory Robinson, PhDSenior Research Associate
Holly Child, PhD ABDResearch Associate