Impetus Our current capacity to evaluate our effectiveness among youth addiction agencies in Ontario is varied and non-systematic

Can’t know if we help or how much

Can’t know if we are getting the right youth to the right program

BUT

Agency level dedication to understanding effectiveness

Ministry will at some point mandate outcome eval

We want to drive the mandate

WHAT WE ARE DOING ‘We’ are a group of Youth Managers

We needed to:

1. Understand what ‘success’ means for youth addiction treatment

2. Measure success without straining therapeutic relationship

3. Measure success in a common way across programs

4. Value our measures at the client-therapist level and for program evaluation

5. Share what we learned with other programs in/out of youth addiction sector

Understanding What ‘Success’ Means Environmental Scan

Varied agency type

Varied client populations

Talked to clients, families, staff

“What does success mean?”

C: “How do you feel about contributing?”

P:“What are the current evaluation practices”

Common Outcomes & IndicatorsSELF WELL BEING

CopingHopeConfidenceInsight

EsteemEmotion ManagementGoalsMotivation

PhysicalMentalSexualSubstance Use

SleepDaily Functioning ExerciseDoctor/Dentist Visits

RELATIONSHIPS COMMUNITY/CULTUREFamilyPeersTherapeuticEmplyers/Tchers

Mentors/EldersLegalLeadership

Connection CmtyConnection CltrVolunteerCultural ActivitiesSpirituality

Resource KnowledgeSchool/WorkComplianceSafety

Finding Tools to Measure Success

Screening

GAIN SS + 5 Questions

Assessment

Gain Q3 MI (ON) + WM-Tool

Outcomes

If only GAIN SS + 5 at pre-treatment, GAIN SS + 5 post

If GAIN Q3 MI (ON) + Other at pre-treatment, GAIN Q3 WM-Tool post

Measuring Success Without Strain

Embed evaluation in therapeutic practice

Ensure process is user-friendly for clients and clinicians

Allow for instant access to results for individual evaluation and planning

Allow for instant access to measures of change over time

Allow for results to roll up to group, program, or agency level instantly

Develop standard dashboard for results

Screening

Assessment

Other Indicators GAIN Q3 MI Work problems

Substance Use

Mental Health

Physical Health

Sources of Stress

Risk Behaviour & Trauma

Crime & Violence

Life Satisfaction

What Would You Like To Talk About?

Where Should We Start?

GAIN – BK – BMI. ………………………………………………………………………17.5GAIN – BK - Are you pregnant? ………………………………………………..…NoGAIN – PH - Last time you: gained 10 or more pounds when you were not trying to? ……………….…Past Month lost 10 or more pounds when you were not trying to? ……………………...Never were worried about your physical health? …………………………….…......Past Month had a lot of physical pain or discomfort? ………………………………….…4-12 Mths Ago had physical health problems kept you from meeting responsibilities? …Past Month saw a doctor /nurse for physical health or took medication for one?.......NeverPast 90 days, how many days: have you exercised for at least 20 minutes per day?............................................5 Days did you have an injury……………………………………………………………..………0 Days you took prescribed medication for a physical health problem?........................0 Days you were bothered by any physical health or medical problems?.....................5 Days have physical health problems kept you from meeting your responsibilities?....10 Days have you smoked or used any kind of tobacco?.................................................90 DaysGAIN – PH - During the past 90 days, how many: times have you had to go to emergency for a physical health problem?.........5 Times nights total did you spend in hospital for a physical health problem?.................5 Nights times did you see a doctor / nurse for a physical health problem?....................1 Time times did you have an outpatient surgery for a physical health problem?.......5 Times

Physical Health

Where Should We Start?

Common + Flexible Basic tools will be provided (GAIN SS, GAIN Q3 MI (ON), WMTool)

Each agency may add tools (e.g., trauma-symptom-specific)

Tablets Collaborative team will be outfit with tablets for each clinician

Wi-fi enabled (not cell)

Will be loaded with data collection software

Capacity for narrative reports for screening & assessment

Working on capacity for ongoing therapeutic notes

Working on tool for ongoing monitoring (e.g., 9 questions, clinician response)

WHERE DOES THE DATA GO? Data stored in CATALYST software

Currently an online system

Working on tablet APP

Owned by MOH housed at CAMH

New release Spring 2015

Scoring will be instant

Dashboards instant

Dashboards:◦ individual◦ time-bound◦ provider-bound (group, program, agency)

Dashboards / Output

EVERYDAY WELL BEING  PRE 6M IN TX 1Y IN TX 3-6M POST 9M-1Y

POSTPHYSICAL HEALTH 3.4 3.5 3.6 3.8 3.8MENTAL HEALTH 2.0 2.2 2.8 3.1 3.2SEXUAL HEALTH 2.5 2.4 2.7 2.8 2.8DAILY FUNCTIONING 1.9 1.8 2.5 2.1 1.9SUBSTANCE USE 2.0 2.1 3.0 3.1 3.2DAILY FUNCTIONING 3.0 3.1 3.8 3.1 3.0

PRE- POST AVERAGES MULTIPLE RESPONDENTS

PRE POST AVERAGE CHANGE MULTIPLE RESPONDENTS

Vetting Four ‘new’ agencies will review the Common Evaluation Plan

Critique, suggest, edit

We will revise as necessary

These agencies may later be offered to be one of the pilot agencies

Implementation Proposed Start July 2015 Each team agency will implement the Common Evaluation Plan

CAMH Implementation team will support ‘front end’ Screening & Assessment

EPYC will train on use for interactive & dashboard outcomes

Multi-level implementation plan (board, exec, management, staff)

Implementation will be collaborative with each agency

Project staff will support on site or phone

Our Support Knowledge Exchange Initiative Grant $19,000

◦ Ontario Centre of Excellence for Child & Youth Health ◦ Started the conversation ‘What Should Define Youth Health’

Addictions Ontario $15,000◦ Workshops in our team agencies with staff◦ What defines youth success, what is current evaluative practice

Ontario Trillium Foundation $327,500◦ Environmental Scan Ontario-wide with clients, parents, staff◦ What defines youth success, current practice, feelings about doing evaluation

Drug Treatment Funding Program $207,000◦ Proposed to support tech & population-specific liaison

Alignment with Provincial Initiatives DTFP (CAMH) – Screening & Assessment, OPOC

DATIS – Involvement with new software platform, possible pilot sites

MCYS – Learning their outcome reporting initiative for YJ

Individual agencies – primarily informative / consultative

Thank you.QUESTIONS?