1

New Path Youth & Family ServicesNew Path Youth & Family Services

Program Evaluation Program Evaluation Activities and Counselling for Teens (ACT) ProgramActivities and Counselling for Teens (ACT) Program

September 1, 2003 September 1, 2003 –– March 31, 2006March 31, 2006

April 2006

Prepared by:

Arleigh Luckett, M.Ed App. Psych.

Larry Shirk, B.Sc CIS

ShirLuck Consultants

2

ACTACT Activities and Counselling for TeensActivities and Counselling for Teens

Activity basedFor youth age 9 –1512 session group programParent program (currently Triple P)Individual and family therapy

3

DemographicsDemographics --||--

Age at AdmissionAge at Admission

0

5

10

15

20

(ACT n=83)

Perc

ent o

f Clie

nts

ACT 0 0 3.6 12 8.4 19.3 13.3 21.7 12 7 1.2 0 0

6yrs 7yrs 8yrs 9yrs 10yrs 11yrs 12yrs 13yrs 14yrs 15yrs 16yrs 17yrs 18yrs

4

DemographicsDemographics --||--

Age GroupsAge Groups

56.6

43.4

0

10

20

30

40

50

60

Preadolescent (<= 12yrs) Adolescent (13-18yrs)

Perc

ent o

f Clie

nts

5

DemographicsDemographics --||--

Gender Gender

66%

34%

Boys Girls

6

Brief Child & Family Phone Brief Child & Family Phone InterviewInterview

Intake Screening

7

BCFPI Question/CalculationBCFPI Question/Calculation

Code Question/Calculation

RAIAp Regulation of Attention, Impulsivity and Activity, population normCOp Cooperativeness - population normsCDp Conduct - population norms.EXp Externalizing - population norms.SPp Separation from parents - population normsMAp Managing Anxiety - population normsMMp Managing mood- 6 item population normsSHp 6 mood + 3 self harm indicators, population normsINp Internalizing, population norms.TMHP Total. 6 mental health domains (5 for Teacher form), Population norms.SocPartP Social participation - population normsQRelP Quality of relationships - Population normsSchoolP School participation and achievement - population normsChFp Global functioning... pop normsFActP Family activities... pop normsFcFp Family comfort- population normsGFsP Global family situation - Population normsPMMp Informant - depression -population normsFADp FAD -population norms

8

010203040506070

Perc

ent o

f Clie

nts

IY 31 44 34 44 19 24 44 51 36 51 60 31 35 58 44 60 61 25 8.9

RAIAp

COp

CDp

EXp

SPp

MAp

MMp

SHp INp TM

HP

SocPart

QRelP

SchoolP

ChFp

FActP

FcFp

GFsP

PMMp

FADp

BCFPI Problem PrevalenceBCFPI Problem Prevalence Percentage of Clients Scoring Above CutoffPercentage of Clients Scoring Above Cutoff

n = 101n = 101

9

BCFPI BCFPI Average ScoresAverage Scores

n = 102n = 102

0.0

20.0

40.0

60.0

80.0

100.0

COp Avg Scr CDp Avg Scr EXp Avg ScrSPp Avg Scr MAp Avg Scr MMp Avg ScrSHp Avg Scr INp Avg Scr TMHP Avg ScrSocPartP Avg Scr QRelP Avg Scr SchoolP Avg ScrChFp Avg Scr FActP Avg Scr FcFp Avg ScrGFsP Avg Scr PMMp Avg Scr FADp Avg Scr

10

CAFAS AssessmentCAFAS Assessment

Completed by staff after second session

11

What are the Types and Extent of What are the Types and Extent of YouthsYouths’’ Impairments at Entry CAFAS?Impairments at Entry CAFAS?

:

n sd (standard deviation)

Youths’ Status at Entry CAFAS: 83 26.84

• Mean: average score

• Median: the middle score

• Mode: most frequently occurring score

12

Rating Total at Entry to TreatmentRating Total at Entry to Treatment

49.64

40 40

0

10

20

30

40

50

60

Mean Median Mode

Scor

e on

CA

FAS

13

CAFAS Total ScoreCAFAS Total Score

• 0-30 likely referred to qualified health professional• 40-70 likely receives outpatient services• 80-100 likely receives outpatient care with additional

services of a supportive or intensive nature• 110 -130 likely receives intensive, community-based

services, although some youths may need acute residential services at some point

• > 140 very intensive services would be required; may be in residential or inpatient settings at some point

14

Severity intervals at Entry to Treatment Severity intervals at Entry to Treatment (ACT n=83)(ACT n=83)

30.1

54.2

12

2.4 1.2 00

10

20

30

40

50

60

0-30 40-70 80-100 110-130 >140 not scored

% o

f clie

ntss

15

CAFAS SubscalesCAFAS Subscales

• School• Home• Community• Behavior Towards Others• Moods / Emotions• Self Harm• Substance Abuse• Thinking

16

Percentage of Youths With Severe or Moderate Impairment on Percentage of Youths With Severe or Moderate Impairment on CAFAS Subscales at Entry to TreatmentCAFAS Subscales at Entry to Treatment

(ACT n = 83)(ACT n = 83)

0

5

10

15

20

25

30

35

40

Per

cent

of C

lient

s

ACT 34.9 16.9 12 25.3 33.7 3.6 2.4 2.4

school home communitybehavior towards others

moods/ emotions self-harm

substance use thinking

17

Percentage of Youths Who Were Rated as Severely or Moderately Percentage of Youths Who Were Rated as Severely or Moderately Impaired on None to 8 of the 8 CAFAS Subscales at Entry to Impaired on None to 8 of the 8 CAFAS Subscales at Entry to

TreatmentTreatment with at least some subscale data, not necessarily on all subscalwith at least some subscale data, not necessarily on all subscaleses

0

5

10

15

20

25

30

35

40

Perc

ent o

f Clie

nts

ACT 36.1 31.3 12 12 3.6 3.6 1.2 0 0

None 1 2 3 4 5 6 7 8

18

Percentage of Youths Who Were Rated as Severely Impaired Percentage of Youths Who Were Rated as Severely Impaired on None to 8 of the 8 CAFAS Subscales at Entry to Treatmenton None to 8 of the 8 CAFAS Subscales at Entry to Treatment

with at least some subscale data, not necessarily on all subscawith at least some subscale data, not necessarily on all subscalesles

0

10

20

30

40

50

60

70

80

% o

f clie

nts

ACT 75 4 3 1 0 0 0 0 0

None 1 2 3 4 5 6 7 8

19

Percentage of Youths With Severe Impairment on CAFAS Percentage of Youths With Severe Impairment on CAFAS Subscales at Entry to TreatmentSubscales at Entry to Treatment

0

1

2

3

4

5

6

7

Perc

ent o

f Clie

nts

ACT 6 6 0 1.2 2.4 3.6 2.4 0

school home communitybehavior towards others

moods/ emotions self-harm substance

use thinking

20

What are the Outcomes for Youth?What are the Outcomes for Youth?

• Outcomes from Entry to Last CAFAS– Includes open and closed cases, and those with an Exit CAFAS

(T14)– n= # of cases with two or more CAFAS evaluations, excluding

CAFAS where it was indicated that no treatment was delivered; • Three Ways of Viewing Outcome

– Change in average scores– Proportion of youth improved 20 points or more_ Proportion of youth improved from above cutoff to below cutoff

21

Outcome: Change in Average CAFAS from Outcome: Change in Average CAFAS from Entry to Treatment to Exit CAFASEntry to Treatment to Exit CAFAS

40

10

49.74

20.39

40

20

0

10

20

30

40

50

60

Entry Exit

Scor

e on

CA

FAS

Mean Median Mode

22

Outcome: Change in Average CAFAS Score from Outcome: Change in Average CAFAS Score from Entry to Treatment to Exit CAFAS Entry to Treatment to Exit CAFAS -- Other StatisticsOther Statistics

Statistics for cases having both Entry and ExitStatistical significance .00

n 76Correlation .592

Standard Deviation 23.056

t Score 11.095

23

Outcome: Change in Average Score on CAFAS SubscalesOutcome: Change in Average Score on CAFAS Subscales from Entry to Treatment to Exit CAFASfrom Entry to Treatment to Exit CAFAS

(cases having both Entry to Treatment and Exit CAFAS completed)(cases having both Entry to Treatment and Exit CAFAS completed)

0

5

10

15

Ave

rage

Sco

re o

n C

AFA

S

ACT Entry 10.39 11.18 2.63 10.92 11.45 1.05 0.66 1.45Exit 5 4.47 1.05 5.53 3.42 0.26 0.39 0.26

school home community behavior towards

moods/ emotions self-harm substance

use thinking

24

Degree of Absolute Change in CAFASDegree of Absolute Change in CAFAS (positive value means improvement) (positive value means improvement)

0

5

10

15

20

25

Increase or Decrease in Score by Percent of Clients

Perc

ent o

f Clie

nts

ACT 0 0 0 0 0 0 0 1.3 13 13 16 24 17 3.9 2.6 3.9 2.6 1.3 0 1.3 0 0

-80 -70 -60 -50 -40 -30 -20 -10 0 10 20 30 40 50 60 70 80 90 100 110 120 130

Worse Improved

25

Youths Improved on Various Outcome MethodsYouths Improved on Various Outcome Methods

Outcome method Improved/PossibleImproved on at Least One of Three Outcome Indicators 56/72

Clinically meaningful reduction in total CAFAS Score (Restricted to youths with Entry to Treatment CAFAS >= 20)

55/72

No Severe impairments at exit (Restricted to youth with one or more severe impairments at Entry to Treatment)

4/8

Improving from SED Region no SED (Restricted to youths with Entry to Treatment CAFAS <=40)

30/37

26

Results on Outcome Measures Percent of Clients Not Improved, Improved or Excluded on Each Outcome

Measure

21.1

73.7

5.3

23.6

76.4

05.3 5.3

89.5

9.2

39.5

51.3

0102030405060708090

100

Not Improved Improved % CasesExcluded

Perc

ent o

f Clie

nts

Improved on One of Three IndicatorsClinically Meaningful ReductionNo Severe Impairments at ExitImproving from SED Region to no SED

27

Caregiver InformationCaregiver Information

NFamily Social Support subscale Entry Exit

Primary Family 74 76Non-Custodial Caregiver 13 7Surrogate Caregiver 0 0

Family Material Needs subscalePrimary Family 74 76Non-Custodial Caregiver 13 7Surrogate Caregiver 0 0

28

61.8

28.9

17.16.6

2.62.6 0

80.3

0%

20%

40%

60%

80%

100%

Entry ExitScore on CAFAS

Perc

ent o

f Car

egiv

ers

0 10 20 Missing

Family/Social Support SubscaleFamily/Social Support Subscale Primary Family Score at Entry and ExitPrimary Family Score at Entry and Exit

29

67.1

23.7

6.62.6

0

10

20

30

40

50

60

70

80

No Change Improved Worse Missing Data

Perc

ent o

f Clie

nts

Family/Social Support SubscaleFamily/Social Support Subscale Change in Care Giving Environment (Primary Family)Change in Care Giving Environment (Primary Family)

30

73.7

23.7

0

10

20

30

40

50

60

70

80

No Reliable Change Reduction by min 10 pts

Perc

ent o

f Clie

nts

Family/Social Support SubscaleFamily/Social Support Subscale CaregivingCaregiving Environment (Primary Family)Environment (Primary Family)

Reliable Change (decrease by at least 10 points)Reliable Change (decrease by at least 10 points)

31

Outcome Status for Closed CasesOutcome Status for Closed Cases

Outcome Status n

Treatment Not Needed: Only one CAFAS evaluation was done and there was a good reason for not providing treatment (i.e., evaluation only, no treatment needed, or referred to other service). This information is provided when case is closed

0

Not Improved: Did not improve on any of 3 outcome indicators 16

Improved: Improved on 1 or more of 3 outcome indicators 56

Entry Was 0 OR 10: Outcome can not be evaluated because the entry CAFAS was a 0 or 10. Since the least “ambitious” outcome indicator requires a reduction of 20 points or more, it is not possible to evaluate outcome if the entry score is less than 20

4

Likely Drop-Outs. Needed Treatment: There was an entry CAFAS but no subsequent CAFAS evaluations, and the case appeared to be appropriate for treatment (i.e., case was not coded as “evaluation only,” “no services needed,” or “referred to other services” when case was closed)

4

32

Three Outcome IndicatorsThree Outcome Indicators

Outcome Indicators nReliable Improvement(20 Pts) on CAFAS – Entry to Exit

55

No Severe Impairments at Exit 4Improving from SED Region non SED 30

33

BCFPI BCFPI Pre and Post ResultsPre and Post Results

September 1, 2005 September 1, 2005 –– March 31, 2006March 31, 2006

34

BCFPI BCFPI Average Pre and Post ScoresAverage Pre and Post Scores

Before n = 17 / After n = 15 FormsBefore n = 17 / After n = 15 Forms

0102030405060708090

100110

Stage: Before 68.2 70.8 72.7 73.6 61.2 63.9 74.7 80.9 69.7 74.7 76.2 69.2 75.3 78.4 69.5 72.6 75.4 61.6 57.8Stage: After 66.7 64.9 60.3 67.9 56.2 57.7 60.7 60 66 69.5 59.3 67.9 68.9 70.5 67.1 71.1 59.3 54.2

RAIAp COp CDp EXp SPp Map MM

p SHp INp TMHP

SocPart

P

QRelP

SchoolP

ChFp

FActP

FcFp

GFsP

PMMp

FADp

35

ClientsClients’’ and Staffand Staff’’s s Perception of ChangePerception of Change

Client Information SystemClient Information System

36

Change Reported by Clients Change Reported by Clients At ClosureAt Closure

n = 85n = 85

4%

53%

43%

No Change

SomeImprovementConsiderableImprovement

37

Change Reported by StaffChange Reported by Staff At ClosureAt Closure

n = 99n = 99

3%

56%

41%No Change

Some Improvement

ConsiderableImprovement

38

RecommendationsRecommendations

• Post CAFAS is done at the end of the program. Delay post BCFPI to measure change 3 to 6 months after exit. This could be done by phone, mail or through a follow-up session.

• The parenting aspect of the program has evolved over the period of this study. Caregiver data could be further analyzed to look for effect of changes in the parenting component over time.

• Recognize counselling and support staff for the ACT program for exellent compliance with data entry protocols.

39

AcknowledgementsAcknowledgements

Analysis and presentation of CAFAS data was modeled on:“Child and Adolescent Functional Assessment Scale Service Outcome for Children Receiving Treatment - December, 2005 “, Melanie Barwick, Cristina Vlad & Liling Xiang, CAFAS in Ontario.

Funding for this project was provided by:The Provincial Centre of Excellence for Child and Youth Mental Health at CHEO (PEG 162606-054)