Keeping it Fresh: 9 Years of ANSA Implementation

Keeping it Fresh: 9 Years of ANSA Implementation

Melissa Villegas

Emily Shapiro

Presentation Overview1. How to adapt the ANSA over time to

reflect the needs and strengths of the clients served.

2. How to conduct a multi-year analysis to inform an internal CQI process.

3. How to keep the ANSA fresh and relevant in the minds of clinicians after they have been using it for longer periods of time.

2

Introduction and Overview of Agency TCOM Implementation

3

Introduction: Jewish Child and Family Services (JCFS)

» Mission: To provide help, healing, and caring services infused with Jewish values to strengthen lives in our community

» 6 Primary Program Areas1. Counseling and Support

2. Support for Communities

3. Education

4. Child Welfare

5. Services for People with Disabilities

6. Employment Services

4

History of JCFS CANS, ANSA, and FAST Implementation

5

DCFS requires use of CANS in System of

Care (SOC) now known as Intensive

Placement Stabilization (IPS)

DCFS expands CANS to Specialized Foster

Care/Residential

JCFS adopts CANS for all Foster Care

contracts

2009 - Outpatient Counseling chooses

ANSA

2010 - Maintaining Adoption

Connections (MAC) chooses FAST

JCFS chooses CANS with FAST for

Outpatient Counseling

MAC shifts to CANS and FAST

ANSA Revised

FAST added to ANSA

JCFS Therapeutic Day School implements TDS CANS

Comprehensive

2003 2005 2009-2010 20132013 2017

*DCFS=Department of Children and Family Services*CANS=Child and Adolescent Needs and Strengths*ANSA=Adult Needs and Strengths Assessment*FAST=Family Advocacy and Support Tool

Current Tools in Use at JCFS

» JCFS CANS Comprehensive• Based on National Child Traumatic Stress Network CANS

• Used in Outpatient Counseling and Maintaining Adoption Connections

• FAST integrated into tool

» ANSA 2.0• Based on the Indiana ANSA

• Used in Outpatient Counseling

• FAST integrated into tool

» DCFS CANS 2.0• Used in Foster Care and Intensive Placement Stabilization

» TDS CANS Comprehensive• Used in Therapeutic Day School 6

Choosing and Revising the ANSA

7

JCFS Outpatient Counseling Program

» Services: Individual, Couples, Family, and Group Counseling

» Top 3 Primary Treatment Approaches:

• Psychodynamic Therapy, Family Systems Therapy, Cognitive Behavioral Therapy

» Serve ~1,100+ clients yearly

8

Program Goal: Increase, strengthen, and promote adaptive functioning and resilience within adults, children and families

All Clients

» Median length of service: 6.4 months

» Majority: female, Caucasian, Jewish, English-speaking, low-income

» Median age at admission: 26 ranging from <1 year to 90+ years old

» Majority are adult clients (58%)

» Top 5 presenting problems: Depression, Anxiety, Adjustment, Prevention/Support, Relationship Problems (Family)

9

JCFS Outpatient Counseling Program Client Characteristics

Adult Clients (19 and Older)

» Median length of service: 6.6 months

» Largest percentages (33-34%) identify as single or married; 20% divorced

» Median age at admission: 46. Largest percentage (31%) 40 to 54followed by 25 to 39 (27%)

All Clients

Adult Clients

Selecting the ANSA in 2009

» Agency merged and moved from a child-serving program to a program serving all ages

» Updated the logic model unifying the program as one

» Embarked on a search for an adult assessment tool

• Using CAFAS, CBCL, YSR for children, wanted an assessment tool for adults

• Sub-committee looked at potential adult assessments for outcomes

» ANSA preferred choice

» Implemented using the Indiana model

10

» Employment

» Developmental Needs

» Trauma • Traumatic Experiences

• Trauma Adjustment

» Substance Use

» Violence• Historical Risk Factors

• Emotional/Behavioral Risks

• Resiliency Factors

» Sexual Aggression

» Criminal Behavior

» Caregiver Strengths and Needs

11

ANSA 1.0

» Strengths

» Life Domain Functioning

» Acculturation

» Behavioral Health Needs

» Risk Behaviors

Core Domains Specialty Individual Domains Caregiver Specialty Domain

How was the ANSA revised in 2013?

» Clinicians volunteered to serve on a short-term assessment committee

» Reviewed the internal ANSA Item Level Analysis Report• Which items to eliminate?

• Which areas were not reflected on the tool?

• What items were not clearly defined by the manual?

• Reviewed changes made by Indiana and their work on Recovery

12

What did we decide to change?

» Clear gaps:

• Did not have any items on parenting

• Did not have any items focused on family needs

» 2013 Modification – Key changes

• Improved definitions

• FAST integrated into assessment (Family Together, Adult, and Child Functioning domains)

• Added Caregiver Role Functioning item with Caregiver Role specialty module

• Changed Acculturation domain to optional specialty module

• Added Suicide Risk as a specialty module

• Moved Emotional/Behavioral Risk items out of the optional specialty Violence module into the Core Risk Behaviors module

13

» Caregiver/Parenting Role

» Employment

» Developmental Needs

» Cultural Identity and Acculturation

» Trauma • Traumatic Experiences

• Traumatic Stress Symptoms

» Substance Use

» Suicide Risk

» Dangerousness

» Sexual Aggression

» Criminal Behavior

» Family Assessment• Family Together

• Adult Functioning

• Child Functioning

» Caregiver Strengths and Needs

14

ANSA 2.0 Today

» Strengths

» Life Domain Functioning

» Behavioral Health Needs

» Risk Behaviors

Core Domains Specialty Individual Domains Family and Caregiver Specialty Domains

Keeping the ANSA Fresh

15

Implementation, Analysis, and Continuous Quality Improvement

ANSA Implementation Today

» In-person reliability training

» Completed at Intake, 6 Month, 1 Year, Annually from Admission, Discharge

» Entered into client information system

» Client level reporting on demand

» Program level reporting

16

ANSA Individual Summary Report Example

• Intake Assessment• 5 Identified Strengths to build on

• 3 Actionable Individual Needs

• 2 Background Trauma Experiences Needs

• 2 Actionable Family Needs

• 6 Usable Strengths

17


• Discharge Assessment• 1 Strength to build on

• 2 Background Trauma Experiences Needs

• No Individual Actionable Needs

• No Actionable Family Needs

• 10 Usable Strengths

18


• Change in Domain Scores Over Time• Reflects Domain Scores for all

Assessments

• Reliable Improvement in:• Life Domain Functioning

• Strengths

• Acculturation

• Family Together

• Career/Employment

19

ANSA for Continuous Quality Improvement

» Twice yearly reporting

» Sample based on the fiscal year

» Calculate yearly Reliable Change Index

» Key Outcome: Reliable improvement in any applicable domain

20

21

LongitudinalANSA Analysis Methodology

9 Year ANSA Analysis –Methodology

» Data cleaning

» Calculate Reliable Change Index

» Determine baseline – highest % actionable

» Refine sample for admission cohort analysis

22

9 Year ANSA Analysis –Data Cleaning

» Started with 9,617 Assessments

» Included only clients admitted 2009 onward (7,797 assessments)

» Included only assessments marked FINAL

» Excluded any data entry errors

• Incorrect program selected

• Assessments dated prior to admission date

» Deleted multiple assessments completed recorded for the same date – kept the most recent based on “Data Entry Date” in the system

23

9 Year ANSA Analysis –Data Cleaning

» Recoded for rating periods

• Intake: Any ANSA completed within the first 90 days

• 6-Month: Any ANSA completed >90 days and less than 9 months

• 1 year: 9 months to 1.49 years

• 2 Years and Onward: [year].50 round up

• Discharge: 31 days from discharge or last date of service

» Deleted multiple assessments based on recoded rating period

» 7,557 ANSAs for 2,837 client episodes

24

9 Year ANSA Analysis –Reliable Change Index (RCI)

» What is it?• The amount of change necessary in the domain score above and beyond

measurement error (reliable change). Jacobson & Truax 1991

» How to calculate:• Calculate a domain score

• Calculate the standard deviation for each domain

• Calculate the average reliability score of all ANSA raters

• Plug your standard deviation and average reliability score into this formula:

25

FormulaRCI=1.28*(standard deviation) x SQRT (1-reliability)

9 Year ANSA Analysis –Domain Score

» How to calculate:

• Add all scores on items in a domain to create a sum. Exclude N/A values.

• Divide by the total items completed for the scale, excluding missing and N/A responses.

• 75% of a domain must be rated to calculate a score.

• Multiply the score by 10 to produce uniform 30 point domain score where:

• 0=‘0’ ratings on all items

• 30=all ‘3’ ratings on every item in the domain.

26

9 Year ANSA Analysis –Reliable Change Index (RCI) –Core Domains

Domain N Mean Minimum MaximumStandard Deviation

RCI

Strengths 2,494 12.27 0.00 30.00 6.05 3.46

Life Domain Functioning 2,483 5.21 0.00 23.33 3.29 1.88

Behavioral Health Needs 2,468 4.75 0.00 21.00 3.05 1.75

Risk Behaviors Pre-2013 2,460 0.54 0.00 13.75 1.29 0.74

Risk Behaviors Post-2013 1,345 1.10 0.00 14.62 1.74 0.99

27

9 Year ANSA Analysis –Reliable Change Index (RCI) –Specialty

Domain N Mean Minimum MaximumStandard Deviation

RCI

Family Together 449 10.89 0.00 30.00 4.87 2.79

Identity and Acculturation 1162 1.46 0.00 26.67 3.32 1.90

Traumatic Stress Symptoms 454 7.61 0.00 27.50 5.07 2.90

Employment 343 9.07 0.00 30.00 7.51 4.30

Caregiver Needs and Strengths 120 3.18 0.00 18.33 4.17 2.39

Caregiving/ Parenting Role 165 9.22 0.00 25.00 5.74 3.28

Substance Use 187 11.49 0.00 30.00 8.39 4.80

Suicide Risk 171 5.00 0.00 14.00 3.20 1.83

28

9 Year ANSA Analysis –Baseline Swap

» Why do we swap baselines?

» Baseline Swap Inclusion Criteria• 3 or more ANSAs completed

• Intake and 6 month ANSAs required

» 38% of episodes fit the criteria (1,092 of 2,837 episodes)

» Total % actionable (# items rated 2 or 3 divided by # rated) increased in 34% of eligible cases (372 of 1,092)

29

9 Year ANSA Analysis –Baseline Swap

After swapping baselines…

1 in 4 clients demonstrated:• 15 increased needs

• 8 decreased strengths

Why the change?• Overestimating client strengths

• Discovery

30

» Final Sample: 1,426 client episodes for clients admitted 2009-2016

» Excluded 2017 and 2018 admission cohorts (<75% have discharged)

» Included only clients with a baseline and discharge ANSA for comparison

31

9 Year ANSA Analysis –Refine Sample for Analysis

AdmissionYear

# Client Episodes

2009 141

2010 185

2011 195

2012 196

2013 215

2014 168

2015 190

2016 136

LongitudinalANSA AnalysisKey Findings

32

ANSA Analysis –Reliable Improvement in Any Domain

68%65% 65%

70%

61%60%

52%

57%

2009 2010 2011 2012 2013 2014 2015 2016

33

ANSA Analysis –Reliable Improvement in Any Domain

68%65% 65%

70%

61% 60%

52%

57%

45%

49%53%

49% 49%52%

36%

46%

2009 2010 2011 2012 2013 2014 2015 2016

34

Improve in Any Domain

Improve without Decline

ANSA Analysis –Reliable Improvement vs. Decrease % Actionable

68%65% 65%

70%

61%

60%

52%

57%61% 60%

64%

68%67%

62%

57%

63%

2009 2010 2011 2012 2013 2014 2015 2016

35

Improve in Any Domain

Decrease Total Percentage Actionable

ANSA Analysis –Admission Cohort vs. Fiscal Year

68%65% 65%

70%

61%60%

52%

57%

2009 2010 2011 2012 2013 2014 2015 2016

36

Fiscal Year

ANSA Analysis –Reliable Improvement by Core Domain

Strengths

Life Domain Functioning

Behavioral Health

15%

20%

25%

30%

35%

40%

45%

2009 2010 2011 2012 2013 2014 2015 2016

37

ANSA Analysis –Reliable Improvement –Risk Behaviors

0%

5%

10%

15%

20%

25%

2009 2010 2011 2012 2013 2014 2015 2016

38

Pre-2013 Items

Post-2013 Items

*In 2013 Frustration Management, Hostility, Paranoid Thinking, Secondary Gains from Anger, and Violent Thinking moved from the Violence Emotional/Behavioral Risks Specialty Domain to Core Risk Behaviors Domain

ANSA Analysis –Reliable Improvement –Family and Parenting

0%

5%

10%

15%

20%

25%

30%

35%

2013 2014 2015 2016

39

Family Together

Caregiver/Parenting Role

ANSA Analysis – Significant Domain Score Change

0.9

0.4

4.3

4.8

10.8

1.5

0.5

4.9

5.4

12.7

Acculturation

Risk Behaviors

Mental Health


Strengths

40

BaselineDischarge

p<.05

Increased Strengths

» Family

» Social Connectedness

» Optimism

» Educational

» Job History

» Talents/Interests

» Spiritual/Religious

» Community

» Natural Supports

» Resiliency

» Resourcefulness

» Volunteering41

ANSA Analysis – Significant Item Level Changes

Decreased Needs


» Family

» Caregiver Functioning

» Employment

» Social Functioning

» Recreational

» Residential Stability

» Legal

» Sleep

» Decision-making

» Cultural Identity

» Depression

p<.05

Decreased Needs

Behavioral Health

» Anxiety

» Interpersonal

» Adjustment to Trauma

» Anger Control

» Eating Disturbance

Risk Behaviors

» Suicide Risk

Specialty Domains

» Job Time

» Cultural Rituals

» Cultural Stress

» Affect Regulation 42

ANSA Analysis – Significant Item Level Changes

Increased Needs


» Involvement in Treatment

Specialty Domains

» Ability to Listen

» Suicide Intent

p<.05

ANSA Analysis – Improvement in Top Diagnostic Categories

57%

61%

65%

62%

22%

21%

14%

23%

21%

18%

21%

15%

Other Conditions – Relational

Anxiety Disorders

Depressive Disorders

Trauma- and Stressor-Related

43

Improvement 1+ Domain No Change Decline

(n=393)

(n=344)

(n=207)

(n=157)

ANSA Analysis – Significant Differences –Length of Stay

76%

70%

71%

70%

66%

51%

13%

10%

15%

15%

19%

25%

12%

20%

14%

15%

15%

24%

4+ Years

3 Year

2 Year

1 Year

6 Months to 1 Year

<6 Months

44


p<.05

ANSA Analysis – Significant Differences –Counseling Modalities

67%

69%

69%

60%

38%

10%

14%

21%

44%

33%

20%

16%

19%

19%

4

3

2

1

0

45


p<.05*Modalities include: Individual, Couples, Family, and Group*0 Modalities=Case Management/Assessment Only

ANSA Analysis – Significant Differences –Couples

41%

62%

41%

18%

17%

21%

Couples Only

Couples + Other

46


(n=141)

(n=29)

p<.05

ANSA Analysis – Significant Differences –Family

33%

70%

33%

12%

33%

18%

Family Only

Family + Other

47


(n=264)

(n=24)

p<.05

ANSA Analysis – Improvement based on Failed Session Percentage

54%

64%

68%

64%

20%

19%

17%

20%

26%

17%

15%

16%

>32%

18% to 32%

8% to 18%

<8%

48


(n=358)

(n=356)

(n=348)

(n=356)

p<.05

ANSA Analysis –Other Significant Findings

Clients who improved:

» Higher total % actionable at baseline

• Life Domain Functioning, Behavioral Health, Risk Behaviors (pre-2013 scale), Acculturation, Trauma Experiences

» Lower % usable strengths at baseline

» Discharged due to “Planned Service Completion”

49p<.05

ANSA Analysis –What happened in 2015 and 2016?

68%65% 65%

70%

61%60%

52%

57%

2009 2010 2011 2012 2013 2014 2015 2016

50

Actionable NeedsUsable Strengths Length of Stay Sessions

Key QuestionsWhat have we learned?

Potential Action Areas for Counseling CQI

» What clinical groups can we offer to supplement other forms of counseling?

» How do we encourage family/partner involvement in treatment?

» How can we increase client engagement in services?

» What does treatment planning look like for clients with fewer needs and more strengths at intake?

» How can we adapt existing strengths to address needs?

» Is it time to revise the ANSA again?

52

Keeping it Fresh

Final Thoughts/Suggestions

1. Convene/engage a subcommittee of program staff to revise the assessment to reflect the needs and strengths of the client population

2. Provide real-time reports for staff to monitor improvement and share with clients

3. Provide booster trainings and additional opportunities for staff to gather and share ideas of how to use these assessments in practice

4. Conduct regular analysis to share significant findings with staff to inform treatment on a program level

53

Melissa Villegas

[email protected]

www.jcfs.org

Thank you!

Questions?

Emily Shapiro

[email protected]

Documents

Keeping it Fresh: 9 Years of ANSA Implementation