The Wraparound Team Monitoring System (Wrap-TMS): Feasibility

The Wraparound Team Monitoring System (Wrap-TMS): Feasibility, Utility, and Initial Psychometrics of a Wraparound-specific Health Information Technology System

Eric J. Bruns, Ph.D., April Sather, MPH, and Hattie Quick, MSW University of Washington School of Medicine, Seattle, WA

Kelly L. Hyde, Ph.D. Social TecKnowledgy, Inc. 26th Annual Research and Policy Conference on Children’s Mental Health Tampa, FL March 4, 2013

1

Funding provided by:

National Institute for Mental Health (MH095516)

Maryland Department of Social and Health Services / Institute for Innovation and Implementation at the University of Maryland School of Social Work

Center for Medicaid and Medicare Service

Thanks to Vista del Mar, Three Rivers Wraparound, Sacramento Children’s Home, Kentucky Impact, Families and Children First of Clermont County, and Maryland Choices for participating in Pilot testing

Overview of presentation

• Present a rationale for and background on the Wraparound Team Monitoring System

• Provide a quick overview of the system

• Present results of an initial pilot of feasibility and psychometrics of Wrap-TMS scales

Why did we develop Wrap-TMS?

• Research shows that a large portion of variance in treatment outcomes are accounted for by:

• clarity around goals

• monitoring and feedback of data on progress, satisfaction, and alliance

• Outcomes of collaborative care are more positive when decisions are informed by reliable & relevant information

Why Wrap-TMS?

• States, initiatives, and providers have often struggled to find ways to manage data and information in a way that is in line with wraparound principles

• Fidelity and process data collected externally doesn’t help individual teams keep on track

• Outcomes / costs may not be monitored at all

• We need ways to facilitate achievement of wraparound principles such as outcomes-based and collaborative

Why did we develop Wrap-TMS?

• Prominent wraparound initiatives have developed excellent MIS systems that are critical to their success

• However, these are locally specific and don’t always translate easily to new sites/states

• NWI aims to provide wraparound sites with a relevant, field-tested, model adherent IT system AND needed supports: • Customization, training, help-desk, ongoing research and reliability testing,

interoperability, Electronic Health Record

Wrap-TMS provides a way to manage

and access key information on the

wraparound process • Individuals engaged in the process

– Youth and family members, team members, providers, natural and community supports, coordination of care

• Key documentation

– Plans of care, strengths, needs, family stories, family history timeline, meeting and appt. times, meeting notes, contact histories, critical incidents, services and costs

• Service processes

– Family Satisfaction, Fidelity, Progress toward needs

• Outcomes

– youth and family support, residential status, educational environment and behavior, youth functioning

Wrap-TMS is intended to facilitate more

effective implementation of wraparound…

at a family, team, and initiative level

• At a youth/family and team level:

– Aid communication between team members

– Allow transparency in information sharing

– Assist in documentation compliance

– Help keep teams on track:

• What are the priority needs? Are we making progress?

• What is the family’s vision for the future? Are we getting there?

• Is the process working for the family and team?

Wrap-TMS is intended to facilitate more

effective implementation of wraparound…

at a family, team, and initiative level

• At higher levels:

– Supports data-driven supervision

– Facilitates evaluation

• Of staff performance

• Of initiative process and outcomes

– Allows managers and administrators (at program, initiative, county, state levels) to monitor implementation, outcomes, service use, and costs in “real time”

Wrap-TMS Data elements are based on input

about what will be most critical to decision

making in team-based integrated care

• Youth, family, and team information

• Plans of care and crisis plans

• Progress notes and incident reports

• Core outcomes such as:

– Family Support and Connectedness

– Progress Towards Needs and Family Vision

– Core Community Outcomes (residential placement, school achievement and suspensions, involvement with JJ, emotional and behavioral functioning)

– Family and Team Perspectives (Fidelity/Satisfaction)

Home Page – Dashboard Staff

Care Coordinator/Facilitator

Specific Dashboards

Customizable Dashboard Views

Home Page –Dashboard

Supervisor

Supervisor Specific Dashboards

Customizable User

Workflow

Customizable Dashboard Options

Add New Case Record

Draft: Replication of slides not permitted without consent from Social TecKnowledgy

12

User identified Required Data Elements

Waiver Funding Data

Add New Demographic Data

Fields

Adding a New Demographic Data Element

Draft: Replication of slides not permitted without consent from Social TecKnowledgy

13

Add/Edit Core Data Assessments

Draft: Replication of slides not permitted without consent from Social TecKnowledgy

14

Core Assessment History

Review/Update all Core Assessments

Add/Edit Customized and or

Required Assessments

Draft: Replication of slides not permitted without consent from Social TecKnowledgy

15

Users can build their own assessments

Other Behavioral Health Assessments available for use

Youth Dashboard with Work Flow

User Specific Dashboards

Customizable User Workflow

Youth Space

Replication of slides not permitted without consent from Social TecKnowledgy

17

Plan of Care:

Available to auto-sign and email

Replication of slides not permitted without consent from Social TecKnowledgy

Embedded Progress and Contact

Notes with Exportable History

Replication of slides not permitted without consent from Social TecKnowledgy

19

Incident Report Form

Replication of slides not permitted without consent from Social TecKnowledgy

20

Multi-Filter Option for Export of Data by

Graphical Reports and or Spreadsheets

Draft: Replication of slides not permitted without consent from Social TecKnowledgy

21

Choice to de-identify all data for export

User Roles-Core set with flexibility

to add roles as needed

Draft: Replication of slides not permitted without consent from Social TecKnowledgy

22

Permissions Set by User Role

Draft: Replication of slides not permitted without consent from Social TecKnowledgy

23

Results of Initial Pilot

Aims of the initial pilot test

• Program Wrap-TMS items into a web-based user input prototype;

• Conduct usability testing with a diverse group of users nationwide:

• Assess psychometrics of Wrap-TMS items and scales:

• Goal setting and progress monitoring

• Fidelity and satisfaction

• Outcomes (school, residential, and community functioning)

• Youth and family social support

• Pilot test and evaluate the utility of the “team dashboard” reporting function of Wrap-TMS with real-world programs serving youth and families participating in actual WSM teams;

Assessments Completed

Assessments

N Youth

Entered

Community

Outcomes

Family

Satisfact

ion

Family

Support

School

Outcomes

Youth

Needs

Youth

Support

Custom

or Other

Provider 1 5 0 0 2 0 3 1 0

Provider 2 5 0 0 3 1 3 3 2

Provider 3 5 1 4 3 2 3 0 7

Provider 4 18 6 2 9 7 10 10 3

Provider 5 5 0 0 2 4 3 2 0

Provider 6 10 11 9 1 3 10 2 0

Total 48 18 15 20 17 32 18 12

18 users entered 48 youth and 132 assessments

Pilot testers

• 12 Facilitators and 6 supervisors

• Mean years experience with wraparound = 4.61 (SD = 1.24)

• 88% had over 1 year experience

• 45% over 3 years.

Results of usability testing

0% 20% 40% 60% 80% 100%

Youth/family/team data entry easy

Enter data on progress easy

Enter information about plan of care easy

Enter data about process/satisfaction easy

Overall high usability

Would be a benefit to our wrap program

Percent strongly agree

Community Outcomes Assessment Based on Strengths and Difficulties Questionnaire (SDQ)

• 18 assessments completed

• Cronbach’s Alpha = .771

Two sections: 1) Rate the degree to which

difficulties interfere with life a) Scales range from 0 to 3 b) “Overall Functioning”

score calculated 2) Enter number and type of juvenile justice contacts

Community Outcomes Assessment

0

2

4

6

8

10

12

14

16

18

Classroom Learning Friendships Home Life Leisure/Fun

Do the youth's difficulties interfere with...

0 - Not at all

1 - A Little

2 - A Medium amount

3 - A Great deal

Mean = 2.07 Mean = 2.21 Mean = 2.39 Mean = 2.06

Community Outcomes Assessment

0

2

4

6

8

10

12

14

16

18

0 - No 1 - Yes, minor difficulties 2 - Yes, definite difficulties 3 - Yes, severe difficulties

Overall Functioning

Family Satisfaction Assessment

• Scales range from 0 “Very Dissatisfied” to 3 “Very Satisfied”

Items Mean (SD)

Service Satisfaction 2.00 (.62)

Progress Satisfaction 1.62 (.96)

Family Support Assessment

• Rate the degree to which family receives support. Scale ranges from 0 to 3

• 19 assessments completed

• Cronbach’s Alpha = .529.

Scale: 0 = Significant need for support, 1 = Need for support 2 = Strength, 3 = Significant strength

Family Support Assessment

Items Mean (SD) Alpha if Item Deleted

Community 1.12 (.49) .552

Family 1.26 (1.00) .467

Friends 1.12 (.60) .546

School/Work 1.42 (1.07) .265

Service 1.68 (.82) .425

Scale: 0 = Significant need for support, 1 = Need for support 2 = Strength, 3 = Significant strength

Family Support Assessment

0

2

4

6

8

10

12

14

16

18

Community Family Friends School/Work Service

0 "Significant Need for for Support"

1 "Need for Support"

2 "Strength"

3 "Significant Strength"

Youth Support Assessment

• Rate degree to which youth receives support. Scale ranges from 0 to 3

• 16 assessments completed

• Cronbach’s alpha = .323

Youth Support Assessment

0

2

4

6

8

10

12

14

16

Community Family Friends School/Work Service

0 "Significant Need for for Support"

1 "Need for Support"

2 "Strength"

3 "Significant Strength"

School Outcomes Assessment

• Tracks percent of school days attended

• 14 assessments completed

School Outcomes assessment

% Days Attendance 87.83 (26.72)

Current Success 1.50 (1.16)

Suspensions 1

Expulsions 0

Other 0

Youth Needs Assessment

• The user enters the specific needs of the youth/family, sorts them into 14 categories, and provides ratings of progress towards meeting those need

• 22 assessments completed

Youth Needs Assessment Needs category: %Yes

Behavior 63.3

Comp Dev 4.5

Culture 4.5

Emotional 63.6

Family 63.6

Finances 9.1

Health 13.6

Legal 4.5

Logistical 4.5

Place to Live 13.6

Safety 13.6

School 59.1

Social Relation 45.5

Spiritual 4.5

Work 9.1

Other 0

Youth Needs Assessment

0

10

20

30

40

50

60

70

80

90

100

Pe

rce

nt

Sele

cte

d

Needs Categories

Dashboards - Youth

Youth-Specific Dashboards

Dashboard Details

Draft: Replication of slides not permitted without consent from Accountability

Solutions, Inc. 43

Item-Level Correlations

Wrap-TMS Items/Scales Pearson r

Interference with Friendships – Youth Support, Friends -.837*

Interference with Home life – Family Support, Service .878*

Youth Support, Family – Family Support, Family .947**

Youth Support, Community – Family Support, Community .853**

Community Outcomes, Overall Functioning - % School Attendance -.524

Community Outcomes, Overall Functioning – Family Support, Friends -.664

Community Outcomes, Overall Functioning – Youth Support, Friends -.575

Community Outcomes, Learning – Current School Success -.680

Community Outcomes, Learning – Family Support, Community -.768

Community Outcomes, Learning – Family Support, School/work -.625

Community Outcomes, Learning – Youth Support, Service .636

Family Satisfaction, Services – Current School Success -.612*

Family Satisfaction, Services – Family Support, School/Work -.866**

Support for construct validity

• Ran 42 correlations between Wrap-TMS scales or items

• Of the 42 pairs, there were 24 meaningful correlations, with 22 in the hypothesized direction

• Although this exercise is exploratory and highly influenced by small sample sizes, we interpret it as initial evidence for the construct validity of our assessment items and scales.

Pilot of Wrap-TMS Conclusions

• Users overall found system to have high usability and promise for improving efficiency and outcomes

• Areas for improvement:

• Entry of data on Plan of care elements

• Fewer mouse clicks per operation

• Calendar system

• Youth and family support connections assessment

• Psychometrics of Wrap-TMS assessments • Items showed good variability

• Most scales showed good internal consistency

• Exception: Youth and family support connections

• Scales largely correlated in hypothesized directions, lending initial support to construct validity

Next steps • Revise Wrap-TMS to respond to concerns

about items in assessments and overall usability

• Enhancing features to include: • Custom reports • Drag and drop reporting • Costs and service tracking and reporting • Child and Adolescent Needs and Strengths

(CANS) reports, and clinical flags

• Wrap-TMS will be available to programs and wraparound initiatives by July 2013