Improving Practice to Wraparound Improving Practice to Wraparound Through Use of Fidelity MeasuresThrough Use of Fidelity Measures

Enrica Bertoldo, Quality Support ManagerEnrica Bertoldo, Quality Support ManagerEleanor Castillo, Director of Outcomes & Quality Assurance Eleanor Castillo, Director of Outcomes & Quality Assurance

Mary Ann Wong, Research SpecialistMary Ann Wong, Research SpecialistVeronica Padilla, Manager of Outcomes & EvaluationsVeronica Padilla, Manager of Outcomes & Evaluations

26th26th Annual California Mental Health Advocates for Children & Youth Annual California Mental Health Advocates for Children & Youth Conference, Asilomar, CaliforniaConference, Asilomar, California

May 4, 2006May 4, 2006

EMQ Children & Family EMQ Children & Family ServicesServices

Our MissionOur MissionTo work with children and their families To work with children and their families to transform their lives, build emotional, to transform their lives, build emotional, social and familial well-being, and to social and familial well-being, and to transform the systems that serve them.transform the systems that serve them.

Our PhilosophyOur Philosophy Family Voice Family Voice Team Based Team Based Collaborative/IntegrativeCollaborative/Integrative Community-Based Community-Based Culturally CompetentCulturally Competent Individualized Individualized Strength-BasedStrength-Based Natural Supports Natural Supports PersistencePersistence Outcomes-Based Outcomes-Based

PurposePurpose

Given the complexity of Wraparound, existing Given the complexity of Wraparound, existing fidelity measures are utilized to improve fidelity measures are utilized to improve practice.practice.

This workshop will share successes and This workshop will share successes and challenges in the implementation of fidelity challenges in the implementation of fidelity measures (WFI-3.0 & WOF-2.0) that affect measures (WFI-3.0 & WOF-2.0) that affect practice and supervision to Wraparound.practice and supervision to Wraparound.

A pilot in utilizing a Wraparound supervisory A pilot in utilizing a Wraparound supervisory adherence measure (W-SAM) will also be adherence measure (W-SAM) will also be discussed.discussed.

What is Wraparound?What is Wraparound?

Team-based, individualized service Team-based, individualized service planning processplanning processPrinciples of community-based, strength-Principles of community-based, strength-based, family-centered, and culturally based, family-centered, and culturally competent competent Wraparound is described as a promising Wraparound is described as a promising practice (Burns, Hoagwood, & Maultsby, practice (Burns, Hoagwood, & Maultsby, 1998)1998)

11 Core Elements of Wraparound11 Core Elements of Wraparound

1.1. Voice and ChoiceVoice and Choice2.2. Youth/Family TeamYouth/Family Team3.3. Community-Based Community-Based

Services/SupportsServices/Supports4.4. Cultural Cultural

CompetenceCompetence5.5. Individualized Individualized

Services/SupportsServices/Supports6.6. Strengths-Based Strengths-Based

Services/SupportsServices/Supports

7.7. Natural SupportsNatural Supports8.8. Continuation of Continuation of

Services/SupportsServices/Supports

9.9. CollaborationCollaboration10.10. Flexible Resources/ Flexible Resources/

Funding Funding11.11. Outcome-Based Outcome-Based

Services/SupportsServices/Supports

Essential Elements of Wraparound Essential Elements of Wraparound

Element 1: Voice and ChoiceElement 1: Voice and ChoiceFamilies must be full and active partners at every level of the wraparound Families must be full and active partners at every level of the wraparound process. If the team cannot reach consensus, the final decision should be process. If the team cannot reach consensus, the final decision should be up to the caregiver.up to the caregiver.

Element 2: Youth and Family TeamElement 2: Youth and Family TeamWraparound is a team-driven process involving caregivers, youth, natural Wraparound is a team-driven process involving caregivers, youth, natural supports, and community services working together to develop, supports, and community services working together to develop, implement, and evaluate the individualized plan.implement, and evaluate the individualized plan.

Element 3: Community-based Services and SupportsElement 3: Community-based Services and SupportsServices and supports that the youth and family receive should be based Services and supports that the youth and family receive should be based in their community. The family should not have to leave their community if in their community. The family should not have to leave their community if more restrictive services are necessary.more restrictive services are necessary.

Element 4: Cultural CompetenceElement 4: Cultural CompetenceThe team should not only be respectful of the family’s beliefs and The team should not only be respectful of the family’s beliefs and traditions, but also actively seek to under-stand the family’s unique traditions, but also actively seek to under-stand the family’s unique perspectives and convey them to others.perspectives and convey them to others.

Element 5: Individualized ServicesElement 5: Individualized Services This means that services and supports are tailored to the unique This means that services and supports are tailored to the unique

situation, strengths, and needs of teach individual, and may involve situation, strengths, and needs of teach individual, and may involve existing categorical services and informal sup-ports; modifying existing existing categorical services and informal sup-ports; modifying existing services and supports; and or creating new services and supports. services and supports; and or creating new services and supports. Further, the team should create a specific plan to meet the family’s goals Further, the team should create a specific plan to meet the family’s goals and crisis/ safety plan to manage potential emergencies.and crisis/ safety plan to manage potential emergencies.

Element 6: Strengths-Based ServicesElement 6: Strengths-Based ServicesThe focus of the team should be on what is working and going well for the The focus of the team should be on what is working and going well for the family. While goals may be drawn up based on the family’s needs, the family. While goals may be drawn up based on the family’s needs, the plan should capitalize on the family’s positive abilities and characteristics.plan should capitalize on the family’s positive abilities and characteristics.

Element 7: Natural SupportsElement 7: Natural SupportsServices and supports should reflect a balance of formal and informal Services and supports should reflect a balance of formal and informal community and family supports rather than a reliance on formal community and family supports rather than a reliance on formal professional services.professional services.

Element 8: Continuation of CareElement 8: Continuation of CareServices and supports must be provided unconditionally. In a crisis, Services and supports must be provided unconditionally. In a crisis, services and supports should be added rather than placing the youth with services and supports should be added rather than placing the youth with a new provider.a new provider.

Element 9: CollaborationElement 9: CollaborationThe team should coordinate services and supports so they seem The team should coordinate services and supports so they seem seamless to the family rather than disjointed.seamless to the family rather than disjointed.

Element 10: Flexible Funding and ResourcesElement 10: Flexible Funding and ResourcesSuccessful wraparound teams are creative in their approach to service Successful wraparound teams are creative in their approach to service delivery and have access to flexible funds and resources to implement delivery and have access to flexible funds and resources to implement their ideas.their ideas.

Element 11: Outcome-Based ServicesElement 11: Outcome-Based ServicesSpecific, measurable out-comes should be monitored to assess the youth Specific, measurable out-comes should be monitored to assess the youth and family’s progress toward goals.and family’s progress toward goals.

The description of each element is taken directly from The description of each element is taken directly from Wraparound Fidelity Index 3.0Wraparound Fidelity Index 3.0 (2002), (2002), produced by the Wraparound Evaluation and Research Team at the University of Vermont. produced by the Wraparound Evaluation and Research Team at the University of Vermont.

What is Fidelity to Wraparound?What is Fidelity to Wraparound?

Treatment Fidelity = The degree to which a Treatment Fidelity = The degree to which a program is implemented as intended (Rast & program is implemented as intended (Rast & Bruns, 2003; Moncher & Prinz, 1991)Bruns, 2003; Moncher & Prinz, 1991)

Adherence to the 11 Core Wraparound Adherence to the 11 Core Wraparound Elements (e.g., WFI-3.0; Suter et al., 2002)Elements (e.g., WFI-3.0; Suter et al., 2002)

Adherence to the Child and Family Team Adherence to the Child and Family Team Process (e.g., WOF-2.0; Epstein et al., 2002)Process (e.g., WOF-2.0; Epstein et al., 2002)

Why Measure Fidelity?Why Measure Fidelity?

Measuring fidelity is essential to families, Measuring fidelity is essential to families, providers, policy makers and researchersproviders, policy makers and researchers

1.1. Without measuring fidelity, how do you ensure the Without measuring fidelity, how do you ensure the Wraparound process is occurring?Wraparound process is occurring?

2.2. Without outcomes, the Wraparound process is just Without outcomes, the Wraparound process is just one more fadone more fad

3.3. High fidelity has been associated with positive High fidelity has been associated with positive outcomes in some studiesoutcomes in some studies

4.4. It improves quality assuranceIt improves quality assurance5.5. It helps agencies secure more funding by proving It helps agencies secure more funding by proving

outcomesoutcomes6.6. It can even help create legislation on how families It can even help create legislation on how families

and kids receive servicesand kids receive services

More Why’s…..More Why’s…..

All nationally recognized Wraparound programs All nationally recognized Wraparound programs have extensive fidelity measurements in place. have extensive fidelity measurements in place.

Why?Why?Without fidelity measurement standards firmly in Without fidelity measurement standards firmly in

place, Wraparound programs and team place, Wraparound programs and team members have difficulty knowing what they are members have difficulty knowing what they are doing well and what they need to improve in doing well and what they need to improve in order to achieve the ideal as presented in the order to achieve the ideal as presented in the Wraparound vision.Wraparound vision.

Fidelity Implementation MeasuresFidelity Implementation Measures

Wraparound Fidelity Index-3.0 (WFI-3.0)Wraparound Fidelity Index-3.0 (WFI-3.0)Measures fidelity to Wraparound principles through Measures fidelity to Wraparound principles through structured interviewsstructured interviews

Administered to Youth, Caregiver, & Resource Administered to Youth, Caregiver, & Resource FacilitatorFacilitator

Higher Score = Greater FidelityHigher Score = Greater Fidelity

Wraparound Observation Form-2.0 (WOF-2.0)Wraparound Observation Form-2.0 (WOF-2.0)Measures fidelity to Wraparound through Measures fidelity to Wraparound through observation of the Child and Family Team meetingsobservation of the Child and Family Team meetings

Higher Score = Greater FidelityHigher Score = Greater Fidelity

Fidelity to WraparoundFidelity to Wraparound

EMQ Sacramento has collected the EMQ Sacramento has collected the Wraparound Fidelity Index (WFI) since Wraparound Fidelity Index (WFI) since 2001 2001

Data from WFI-3.0: Data from WFI-3.0: January 2003 – September 2005January 2003 – September 2005

A Comparison of EMQ Data & A Comparison of EMQ Data & National DataNational Data

WFI Total Scores by Respondent: WFI Total Scores by Respondent: EMQ & National DataEMQ & National Data

77

74

74

81

73

67

67

79

0 20 40 60 80 100

Total

Youth

Caregiver

Facilitator

EMQ

National

*National WFI-3 Dataset came from the Wraparound Evaluation and Research Team presentation for the 18th Annual Research Conference on Systems of Care and Children's Mental Health in Tampa on 3/7/05 by Eric J. Bruns, Ph.D. for "Is it Wraparound Yet?" - Bootstrapping wraparound fidelity standards using the WFI .

WFI Elements: EMQ & National DataWFI Elements: EMQ & National Data

81

83

88

73

66

87

80

86

83

62

56

83

0 20 40 60 80 100

Strengths

Individualized

Cultural Competence

Community

Youth/Family Team

Voice/Choice

EMQ

National

*National WFI-3 Dataset came from the Wraparound Evaluation and Research Team presentation for the Technical Assistance Partnership Webinar on 4/5/04 by Eric J. Bruns, Ph.D. for "Ensuring High-Quality Wraparound"

WFI Elements: EMQ & National DataWFI Elements: EMQ & National Data

77

68

68

85

63

85

66

76

76

47

0 20 40 60 80 100

Outcome-Based

Flex Funds

Collaboration

Continuation

NaturalSupports

EMQ

National

*National WFI-3 Dataset came from the Wraparound Evaluation and Research Team presentation for the Technical Assistance Partnership Webinar on 4/5/04 by Eric J. Bruns, Ph.D. for "Ensuring High-Quality Wraparound"

The Relationship Between The Relationship Between Fidelity & Outcomes Fidelity & Outcomes

The Relationship Between The Relationship Between Fidelity & Outcomes Fidelity & Outcomes

Previous research has found an association Previous research has found an association between greater Wraparound fidelity and between greater Wraparound fidelity and better child and family outcomes (Bruns, better child and family outcomes (Bruns, 2004), but the relationship has not been clearly 2004), but the relationship has not been clearly understoodunderstood

EMQ (2005) completed a study to further EMQ (2005) completed a study to further understand the relationship between fidelity to understand the relationship between fidelity to the 11 core elements of Wraparound and the 11 core elements of Wraparound and treatment outcomes treatment outcomes

DemographicsDemographics

Average Age at Admission: 14 yearsAverage Age at Admission: 14 years

63% Male; 37% Female63% Male; 37% Female

62% Caucasian; 26% African-American; 62% Caucasian; 26% African-American; 8% Latino; 2% Asian/Pacific Islander; 8% Latino; 2% Asian/Pacific Islander; 1% Native-American; 1% Other1% Native-American; 1% Other

Average Length of Stay: 15 monthsAverage Length of Stay: 15 months

Fidelity to WraparoundFidelity to Wraparound

Implementation Measure: Implementation Measure:

Wraparound Fidelity Index – 3.0 (WFI-3.0)Wraparound Fidelity Index – 3.0 (WFI-3.0)

Collection since 2003Collection since 2003

Sample in study included the following:Sample in study included the following:

146 WFI’s collected from Youth146 WFI’s collected from Youth

124 WFI’s collected from Caregivers124 WFI’s collected from Caregivers

183 WFI’s collected from Resource 183 WFI’s collected from Resource FacilitatorsFacilitators

Outcome Measures at DischargeOutcome Measures at Discharge

Child and Adolescent Functional Assessment Scale Child and Adolescent Functional Assessment Scale (CAFAS; Hodges, 2000)(CAFAS; Hodges, 2000)

Measures youth’s level of functioning; completed by Resource FacilitatorMeasures youth’s level of functioning; completed by Resource FacilitatorLower Score = Greater FunctioningLower Score = Greater FunctioningExit Total CAFAS ScoreExit Total CAFAS Score

Child Behavior Checklist (CBCL; Achenbach, 2001)Child Behavior Checklist (CBCL; Achenbach, 2001)Measures youth’s level of behavior problems; administered to CaregiverMeasures youth’s level of behavior problems; administered to CaregiverLower Score = Lower Behavior ProblemsLower Score = Lower Behavior ProblemsExit Total, Externalizing, & Internalizing CBCL ScoresExit Total, Externalizing, & Internalizing CBCL Scores

Living Arrangement at Discharge: Community or Living Arrangement at Discharge: Community or FacilityFacility

CBCLTotal

CBCL Externalizing

CBCL Internalizing

CAFAS Total

Living in Community

1. Voice and Choice .12 .07 .12 .15* .47

2. Youth and Family Team -.04 -.04 .01 .07 .10

3. Community-Based Services

-.19* -.25** -.21* -.40** .21**

4. Cultural Competence .06 .01 .02 .10 .15

5. Individualized Services .09 .06 .01 .11 .05

6. Strengths-Based -.10 -.09 -.14 .08 .05

7. Natural Supports -.19* -.12 -.13 -.07 .13

8. Continuation of Services -.06 -.02 -.12 .12 -.02

9. Collaboration -.04 -.03 -.17* .13 -.11

10. Flexible Resources/Funds -.06 .04 -.14 .05 .20**

11. Outcome-Based Services -.16 -.04 -.18* .14* .21**

12. RF Total Fidelity -.09 -.20 -.08 -.15 .16

13. Caregiver Total Fidelity -.04 -.02 -.12 .04 -.15

14. Youth Total Fidelity -.03 -.01 -.05 .02 -.05

15. WFI Total Fidelity -.10 -.09 -.15 .05 .04

Correlations Among WFI Scores and Outcomes at Discharge

Note: *p<.05; **p<.01

Significant ResultsSignificant Results

Greater community-based services is related Greater community-based services is related to lower impairment on the CBCL and CAFAS to lower impairment on the CBCL and CAFAS at dischargeat dischargeGreater community-based services is related Greater community-based services is related to community living arrangement at dischargeto community living arrangement at discharge

Community-Based ServicesCommunity-Based Services

CBCL TotalCBCL Total -.19*-.19*

CBCL ExternalizingCBCL Externalizing -.25**-.25**

CBCL InternalizingCBCL Internalizing -.21*-.21*

CAFAS TotalCAFAS Total -.40**-.40**

Living in CommunityLiving in Community .21**.21**

Note: *p<.05; **p<.01

ResultsResults

Greater fidelity to the provision of Community Greater fidelity to the provision of Community Services/Supports in Wraparound is related to Services/Supports in Wraparound is related to positive outcomespositive outcomes

No other elements had as significant correlations No other elements had as significant correlations to outcomes as Community Services/Supportsto outcomes as Community Services/Supports

No significant relationship found between Total No significant relationship found between Total Fidelity Scores and outcomes at dischargeFidelity Scores and outcomes at discharge

Correlations were consistently low even when Correlations were consistently low even when significantsignificant

ImplicationsImplications

Implications for teams to support a Implications for teams to support a youth’s school attendance and youth’s school attendance and involvement in work, training, and other involvement in work, training, and other community activities (e.g., church, community activities (e.g., church, sports, art, etc.) as well as support sports, art, etc.) as well as support community-based living situations for community-based living situations for youth (as an alternative to residential or youth (as an alternative to residential or institutional care) in order to improve institutional care) in order to improve outcomes.outcomes.

Challenges of the WFIChallenges of the WFI

Direct feedback to teams was not provided Direct feedback to teams was not provided

The utility of the WFI in its relationship to The utility of the WFI in its relationship to outcomes was limited based on EMQ’s outcomes was limited based on EMQ’s analysis of the dataanalysis of the data

Time intensive and limited resources Time intensive and limited resources available to conduct 3 structured available to conduct 3 structured interviews per youthinterviews per youth

EMQ Practice Standards Initiative EMQ Practice Standards Initiative

Initiative to Improve PracticeInitiative to Improve Practice

Quality Improvement Project launched in Quality Improvement Project launched in January 2004 to implement practice standards January 2004 to implement practice standards and feedback tools across positions and feedback tools across positions

Family FacilitatorFamily Facilitator Family SpecialistFamily Specialist Family PartnerFamily Partner

Goal was to standardize job positions, provide Goal was to standardize job positions, provide role clarity, and improve consistency in job role clarity, and improve consistency in job performance and supervisionperformance and supervision

Example of EMQ Feedback Tool:Example of EMQ Feedback Tool:Family FacilitatorFamily Facilitator

Scoring:Scoring:Opportunity for Improvement (1)Opportunity for Improvement (1)Acceptable (2)Acceptable (2)Exemplary (3)Exemplary (3)Not Observed (*)Not Observed (*)

3 Dimensions:3 Dimensions:A.A. Families receive services that are consistent with Families receive services that are consistent with wraparound valueswraparound valuesB.B. Families receive services that are collaborative, Families receive services that are collaborative, integrated and adhere to best practicesintegrated and adhere to best practicesC.C. Children and Families served are safe and stable at Children and Families served are safe and stable at home and in the communityhome and in the community

Facilitator Field Feedback ToolFacilitator Field Feedback ToolA.A. Families receive services that are consistent with Families receive services that are consistent with

wraparound valueswraparound values1.1. Arrived on time and was prepared for the CFTArrived on time and was prepared for the CFT2.2. Started the CFT with “what’s working” to maintain strength based Started the CFT with “what’s working” to maintain strength based

focus and future orientationfocus and future orientation3.3. Facilitated the CFT to validate and incorporate ideas, requests, and Facilitated the CFT to validate and incorporate ideas, requests, and

concerns of all membersconcerns of all members4.4. Evidence that needs and strategies were reviewed within the CFT Evidence that needs and strategies were reviewed within the CFT

to assure the family voice and preference guide all activitiesto assure the family voice and preference guide all activities5.5. Evidence that plans were reviewed for progress on current Evidence that plans were reviewed for progress on current

measurable, observable goals and updated as needed to address measurable, observable goals and updated as needed to address new and changing needsnew and changing needs

6.6. Evidence that the next CFT and staff visits were scheduled Evidence that the next CFT and staff visits were scheduled according to family preferenceaccording to family preference

Facilitator Field Feedback ToolFacilitator Field Feedback ToolB.B. Families receive services that are collaborative, integrated and Families receive services that are collaborative, integrated and

adhere to best practicesadhere to best practices1.1. For children open six months or longer, evidence that the CFT was comprised of For children open six months or longer, evidence that the CFT was comprised of

at least 50% non-paid informal membersat least 50% non-paid informal members2.2. Facilitated the CFT to allow pertinent agenda items to be covered according to Facilitated the CFT to allow pertinent agenda items to be covered according to

family preference and time availabilityfamily preference and time availability3.3. Demonstrated engagement and rapport with all team members including system Demonstrated engagement and rapport with all team members including system

partnerspartners4.4. Therapy, medications and adjunctive services were evaluated relative to plan Therapy, medications and adjunctive services were evaluated relative to plan

goals as needed goals as needed 5.5. Specific tasks were assignedSpecific tasks were assigned

C.C. Children and Families served are safe and stable at home and in Children and Families served are safe and stable at home and in the communitythe community

1.1. Evidence that team has a plan B to back up plan AEvidence that team has a plan B to back up plan A2.2. If the CFT was for a child in out of home placement, this was addressed as a If the CFT was for a child in out of home placement, this was addressed as a

priority with specific plan, tasks, and target datepriority with specific plan, tasks, and target date3.3. Evidence that strategies and interventions developedEvidence that strategies and interventions developed emphasize the use of emphasize the use of

typical resources in the child’s communitytypical resources in the child’s community

EMQ Feedback ToolsEMQ Feedback Tools

Data analysis showed that the tools were not Data analysis showed that the tools were not measuring what was intended; therefore, tools measuring what was intended; therefore, tools were discardedwere discarded

Led to an examination of the national Led to an examination of the national standards for Wraparoundstandards for Wraparound

EMQ Sacramento decided to collect data on EMQ Sacramento decided to collect data on the Child and Family Team process using the the Child and Family Team process using the Wraparound Observation Form – Version 2.0 Wraparound Observation Form – Version 2.0 (WOF-2.0)(WOF-2.0)

Wraparound Observation Form – Wraparound Observation Form – Second Version (WOF-2)Second Version (WOF-2)

Measure to assess the implementation of the Wraparound Measure to assess the implementation of the Wraparound approach by direct observation of youth and family team approach by direct observation of youth and family team meetingsmeetings

WOF has demonstrated good inter-rater reliability WOF has demonstrated good inter-rater reliability (kappa=.886; Nordess & Epstein, 2003)(kappa=.886; Nordess & Epstein, 2003)

48 items that measures 8 characteristics: 48 items that measures 8 characteristics:

Community-BasedCommunity-Based Unconditional Care Unconditional Care IndividualizedIndividualized Measurable Outcomes Measurable Outcomes Family-DrivenFamily-Driven Management of Team Meeting Management of Team Meeting Interagency Collaboration Interagency Collaboration Care Coordinator Care Coordinator

Implementation of the WOFImplementation of the WOF

WOF data collection began July 2005WOF data collection began July 2005

Feedback provided in individual Feedback provided in individual supervision if completed by direct supervision if completed by direct supervisorsupervisor

Verbal feedback provided to facilitators Verbal feedback provided to facilitators after the child and family team meeting if after the child and family team meeting if completed by neutral observer. Copy of completed by neutral observer. Copy of tools also provided to supervisorstools also provided to supervisors

Wraparound Observation Form – Wraparound Observation Form – Second Version (WOF-2)Second Version (WOF-2)

Community-based resources (5 items)Community-based resources (5 items)

1. Information about resources / interventions in the area is offered to 1. Information about resources / interventions in the area is offered to the team. the team.

2. Plan of care includes at least one public and/or private community 2. Plan of care includes at least one public and/or private community service/resource.service/resource.

3. Plan of care includes at least one informal resource.3. Plan of care includes at least one informal resource.4. When residential placement is discussed, team chooses 4. When residential placement is discussed, team chooses

community placements for child(ren) rather than out-of-community community placements for child(ren) rather than out-of-community placements, whenever possible.placements, whenever possible.

5. Individuals (non-professionals) important to the family are present 5. Individuals (non-professionals) important to the family are present at the meeting.at the meeting.

Individualized services for the family (9 items)Individualized services for the family (9 items)

6. If an initial plan of care meeting, the parent is asked what 6. If an initial plan of care meeting, the parent is asked what treatments or interventions he/she felt worked/didn’t work prior to treatments or interventions he/she felt worked/didn’t work prior to WRAP.WRAP.

7. Facilitator advocates for services and resources for the family 7. Facilitator advocates for services and resources for the family (e.g., identifies and argues for necessary services). (e.g., identifies and argues for necessary services).

8. All services needed by family are included in plan (e.g., no 8. All services needed by family are included in plan (e.g., no needed services were not offered).needed services were not offered).

9. Barriers to services or resources / interventions are identified and 9. Barriers to services or resources / interventions are identified and solutions discussed.solutions discussed.

10. The steps needed to implement the plan of care are clearly 10. The steps needed to implement the plan of care are clearly specified by the team.specified by the team.

11. Strengths of family members are identified and discusses at the 11. Strengths of family members are identified and discusses at the meeting.meeting.

12. Plan of care that includes life domain(s) goals, objectives, and 12. Plan of care that includes life domain(s) goals, objectives, and resources/interventions is discussed (or written).resources/interventions is discussed (or written).

13. Plan of care goals, objectives, or interventions are based on 13. Plan of care goals, objectives, or interventions are based on family/child strengths.family/child strengths.

14. Safety plan/Crisis plan developed/reviewed.14. Safety plan/Crisis plan developed/reviewed.

Family-driven services (10 items)Family-driven services (10 items)

15. Convenient arrangements for family’s presence at meeting are 15. Convenient arrangements for family’s presence at meeting are made (e.g., location, time, transportation, day care made (e.g., location, time, transportation, day care arrangements).arrangements).

16. The parent/child is seated or invited to sit where he/she can be 16. The parent/child is seated or invited to sit where he/she can be included in the discussion.included in the discussion.

17. Family members are treated in a courteous fashion at all times.17. Family members are treated in a courteous fashion at all times.18. The family’s perspective is presented to professionals from other 18. The family’s perspective is presented to professionals from other

agencies. agencies. 19. The family is asked what goals they would like to work on.19. The family is asked what goals they would like to work on.20. The parent is asked about the types of services or 20. The parent is asked about the types of services or

resources/interventions he/she would prefer for his/her family.resources/interventions he/she would prefer for his/her family.21. Family members are involved in designing the plan of care.21. Family members are involved in designing the plan of care.22. In the plan of care, the family and team members are assigned 22. In the plan of care, the family and team members are assigned

(or asked) tasks and responsibilities that promote the family’s (or asked) tasks and responsibilities that promote the family’s independence (e.g., accessing resources on own, budgeting, independence (e.g., accessing resources on own, budgeting, maintaining housing).maintaining housing).

23. The team plans to keep the family intact or to reunite the family.23. The team plans to keep the family intact or to reunite the family.24. Family Members voice agreement/disagreement with plan of 24. Family Members voice agreement/disagreement with plan of

care.care.

Interagency collaboration (7 items)Interagency collaboration (7 items)

25. Staff from other agencies who care about or provide 25. Staff from other agencies who care about or provide resources/interventions to the family are present at the meeting.resources/interventions to the family are present at the meeting.

26. Staff from other facilities or agencies (if present) have an 26. Staff from other facilities or agencies (if present) have an opportunity to provide input opportunity to provide input

27. Informal supports (if present) have an opportunity to provide 27. Informal supports (if present) have an opportunity to provide input.input.

28. Problems that can develop in an interagency team (e.g., turf 28. Problems that can develop in an interagency team (e.g., turf problems, challenges to authority) are not evident or are resolved.problems, challenges to authority) are not evident or are resolved.

29. Staff from other agencies describe support resources 29. Staff from other agencies describe support resources interventions available in the community.interventions available in the community.

30. Statement(s) made by a staff member or an informal support 30. Statement(s) made by a staff member or an informal support indicate that contact/ communication with another team member indicate that contact/ communication with another team member occurred between meetings.occurred between meetings.

31. Availability of alternative funding sources is discussed before 31. Availability of alternative funding sources is discussed before flexible funds are committed.flexible funds are committed.

Unconditional Care (3 items)Unconditional Care (3 items)

32. Termination of Wrap services is discussed because of the 32. Termination of Wrap services is discussed because of the multiplicity or severity of the child’s/family’s behaviors /problems.multiplicity or severity of the child’s/family’s behaviors /problems.

33. Termination of other services (not Wrap) is discussed because of 33. Termination of other services (not Wrap) is discussed because of the multiplicity or severity of the child’s/family’s behaviors the multiplicity or severity of the child’s/family’s behaviors /problems./problems.

34. For severe behavior challenges (e.g., gangs, drugs), discussion 34. For severe behavior challenges (e.g., gangs, drugs), discussion focuses on safety plans/crisis plans (e.g., service and staff to be focuses on safety plans/crisis plans (e.g., service and staff to be provided) rather than termination.provided) rather than termination.

Measurable Outcomes (3 items)Measurable Outcomes (3 items)

35. The plan of care goals are discussed in objective, measurable 35. The plan of care goals are discussed in objective, measurable terms.terms.

36. The criteria for ending Wrap involvement are discussed.36. The criteria for ending Wrap involvement are discussed.37. Objective or verifiable information on child and parent functioning 37. Objective or verifiable information on child and parent functioning

is used as outcome data.is used as outcome data.

Management of team meeting (5 items)Management of team meeting (5 items)

38. Key participants are invited to the meeting (e.g., family members, 38. Key participants are invited to the meeting (e.g., family members, CPS worker, teacher, therapist, others identified by family).CPS worker, teacher, therapist, others identified by family).

39. Current information about the family (e.g., social history, 39. Current information about the family (e.g., social history, behavioral and emotional status) is gathered prior to the meeting behavioral and emotional status) is gathered prior to the meeting and shared at meeting (or beforehand).and shared at meeting (or beforehand).

40. All meeting participants introduce themselves (if applicable) or are 40. All meeting participants introduce themselves (if applicable) or are introduced.introduced.

41. The family is informed that they may be observed during the 41. The family is informed that they may be observed during the meeting.meeting.

42. Plan of care is agreed on by all present at the meeting.42. Plan of care is agreed on by all present at the meeting.

Facilitator (6 items)Facilitator (6 items)

43. Facilitator makes the agenda of meeting clear to participants.43. Facilitator makes the agenda of meeting clear to participants.44. Facilitator reviews goals, objectives, interventions and/or 44. Facilitator reviews goals, objectives, interventions and/or

progress of plan of care.progress of plan of care.45. Facilitator directs (or reflects) team to discuss family/child 45. Facilitator directs (or reflects) team to discuss family/child

strengths.strengths.46. Facilitator directs (or reflects) team to revise/update plan of care.46. Facilitator directs (or reflects) team to revise/update plan of care.47. Facilitator summarizes content of the meeting at the conclusion of 47. Facilitator summarizes content of the meeting at the conclusion of

the meeting.the meeting.48. Facilitator sets next meeting date/time.48. Facilitator sets next meeting date/time.

EMQ WOF-2 Results:EMQ WOF-2 Results:Total Element Scores (N=30)Total Element Scores (N=30)

7795

63

88929077

57

235

37128

102343

0

20

40

60

80

100

Comm

unity

Indiv

idua

lized

Family

-Driv

en

Collabo

ratio

n

Uncond

itiona

l Car

e

Out

com

es

Man

agem

ent

Facilit

ator

Undesired

Desired

Results of WOFResults of WOF

EMQ Sacramento shows higher scores on EMQ Sacramento shows higher scores on elements of Family-driven, Collaboration, elements of Family-driven, Collaboration, Unconditional Care & Management of CFT Unconditional Care & Management of CFT MeetingsMeetingsEMQ Sacramento continues to be challenged EMQ Sacramento continues to be challenged by lower scores on Community-based by lower scores on Community-based Resources, inclusive of natural supportsResources, inclusive of natural supportsIn contrast to the results of the WFI for In contrast to the results of the WFI for Outcomes (high scores), the WOF showed low Outcomes (high scores), the WOF showed low scores on this element scores on this element Slight increase in Total WFI scores since the Slight increase in Total WFI scores since the implementation of the WOFimplementation of the WOF

Back to the WFIBack to the WFI

Could the use of the WOF impact Could the use of the WOF impact WFI scores?WFI scores?

A look at WFI scores across timeA look at WFI scores across time

WFI Total Scores Across Time WFI Total Scores Across Time

75 7570

76

0

20

40

60

80

100

1/04-6/04 7/04-12/04 1/05-6/05 7/05-12/05

WOF Implementation Begins 7/05

WFI Respondent Scores WFI Respondent Scores Across TimeAcross Time

81 79 82 84

7264

74 7771 68

72 71

0

10

20

30

40

50

60

70

80

90

Facilitator Caregiver Youth

1/04-6/04

7/04-12/04

1/05-6/05

7/05-12/05

High Scores on Voice and Choice, Cultural Competence, Individualized Services, Strengths, Outcome-Based Services Low Scores on Natural Supports, Community-Based Services, and CFT Element

Challenges of the WOFChallenges of the WOF

Limited resources in staff to become neutral Limited resources in staff to become neutral observersobserversManagers wanted to use the WOF as a Managers wanted to use the WOF as a supervision toolsupervision toolWillingness and comfort level of staff to Willingness and comfort level of staff to participate in direct observationparticipate in direct observationStaff perception of fear of families being Staff perception of fear of families being judged by observersjudged by observersSponsorship by management Sponsorship by management Competing priorities in data collectionCompeting priorities in data collection

Next StepsNext Steps

Supervisors will continue to utilize the Supervisors will continue to utilize the WOF to provide feedbackWOF to provide feedbackOperational realities limit the use of Operational realities limit the use of neutral observersneutral observersEMQ will no longer collect the WFI due EMQ will no longer collect the WFI due to limitations in resources, the utility of to limitations in resources, the utility of the tool for feedback, and its relationship the tool for feedback, and its relationship to outcomesto outcomesContinue to develop practice standardsContinue to develop practice standards

ReferencesReferences

Achenbach, T. M., & Rescorla, L.A. (2001). Achenbach, T. M., & Rescorla, L.A. (2001). Manual for the ASEBA Manual for the ASEBA school-age forms & profiles. school-age forms & profiles. Burlington, VT: University of Vermont, Burlington, VT: University of Vermont, Research Center for Children, Youth, & Families. Research Center for Children, Youth, & Families.

Epstein, M.H., Nordess, P.D., & Hertzog, M. (2002). Epstein, M.H., Nordess, P.D., & Hertzog, M. (2002). Wraparound Wraparound Observation Form-Second VersionObservation Form-Second Version..

Epstein, M.H., Nordess, P.D., et al. (2003). Epstein, M.H., Nordess, P.D., et al. (2003). Assessing the Wraparound Assessing the Wraparound Process During Family Planning Meetings.Process During Family Planning Meetings.

Hodges, K. (2000). Child and Adolescent Functional Assessment Scale Hodges, K. (2000). Child and Adolescent Functional Assessment Scale (CAFAS). (CAFAS). CAFAS Self-Training Manual and Blank Scoring Forms CAFAS Self-Training Manual and Blank Scoring Forms (2003)(2003). Ann Arbor, MI . Ann Arbor, MI

Nordness, P.D., & Epstein, M.H.(2003). Nordness, P.D., & Epstein, M.H.(2003). Reliability of the Wraparound Reliability of the Wraparound Observation Form Second Version.Observation Form Second Version.

Suter, J., Force, M., Bruns, E., Leverentz-Brady, K., & Burchard, J. Suter, J., Force, M., Bruns, E., Leverentz-Brady, K., & Burchard, J. (2002). (2002). Manual for training, administration, and scoring of the WFI 3.0.Manual for training, administration, and scoring of the WFI 3.0. Burlington: University of Vermont. Burlington: University of Vermont.

Contact InformationContact Information

Enrica BertoldoEnrica Bertoldo ebertoldo@emq.orgebertoldo@emq.org Quality Support Manager, SacramentoQuality Support Manager, Sacramento

Mary Ann WongMary Ann Wongmwong@emq.orgmwong@emq.org

Research Specialist, SacramentoResearch Specialist, Sacramento

Wraparound Supervisor Wraparound Supervisor Adherence Measure (W-SAM): Adherence Measure (W-SAM):

A Pilot A Pilot

Eleanor Castillo, Ph.D.Eleanor Castillo, Ph.D.

Veronica PadillaVeronica Padilla

RationaleRationale

Supervision identified as a critical Supervision identified as a critical aspect of Wraparoundaspect of Wraparound

1.1. No tool to date to measure this aspect of No tool to date to measure this aspect of WraparoundWraparound

Concept adapted from Multisystemic Concept adapted from Multisystemic Treatment’s (MST) Supervisor Treatment’s (MST) Supervisor Adherence Measure (Adherence Measure (SAMSAM))

RationaleRationale

WFI-3.0 measures adherence to WFI-3.0 measures adherence to Wraparound principles from the Wraparound principles from the Youth, Caregiver, and Resource Youth, Caregiver, and Resource Facilitator perspectiveFacilitator perspective

WOF-2.0 measures the Child and WOF-2.0 measures the Child and Family Team aspect of WraparoundFamily Team aspect of Wraparound

DescriptionDescription

Facilitator rates the Clinical Program Facilitator rates the Clinical Program Manager (CPM)Manager (CPM)

40 Items40 Items

Likert Scale (1 = Never; 5 = Almost Likert Scale (1 = Never; 5 = Almost Always)Always)

Administered quarterlyAdministered quarterly

Highly reliable (alpha = .84-.98)Highly reliable (alpha = .84-.98)

Process ImplementationProcess Implementation

Sponsorship from the DirectorsSponsorship from the Directors

Program quality improvement versus Program quality improvement versus “mandate”“mandate”

Immediate feedbackImmediate feedback

Quality ManagementQuality Management

TrainingTraining

EvaluationEvaluation1.1. Overall Program ResultsOverall Program Results

2.2. Individual CPMIndividual CPM

Performance AppraisalPerformance Appraisal1.1. 360360

2.2. AnnualAnnual

Quality ManagementQuality Management

3-Tiered: Agency-wide, Program, and 3-Tiered: Agency-wide, Program, and IndividualIndividual

1.1. Training and CoachingTraining and Coaching2.2. EvaluationEvaluation

Performance AppraisalPerformance Appraisal1.1. Quarterly 360 feedbackQuarterly 360 feedback2.2. AnnualAnnual

Correlation between supervisor focus on Correlation between supervisor focus on the Wraparound process and youth the Wraparound process and youth outcomesoutcomes

Quality ManagementQuality Management

Trend analysesTrend analyses1.1. Fidelity consistency over timeFidelity consistency over time

2.2. What factors affect fidelityWhat factors affect fidelityLearning curve with new manager or Learning curve with new manager or facilitatorsfacilitators

Next StepsNext Steps

Continue pilotContinue pilot

Conduct further analyses on the Conduct further analyses on the psychometric properties of the psychometric properties of the scalescale

1.1. Factor analysesFactor analyses

2.2. Items reductionItems reduction

Contact Information:Contact Information:

Eleanor Castillo, Ph.D.Eleanor Castillo, Ph.D.

Director of Outcomes & Quality Director of Outcomes & Quality AssuranceAssurance

Email: ecastillo@emq.orgEmail: ecastillo@emq.org