Data Driven Decision Making within a Multi-tiered System of

SupportLucille Eber, Ed.D.

Midwest PBIS Network and PBIS National TA CenterNovember 4, 2015

School Mental Health Capacity Building Meeting20th Annual Conference on Advancing School Mental Health

New Orleans, LA

The CSMH’s Meeting Goals for Today

1. Learn how to use the national census and SMH performance measures;

2. Explore engagement strategies to promote cross agency and stakeholder partnerships;

3. Learn about best practices, policies, financing strategies that will help improve your comprehensive SMH system;

4. Obtain innovative resources and tools related to advancing quality and sustainability in SMH to take back to your state/community;

5. Begin an action plan to improve your state/district/school comprehensive SMH system.

This DDDM Session

– Define DDDM in context of school mental health– Provide system and practice examples of DDDM in

schools that can inform – Consider opportunities for advancing SMH

innovations through DDDM

Acknowledgements

Rob Horner, UOKent McIntosh, UOKelsey Morris, UO Tamika LaSalle, UConnSusan Barrett , Mid-Atlantic PBIS NetworkMark Weist, USCOthers from all of our Centers…..

DDDM: A Key Feature of Effective Systems of Support in Schools

– Tiered prevention logic– Cross-system problem solving teams – Use of data to decide which evidence based

practices to implement.– Progress monitoring for both fidelity and

impact. – Active involvement by youth, families, and

other school and community stakeholders.

DDDM: A Key Feature of Continuous Quality Improvement

Plan

Implement

Evaluate

DDDM: A Key Feature of Sustainability

Plan

Implement

Evaluate

What is most related to high sustainability? The frequency that data are presented to all school staff.McIntosh, K., Predy, L., Upreti, G., Hume, A. E. & Mathews, S. (2014).

• Data are the many sources of information we use to make decisions about how to allocate resources (e.g. time and attention) for implementing EBPs to achieve desired outcomes.

• In Schools, data come in many forms such as office referrals, attendance records, grades, visits to the nurse/counselor, surveys (student, teacher, families), verbal feedback, and observations.

• Data is only useful if it is documented, shared, and applied to specific action planning (plan, implement, evaluate, refine) .

What is Data?

What is Data-Driven Decision Making?

• DDDM is the process of using relevant information to make objective decisions.

• DDDM can help inform decisions related to systems development/improvement and student outcomes.– Student Performance Data (academic,

social/emotional)– Fidelity Data (measuring core feature

implementation)

Data Collection vs Data Analysis

Data Collection• Systematic gathering of

required information.

Data Analysis• Examination of data such

that relationships, patterns, trends, etc. emerge and foster hypotheses and decision making.

Value and Utility of Data

Repeatedly giving people the right information, at the right time, in the right format is the single most effective way to improve decision making and achieve valued outcomes (Gilbert, 1978).

Data

PracticesSystems

Valued Outcomes

Improving Decision Making

Problem

Problem

Solving

Solution

Action Plannin

g

Problem

Solution

Problem-solving Steps : Systems or Student Level

1. Determine problem

2. Conduct Assessment

3. Obtain Data

4. Analyze Data

5. Determine Conclusion

6. Implement Strategy

Rationale

• Data allows stakeholders/teams to look at a problem more objectively.

• Without data, teams (or individuals) are more likely to make ambiguous, or emotionally driven decisions.

• Data is used for identifying and planning to address problems, celebrating successes, and accountability.

• Data helps teams ask the right questions…it does not provide the answers.

• Use data to:– Identify problems– Refine problems– Define the questions that lead to solutions

• Data helps place the “problem” in the context rather than in the students.

Why Use Data For Decision Making?

Problem Solving with Precision

• The statement of a problem is important for team-based problem solving.– Everyone must be working on the same problem with

the same assumptions.

• Problems are often framed in the “primary” form. – Raises awareness– Not useful for problem solving

• Precise problem statements result from a detailed data review and are solvable.

Problem Solving with Precision

Primary Statements• There are too many

referrals

• Gang behavior is increasing

• The cafeteria is out of control

• Student disrespect is a big problem

Precision Statement

• There are more ODRs for aggression on the playground than last month. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

Problem Solving with Precision

There are more ODRs for aggression on the playground than last month. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.What? Where? When? Who? Why?

Aggression Playground 1st Recess Large number of students

To get new playground equipment

Student Outcomes: Data Examples

Attendance Academic performance Social/emotional Discipline referrals

Suspensions Placements outside of natural environments

Expulsion Arrests Hospitalizations

Student Perception Safety Surveys School Climate Surveys

Other…

Recognize and RespondMultiple Data Points as Natural Screeners - LOW Threshold

– Attendance– Office Referral– Suspension– Instructional Time– Tardy– HW Completion– GPA– Credit Accrual

Example: DDDM Guides Necessary Conversations within a MTSS

CICO

SAIGGroup w. individual

feature

Complex

FBA/BIP

Problem Solving Team

Tertiary Systems Team

Brief

FBA/BIP

Brief FBA/BIP

WRAP

RENEW

Secondary Systems Team

Plans SW & Class-wide supports

Uses Process data; determines overall

intervention effectiveness

Standing team with family; uses FBA/BIP process for one youth

at a time

Uses Process data; determines overall

intervention effectiveness

UniversalTeam

Universal Support

Family and community Family and

community

Family and community

Community

School Data Community Data

• Academic (Benchmark, GPA, Credit accrual etc)

• Discipline• Attendance• Climate/Perception• Visits to Nurse,

Social Worker, Counselor, etc

• Screening from one view

• Community Demographics

• Food Pantry Visits• Protective and Risk

Factors• Calls to crisis centers,

hospital visits• Screening at multiple

views

School Climate Data as a Meaningful Resource

• Efficient & relatively easy to assess• Provides information about the school

environment from multiple perspectives• Allows for the comparison of data across

participants, time, settings, etc. • Adaptable

Teacher Perceptions

Parent PerceptionsCT Georgia Parent School Climate Survey Overall Average and Average by Subscale

Secondary Student Perceptions

Example: Tier 2/3 Data Decision Rules

• Staff need to know how students are eligible for the intervention (IN), how progress is monitored (ON), and how students exit the intervention (OUT)

– IN = Identification– ON = Progress Monitoring– OUT = Exiting/Transitioning

• Examples of data used to progress monitor– Points earned on Daily Progress Report (DPR)– Reduction in ODRs– Attendance improvement– Reduction in In School Suspensions– Reduction in Out of School Suspensions– Improvement in grades– Reduction of tardies– Student survey

• Reverse Request for Assistance Form– Secondary systems team gives RRA Form to student’s teacher(s) to determine any

intervention changes

Student perspective Survey Example‘I feel connected to my school’

44%

11%11%

11%

22%

Pre-TestStrongly Disagree Disagree

Neither Agree nor Disagree Agree

Strongly Agree

33%

22%

45%

Post-TestStrongly Disagree Disagree

Neither Agree nor Disagree Agree

Strongly Agree

U-46 SD,IL

Daily Progress Report (DPR) SampleNAME:______________________ DATE:__________________

EXPECTATIONS1st block 2nd

block3rd block 4th block 5th block 6th block 7th

block

Be Safe 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0

Be Respectful 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0

Be Responsible 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0 2 1 0

Total Points

Teacher Initials

Mark will keep hands to self

Mark will hold up a yellow card to

indicate needing a break

Mark will fill out assignment notebook

“Individualized Student

Card for Mark” (FBA/BIP)

Replacement behaviorPossible behaviors taught in previous SAIG groups

“Social & Academic

Instructional Groups”

(sample academic skills group)

Walk to classKeep hands to

self

Use appropriate language

Raise hand to speak

Bring materials Fill out

assignment notebook

Examples of Data to Guide System Development

Readiness Surveys Resource Maps/System Audits Fidelity of Implementation

Systems Fidelity of Specific Practices

CICO Fidelity Tools CBITs Fidelity measures Assessing Core features of FBA/BIP Wraparound Fidelity Tools

DDDM Applications

• System Readiness Examples• Fidelity-Systems

– Measures extent to which core features are in place

– Self-assessment or outside evaluator

Interconnected System Framework Readiness Survey (SMH & PBIS)

• 63-item survey • All school faculty and

staff• Used to determine

readiness to integrate PBIS and SMH

• Pilot study conducted by Vittoria Annello and Mark Weist (publication forthcoming)

Westchester NY Example Readiness Survey (District Application)

Westchester NY Example Readiness (Ditrict Application)

Westchester Readiness Survey Results

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q80

10

20

30

40

50

60

70

80

1 2 3

Fidelity Tools

40

Fidelity tools are the GPS, MapQuest, Google maps of our journey to support all teachers, students, and instructional leaders on their pathway to success.

Erin A. Chaparro, University of OregonKimberly St. Martin, Michigan’s Integrated Behavior & Learning Support Initiative, MiBLSi

FIDELITY“Fidelity of implementation of academic and

behavior interventions is documented before measuring effects and assigning worth to them or blaming children for continuing failure. Continuing emphasis and dialogue on the importance of treatment fidelity speaks to the need for documenting the extent to which essential features of effective core instruction are in place and implemented with integrity

(cf. Miller, 2010)” Algozzine, Putnam and Horner 2012

• Primary purpose of the instrument is to help school teams improve

• Primary audience for instrument results is the team, faculty, families and administrators of the school.

• Effective use of the instrument requires multiple administrations (progress monitoring)

Example: SWPBIS Tiered Fidelity Inventory ( a self-assessment example)

The purpose of the School-wide PBIS Tiered Fidelity Inventory is to provide an efficient and valid index of the extent to which PBIS core features are in place within a school.

Fidelity Assessment: Assessing Core Elements

Intensive SWPBIS Features• Teams• Resources• Support Plans• Evaluation

Targeted SWPBIS Features• Teams• Interventions• Evaluation

Universal SWPBIS Features• Teams• Implementation• Evaluation

An Example of an Innovation (Today’s Meeting Goal #4)

• ISF Enhancement Action Planning (Companion Guide to SWPBIS TFI)– The purpose is to guide action planning for integration of

Mental Health into PBIS

Implementation Inventory for an Interconnected Systems Framework

• Purpose:– To assist school and community partners in their

installation and implementation of interconnected PBIS and SMH at the building level

– To assess baseline and/or ongoing progress on critical features of SMH/PBIS interconnected systems framework

– To guide action planning for a PBIS/SMH interconnected systems framework

Action Planning Companion Guide

Action Planning Companion Guide

Example of Action Planning from I3 Tool Data:

Data-Driven Decision Making• Effective teams use data to document progress and outcomes, guide

decisions, and inform stakeholders (Boudett, City, & Murnane, 2006; Burke, 2010; Deno, 2005; Hill 2010; Newton, Algozzine, Algozzine, Horner, & Todd, 2011; Newton, Horner, Algozzine, Todd, & Algozzine, 2009; Pidgeon & Gregory, 2004; Renfro & Grieshaber, 2009)

• Fidelity and student outcome data are essential (Fixsen, Blase, Metz, & Van Dyke, 2013)

• Continues to be a struggle for schools (Dunn, Airola, Lo, & Garrison, 2013; Schildkamp, Ehren, & Lai, 2012; Telzrow, McNamara, & Hollinger, 2000)

• Advances in computer technology could provide efficient means for data management (Wayman, 2005)

• A critical predictor of sustained implementation of SWPBIS (Coffey & Horner, 2012; McIntosh et al., 2013)