ActivitySelf Perception

Coaching teams to use data for decision making

Coaching

What is a PBIS Coach?

What do we mean by “Coach”PBIS Coaches are not “trainers”, they support teams who have basic training in PBIS. PBIS Coaches support teams to make data-based decisions toward quality improvement of student, staff and family outcomes.

What do I need to be doing as a coach?

1. Prevent team members from launching into solutions before they are ready by engaging in active problem solving

2. Get team members to ask questions, even if they don’t have all the information

3. Don’t move forward until a measureable goal is identified and a solution is designed

What do I need to know to coach well?

1. Desired Outcomes – how will we know if what we’re doing is having a positive effect on students, staff, and families?

2. Practices – what PBIS Interventions are in place? 3. Data – what data do we have and what tools do we have to collect &

summarize data? 4. Systems – what do we have in place to support teams to look at our

data and use it for quality improvement?

Data-Based Decision Making

Here’s what we know…

Decisions are more likely to be effective and efficient when they are based on data.

The quality of decision making depends most on the first step (defining the problem to be solved).

Define problems with precision and clarity

Data help us ask the right questions…they do not provide the answers. Use data to:

Identify problemsRefine problemsDefine the questions that lead to solutions

Data help place the “problem” in the context rather than in the students.

School-wide PBIS

Primary Prevention: School-wide & Classroom-wide systems for all students and all staff in all settings.

Universal, Tier I

Secondary Prevention: Systems for targeted or group-based interventions for students needing additional support beyond the Universal or Tier I system.

Targeted, Tier II

Tertiary Prevention: System for students requiring more intensive & individualized supports for academic, social, or mental health services.

Individualized,Tier III

What is DBDM?

The process of planning for student success (both academic and behavioral) through the use of ongoing progress monitoring and analysis of data

Douglas County School District (Colorado)

Why Do it?

The value of Data-Based Decision Making is: Quality Improvement Cycle of continuous Improvement

Improving what? fidelity of implementation, social climate, learning environment, student learning, attendance, grades)

How do we do it?

Right Data/Format/Time/PeopleRight QuestionsSolution Development & Action Planning

Hallway Noise Study

A brief vignette to demonstrate how SWIS data is used to support data-based decision making.

Kartub, D., Taylor-Greene, S., March, R., Horner, R.H. (2000). Reducing Hallway Noise: A Systems Approach. Journal of Positive Behavior Interventions, 2(3). 179-182

Using SWIS Data for Active Decision Making

Problem

Staff at a middle school (Grades 6-8) in a rural school district with 520 students have identified an issue with student noise in the hallways.

Teachers complain that hallway noise is significantly disruptive around lunch.

Three lunch periods (by grade)

Students required to walk past classrooms still in session to access cafeteria.

Problem Solving Process

a. Team Assesses the Extent of the Problem• Vote during faculty meeting confirmed as a priority

to address

b. Review Existing Practices• Students were taught school-wide expectations• Teaching Assistant in hall gives out detentions &

office referrals for loud noise.

c. Review Existing Data• Referrals by location• Hallway ODR per student

d. Build a hypothesisNoise is occurring because • Students have been in class all morning (low blood

sugar) and want to socialize (peer attention)• Hallway is loud at beginning and end of day

e. Define problem-solving logic• Small number of kids = address group/individually

Large number of kids = address system• Define, teach, monitor, and reward BEFORE

increasing use of punishment.

Problem Solving Process (cont.)

Office Referrals by Location

Students: 173 Referrals: 530

Office Discipline Referrals by Student

Drill Down into the Problem

Who? Large number of students across grade levels

What? Disruptive (loud, rowdy) behavior

When? After morning class

Where? Hallway

Why? (a) To gain peer attention, and (b) behavior is similar to what they do before and after school.

*Teaching Assistant’s consequences are not proving effective

Solution (keep it simple)Make lunch hallways look different from hallways in morning and afternoon.

Change lighting

Review school-wide expectations for hallwayFive-minute review of “quiet”

Build reward for valued behaviorThree days of quiet in hallway results in an extra five minutes of social time (at lunch or at end of school)

Remind students to be quiet just before they are released for lunch

Measure and ImplementUse a decibel meter to measure noise levelPublic posting of results

Build Action Plan

Actions Who When1. Build “Quiet” Curriculum Ben and

MaryNov 12

2. Buy Decibel Meter Rob Nov 10

3. Teach Hallway Expectations/ Reminders

Team Dec 2-3

4. Collect and Post Data Reiko Ongoing

5. Schedule Lunch Times Ms. Green Ongoing

6. Graph and Report Data Reiko Ongoing

7. Report to Staff Team Staff Meeting

Sixth Grade Lunch Noise

Seventh Grade Lunch Noise

Eighth Grade Lunch Noise

Improving Decision Making

Problem Solution

From

To

Problem

Problem

Solving

SolutionAction Plannin

g

How?

Right Data/Format/Time/PeopleWhat is the right data? What would be the right format? What is the right time (schedule) to bring the data? Who are the right people to be discussing and using this data to address issues?

How?

Right Questions

The statement of a problem is important for team-based problem solving.

Everyone must be working on the same problem with the same assumptions.

Problems often are framed in a “Primary” form. That form creates concern, but is not useful for problem-solving.

Frame primary problems based on initial review of data

Use more detailed review of data to build “Solvable Problem Statements.”

What are the data we need for a decision?

Precise problem statements include information about the following questions:

What is the problem behavior?How often is the problem happening?Where is the problem happening?Who is engaged in the behavior?When is the problem most likely to occur?Why is the problem sustaining?

Primary versus Precision Statements

Primary StatementsToo many referrals

September has more suspensions than last year

Gang behavior is increasing

The cafeteria is out of control

Student disrespect is out of control

Precision StatementsThere are more ODRs for aggression on the playground than last year. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

Primary versus Precision Statements

Precision Statement:

What? More ODRs for disruption.Where? In the hallway.Who? A large number of students across grade levels.When? After morning class.Why? To get access to peer attention.

There are more ODRs for disruption (loud, rowdy behavior) in the hallway. These are most likely to occur during after morning class, with a large number of students across grade levels, and the disruption is related to getting peer attention.

How? Solution Development & Action Planning

Prevention— how can we avoid the problem context?Who? When? Where?Schedule change, curriculum change, etc.

Teaching— how can we define, teach, and monitor what we want?Teach appropriate behaviorUse problem behavior as negative example

Recognition— how can we build in systematic rewards for positive behavior?

Extinction— how can we prevent problem behavior from being rewarded?

Consequences— what are efficient, consistent consequences for problem behavior?

How will we collect and use data to evaluate:Implementation fidelity?Impact on student outcomes?

Solution DevelopmentSolution

Component Action Step(s)

PreventionHow can we avoid the problem context?

Example: Schedule Lunch Times, Change Lighting

TeachingHow can we define, teach, and monitor what we want?

Example: Build “Quiet” Curriculum, Buy Decibel Meter, Teach Hallway Expectations/ Reminders

RecognitionHow can we build in systematic rewards for positive behavior?

Example: Three days of quiet in hallway results in an extra five minutes of social time (at lunch or at end of school)

ExtinctionHow can we prevent problem behavior from being rewarded?

Example: Public posting of results

Corrective Consequence

Consequences—what are efficient, consistent consequences for problem behavior?Example: Continue current system (Minor/Major ODR)

Data collection

Implementation fidelity?Example: Walkthrough report, observation, self-assessment

Impact on student outcomes?Example: SWIS ODR Data

Solution Components

What are the action steps? Who is Responsible? By When? How will fidelity be

measured? Notes/Updates

Prevention

Schedule Lunch Times, Change Lighting

Custodial staff to adjust lightingPrincipalto adjust schedule

Ongoing

Nov 10

New lunch scheduleWalkthrough report

Teaching

Build “Quiet” Curriculum, Buy Decibel Meter, Teach Hallway Expectations/ Reminders

Ben & Mary Nov 12 Permanent productStaff Self Assessment

Recognition

Continue current acknowledgment system and add an extra five minutes of social time (at lunch or at end of school) after three days of quiet in hallway

Reiko & Principal Nov 9 (announcements & chart up)

Announcement madeChart made

Extinction Public posting of results of decibel readings

Reiko Ongoing Posted chart

Corrective Consequence

Continue current system (Minor/Major ODR)

Hallway and Cafeteria supervisors

Ongoing SWIS ODR Reports

What data will we look at?

Who is responsible for gathering the data?

When/How often will data be gathered?

Where will data be shared? Who will see the data?

Data Collection

ODR recordSupervisor weekly report

SWIS Data Entry person and Principal share report with supervisors

Weekly In supervisor meeting and posted in the faculty lounge on PBIS bulletin board

All staff

Precise Problem Statement: Many students across grade levels are engaging in disruptive (loud, rowdy) behavior in the hallway after morning class, and the behavior is maintained by peer attention.

Goal: Reduce hallway ODRs by 50% per month (currently 24 per month average)

Most frequently misunderstood and overlooked component!Example: public posting of results will reduce likelihood of

payoff that previously reinforced this behavior

Value of this work/process

Quality Improvement is Continuous Improvement

The value of Data-Based Decision Making is: Quality Improvement Cycle of continuous Improvement

Improving what? fidelity of implementation, social climate, learning environment, student learning, attendance, grades)

Example: Hallway Study, recoup teaching time, improving social climate (for staff and students)

Show Videos

ActivityReflection Activity - Shapes

Lunch BreakGo eat!

ActivityData Treasure Hunt

Different Data for Different Decisions

Decision-Making for Quality Improvement

Outcome DataDiscipline Data for Short-Term Improvement

Progress Monitoring (formative)Universal Screening

Discipline Data for Long-Term ImprovementAnnual summarization and review of Strengths, Weaknesses, and Planning (summative)

Decision-Making for Quality Improvement

Fidelity DataFidelity Data for Short-Term Improvement

Progress MonitoringUniversal Screening

Fidelity Data for Long-Term ImprovementAnnual Assessments

Decision-Making for Quality Improvement

Outcome Data• Discipline (e.g., referrals)• Academic• Attendance• Climate/Culture• School Safety

Fidelity Data• Team/Self Assessments• Walk-through reports• PBIS Assessment (e.g.,

SET, Self Assessment, BoQ, TIC)

Decision-Making for Quality Improvement

Outcome Data• Discipline Data for Continuous

System Improvement and Progress Monitoring (formative)

• Discipline Data for Universal Screening of Student Needs

• Discipline Data for Evaluation (summative) of Strengths, Weaknesses, and Planning

Fidelity Data• Fidelity Data for Continuous

Improvement• Fidelity Data for Evaluation

Connecting Outcomes & Fidelity

Lucky Sustaining

Positive outcomes, low understanding of how we achieved them

Replication of success unlikely

Positive outcomes, high understanding of how we achieved them

Replication of success likely

Losing Ground Learning

Undesired outcomes, low understanding of how we achieved them

Replication of failure likely

Undesired outcomes, high understanding of how we achieved them

Replication of mistakes unlikely

Out

com

es

Fidelity

Sustaining

Positive outcomes, high understanding of how we achieved them

Replication of success likely

Discipline Data for a cycle of Continuous quality improvement

Short-Term vs. Long-Term Quality Improvement

Quality improvement requires two levels of analysis/use:

Short-term (sometimes called progress monitoring) is using data regularlyLong-term (sometimes called evaluation or annual assessment) is summarizing data for a big picture

Harbor Haven Middle School

565 studentsGrades 6, 7, 8

Harbor Haven Middle School

Is there a problem?

If so, what is it?

Problem

Dashboard

Harbor Haven Middle School

Median

Harbor Haven Middle School

HallwayCaféBusGym

10:0011:00-12:30

DefianceHarassmentTheft

School-wide Data

Harbor Haven Middle School

School-wide Data

Majority of referrals come from 6th and 7th grades.

11 students have more than 2 referrals.

What Do I Know?

What? Defiance and harassment.

Where? Hallways and cafeteria.

Who? A large number of students. Majority of referrals are from 6th grade.

When? 10:00 and 11:00-12:30.

What Do I Know?

I know pieces of information.

But, I do not know if any of this information is connected.

I need to drill down to look for connections.

Data Drill Down

Use the information from the SWIS Dashboard to drill

down and analyze data.

Change the graph type to change the

analysis.

Data Drill Drown

Cafeteria and harassment are connected.

Change the graph type to change the

analysis.

Data Drill Down

Cafeteria, harassment, and the time range 11:30-12:00

are connected.

Change the graph type to change the

analysis.

Many students are engaging in harassment in the cafeteria during the 11:30-12:00 time range (6th grade lunch), and the behavior is maintained by peer attention.

Data Drill Down for Connections

Precise Problem Statement & Solution Development

Cafeteria Harassment 11:30-12:00

Obtain Peer Attention

Solution DevelopmentTarget Area(s): Harassment in the cafeteriaGoal: Reduce referrals for harassment in the hall & cafeteria by 50%

Solution Component Action Step(s)

Prevention Maintain the current lunch schedule, but shift 6th grade classes to balance numbers.

Teaching Teach behavioral expectations in cafeteria

Recognition Establish “Friday Five”—an extra 5 minutes of lunch on Friday for five good days.

Extinction Encourage all students to work for “Friday Five”, making a reward for problem behavior less likely.

Corrective Consequence Active supervision and continued early consequence (ODR)

Data collection Maintain ODR record and supervisor weekly report

Solution Components

What are the action steps? Who is Responsible? By When? How will fidelity be

measured? Notes/Updates

Prevention

Maintain current lunch schedule, but shift 6th grade classes to balance numbers.

Principal to adjust schedule and send to staff

January 10 Email to staff

Teaching

Teach behavioral expectations in the cafeteria

Teachers will take class to cafeteria; cafeteria staff will teach expectations

Rotating schedule on January 15

Sign-up sheet for scheduled times

Recognition

Establish “Friday Five”—extra 5 min. of lunch on Friday for 5 good days

School counselor and Principal will create a chart and staff extra recess

Principal to give announcement on intercom on Monday

Announcement madeChart made

Extinction

Encourage all students to work for “Friday Five”—make reward for problem behavior less likely

All staff Ongoing

Corrective Consequence

Active supervision and continued early consequence (minor/major ODR)

Hallway and Cafeteria supervisors

Ongoing

What data will we look at?

Who is responsible for gathering the

data?

When/How often will data be gathered?

Where will data be shared?

Who will see the data?

Data Collection

ODR recordSupervisor weekly report

SWIS Data Entry person and Principal share report with supervisors

Weekly In supervisor meeting and posted in the faculty lounge on PBIS bulletin board

All staff

Precise Problem Statement: Many students are engaging in harassment in the cafeteria during the 11:30-12:00 time range (6th grade lunch), and the behavior is maintained by peer attention.

Goal: Reduce cafeteria ODRs by 50% per month (currently 24 per month average)

Discipline Data for Universal Screening

Using SWIS Data for Decision Making

Universal Screening ToolProportion of students with

0-1 Office Discipline Referrals (ODRs)2-5 ODRs6+ ODRs

Progress Monitoring Tool

Compare data across timePrevent previous problem patterns

Using the Referrals by Student as a Universal Screening Tool

Research Study on Early Intervention

Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun0

2

4

6

8

10

12

0-12-56+

Cum

ulati

ve M

ean

OD

Rs

Cumulative Mean ODRs Per Month for 325+ Elementary Schools 08-09

Jennifer Frank, Kent McIntosh, Seth May

Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun0

2

4

6

8

10

12

0-12-56+

Research Study on Early InterventionCu

mul

ative

Mea

n O

DRs

Cumulative Mean ODRs Per Month for 325+ Elementary Schools 08-09

Jennifer Frank, Kent McIntosh, Seth May

The “October Catch”

Discipline Data for Long-Term Continuous Improvement

SWIS Year-End Report

Fidelity Data for Short-Term Continuous Quality Improvement

PBIS Assessment Reports

Fidelity Data for Universal Screening

How would we ensure that the Universal Screening occurred?

Who is responsible to gather the data for the teamWhat is the schedule for reviewing this data (for purposes of universal screening)?

Fidelity Data for Long-Term Continuous Improvement

Two-fold

TBD

Activity