Joseph D. Otter LMSW, Regional PBIS Specialist

Eastern Region

Communities of One Project SPSE New Coaches Training Day 2

March 24, 2009 Mineville, NY

AcknowledgementsOSEP Center on Positive Behavioral Interventions & Supports Technical Assistance Center at University of Oregon

National Network of Partnership Schools (NNPS) Johns Hopkins University

Illinois EBD/PBIS Network

NYS-PBIS Initiative

Today’s Objectives• Discuss best practices of data collection and use

• Identify and describe data sources to be used by PBIS teams

• Distinguish between different kinds of data

• Become better data-based decision makers

• Network and problem solve

Today’s Agenda

• Welcome, Introductions & Evaluation Data• Warm Up• Kinds and types of data• Data Sources: SET, TIC, CTIC, EBS• Data-based Action Planning• Office Discipline Referral Data as a microcosm• Presenting Data• Networking/Closing

PBIS Competency Areas:What is PBIS?

Systems, Data, Practices Three Tiers of Support

Data Competencies Component 1: On-Going Assessment, Planning,

Evaluation Component 2: Intervention Planning at all three tiers

Systems Competencies Component 1: Triangle – Three tiers of support Component 2: District support & visibility

Practices Competencies School-Wide System Targeted System Intensive System

Principles of Data-based Decision-Making

• When you collect data from people, always show people how you used it – you asked them to work or change for you to get it

• Data-based decision making in an ethical practice

• Only collect what you plan on using

Principles of Data-based Decision-Making

• Whenever possible, use data for decision-making

• Data must be used ethically and factually (people can use it to “say anything” with enough twisting)

• Using data often results in needing to look at more data and ask more questions

Process Data• Tools & instruments that measure action

steps taken and the impact of those action steps

• A way of determining if what needs to occur to “set the stage” for outcome data is occurring

• Good for showing some results to vested stakeholders on the way to outcome data

• Data on if the implementers behavior has changed

Outcome Data• The ways of measuring achievement of

desired goals

• The data that stakeholders are truly interested in and that will get funding & district support

• Can be– Changes in perception– Changes in behavior– Changes in achievement

Data for decision-making in the adoption and maintenance of

School-wide PBISSYSTEM PROGRESS

– Yearly EBS/PBIS Survey (EBSSAS)– School-wide Evaluation Tool (SET)– Coaches Checklist CTIC– Team Checklist TIC– Other

STUDENT PROGRESS– Office Referrals (ODRs) and other discipline data – Academic Data

School-wide Evaluation Tool

• Conducted by trainer evaluator

• Interviews with staff and students

• Observation

• Interview with administrators

The interviews…

1. Focus on definitive answers, not perception

2. Ask about behaviors and knowledge

3. Emphasize being quantifiable (can be scored)

4. Seek specifics

System-wide Evaluation Tool (SET) Categories

1. Expectations defined

2. Behavioral Expectations Taught

3. On-going Reward System

4. System for Responding to violations

5. Monitoring & Decision-making

6. Management

7. District-Level Support

ES Cohort 1School-wide Evaluation Tool (SET)

% “In-Place” per SET Component 02-03 vs. 03-04

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 ts

SET Components

1) Expectations Defined

2) Expectations Taught

3) Rewards System

4) Violations System

5) Monitoring

6) Management

7) District Support

ts = Average % “In- Place” for ALES or “Total Score”

Cohort 1 SchoolsSchool-wide Evaluation Tool (SET)

% “In-Place” per SET Component 02-03 vs. 03-04

0

10

20

30

40

50

60

70

80

90

1 2 3 4 5 6 7 8

2002-2003

2003-2004

Team Implementation Checklist (TIC)

• Monitors & guides activities for implementation and assist in development of action plan– Start-up Activities

– On-going Activities

– School, Family & Community Partnerships

• Self-report data• Factual• Objective

TEAM IMPLEMENTATION CHECK LISTSTART UP ACTIVITY

NYS

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

EstablishCommitment

Establish &Maintain Team

Self Assessment Establish School-wide Expectations

EstablishInformationSystems

Build Capacity forFunction-based

Support

START UPACTIVITY TOTAL

SCORE

Perc

enta

ge Im

plem

ente

d

October December March May

Total Teams:Respondents

Oct: Dec: March: May

2002-Cohort: 1

Capital/North 0 8 6 6

TEAM IMPLEMENTATION CHECK LISTSTART UP ACTIVITY

NYS

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

EstablishCommitment

Establish &Maintain Team

Self Assessment Establish School-wide Expectations

EstablishInformationSystems

Build Capacity forFunction-based

Support

START UPACTIVITY TOTAL

SCORE

October January April

Total Teams:

RespondentsOct: Jan: April:

2003-

Cohort 1Capital/North

10

9 7 8

TEAM IMPLEMENTATION CHECK LISTON GOING ACTIVITY

NYS

0%

20%

40%

60%

80%

100%

120%

PBIS team meetsat least monthly

PBIS team givenstatus report tofaculty monthly

Activities PBISaction plan

implemented

PBIS action planassessed for

accurate.Implementation

Effectiveness ofaction plan

implementationassessed

PBIS dataanalyzed

ON GOINGACTIVITY TOTAL

SCORE

Perc

enta

ge Im

plem

ente

d

October December March May

Total Teams:Respondents

Oct: Dec: March: May

2002-

Cohort 1

Capital/North

10

0 8 6 6

TEAM IMPLEMENTATION CHECK LISTON GOING ACTIVITY

NYS

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

PBIS team meets atleast monthly

PBIS team given statusreport to faculty monthly

Activities PBIS actionplan implemented

PBIS action planassessed for accurate.

Implementation

Effectiveness of actionplan implementation

assessed

PBIS data analyzed ON GOING ACTIVITYTOTAL SCORE

October January April

Total Teams:

Respondents

Oct: Jan: April:

2003-Cohort: 1

Capital/North

10

9 7 8

TEAM IMPLEMENTATION CHECK LISTFAMILY INVOLVEMENT

NYS

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Family rep involved at

all levels of PBIS

Family rep involved in

all PBIS trainings

Families informed of

school-wide

expectations

Families informed of

school-wide rewards

positive behavior

Families informed of

school-wide

consequence

undesirable behavior

Commune with fam is

positive and strength

based

Families involved as

active participants in

interventions

Families received info

on community

resources

FAMILY

INVOLVEMENT TOTAL

SCORE

October January April

Total Teams:Respondents

Oct: Jan: April:

2003-

Coho 1Capital/North

10

9 7 8

COACHES' IMPLEMENTATION CHECK LIST (CTIC) TEAM SUSTAINABILITY

NYS

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

TEAM SUSTAINABILITY TOTALSCORE

1. Facilitates Admin supupport,awarness, involvement in PBIS

2. Support Internalcoach w/duties: mtgs, prep

3. Increase bldg & cmmtyawrness PBIS

Per

cen

tag

e Im

ple

men

ted

October January April

Capital/North Country

Cohort: 1 &

2003-2004

Total Coaches:

Respondents

4

Oct: Jan: April:35 6

Effective Behavior Support School-wide Assessment Survey (EBSSAS)

• Initial & annual assessment of effective behavior support systems in your school

• Examines status & need to improvement• Looks at four behavior support systems

~ school-wide discipline systems

~ non-classroom management systems

~ classroom management systems

~ individual students

Survey Completion – In Person

• All staff at staff meeting

• Individuals from a representative group

• Team member-led focus group

• Done independently

• 20-30 minutes

• Check current status on left

• Check priority for improvement on right

Survey Completion – Online

• Either at staff convenience during a survey period or all at once (computer lab)

• Done independently

• 20-30 minutes

EBS Survey results

• Annual action planning

• Internal decision making

• Assessment of change over time

• Awareness of building staff perceptions

• Team validation

• Buy-In

Staff receive regular feedback on behavior data.

0

5

10

15

20

25

30

35

HP Med Low InP Part Not

Options exist to allow classroom instruction to continue when problem

behavior occurs.

0

5

10

15

20

25

30

35

40

HP Med Low InP Part Not

Problem behaviors receive consistent consequences.

0

5

10

15

20

25

30

35

HP Med Low InP Part Not

Effective Behavior Support School-wide Survey (Current Status)

48% 50%

38%33%

38%42%

46%

41%

14%8%

16%26%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

School-w ideSystems

ClassroomSystems

NonclasroomSettings

IndividualStudentSystems

Not in Place

Partially In-Pace

In-Place

Effective Behavior Support School-wide Survey (Priority for Improvement)

42% 43% 47%55%

37% 34%35%

33%

22% 23%18%

12%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

School-w ideSystems

ClassroomSystems

NonclasroomSettings

IndividualStudentSystems

Low Need

Medium Need

Hi-Need

1. School-wide expectations apply to non-classroom settings.2. School-wide expected behaviors are taught in non-classroom settings.3. Supervisors actively supervise(move, scan, interact) students in non-classroom settings.4. Rewards exist for meeting expected student behaviors in non-classroom settings.5. Physical/architectural features modified to limit unsupervised settings, unclear traffic patterns, inappropriate entrance/exit from school.6. Scheduling of student movement ensures an appropriate # of students in non-classroom spaces.7. Staff receive regular opportunities for developing and improving active supervision skills.8. Status of student behavior and management practices are evaluated quarterly from data.9. All staff are involved directly or indirectly in management of non-classroom settings.

Non Classroom Systems at LHS

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 9

In place

Partially in place

Not in place

LHS Non Classroom Staff Proirites

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 9

Low

Medium

High

Organizing Behavioral Data

Use whatever data already exists

THE BIG FIVE1. Problem Behavior2. Location3. Time of Day4. Referrals by Student5. Average Per Day/Per Month

Central Region ~ Rural School (gr 3-6) ~ 2002-20032002-2003 Totals by Incident Type

0

20

40

60

80

100

120

140

160

180

Type of Incident

To

tal N

um

ber

of

Refe

rrals

Total

0

200

400

600

800

1000

1200

1400

Classroom Bus

Location

Office Referrals by Location1994-1995

Infractions by time of day

0

2

4

6

8

10

12

14

8:30 9:30 10:30 11:30 12:30 1:30 2:30 3:30

NovemberDecember

Discipline Referrals by StudentMiddle School - Southern Illinois

0

50

100

150

200

250

1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52

Number of Referrals

Num

ber

of

Stu

den

ts

• 50% of students account for all behavior referrals

• 10% of students account for 61% of all behavior referrals

• 5% of students account for 41% of all behavior referrals

0

100

200

300

400

500

Sep Oct Nov Dec Jan Feb Mar Apr May

Months

Office Referrals by Month1994-1995

0

5

10

15

20

25

Sep Nov Jan Mar May

Months

Office Referrals/School Day by Month1994-1995

Data-Based Decision Making• Office discipline referral data

• Staff/Faculty input/survey data

• Student input/survey data

• Nurses office

• Family/Community input/survey data

• Academic data

• Kid to Kid

• Kid to Adult

Survey of Respectful Behaviordone by Dr. Rob March

• Participants– 980 middle school students– Chicago, IL– Approximately 63% of students at school

receive free or reduced lunch

• Survey asked students, “What are some ways that teachers show you respect?”

Top 12 Answers

• Of the over 2900 responses, the ones listed were written by 50 students or more.

• 1. Talk privately to students when a problem occurs.

• 2. Use a calm tone of voice, even when they are upset. (No yell)

• 3. Respect personal space (Don't touch, grab, eyeballing, crowd)

Top 12 Answers (continued)

• 4. Listen without interrupting.• 5. Have a sense of humor.• 6. Display student work around the

classroom/school.• 7. Prepare exciting lessons. • 8. Let parents/guardian know student did a

good job sometimes (see a balanced picture).

Top 12 Answers (continued)

• 9. Use student's name when talking to them• 10. Be available during non-classroom times• 11. Return work promptly• 12. Talk sincerely - no sarcasm or “eye

rolling”

• Worth noting: Acknowledge birthday received multiple mentions.

Use Data to Drive Decisions

• Data makes you ask questions• What are the right questions?

– Who, What, Where, When, Why & How?

• What information is needed to answer the questions?

• What are the possible interventions?

What is the smallest change that can produce the biggest impact?

Determine Needs

Analyze data and ask questions

WHO are the students committing this infraction?WHAT is the most efficient solution?WHERE are we experiencing the most problems?WHEN is this behavior not occurring?WHY is this a priority for us?HOW much more data needs to be collected?

Question #1: Are we doing what we should be doing?Question #2: Is it working?

• Are we satisfied with the behavior patterns of students?

• Are we using best practices of DEFINE, TEACH, REMIND, ACKNOWLEDGE & RETEACH in school-wide behavior support?– What are we doing that is working and should

be retained?– What are we NOT doing that would fit our

setting and make a big difference?

Intervention (Course of Action)

What do we want instead (what changes do we want to see?)

Are we trying to change the behavior of children or adults?

What other data do we need to collect (what sources do we have and what are we using)?

Determine priorities Brainstorm ideas (at least 10) - keep it

simple Plan implementation

Who will be involved? When will it happen?

Always Remember the Three Questions for

Active Decision-making:

• Are we doing what we should be doing?

• Is it making a difference?• What is the smallest change that

will make the biggest difference?

Determine Effectiveness and Success

What is our definition of success? What will it look like?

Set achievable, reasonable goals How and how often will we

measure success and effectiveness?

Do we need to modify the plan? When do we move on?

Steps of Data Collection, Analysis, and Use

1. Identify sources of information and data

– Office discipline referrals– Attendance, tardies– Detentions, in-school-suspensions, out-of-school

suspensions, expulsions– Academic performance (classwork, homework,

grades, classroom tests) – EBS/PBIS survey– TIC Team Implementation Checklist– SET Systems-wide Evaluation Tool– Other

Data

Steps of Data Collection, Analysis, and Use

2. Summarize/Organize Data• Systematic, consistent collection

3. Continual Data Analysis• Data points at every meeting

4. Build action plan based on data trends

Data

Initial Work on DisciplineData Collection for Schools

• Office Discipline Referral Form – does it contain all needed information? What is missing? What needs to be changed? (see handout)

• What is our current data collection system? Does it have the capacity to provide graphs and charts for the “Big 5” plus other areas we need?

• Do we have a clear definition of infractions? Is there a clear definition for all staff what behaviors are classroom managed vs. office managed?

• Review EBS survey. Discuss plans for involving staff in survey.

Presenting data• It is best to provide a few well designed

examples of data as opposed to a lot of data that is hard to navigate

• Graphing makes an immeasurable difference!

• Always be prepared to discuss the data sample & questions and know the example intimately

• Ensure that the sample is relevant to audience

Greenbush AcademyPhysical Restraint Data Comparisons

• School Year - Number of Physical Restraints

• 2003-04 243 • 2004-05 224• 2005-06 150• 2006-07 78• 2007-08** 4• ** thru December 31, 2007

Restraints by Year

0

50

100

150

200

250

2003-04 2004-05 2005-06 2006-07 9-07-12-07

SPSE Consultant

Joseph D. Otter LMSW

Capital Region BOCES

Suite 102

900 Watervliet Shaker Road

Albany, NY 12205

Phone: (518) 464-3974

Fax: (518) 464-3975

Email: jotter@gw.neric.org