Susan BarrettJerry Bloom

Sheppard Pratt Health Systemwww.pbis.org

www.pbismaryland.org

Building Evaluation Capacity for States and Districts

2

PBS Systems Implementation Logic

Leadership Team

Management Team

FundingMarketing

Visibility Political Support

Training Coaching Evaluation

Active Coordination

Local School Teams/Demonstrations

Braiding Initiatives

EvaluationWhat are your questions?Do you have the tools to answer?Can you get the answer quickly?Easy, Efficient, RelevantEconomic BenefitsBehaviorAchievementRegular Feedback to all Stakeholders- MD

example

•How many schools are implementing SW-PBS ? 1.Number by cohort2.Number by local district3.Number by implementation phase4.Number of Active and Inactive by local district5.Number of schools reporting data (rate of return)

•Are schools implementing SW-PBS? 1.State and District needs to determine process tool (i.e. team implementation checklist, implementation inventory, benchmarks of quality, EBS self assessment)

•Are schools in beginning or advances stages of implementation?1.Implementation Phase Inventory used to track level

•How do reported rates of problem behavior in local district or state compare to the national average? (SWIS)1.by Elementary, Middle, High, Alternative

•What is the percentage of students who receive 0 or 1 Office referrals/year? (SWIS)1.By Elementary, Middle, High, Alternative

•What are the problem behaviors? (SWIS)1.Elementary, Middle, High, Alternative

•Where are problem events occurring? (SWIS)1.By Elementary, Middle, High, Alternative

•What time do problem events occur? (SWIS)1.By Elementary, Middle , High, Alternative

•What are the discipline patterns by gender? (SWIS)•What are the discipline patterns by ethnicity? (SWIS)•What are the discipline patterns for students on an IEP? (SWIS)

Evaluation TemplateFairfax County Example

Forms/Surveys and ChecklistsTier I

Surveying Staff- EBS Survey

Fidelity Checks- Team Implementation Checklist, SET, BOQ, IPI

Safety Surveys- Oregon Safety Survey

Outcome Measures- -ODR, Suspension, Attendance, Special

Ed, GPA, Honor Roll, AYP

Forms /Checklists to TRAINING MATERIALS

EXAMPLE– MD Training Materials

Tools

State Database-Data Entry, Report Generation

Various Levels POC Coach State Team

www.pbssurveys.orgwww.pbseval.orgwww.swis.org

Marketing and VisibilityWho are your stakeholders?Do you have a spokesperson?Using the data to create newsletters,

presentations, fact sheets, elevator business cards- important you can get access to what you need to make your case on the fly!!

www.pbismaryland.orgMaryland Newsletter

Be Careful

11

Multiple levels of VisibilityState and Local Level: Presentations,

Trainings, Stakeholder meetings, Interagency efforts, (Transformation; Mental Health Integration; Wraparound)

Multiple Media: Visual, Face to Face, Written, Website

Multiple Audiences: School Administrations and Instructional Leaders; University staff; Legislators, Potential alternative funders; State and Local Political appointees; Juvenile Justice; Vendors in the System of Care; Parent and other advocacy organizations; Community Members

PBIS Maryland Partner Meeting

Management Team meets with Assistant Superintendent of Student Services and Special Education

7

18

30

5257

81

90

116110

310

18

47 47

64

82

113110

43%

56%60%

90%

82%79%

91%

97%100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0

20

40

60

80

100

120

140

1999 2000 2001 2002 2003 2004 2005 2006 2007

Trained and Implementing by Cohort

Trained Implementing % of Trained Implementing

NOTE: Retrained schools will be counted in the most recent retraining year only.

13

65

50

72

18

5

19

9

31

7

27

5

11

48

4

32

50

7 8

30

12

2

1412

1013

64

33

70

18

2

17

6

30

1

27

3

11

45

4

31

37

7 6

24

9

2

1311 10

100% 98%

66%

97%100%

40%

89%

67%

97%

14%

100%

60%

100%

94%

100%97%

74%

100%

75%80%

75%

100%

93% 92%

100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0

10

20

30

40

50

60

70

80

Alle

gany

Anne

Aru

ndel

Balti

mor

e Ci

ty

Balti

mor

e Co

.

Calv

ert

Caro

line

Carr

oll

Ceci

l

Char

les

Dor

ches

ter

Fred

eric

k

Gar

rett

Har

ford

How

ard

Kent

Mon

tgom

ery

Prin

ce G

eorg

es

Que

en A

nnes

Som

erse

t

Spec

ial

St. M

arys

Talb

ot

Was

hing

ton

Wic

omic

o

Wor

cest

er

Implementing/Inactive by LSS

Trained Implementing % of Trained Implementing

91% Rate of Return

0%10%20%30%40%50%60%70%80%90%

100%

1999 2000 2001 2002 2003 2004 2005 2006 2007

% of schools submitting data

30

60

97286

IPI Fall 2007

Preparation Initiation Implementation Maintenance

473

# of schools reporting:

Evaluation Question: Are schools in beginning or advanced stages of implementation?Data Source: Implementation Phases InventoryOverall, the IPI data suggest a relatively advanced level of self-reported implementation among the schools in Maryland

Evaluation Question: Are schools implementing School-wide PBIS?Data Source: Team Implementation Checklist (Target Criterion = 80%)Based on the total average of the 327 checklists submitted, school teams report that 78% of items are in place.

T eam Implementation C hec klis tS pring 2007

85.6 86.874.8 76.7 71.9

95.0

80.8 77.0 75.5

51.9

78.3

63.170.6

90.0 85.0 85.0 90.0100.0

83.394.0

72.5

0.010.020.030.040.050.060.070.080.090.0

100.0

Alle

gany

Ann

eA

rund

elB

altim

ore

City

Bal

timor

eC

o.

Cal

vert

Car

olin

e

Car

roll

Cha

rles

Fre

deric

k

Har

ford

How

ard

Mon

tgom

ery

Prin

ceG

eorg

esQ

ueen

Ann

es

Som

erse

t

Spe

cial

St.

Mar

ys

Tal

bot

Was

hing

ton

Wic

omic

o

Wor

cest

er

LS

S A

vera

ges

Evaluation Question: Are schools implementing SW-Positive Behavior Support?Data Source: SET (Objective Criterion = 80%)

-All regions met objective criterion-A pre-post comparison regional average shows a 69% increase.

K-6 6-9 9-120

0.2

0.4

0.6

0.8

1

1.2

0.22

0.600000000000001

0.5

0.19

0.45 0.48

0.34

0.980.93

Office Discipline Referrals (ODRs)Per 100 Students Per School Day

MD Average 2006 - 2007 MD Average 2007 - 2008 National Average

Data from the School-Wide Information System (SWIS)

Evaluation Question: Do 80% of the students in MD Elementary PBIS schools using SWIS receive 0 or 1 office referrals?Data Source: SWIS PBIS Elementary schools in Maryland using SWIS report that 92% of their students receive 0 or 1 office referral.

Evaluation Question: Where is the location of behavior problems in Middle Schools?Data Source: SWISThe majority of the problem behaviors reported occur in the classroom followed by hallways.