Catherine Bradshaw, Ph.D., M.Ed. Johns Hopkins Center for the Prevention of Youth Violence (CDC) Johns Hopkins Center for Prevention and Early Intervention

Catherine Bradshaw, Ph.D., M.Ed.Johns Hopkins Center for the Prevention of Youth Violence (CDC)

Johns Hopkins Center for Prevention and Early Intervention (NIMH)

Department of Mental Health

Penn State IES Fellows: May 17, 2011

Promises and Pitfalls in School-wide Positive Behavioral Interventions and

Supports (PBIS)

PBIS Model: Whole-school PreventionApplication of behavioral, social learning, &

organizational behavioral principles Clear behavioral expectations (e.g., ready, responsible, and respectful)Positive rewardsProcedures for managing disruptions

(Horner & Sugai, 2001; Lewis & Sugai, 1999; Sugai & Horner, 2006)

Be Responsible

Respect Yourself

Respect Others

Eagle Tickets



Focus on changing adult behaviorEmphasizes staff buy-inTeam-based & data-based processConsistency in discipline practices


Observe Problem Behavior

Warning/Conference with Student

Use Classroom Consequence

Complete Minor Incident Report

Does student have 3 MIR slips

for the same behavior in the same quarter

•Preparedness•Calling Out•Classroom Disruption•Refusal to Follow a Reasonable Request (Insubordination)•Failure to Serve a Detention•Put Downs•Refusing to Work•Inappropriate Tone/Attitude•Electronic Devices•Inappropriate Comments•Food or Drink

•Weapons•Fighting or Aggressive Physical Contact•Chronic Minor Infractions•Aggressive Language•Threats•Harassment of Student or Teacher•Truancy/Cut Class•Smoking•Vandalism•Alcohol•Drugs•Gambling•Dress Code•Cheating•Not w/ Class During Emergency•Leaving School Grounds•Foul Language at Student/Staff

Write referral to office

Administrator determines

consequence

Administrator follows through

on consequence

Administrator provides teacher

feedback

Write the student a

REFERRAL to the main office

•Issue slip when student does not respond to pre-correction, re-direction, or verbal warning

•Once written, file a copy with administrator

•Take concrete action to correct behavior (i.e. assign detention, complete behavior reflection writing, seat change)

SIDE BAR on Minor Inc ident Repor t s

•Issue slip when student does not respond to pre-correction, re-direction, or verbal warning

•Once written, file a copy with administrator

•Take concrete action to correct behavior (i.e. assign detention, complete behavior reflection writing, seat change)

SIDE BAR on Minor Inc ident Repor t s

Is behavior office

managed?

ClassroomManaged

Office Managed

No YesOffice

vs.

Class

room

Man

aged

0

50

100

150

200

250

300

1 5 9 13 17 21 25 29 33 37 41 45 49

Teachers

Office Referrals by Teacher1994-1995

0

20

40

60

80

100

1 7 13 19 25 31 37 43 49 55 61 67 73 79

No. of Referrals

Office Referrals by Student1994-1995



Focus on changing adult behaviorEmphasizes staff buy-inTeam-based & data-based processConsistency in discipline practices

Can be implemented in any school level, type, or settingNon-curricular model – flexible to fit school culture & context

Coaching to ensure high fidelity implementationOn-going progress monitoring

Public health approach (universal / selective / indicated)Requires a shift from punitive/reactive to preventive


Academic Systems Behavioral Systems

1-5% 1-5%

5-10% 5-10%

80-90% 80-90%

Intensive, Individually Designed Interventions• Address individual needs of student• Assessment-based• High Intensity

Intensive, Individually Designed Interventions• Strategies to address needs of individual students with intensive needs• Function-based assessments• Intense, durable strategies

Targeted, Group Interventions• Small, needs-based groups for at risk students who do not respondto universal strategies• High efficiency• Rapid response

Targeted, Group Interventions• Small, needs-based groups for at- risk students who do not respond to universal strategies• High efficiency/ Rapid response• Function-based logic

Core Curriculum and Differentiated Instruction• All students• Preventive, proactive• School-wide or classroomsystems for ALL students

Core Curriculum and Universal Interventions• All settings, all students• Preventive, proactive• School-wide or classroom systems for ALL students and staff

Maryland’s Tiered Instructional and Positive Behavioral Interventions and Supports (PBIS)

Framework

(MSDE, 2008)

Maryland’s PBIS Organizational ModelSchool Level

826 PBIS Teams (one per school) ≈ 59% Team leaders (one per school) Behavior Support Coaches (560)

District Level (24)District Coordinators

State LevelState Leadership Team

Maryland State Department of Education (MSDE) Sheppard Pratt Health System Johns Hopkins University 24 Local school districts Department of Juvenile Services, Mental Hygiene Administration University of Maryland

Management TeamAdvisory Group

National LevelNational PBIS Technical Assistance Center

University of Oregon, University of Connecticut, & University of Missouri

State

District

School

Classroom

Student

(Barrett, Bradshaw & Lewis-Palmer, 2008; JPBI)

Group Randomized Trial of SW-PBIS

Funding Centers for Disease Control & Prevention National Institute of Mental Health Institute of Education Sciences

Sample37 voluntary elementary schools across 5 school districts

Enrollment 227-983; 60% Caucasian; 48% suburban; 41% urban fringe; 49% Title I

DesignGroup randomized effectiveness trial

21 PBIS & 16 “Focus/Comparison”Baseline plus 4 years (spring 2002 - spring 2007)

Data from 29,423 students & 3,563 staff Project Target

(Bradshaw et al., Prevention Science, 2009; School Psychology Quarterly, 2008; JPBI, 2010)

SET: PBIS Implementation Fidelity

43%

56%61%

49% 48%

37%

84%90% 91%

95%

0%

20%

40%

60%

80%

100%

Baseline Year 1 Year 2 Year 3 Year 4

Mea

n Sc

ores

(%

)

Comparison PBISNotes. No significant differences between groups at baseline, but differences at all other years at p<.05. Overall SET score: Wilks’ Λ = .38, F (4,32) = 13.36, p <.001, partial η2 = .63, d = 3.22. (Bradshaw, Reinke et al., ETC, 2007)

Brief Summary of SW-PBIS Training Effects on Fidelity

• High fidelity implementation and sustainability of PBIS

All trained schools reached high fidelity within 4 years (66% by the end of year 1)

All trained schools sustained high fidelity

(Bradshaw, Reinke et al., ETC, 2007; Bradshaw et al. Prevention Science, 2009)

Analysis of Climate Data: Organizational Health

Inventory (OHI)OHI: 37 item staff-report measure of 5 aspects of a healthy

functioning school (Hoy et al., 1991)academic emphasis - students are cooperative in the classroom, respectful of other

students who get good grades, and are driven to improve their skills staff affiliation - warm and friendly interactions, commitment, trust collegial leadership - principal’s behavior is friendly, supportive, openresource influence - principal’s ability to lobby for resources for the school institutional integrity - teachers are protected from unreasonable community and

parental demandsoverall OHI score (average of all items)

AnalysesLongitudinal analyses were conducted using a 3-level approach in Mplus 4.21

(Muthén & Muthén, 2006) Intent to treat approach (Lachin, 2000) & moderated by fidelity (SET) Adjusted for staff (sex, race, age) and school (FARMs, student mobility, faculty

turnover, & school enrollment) covariates on intercept and slope

0 1 2 3 42.25

2.50

2.75

3.00

3.25

3.50

Study Year

Ove

rall

OH

I

ComparisonPBIS

Effect of PBIS on Overall OHI

Note. Adjusted means from 3-level model. * Intervention effect on slope of overall OHI significant at p<.05.

*

Sig. change (.05)

Note. Adjusted means from 3-level model. * Intervention effect on slope of overall OHI significant at p<.05.

*

0 1 2 3 42.25

2.50

2.75

3.00

3.25

3.50

Study Year

Col

legi

al L

eade

rshi

p

ComparisonPBIS

Effect of PBIS on Collegial Leadership

Note. Adjusted means from 3-level model. * Intervention effect on slope significant at p<.05.

*Sig. difference (.05)

Brief Summary of SW-PBIS Training Effects on Climate

• PBIS training associated with significant improvements in staff members’ report of school climate / organizational health • Principal leadership, collegial relationships, academic emphasis,

recourse influence, institutional integrity, and overall OHI• Effect sizes ranged from .24 (AE) to .35 (RI)• OHI intercept and slope negatively correlated

Schools starting with lower levels of OHI tended to take longer to reach high fidelity, but improved the most

(Bradshaw, et al., SPQ, 2008; Bradshaw et al. Prevention Science, 2009)

Brief Summary of Impacts of SW-PBIS

on Student Outcomes

• Significant impacts for students: Significant reduction in school-level suspensions among the

PBIS schools Students in PBIS schools were 32% less likely to receive an

office discipline referral A positive trend for school-level MSA academic performance

was observed

(Bradshaw et al., JPBI, 2010)

Impact of SW-PBIS on Bullying and Rejection: HLM 3-Level Results

Teacher-Reported Bullying

Teacher-Reported Rejection

Coefficient SE Coefficient SEIntercept

Intercept 1.4029*** 0.0242 1.8174*** 0.0359Mobility 0.0001 0.0028 0.0021 0.0032Student Teacher Ratio -0.0057 0.0053 -0.0011 0.0088Faculty Turnover -0.0016 0.0034 0.0018 0.0034Enrollment 0.0089 0.0115 0.0309* 0.0141Special Education Status 0.1176*** 0.0268 0.3646*** 0.0367Ethnicity (Black) 0.2317*** 0.0316 0.1545*** 0.0288Grade Cohort -0.0564 0.0473 -0.1095* 0.0481FARMS 0.0846*** 0.0165 0.2347*** 0.0241Gender 0.2261*** 0.0183 0.2127*** 0.0176

Slope (Growth)Intercept 0.0326*** 0.0099 0.0767*** 0.0158SWPBIS Intervention -0.0230* 0.0088 -0.0339* 0.0145Mobility 0.0015 0.0009 0.0011 0.0009Student Teacher Ratio 0.0043* 0.0016 0.0042 0.0022Faculty Turnover 0.0023 0.0013 0.0028* 0.0013Enrollment -0.0114* 0.0037 -0.0110* 0.0048Special Education Status -0.0043 0.0089 0.0187 0.0121Ethnicity (Black) 0.0333*** 0.0059 -0.0024 0.0078Grade Cohort 0.0127 0.0161 0.0121 0.0181FARMS 0.0218*** 0.0051 -0.0018 0.0079Gender (male) 0.0188*** 0.0064 0.0056 0.0077

Note. Measure is Teacher Observation of Classroom Adjustment TOCA (teacher-reported) * p<.05, ** p <.01, *** p <.001

Impact of SW-PBIS on Teacher-Reported Perpetration of Bullying

Behaviors

(N = 12,334, γ = -.02, t = -2.60, p<.05, ES= -.11)

Impact of SW-PBIS on Teacher-Reported Student

Rejection

(N = 12,334, γ = -.03, t = -2.32, p<.05, ES= -.14)

SW-PBIS Intervention Status by Grade Cohort on Rejection

Control Younger

Impact of SW-PBIS on Teacher-Reported

Disruptive Behaviors

(N = 12,334, γ = -.02, t = -2.39, p<.05, ES = -.12)

Impact of SW-PBIS on Teacher-Reported

Concentration Problems

(N = 12,334, γ = -.03, t = -2.08, p=.046, ES=.09)

Impact of SW-PBIS on Teacher-Reported Emotion

Regulation Skills

(N = 12,334, γ = .03, t = -2.30, p=.045, ES = .10)

Impact of SW-PBIS on Teacher-Reported Positive

Behaviors

(N = 12,334, γ = .03, t = -2.11, p<.05, ES = .15)

Variations in the Impact of SW-PBIS

Is there variation in the impact of SW-PBIS based on the child’s baseline pattern of risk?Is SW-PBIS more impactful for some students than others?Parallel to ‘green’, ‘yellow’, and ‘red’ zone framework?

Used Latent Class Analysis (LCA) to examine variationLCA is a person centered approach

Groups participants with similar patterns among indicator variables into latent classes (McCutcheon, 1987)

Analysis conducted in Mplus 6.1 (Muthén & Muthén, 1997-2010)

Fitting the classesSubstantive theory as well as statistical support (Nylund et al., 2007)

5 indices: Akaike Information Criterion (AIC), Baysian Information Criterion (BIC; Schwartz, 1978), Sample Size Adjusted Baysian Information Criterion (SSA BIC; Sclove, 1987), Lo-Mendell-Rubin adjusted likelihood ratio test (LMR; Lo, Mendell, & Rubin, 2001), and the sample size adjusted LMR (Muthén & Muthén, 1997-2008)

Latent Class Analyses: Mean Baseline TOCA Scores

Baseline Teacher Ratings of Student Behavior (TOCA) N = 12,334

Mea

n (C

ente

red)

Variation in Impact by LCA Membership:

% within Class Experiencing (Untoward) Outcome

High Risk At Risk NormativeSocially-

Emotionally Skilled

TOCA item PBIS Control PBIS Control PBIS Control PBIS Control

Sent to Principals’ Office

71A 78 A 41 B 46 B 22 20 10 11

Rcvd Counseling for Inappropriate Behavior

57 54 25 C 30 C 11 11 5 5

Rcvd Counseling for Social Skills Development

50 56 30D 36D 18 18 9E 13E

Grade Retained 8 7 5 7 3 4 2F 4F

Referred to Special Education

36 39 23G 27G 13 13 8 6

Note. Numbers represent percents experiencing that outcome. Those sharing letters are significantly different at p<.05. N = 12,334

Conclusions of Main Effects

Several significant impacts of SW-PBIS on the school environment, staff, and students

Main effects on teacher reported adjustment (i.e., concentration problems, aggressive/disruptive behavior, bullying, rejection, emotion regulation, prosocial behavior)Although students in both groups (PBIS and Comparison) tended to

display higher rates of problem behaviors over time, students in PBIS schools faired better than those in comparison schools

Some indication that the intervention effects are strongest the earlier students are exposed to PBIS

Conclusions from Variations Results

Four classes (not 3) of risk patterns emergedHigh risk (6.6%), At risk (23.3%), Normative (36.5%), Social-

emotional Skilled (33.6%)Those in the ‘high risk’ and the ‘at-risk’ faired the worst

in the Control schools than in the SW-PBIS schoolsShows that main effects models may ‘wash out’ some

significant program impactsConsistent with an RtI framework, however even the

higher risk students are doing ‘better’ in an SW-PBIS environment

Examining Contextual Factors Associated with

Scale-upResearch AimsTo identify school- and district-level characteristics

which predict initial training, adoption, and the quality with which PBIS is implemented.Greater need at the school- and district-level would be

associated with initial training, but lower odds of adoption or quality implementation.

Greater district infrastructure and wealth would be associated with greater odds of all three outcomes.

(Bradshaw & Pas, accepted pending revision)

Participating SchoolsAll 24 MD districts participate in the PBIS initiative.825 traditional elementary schools across 22 districts

in the state were eligible to be included (i.e., two districts had three or fewer schools trained) of which 312 were trained.

236 schools across 17 districts had been trained AND provided data, indicating active participation and could be analyzed for the implementation outcome.

Outcome VariablesTraining in school-wide PBISA school team of at least 4 individuals, including an

administrator, attended the state’s two-day training event.

Adoption of school-wide PBISSchool was trained AND submitted implementation data in the

spring of 2008

Implementation of school-wide PBISThe Implementation Phases Inventory (IPI; Bradshaw, Debnam,

Koth, & Leaf, 2009):implementation and maintenance scales and overall score.

ConclusionsSchools with greater need were more likely to receive training,

and in some cases also adopt SW-PBIS.Maryland schools self-identify for training; lower-performing schools

seem to seek SW-PBIS as a way to improve their school.

Indicators of school disorganization do not serve as an obstacle to successful implementation of SW-PBIS.

The number of years since training and percent of certified teachers is associated with better implementation.

District variables are related to training and adoption, but not implementation.

(Bradshaw & Pas, accepted pending revision)

Next Steps: Integrating PBIS with…

PATHS to PAX & PBIS (NIMH, IES; Ialongo & Bradshaw) Integrate an evidence-based classroom management program (PAX/Good Behavior Game) and a social-

emotional learning curricula (PATHS) with PBIS Pilot work in 6 Baltimore City PBIS schools over 2 years

PBISplus Project (IES; Leaf & Bradshaw) 45 MD elementary schools all implementing school-wide PBIS Provides support to SSTs and teachers related to selection of evidence-based practices 3 years of on-site support and ‘coaching’ provided through a PBISplus Liaison

Safe and Supportive Schools Grant (MDS3) (USDOE; MSDE) Collect data on climate and student safety Conduct a 60 high school randomized trial of the integration PBIS with evidence-based violence

prevention, drug prevention, and truancy prevention programs

Double Check Cultural Proficiency & Student Engagement (IES; Bradshaw) Develop school-wide and classroom –based management system s to reduce disproportionality in PBIS

elementary and middle schools Provide professional development, adapt the Classroom Check-up (Reinke, 2007), support data-based

decision-making

Potential Challenges to the PBIS Model

What impact does the focus on a team-based adaptive process have on variation in implementation quality?

What are some strengths and limitations of the training model?What is the effect on the classroom?Is it a process or a program?Controversial role of rewards What is changing – students or environment?Too behaviorally focused? What about internalizing or social-

emotional factors?Is it too adult focused/driven?Cost benefit analysis – effect size vs. investment What to do when the universal isn’t enough?Will this work for urban schools or high schools?

AcknowledgementsJohns HopkinsPhil LeafKatrina DebnamChrissy KothMary MitchellElise Pas

Maryland State Department of Education Milt McKenna Andrea Alexander

Sheppard Pratt Health System Susan Barrett Jerry Bloom

Supported by NIMH (1R01MH67948-1A, P30 MH08643), CDC (1U49CE 000728 & K01CE001333-01), IES (R324A07118, R305A090307, & R324A110107 ) & USDOE

Contact InformationCatherine Bradshaw, Ph.D.Johns Hopkins Center for the Prevention of Youth ViolenceJohns Hopkins Bloomberg School of Public [email protected]

PBIS Resourceswww.PBIS.orgwww.PBISMaryland.org

Documents

Catherine Bradshaw, Ph.D., M.Ed. Johns Hopkins Center for the Prevention of Youth Violence (CDC) Johns Hopkins Center for Prevention and Early Intervention