Using PBIS Assessment to Create School and District

PBIS Evaluation Plans

Northern California PBIS SymposiumNovember 18, 2013

Are we doing what we said we would do?

Agenda

Overview of available surveys/tools on the PBIS Assessment website

Developing an Evaluation Action PlanDecisions on which tools to chooseDecisions on timelines

Assessment Example Plans

Developing Evaluation Reports

Questions/Comments

Acknowledgments

School-wide Positive Behavior Support: Implementers' Blueprint and Self-Assessment

Sugai, G., Horner, R.H., Algozzine, R., et.al. (2010).

Evaluation Blueprint for School-Wide Positive Behavior Interventions & Support

Algozzine, R., Horner, R.H, Sugai, G., et.al. (2010).

Fidelity Measures to Improve Implementation of Positive Behavioral Support

Tobin, T. J., Vincent, C. G., Horner, R. H., Rossetto Dickey, C., & May, S. A. (2012).

PBIS Evaluation Blueprint

“A hallmark of School-wide Positive Behavior Support (SWPBS) is a commitment to formal evaluation.”

“Evaluation is the process of collecting and using information for decision making.

Foundations

Evaluation is the process of gathering data toAnswer questionsMake decisions

Define the core questions/decisions, before selecting measures.

Provide an adequate context for interpreting the data.

Effective evaluation is cyclical (repeated)Use the evaluation process for improvement

Evaluation Cycle

Core Questions

Evaluation Plan

Data Collection

Evaluation Report

Core Evaluation QuestionsContext

What are/were the goals and objectives for SWPBIS implementation? (state/district capacity; school adoption; student outcomes)

Who delivered the training and technical assistance for SWPBIS implementation?

InputWhat professional development events were part of SWPBIS implementation support?

Was projected level of TA capacity provided (training/coaching)?

Who received the professional development (schools/ cohorts)?

Core Evaluation QuestionsFidelity

To what extent was Tier I SWPBIS implemented with fidelity ?

To what extent were Tier II and Tier III SWPBIS implemented with fidelity?

To what extent is the Leadership Team establishing core functions?

Leadership TeamActive Coordination

FundingVisibility Political

Support

Training Coaching Evaluation

Local School/District Teams/Demonstrations

BehavioralExpertise

Policy

Sugai et al., www.pbis.org

Core Evaluation QuestionsImpact

To what extent is SWPBIS associated with changes in student behavioral outcomes?

To what extent is SWPBIS associated with changes in academic performance, and dropout rates?

To what extent is district/state capacity established? (local training, coaching, evaluation, behavioral expertise)

To what extent is leadership and policy structure established?

Core Evaluation Questions

Replication, Sustainability, and ImprovementTo what extent do Fidelity and Impact outcomes sustain across time?

To what extent does initial SWPBIS implementation affect Implementation with later cohorts?

To what extent did SWPBIS implementation change educational/behavioral capacity/policy?

To what extent did SWPBIS implementation affect systemic educational practice?

Evaluation Questions

FidelityTo what extent was SWPBS implemented as designed?

To what extent was SWPBS implemented with fidelity?

“Are we doing what we said we would do?”

To What Extent Can

PBIS Applications.orgPBIS AssessmentPBIS Evaluation

Help us answer the question:“Are we doing what we said we would do?”

Current Tools in PBIS Assessment

Research Tool

Annual Assessment

Tool

Progress Monitoring

Tool

UniversalSystem

School-wide Evaluation Tool

(SET)

Benchmarks of Quality (BoQ)

Early Childhood BoQ (ECBoQ)

PBIS Self-Assessment Survey (SAS)

Team Implementation

Checklist (TIC)

Secondary &TertiarySystems

Benchmarks of Advanced Tiers (BAT)

PBIS Self-Assessment Survey (SAS)

Monitoring Advanced Tiers

Tool (MATT)

Outcome Measures: School Safety Survey

Current Tools in PBIS Assessment

Research Tool

Annual Assessment

Tool

Progress Monitoring

Tool

UniversalSystem

School-wide Evaluation Tool

(SET)

Benchmarks of Quality (BoQ)

Early Childhood BoQ (ECBoQ)

PBIS Self-Assessment Survey (SAS)

Team Implementation

Checklist (TIC)

Secondary &TertiarySystems

Individual Student School-wide Evaluation

Tool(ISSET)

Benchmarks of Advanced Tiers (BAT)

PBIS Self-Assessment Survey (SAS)

Monitoring Advanced Tiers

Tool (MATT)

Outcome Measures: School Safety Survey

www.pbisapps.org

A Tour of

at

Evaluation Action PlanningDecisions:

Evaluation ToolsWhich tools answer your district/schools’ evaluation questions?

Evaluation TimelineWhen/how often will you administer the tools?

Evaluation ReportingHow will you report outcomes to schools and to the District?How often will you report outcomes to schools and to the District?

Evaluation Action PlanningDecisions about Evaluation Tools

Does your District/School need a research tool?(SET/ISSET)

Which annual assessment tool(s) will you use?BoQ, SAS, SSS, BAT, ECBoQ

Which progressing monitoring tool(s) will you use?TIC, MATT

Does your District/School need to evaluate Tier I or Tier II & III implementation or all three?

Evaluation Action Planning

It is about…

CONTEXTYour settings => Your needs

Which Surveys Meet Your

District’s Needs?

Example District A

Evaluation Questions What is our baseline, and what has been our progress through the year?

Are we improving on our implementation of PBIS?

How do our individual staff members rate our implementation?

Are our self-assessments consistent with data from outside evaluators?

Newly implementing, grant needing specific research data as well as fidelity of implementation data

District A: Evaluation PlanMeasure Year One Year Two Year Three

Fall Winter Spring Fall Winter Spring Fall Winter Spring

SET X X X X

TIC X X X X X X X X X

Safety Survey X X X X

SAS X X X X

Example District B

Evaluation Questions In what specific PBIS areas are we strong, and in which areas do we need more training?

Are we improving on our implementation of PBIS?

How do our staff members actually rate our PBIS implementation?

How is our implementation of Tiers II and III?

Implementing PBIS for three years, have been using the TIC/SET, and have been consistently over 80%.

District B: Evaluation Plan

Measure Year One Year Two Year Three

Fall Winter Spring Fall Winter Spring Fall Winter Spring

BoQ X X X

SAS X X X

Safety Survey X X X

ATT X X X X X X X X X

Example District C

Evaluation Questions Which schools are not meeting the SET benchmark?

In what specific PBIS areas are we strong, and in which areas do we need more training?

How can we tailor our training and evaluation to meet the needs of various schools?

Are we improving on our implementation of PBIS?

How do our individual staff members rate our implementation?

Implementing for five years, some schools are doing very well, but others are not making progress.

District C: Evaluation Plan: Cohort 1

Measure Year One Year Two Year Three

Fall Winter Spring Fall Winter Spring Fall Winter Spring

SASSurvey X

SET X X X

Safety Survey X X X

TIC X X X X X X X X X

Schools still not performing well

District C: Evaluation Plan: Cohort 2

Measure Year One Year Two Year Three

Fall Winter Spring Fall Winter Spring Fall Winter Spring

SASSurvey X

BoQ X X X

Safety Survey X X X

MATT X X X X X X X X X

Schools performing well

Example District D

Evaluation Questions In what specific PBIS areas are we strong, and in which areas do we need more training?In what academic areas are we strong, and in which areas do we need more improvement?How can we tailor our training/evaluation to meet the needs of various schools?

How can we schedule our evaluations so that we are not overwhelming our schools?

The district wants to bring RtI Behavior and Academics together and create one evaluation plan.

Assessment Instrument

Fall (August/Sept/Oct)

Winter (Dec/Jan)

Spring(April/May) Purpose

Team Implementation Checklist (TIC)

X X X Fidelity of Implementation

Self-Assessment Survey

X Fidelity of implementation

School Safety Survey

X Outcomes related to

Protective and Risk Factors

SET X Fidelity of Implementation

AIMsWeb X X XScreening

Progress Monitor

SAT10 X Summative Evaluation

DIBELS X X X Screening

Progress monitoring

Nieves Flores, 2012

Task: Developing Your PlanQuestions to Consider:

Which surveys are your schools/districts currently using?Do they answer your evaluation questions:

Are we implementing SWPBIS with fidelity?Is there more information that we need?

Pbisapps.org > PBIS Assessment > Find Out More about Surveys

Do your surveys assess the Tiers you are implementing?Do you need a research tool?

Task: Developing Your Plan

Are you “topping out” of the surveys you are currently taking?

Do you need to customize your evaluation plan for different schools, or will one evaluation plan fit for all schools?

Do you want to assess all tiers of PBIS implementation?

Is your Evaluation Plan manageable for schools?

District Evaluation Planning Template

Measure Year One Year Two Year Three

Fall Winter Spring Fall Winter Spring Fall Winter Spring

SWPBIS School or District Evaluation Reports

Basic Contents of SWPBIS Evaluation Reports

From the PBIS Evaluation BlueprintContext

Input

Impact on SWPBIS Fidelity

Impact on Student Outcomes

Implications for the Future

Context

Define SWPBIS

Define goals of the specific project

Define stakeholders and evaluation questions

Input

Who received what support and from whom?Leadership teamLocal Capacity Building

Training, Coaching, Behavioral Expertise, and Evaluation

School teams

Impact on SWPBIS Fidelity

Leadership TeamSWPBIS District Leadership Team Self-Assessment

Capacity DevelopmentNumber of trainers/coaches available to support school teams/districtsBehavioral expertise available to support Tier II and Tier III implementationEvaluation capacity

School TeamsTier I Implementation (TIC, BoQ, SET, SAS)

Collectively and/or by training cohortTier II / Tier III Implementation (MATT, BAT, ISSET)

Collectively and/or by training cohort

Impact on Student Outcomes

Behavioral OutcomesOffice Discipline ReferralsSuspension/ExpulsionAttendanceDrop Out/GraduationDisproportionality in ODR and suspension/expulsion

Academic outcomesCurriculum Based Measures

Oral Reading Fluency, Dibels, AimsWebStandardized Measures

State and NationalMath, reading, writing assessments

Impact and Future Directions

Advances to datePlans for next three yearsAreas in need of adaptation

Narrowing of evaluation questions and concerns

Sharing Evaluation Reports

State and District examples at PBIS.orgSend in your district examples, so we can all learn!

Support and Resources

Portal Support SectionSearch FunctionCommunity ForumChat with us online

Account Questions:accounts@pbisassessment.org

Technical Assistance Questions:

support@pbisassessment.org

New phone number1-855-455-8194

Questions & Answers

When you leave the webinar, please complete the survey.

THANK YOU!