AN EVALUATION OF THE IMPLEMENTATION OF A POSITIVE BEHAVIOR
SUPPORT APPROACH IN AN URBAN PRESCHOOL SETTING
A DISSERTATION
SUBMITTED TO THE FACULTY
OF
THE GRADUATE SCHOOL OF APPLIED AND PROFESSIONAL PSYCHOLOGY
OF
RUTGERS,
THE STATE UNIVERSITY OF NEW JERSEY
BY
SUZETTE J. SULARSKI
IN PARTIAL FULFILLMENT OF THE
REQUIREMENTS FOR THE DEGREE
OF
DOCTOR OF PSYCHOLOGY
NEW BRUNSWICK, NEW JERSEY May 2010
APPROVED: ___________________________ Charles A. Maher, Psy.D.
___________________________ Russell J. Kormann, Ph.D.
DEAN: ___________________________ Stanley Messer, Ph.D.
ii
ABSTRACT
This dissertation documents the process of planning and conducting an evaluation of a
positive behavior support (PBS) approach, which was aimed at reducing challenging
behaviors and increasing of prosocial behaviors among preschool students in an urban
public school setting. The program evaluation that was planned and conducted utilized
Maher’s (2000) program planning and evaluation framework. The focus of the
dissertation was to follow through with evaluation of aspects of the PBS approach so that
useful evaluation information could be provided to the supervisors of the Office of Early
Childhood Education (OECE), with the intent of determining strong points of the
program and areas in need of improvement. Another important dissertation intent was to
provide a program evaluation plan that could be incorporated into the operational routine
of the OECE so that program evaluation could become an integral part of the program.
Furthermore, it was a basic contention of this dissertation that behavioral programs and
services in public schools, such as PBS, should be incorporated into organizational
routines and thereby be evaluated routinely so that informed judgments can be made
about the value of the program, which will subsequently contribute to program
development and improvement. This program evaluation was planned and conducted by
this investigator during the 2008-2009 academic year. For this evaluation of the
preschool PBS approach, four program evaluation questions were delineated in the
program evaluation plan. Results of a formative implementation of the program
evaluation plan revealed that teachers were generally satisfied with the PBS approach,
that the program was implemented most consistently in its earliest years of
implementation, that there was a discrepancy between teachers’ self-assessment of their
iii
skills and their ability to document the manifestation of these skills, and that professional
development and coaching were among the most helpful strategies for increasing
teachers’ skills. Findings of the dissertation were that the evaluation plan is feasible, key
stakeholders found the evaluation information useful, and there was a desire to continue
to use the evaluation plan for ongoing evaluations of the PBS program.
Recommendations are offered for the development, improvement, and ongoing
evaluation of the PBS program.
iv
ACKNOWLEDGEMENTS
It is a great pleasure for me to thank a number of people who have supported me
during the dissertation process. First, it would have been impossible to write this
dissertation without my committee chairperson, Dr. Charles Maher, whose enthusiasm
for program planning and evaluation was the inspiration for this dissertation. His expert
guidance, intellectual discipline, and mentorship helped keep me focused on the big
picture. It is an honor and privilege to be part of his program planning and evaluation
world! I would also like to express my sincere appreciation to my committee member,
Dr. Russell Kormann, for his interest in my work and his willingness to share his
knowledge and experiences in working with behavioral challenges.
I would also like to thank Dr. Geraldine Oades-Sese, Dr. Lucinda Seares-Monica,
Dr. Nancy Boyd-Franklin, Dr. Susan Forman, and Dr. Kenneth Schneider for their
general support and encouragement during my GSAPP experience. They unknowingly
contributed to the successful completion of this dissertation with their expressions of
interest and encouraging words. My deep appreciation is extended to Sylvia Krieger and
Kathy McLean for their generous gifts of time and never-ending patience. They have
cheerfully answered countless questions and organized and monitored mounds of
paperwork in their commitment to students and GSAPP. Further, a sincere thank you is
extended to my GSAPP friends and colleagues for sharing this experience with me.
Special gratitude is reserved for Denese Sanders, whose desire to improve
preschool programming and interest in my success turned out to be the foundation of an
unexpected friendship. This dissertation could not have been completed without her
support. I also extend my sincere appreciation to my colleagues and dear friends in the
v
early childhood office, Marge Bailey, Kamini Patel, and Elizabeth Vastano, who not only
provided valuable insights and assistance into the development and implementation of
this program evaluation, but also cheered me on when I needed it most. My gratitude is
also extended to the directors and teachers in the early childhood program for their
participation in the program evaluation.
For some, writing a dissertation can be an isolating experience. I, however, have
been fortunate to be surrounded by numerous friends who remained by my side during
this journey. Special thanks is extended to each and every one of them for their
willingness to remain my friend even after countless unaccepted phone calls and
postponed plans while I was in the “dissertation zone.”
To my sister, Andrea Reed, and my mother, Blanche Jarushewsky I extend my
love, appreciation, and respect for being strong women in the face of intensely difficult
times, and for patiently waiting for me to emerge from the “zone.”
Most significantly, my deepest and most passionate gratitude goes to my husband,
Allan, for without his unconditional love and sustenance (and willingness to do laundry,
food shopping, and cooking!) I would not have been able to complete this dissertation.
To my children, Joshua and Rebecca, thank you for knowing that love transcends a
dissertation.
Finally, I would like to dedicate this dissertation to my brother-in-law, Bruce
Reed, my sister, Leslie J. Adler, and my father, Boris Jarushewsky. I know that I am
eternally embraced by their love.
vi
TABLE OF CONTENTS
ABSTRACT................................................................................................................... ii
ACKNOWLEDGEMENTS........................................................................................... iv
LIST OF TABLES......................................................................................................... xi
LIST OF FIGURES ....................................................................................................... xiii
CHAPTERS
I. INTRODUCTION AND OVERVIEW......................................................... 1
Overview................................................................................................ 1
Rationale for Evaluating a Positive Behavior Support Approach in a
Preschool Program..................................................................... 1
Dissertation Context............................................................................... 2
Dissertation Task ................................................................................... 10
Summary ................................................................................................ 12
II. REVIEW OF LITERATURE ................................................................ 13
Overview................................................................................................ 13
History of Approaches for Addressing Problem Behaviors .................. 15
Positive Behavior Supports.................................................................... 19
Application of Positive Behavior Supports............................................ 21
Behavior Problems of Preschool Children............................................. 24
Positive Behavior Support Model for Preschool Programs ................... 31
Evaluating Positive Behavior Support in Preschool Programs.............. 35
Summary ................................................................................................ 42
vii
III. APPROACH TO PROGRAM EVALUATION .................................... 45
Overview................................................................................................ 45
The Program Evaluation Framework..................................................... 45
Clarification Phase ..................................................................... 46
Design Phase.............................................................................. 46
Implementation Phase................................................................ 47
Program Evaluation Phase ......................................................... 48
Description of the Program Design........................................................ 60
Target Population....................................................................... 60
Statement of Purpose ................................................................. 61
Program Goals ........................................................................... 63
Eligibility Standards................................................................... 63
Phases......................................................................................... 63
Components ............................................................................... 65
Relevant Organizational Context........................................................... 71
Ability of the OECE to Commit Resources............................... 74
Values of the Organizational Members ..................................... 74
Ideas People Have About the Current Situation ........................ 75
Circumstances in the OECE with Respect to its Structure and
Direction ........................................................................ 75
Timing of Using a Programmatic Approach in the
Organization................................................................... 75
Obligation of Individuals and Groups........................................ 76
viii
Resistance Expected by Individuals and Groups ....................... 76
Yield, or Value, of the Information ........................................... 77
Summary ................................................................................................ 77
IV. PROGRAM EVALUATION PLAN ..................................................... 78
Overview of the Chapter........................................................................ 78
Overview of the Program Evaluation Plan ............................................ 78
Program Evaluation Protocols ............................................................... 80
Protocol No. 1 ............................................................................ 80
Protocol No. 2 ............................................................................ 82
Protocol No. 3 ............................................................................ 83
Protocol No. 4 ............................................................................ 84
Communication of Program Evaluation Information ............................ 85
Evaluation of the Program Evaluation................................................... 85
V. RESULTS OF THE PROGRAM EVALUATION ............................... 87
Overview................................................................................................ 87
Results of Program Evaluation Question 1............................................ 87
Relevant Characteristics of Teachers......................................... 89
Relevant Characteristics of Preschool Students......................... 91
Results of Program Evaluation Question 2............................................ 97
Professional Development ......................................................... 98
Provision of Supports to Teachers ............................................. 101
Implementation of Specific Strategies ....................................... 103
Results of Program Evaluation Question 3............................................ 106
ix
Teacher Reactions...................................................................... 107
Results of Program Evaluation Question 4............................................ 118
Summary ................................................................................................ 124
Communication of Program Evaluation Information ............................ 126
VI. EVALUATION OF THE PROGRAM EVALUATION....................... 127
Overview................................................................................................ 127
Practicality ............................................................................................. 128
Utility ..................................................................................................... 130
Propriety................................................................................................. 130
Technical Defensibility.......................................................................... 131
Summary ................................................................................................ 132
VII. CONCLUSIONS AND RECOMMENDATIONS134
Overview................................................................................................ 134
Conclusions............................................................................................ 135
Findings of the Program Evaluation .......................................... 135
Findings of the Dissertation ....................................................... 137
Recommendations.................................................................................. 138
Recommendations for Improvement to the PBS Program......... 138
Recommendations for Implementation of the Evaluation Plan . 141
Summary ............................................................................................... 144
REFERENCES .............................................................................................................. 146
x
APPENDICES ............................................................................................................... 153
A. PROGRAM INSTRUMENTATION........................................................... 153
Instrument 1 Positive Behavior Support Teacher’s Questionnaire..... 154
Instrument 1.1 Preschool Teacher Statistics .......................................... 160
Instrument 1.2 Preschool Student Statistics........................................... 161
Instrument 2 Preschool Positive Behavior Support Classroom
Implementation Checklist .............................................. 162
Instrument 2.1 Review of Professional Development on Positive
Behavior Support ........................................................... 163
Instrument 2.2 Provision of Support to Teachers .................................. 164
xi
LIST OF TABLES
Table 1. Characteristics of Preschool Classroom Teachers ....................................... 90
Table 2. Characteristics of Preschool Students Referred to the PIRT ....................... 92
Table 3. Professional Development Trainings Provided During 2004-2005............. 99
Table 4. Professional Development Trainings Provided During 2005-2006............. 100
Table 5. Professional Development Trainings Provided During 2006-2007............. 100
Table 6. Professional Development Trainings Provided During 2007-2008............. 101
Table 7. Aggregated Estimates of Frequency of Direct Supports to Preschool
Teachers ....................................................................................................... 102
Table 8. Results of Positive Behavior Support Implementation Checklist................ 104
Table 9. Results of Positive Behavior Support Teacher’s Questionnaire: Part IV,
Item 1 ........................................................................................................... 109
Table 10. Results of Positive Behavior Support Teacher’s Questionnaire: Part IV,
Item 2 ........................................................................................................... 110
Table 11. Results of Positive Behavior Support Teacher’s Questionnaire: Part IV,
Item 3 ........................................................................................................... 110
Table 12. Results of Positive Behavior Support Teacher’s Questionnaire: Part IV,
Item 4 ........................................................................................................... 111
Table 13. Results of Positive Behavior Support Teacher’s Questionnaire: Part IV,
Item 5 ........................................................................................................... 111
Table 14. Results of Positive Behavior Support Teacher’s Questionnaire: Part V,
Items 1-8 ..................................................................................................... 113
xii
Table 15. Results of Positive Behavior Support Teacher’s Questionnaire: Part VI,
Weighted Rankings...................................................................................... 117
Table 16. Results of Positive Behavior Support Teacher’s Questionnaire: Part II,
Item 1 ........................................................................................................... 120
Table 17. Ratings of Teacher Responses to Positive Behavior Support Teacher’s
Questionnaire: Part III.................................................................................. 120
Table 18. Results of Positive Behavior Support Teacher’s Questionnaire: Part II,
Item 2 ........................................................................................................... 121
Table 19. Results of Positive Behavior Support Teacher’s Questionnaire: Part II,
Item 3 ........................................................................................................... 122
Table 20. Results of Positive Behavior Support Teacher’s Questionnaire: Part II,
Item 4 ........................................................................................................... 123
Table 21. Results of Positive Behavior Support Teacher’s Questionnaire: Part II,
Item 5 ........................................................................................................... 124
xiii
LIST OF FIGURES
Figure 1. Distribution of Classrooms in the Early Childhood Program During
2008-2009 .............................................................................................. 60
Figure 2. Referrals to PIRT and CST for Four Consecutive Years ...................... 91
1
CHAPTER I
INTRODUCTION AND OVERVIEW
Overview
This chapter discusses the importance of evaluating positive behavior support
approaches in preschool programs. It also details the context in which the program
evaluation was conducted, which includes a review of the evolution of preschool
programming in New Jersey, the role of the New Jersey Department of Education
(NJDOE) in preschool programming, and information about the structure of the Local
Education Agency (LEA) in which the program evaluation was conducted. The chapter
concludes with a description of the dissertation task and delineates four program
evaluation questions.
Rationale for Evaluating a Positive Behavior Support Approach in a Preschool Program
The group of students most likely to be excluded from regular education programs
due to behavioral difficulties is preschoolers. Data collected as part of the National
Prekindergarten Study indicated that the expulsion rate for preschool students is 6.7 per
1000 prekindergarten children enrolled, a rate that is 3.2 times higher than the national
rate of expulsion for students at all other grade levels (Gilliam, 2005). This finding is
particularly disturbing since expulsion is simply a reactive approach to challenging
2
behaviors. It fails to teach preschool children the skills necessary to replace inappropriate
behaviors. Moreover, many preschool children have not yet developed the cognitive
ability to connect the reason for not being permitted in school with the problem behavior
they exhibited. Collectively, this suggests that the need for worthwhile effective
behavioral programs and services at the preschool level is even greater than the need for
behavioral interventions in other grades.
Positive Behavior Support (PBS) models are being used in school settings as a
method of choice for reducing challenging behaviors and promoting prosocial behaviors.
The PBS approach was originally developed as an alternative to aversive interventions
for students exhibiting severe forms of aggression and self-injurious behaviors (Carr,
2007; Carr et al. 2002). PBS is now being used with a wide range of students across a
variety of contexts.
Best practice models for implementing the PBS framework in schools are
developing. Behavioral programs and services, such as PBS, used in schools should be
evaluated so that judgments can be made about the value of the program, which will
subsequently contribute to program development and improvement (Maher, 2000).
Dissertation Context
The purpose of the current program evaluation was to evaluate the
implementation of a PBS approach for reducing challenging behaviors and promoting
prosocial behaviors among preschool students in an Abbott preschool program in an
urban public school district in New Jersey during the 2008-2009 academic year.
3
Abbott was the first named plaintiff in a landmark class action lawsuit that was
filed in 1981 on behalf of urban children living in economically disadvantaged
municipalities. The lawsuit claimed that school funding for the poorest public school
districts was unconstitutional in comparison to the funding received by wealthier
suburban school districts and that schoolchildren in the poorest public school districts
were not receiving a thorough and efficient education. The criterion that was established
by the New Jersey State Supreme Court to determine if students living in poorer districts
were receiving a constitutionally guaranteed public education was whether or not they
were mastering the New Jersey Core Curriculum Content Standards to the same degree as
students in wealthier suburban districts. After a series of New Jersey Supreme Court
Rulings, 28 public school districts were identified as “Abbott” districts. By 2004, three
more public school districts were so designated.
As a result of the New Jersey Supreme Court Abbott decisions, new initiatives
have emerged to remedy the lack of parity between wealthy suburban and poorer urban
public school districts. In 1997, one of the first important Abbott rulings mandated that
Abbott public school districts receive state funding that provides them with the same per-
pupil operating budget as found among the state’s wealthiest public school districts. In
1998, the New Jersey Supreme Court ordered the New Jersey Department of Education
(NJDOE) to provide universal high quality preschool programs for all three- and four-
year-old children residing in Abbott school districts. The Court defined basic standards
for high quality preschool, which included a certified teacher and an assistant teacher for
each classroom, a maximum class size of 15 students, a developmentally appropriate
curriculum, and adequate facilities.
4
Most recently, the New Jersey School Funding Reform Act of 2008 expanded the
mandate for high quality preschool programming. Under this School Funding Reform
Act, all eligible at-risk three- and four-year-old children, regardless of the public school
district in which they live, must be offered placement in a high quality preschool program
at public expense (NJDOE, 2008a). Children are considered at-risk as defined by income
eligibility. This means that every public school district in New Jersey must offer a high
quality preschool program even if only one child living in the school district meets the
family income eligibility requirements. As a result of these preschool expansion
guidelines, public school districts that were once referred to as Abbott school districts,
are now referred to as having “universal” preschool programs since all three- and four-
year-old children residing in these districts are offered placement in a high quality
preschool program. Other non-Abbott school districts are referred to as having
“targeted” preschool programs since these school districts must only offer placement in a
high quality preschool program to targeted children based on income eligibility
requirements. The goal of these programs is to serve every eligible at-risk preschool
child. No eligible child may be excluded from participation in a high quality preschool
program for any reason including toilet training, immigration status, illness, or any other
individual circumstance (NJDOE, 2008a).
The Division of Early Childhood Education (DECE) of the NJDOE has
programmatic responsibility for preschool through 3rd grade programs. The DECE is
responsible for the development, implementation, and alignment of program components
with a focus on standards, curricula, and assessment. The DECE oversees all preschool
programming in the state. Accordingly, it has developed regulations for Elements of High
5
Quality Preschool Programs (NJDOE, 2008b) that are mandated to be followed by every
public school district. The DECE document describes the elements necessary for
implementing a high quality preschool program and includes specific guidelines for
eligibility, program planning, program staffing, curriculum and assessment, transition,
facilities, program evaluation, contracts with private providers and Head Start, fiscal
oversight, and appeals.
The DECE also produced Preschool Program Implementation Guidelines (NJDOE,
2008a). Together these documents provide guidance to school districts in the planning and
implementation of high quality preschool programs. Each school district is mandated to
have a preschool program that is driven by research-based best practices, and the plans
must be updated annually. Preschool programs may be provided by a mix of in-district,
private provider, and local Head Start classrooms. State funded preschool programs must
include all of the major features as outlined in the Elements of High Quality Preschool
Programs. The overall purpose of preschool programming for all eligible three- and four-
year-old children is to close the achievement gap between children of families with lower
income and higher income levels.
Each Local Education Agency (LEA) has a district level Office of Early
Childhood Education (OECE) to oversee district level preschool programming. The
anonymity of the district in which this program evaluation was conducted must be
maintained. For the purposes of this dissertation, therefore, the LEA district level office
will be referred to as the OECE. The OECE in which this program evaluation was
conducted is managed by two Supervisors of Early Childhood Education. The primary
responsibilities of the supervisors are to develop and implement the preschool five-year
6
plan, oversee the budget and coordination of services, and supervise administrative and
OECE staff. Per NJDOE guidelines, the OECE staff includes a number of specialized
positions, each of which will be reviewed below.
The fiscal specialist is required to have auditing, budgeting, and accounting
experience. She reports to the supervisors of the OECE as well as to the district level
School Business Administrator. The fiscal specialist’s responsibilities include financial
management assistance to private providers, compliance monitoring, tracking and
reporting of teacher certification, reviewing and expediting adjustments to quarterly
expenditure reports, and providing assistance with fiscal corrective action plans in
response to audits.
One master teacher is budgeted for a maximum of every 20 classrooms. In
districts, however, that support students who are English Language Learners (ELL) the
ratio is reduced to one master teacher for every 15 classrooms. The ratio for master
teachers to classrooms is further adjusted based upon the level of classroom teacher
certification and the number of years of classroom teacher experience. As a result of
these guidelines, the OECE has six master teachers. Master teachers are required to have
three to five years of teaching experience in preschool programs. The primary roles of
the master teacher are to provide modeling, coaching, and informal observations of
classroom teachers with the aim of providing feedback. The master teachers are also
responsible for assisting with the implementation of the curriculum selected by the
OECE, providing staff development, and implementing performance-based assessment.
In the OECE, one master teacher is the bilingual specialist and another master teacher is
the inclusion specialist.
7
The OECE is also required to have one school nurse for every 300 preschool
students. School nurses are responsible for conducting vision, hearing, dental, height,
and weight screenings upon entry into the program. Other responsibilities include
maintaining and following up on preschool student health records, communicating with
parents and staff about health issues, and assisting parents with locating medical
resources. The OECE staffs four school nurses to serve the students in the district
preschool program.
Each district OECE is required to have one Community Parent Involvement
Specialist (CPIS). The CPIS is responsible for coordinating the Early Childhood
Advisory Council, conducting needs assessments regarding families, and coordinating
work with other social services personnel. The Early Childhood Advisory Council
reviews preschool program implementation and supports transition activities as children
move from preschool through third grade.
State funded early childhood programs are also required to establish one four-
member Preschool Intervention and Referral Team (PIRT) for every 750 preschool
students enrolled in the preschool program. The PIRT may include a combination of
psychologists, learning disabilities teacher consultants, social workers, and speech
language specialists. The PIRT in the OECE consists of one psychologist, one learning
disabilities teacher consultant, three social workers, and two part-time speech and
language specialists. The PIRT is required to use a consultation model to assist staff with
modifying children’s challenging behaviors so that all preschoolers may successfully
participate in the general education classroom. PIRT responsibilities include, but are not
limited to, providing written strategies to teachers, modeling, providing consultation to
8
teachers, parents, administrators, and master teachers, and coordinating screenings aimed
at identifying preschool children at-risk for future school difficulties. The overarching
aim of the PIRT is to reduce referrals to the Child Study Team (CST) and increase
classroom teachers’ ability to support students with special needs in the general education
classroom. Classroom teachers may request assistance from the PIRT for support
regarding specific children. The PIRT in the OECE receives 220-240 such requests each
academic year.
As mandated by the DECE, one of the main functions of the PIRT is to provide
support to teachers, staff, and administrators on the implementation of the PBS approach
as a means of building teachers’ skills for addressing the needs of students with
challenging behaviors. This support is offered in a number of modalities including
professional development on the PBS approach, observing teacher-student interactions in
the classroom, collaborating with parents and teachers, providing written strategies as
needed, modeling and coaching on the use of suggested strategies, and developing and
monitoring behavior support plans for individual children. In order to develop the
abilities of PIRT members to provide training and support on the PBS approach, the
DECE provided 14 days of training on the PBS approach for all PIRT members statewide
(10 days in 2003-2004 and an additional 4 days in 2004-2005).
In the five years that the PIRT has been providing support to teachers, staff, and
administrators on PBS, no evaluation of the PBS program has been conducted. The
NJDOE requires every public school district providing preschool programs to participate
in the Self Assessment and Validation System (SAVS) in an effort to maintain the quality
of the preschool programs. Components of the SAVS were designed by the NJDOE and
9
are aimed to broadly assess how the components of the program work together to support
each child’s learning and development (NJDOE, 2008a). The results of the SAVS are
used to inform the five-year preschool program improvement plan and annual updates
(NJDOE, 2008b), which must identify areas in need of improvement, steps to be taken
for improvement, and timelines for such improvements. Although the SAVS includes a
set of criteria for assessing intervention and support services provided by the PIRT, it
fails to include any criteria for evaluating the extent to which the PBS approach is
impacting upon the manner in which teachers are able to address the needs of children
exhibiting challenging behaviors.
According to Maher (2000), a sound program evaluation is one that is practical,
useful, proper, and technically defensible. The purpose of an evaluation of a human
services program is to enable judgments to be made about the value of the program which
will contribute to program development and improvement. A significant portion of
resources has been devoted toward implementing the PBS program in the OECE, as well
as in a number of other districts across New Jersey and the nation. It was important to
evaluate the effectiveness of the PBS program in order to determine which components
of the approach were successful and which were in need of improvement so that relevant
stakeholders could make decisions about the merit and worth of the program based on
accurate information. Additionally, information about the effectiveness of the PBS
program will add to the growing body of literature examining the value of the PBS
approach. Furthermore, the process by which the PBS program was evaluated was
assessed to identify the strengths and weaknesses of the evaluation process, which will
serve to strengthen future program evaluations in public school settings.
10
Dissertation Task
One supervisor of the OECE was interested in knowing if the PBS program
increased teachers’ abilities for addressing the needs of preschool children exhibiting
challenging behaviors. In order to accomplish this task, the main focus of this
dissertation was the application of the program evaluation process as described by Maher
(2000). The Evaluation Phase is one of four phases in Maher’s framework for program
planning and evaluation. The other phases of the framework are the Clarification Phase,
the Design Phase, and the Implementation Phase, each of which will be discussed in
detail in Chapter III.
As the school psychologist on the PIRT, the current investigator was a participant
observer, which provided the opportunity to have direct knowledge of the PBS
implementation process and the daily workings of the OECE. In collaboration with one
OECE supervisor, the investigator was able to place the PBS program into an evaluable
form, which is one of the early steps in the Evaluation Phase of Maher’s (2000) Program
Planning and Evaluation framework. It is important to place a program that is being
evaluated into an evaluable form so that there will be clarity for all stakeholders
regarding what will actually be evaluated. This will also be discussed in greater detail in
Chapter III.
The current investigator also collaborated with the supervisor in the OECE and
other PIRT members to develop a program evaluation plan for the PBS program in
accordance with the framework presented by Maher (2000). Accordingly, four program
evaluation questions were developed:
11
1. Who has participated in the PBS approach? 2. How has the PBS approach been implemented?
3. What were the reactions of preschool teachers to the PBS approach in terms of
strengths, adequacies, and areas for improvement?
4. To what extent were the goals of the PBS program attained?
In order to successfully address these questions, and to make sound judgments
about the value of the PBS program, the current investigator and other key stakeholders
engaged in a series of sequential, interrelated, and reflexive activities as outlined by
Maher (2000). Accordingly, specific program evaluation protocols were developed that
detailed the program evaluation questions, data collection variables, data collection
methods, instruments and procedures, methods and procedures for data analysis, and
guidelines for communication and use of evaluation information. Evaluation data were
collected via two instruments designed specifically for evaluation of the PBS program.
The first was a teacher’s questionnaire and the second was a classroom observation
checklist. Each instrument was designed by the current investigator to capture variables
that are specific to the PBS program. A permanent product review was also conducted
in order to determine whether and to what extent there was a relationship between the
PBS program and the degree to which teachers requested assistance with addressing
challenging behaviors and promoting positive behaviors in the classroom. Program
evaluation questions, as well as corresponding program evaluation protocols, will be
reviewed in Chapter IV. The evaluation of the PBS program and consequent results will
12
be discussed in Chapter V. Chapter VI will review the evaluation of the program
evaluation. Chapter VII will present conclusions and recommendations.
Summary
It is important to evaluate PBS approaches used in public preschool programs so
that judgments can be made about the value of the program, which will subsequently
contribute to continued program development and improvement (Maher, 2000). This is
particularly important as best practice models continue to emerge. One supervisor in the
OECE was interested in knowing if the PBS program contributed to teachers’ abilities for
addressing the needs of preschoolers exhibiting challenging behaviors. Four program
evaluation questions were developed in accordance with the program evaluation
framework presented by Maher (2000). A series of sequential, interrelated, and reflexive
activities were conducted in an effort to address the program evaluation questions.
13
CHAPTER II
REVIEW OF LITERATURE
Overview
The PIRT members of the OECE implemented the PBS program mandated by the
NJDOE Early Childhood Office to assist teachers with addressing the needs of children
with challenging behaviors. To facilitate the application of PBS in preschool programs,
the Division of Early Childhood Education provided 14 days of training on the PBS
approach for PIRT members (10 days in 2003-2004 and an additional 4 days in 2004-
2005). PIRT members were expected to train teachers and other early childhood staff on
the PBS approach and to facilitate implementation of the program.
The task of this dissertation was to evaluate the PBS approach in the OECE so
that sound judgments about the value of the program can be made, which will
subsequently contribute to the development and continuous improvement of the program.
The purpose of the current chapter is to review the literature relevant to the dissertation.
Sections of this chapter include the following: History of Approaches to Addressing
Behavior Problems in Schools; Positive Behavior Supports; Application of Positive
Behavior Supports; Behavior Problems of Preschool Children; Positive Behavior Support
14
Model for Preschool Programs; and Evaluating Positive Behavior Supports in Preschool
Programs.
The first section, History of Approaches for Addressing Problem Behaviors in
Schools, reviews the history of school discipline practices and provides a timeline for the
development of the PBS approach. This section is relevant to the dissertation because it
offers the reader additional insight into the context in which the development of PBS
occurred, as well as an explanation as to why innovative frameworks for addressing
problem behaviors in schools were needed.
The second section, Positive Behavior Supports, describes the aims of PBS and
the specifics of how PBS emerged. A widely accepted framework for delivering PBS in
schools is also presented. This section is relevant to the dissertation because it is
important to understand the foundation upon which behavior support initiatives are based.
The reader must have a general understanding of the levels of intervention that are a part
of the PBS framework.
The Application of Positive Behavior Supports is the third section of this chapter.
This section reviews the elements of PBS as they are applied in schools and alternate
settings. Most of the current literature on PBS is focused on implementation at school
age grade levels (kindergarten through high school) and not on preschool. This
discussion, therefore, is relevant to the dissertation because it lays the groundwork for
expanding the application of PBS to early education programs.
The fourth section, Behavior Problems of Preschool Children, discusses the
prevalence and impact of preschool problem behaviors. There is also a review of ten
summary statements reflecting the current knowledge level of preschool behavior
15
problems. This supports the task of the dissertation because it underscores the need to
implement early intervention behavioral programs in early childhood settings.
The fifth section of the literature review is the Positive Behavior Support Model
for Preschool Programs. The PBS framework mandated for use in early childhood
programs by the NJDOE is presented and each level of support in the framework is
reviewed in this section. An understanding of this particular PBS model is relevant to the
dissertation because this is the model that is being evaluated in the dissertation task.
The final section of this chapter, Evaluating Positive Behavior Supports in
Preschool Programs, presents examples of case studies that support the use of the PBS
approach with young children. The strengths and limitations of specific tools designed
for evaluating PBS in preschool programs are also discussed. This is a relevant aspect of
the dissertation since it provides an underlying basis for understanding the need to further
evaluate PBS programs in early childhood settings. It provides the foundation for
selecting an evaluation approach which may extend the literature in this area.
History of Approaches for Addressing Problem Behaviors in Schools
Teachers have been faced with students exhibiting problem behaviors since the
days of the one room schoolhouse. Numerous anecdotal reports of children standing in
corners and wearing dunce caps have been shared by generations of grandparents. Since
the turn of the last century, however, the number and intensity of discipline problems in
schools has increased dramatically. The days in which educators’ biggest concerns were
of placing frogs in the teacher’s water glass or dipping pig-tails in ink wells are gone.
School discipline problems appear in the news on an all too frequent basis, citing
16
physical violence, property damage, homicide, and suicide. Today, school violence is so
prevalent that there have even been incidents of preschoolers making serious violent
threats.
As a result of the increase in intensity and frequency of serious behavior
problems, school districts have developed comprehensive discipline procedures that
include catching future instances of problem behaviors via close monitoring, restating
rules and consequences for undesirable behaviors, having a continuum of consequences
for repeat offenders, exercising consistency in how staff responds to problem behaviors,
and emphasizing final consequences to inhibit future problem behaviors (Sugai &
Horner, 2002a).
When the school discipline approaches listed above fail to reduce serious
challenging behaviors, and the number and intensity of problem behaviors continue to
rise, school districts have reacted by initiating zero tolerance policies, hiring security
guards, installing surveillance cameras and metal detectors, mandating school uniforms,
implementing detention, suspension, and expulsion procedures, and assigning alternative
school placements (Sugai & Horner, 2002a).
These discipline practices have been based on an underlying assumption that the
behavior problems displayed by students are discrete; each instance of problem behavior
is viewed and addressed as a separate, unrelated issue. Such practices fail to consider the
context in which the behaviors occur, and fail to incorporate a systems level approach for
reducing challenging behaviors. Moreover, these disciplinary interventions fail to teach
students new skills for dealing with intense emotional situations that trigger undesirable
behavior outbursts.
17
During the period of the 1960s through the 1980s, strategies for behavior change
were based on positivistic research grounded in operant learning punishment principles.
The use of aversive techniques was widely accepted in treating individuals with severe
disabilities and severe behavior problems (Lovaas, Schaeffer, & Simmons, 1965 as cited
in Dunlap, Sailor, Horner, & Sugai, 2009). However, when the movement toward
deinstitutionalization appeared in the 1980s, a mismatch between the use of aversive
measures to reduce challenging behaviors and the moral values of the community
emerged. By the early 1980s, positivistic research began to examine the secondary
effects of aversive treatment such as outbursts, anxiety, and avoidance (Favell &
Rincover, 1983 as cited in Dunlap et al., 2009).
The 1980s saw a paradigm shift in the treatment of severely challenging
behaviors. Not only were aversive treatments thought to be morally distasteful, but there
was also recognition that in order to increase quality of life, challenging behaviors must
not only be reduced, but positive behaviors must also be increased. Research began to
focus on why problem behaviors occurred, which led to applied behavior analysis (ABA)
and functional analysis of behaviors (Dunlap et al., 2009). Rather than an emphasis on
punishment to alter problem behaviors, an emphasis on the need for preventative
practices emerged. By 1987, the United States Department of Education funded research
for non-aversive behavior interventions, which led to the coining of a new term, positive
behavior support (PBS). PBS was focused on addressing severe disabilities and
behaviors, and was based on research regarding applied behavior analysis from the early
1980s. By the early 1990s, the application of PBS expanded to not only the treatment of
18
individuals with severe disabilities and behaviors, but it was also applied to the treatment
of emotionally and behaviorally disturbed individuals.
The event that spiraled PBS into school-based interventions was the 1997
Individual with Disabilities Education Act (IDEA), which required the use of functional
behavioral analysis (FBA) and the use of positive interventions as behavioral change
strategies (Sugai and Horner, 2002b; Turnbull, Wilcox, Stowe, and Turnbull, 2001). As a
result of this mandate, there was so much interest in and research on PBS that in 1999 a
new peer reviewed publication emerged, the Journal of Positive Behavior Interventions.
By the late 1990s and early 2000s, PBS was applied to early intervention with young
children (Fox, Dunlap, Hemmeter, Joseph, and Strain, 2003). Sugai and Horner (2002a)
introduced a now widely accepted, multi-tiered framework for PBS. The application of
PBS continued to expand. In 2001, the NJDOE Division of Early Childhood Education
mandated the use of PBS in all Abbott preschool programs (NJDOE, 2001). In 2003, the
Association for Positive Behavior Support, which focused on promoting research
strategies, person-centered values, and systems change to increase quality of life and
decrease problem behaviors, was founded.
The increase in interest and use of PBS continued to soar. The requirements to
use FBA and PBS that were introduced in IDEA 1997 remained in IDEA 2004 (Office of
Special Education Programs, OSEP, 2009). As part of the IDEA initiative, each state was
required to establish a PBS technical assistance center and develop a website to support
PBS implementation throughout school districts in their state. These technical assistance
centers and websites are funded by IDEA 2004. Most recently, the NJDOE Preschool
Program and Implementation Guidelines (NJDOE, 2008a) strengthened the mandate for
19
the use of PBS by citing a specific PBS model: the Social Emotional Teaching Pyramid
Model proposed by Fox, Jack, & Broyles (2005). This PBS model will be reviewed later
in this chapter.
Positive Behavior Supports
The aim of PBS is to decrease problem behaviors and increase positive behaviors
in order to influence the quality of life for individuals with behavioral disabilities or other
disorders that impact behavior (Carr, 2007; Carr, et al. 2002; Office of Special Education
Programs Technical Assistance Center on Positive Behavior Interventions and Supports,
2009). PBS describes an approach for meeting the needs of children exhibiting
challenging behaviors that examines the purpose of the behavior and focuses on teaching
new skills to replace challenging behaviors. PBS originally focused on the development
of behavior support plans to help children and adults who exhibited challenging behavior.
More recently, PBS has been implemented at school-wide and program-wide levels. In
these settings, all school staff work together to teach behavioral expectations and social
skills, and to provide individualized interventions to those students most at risk for future
problems (Fox, et al., 2005).
Carr and colleagues highlighted that PBS emerged from three major areas. First,
applied behavior analysis provided basic terminology and concepts that have contributed
to the formulation of PBS such as stimulus-response, setting events, reinforcing
consequences, shaping, and prompting. Applied behavior analysis also gave rise to
functional analysis, which serves to identify the purpose of behaviors via specific
assessment procedures. Second, the normalization/inclusion movement that emerged
20
over the past 150 years (Carr et al., 2002) extended rights to individuals and groups who
have been marginalized by mainstream society. This has led to current educational
practices of including students with disabilities with regular education students rather
than segregating them into self-contained special education classrooms. Third, PBS
looks to person-centered values to inform strategies that serve to “enhance personal
dignity and opportunities for choice” (p. 6, Carr et al., 2002).
Carr and colleagues (2002) outlined how applied behavior analysis, the
normalization/inclusion movement, and person-centered values have given birth to PBS
as a new, but still evolving applied science. It is the manner in which the critical features
of PBS are integrated, however, that make PBS a unique approach to addressing
challenging behaviors. PBS offers a comprehensive lifestyle change for behaviorally
challenged individuals and their families, via a life-span perspective rather than a short-
term approach. It also offers ecological validity and meaningful application in real-world
settings. The application of PBS is a collaborative process involving participants from
different systems in conjunction with interventions that are practical and desirable for
stakeholders. Further, PBS interventions are proactive rather than reactive. Another
factor that is unique to PBS as a behavioral intervention approach is that participants
accept alternate scientific practices such as qualitative measures, self-reports, and
interviews rather than requiring traditional experimental research methods. Moreover,
PBS embraces multiple theoretical perspectives from ecological, environmental, and
community psychology (Carr, et al., 2002).
Sugai and Horner (2002a) developed a continuum of behavior supports based on a
public health and disease prevention model. The PBS continuum was designed to
21
provide supports based on the relative needs of the students. In this framework, primary
prevention refers to school- or classroom-wide supports for all students to reduce the
likelihood of the development of problem behaviors. According to Sugai and Horner,
primary prevention may include an emphasis on teaching appropriate behaviors and
teaching practices that boost academic success, and may prevent problem behaviors
among 80% of students. Secondary prevention will be needed by 15% of students and is
aimed at reducing risk factors, such as poverty, and strengthening protective factors, such
as additional school supports and family assistance. Tertiary prevention is aimed at the
remaining 5% of students with high risk for problem behaviors and involves
individualized systems supports.
The Application of Positive Behavior Supports
One focus of the Individuals with Disabilities Education Act (IDEA) was on the
use of PBS and functional behavior assessments (FBA) for students with disabilities.
IDEA also required that schools apply the PBS approach to students who have not
already been identified as eligible for special education if the school had knowledge that
the student is at risk for needing special education services due to their behaviors (IDEA,
1997). The basis of such knowledge can be from parent or teacher reports, or if the
behavior of the student has demonstrated the need. This broad basis underscored the
need for schools to adopt the PBS approach for all students, which resulted in the
development of school-wide PBS (SWPBS; Sugai & Horner, 2002a).
Sugai and Horner (2002a) identified four key elements of PBS: outcomes defined
and valued by stakeholders; research validated practices; data-driven decision making;
22
and a process level perspective (committees, families, administrative leadership, etc.). To
make these elements more manageable and behavioral changes more sustainable, Sugai
and Horner emphasized the need to organize a multi-systems approach that includes
school-wide, classroom, non-classroom, and individual student perspectives. In the PBS
framework, behavior supports are placed on a continuum based upon the needs of the
student, and a model of prevention is embraced by all stakeholders and applied to all
students. This model points toward examining how the school functions as a whole,
rather than looking to classroom management styles of one teacher or the behavior of one
student. Accordingly, Sugai and Horner (2002a) outlined a five-step process for
implementing SWPBS. These steps are (1) Establish a school leadership team; (2)
Secure school-wide supports from staff; (3) Develop data-based action plans; (4) Arrange
for high fidelity of implementation; and (5) Conduct formative data-based monitoring.
In 2002, there were about 500 schools across the nation implementing SWPBS
(Sugai & Horner, 2002a). By 2008, more than 5300 schools were implementing SWPBS
(Frey, Lingo, & Nelson, 2008). A preponderance of case-based literature has indicated
general success for SWPBS. For example, at an elementary school in Oregon, two third
grade boys with serious behavior problems (i.e. hitting, self-injurious behavior, eating
staples, poking others with scissors, running away from school) and significant
disabilities (i.e., emotionally disturbed, Attention Deficit Hyperactivity Disorder, Autism
Spectrum Disorder) exhibited significant reductions in problem behaviors when
secondary tier interventions were used to support all third grade students in addition to
the individualized behavior support plans that were developed specifically for these boys
(Freeman, et al., 2006).
23
At an elementary school in Illinois, SWPBS was credited with preventing another
3rd grade boy from being classified as a student with a disability (Freeman, et al., 2006).
In this case, the boy was exhibiting a variety of problem behaviors and academic
struggles. The support team developed a behavior support plan that closely involved
family input and targeted teaching and reinforcing social and academic skills. A Child
Study Team (CST) evaluation that was initiated as part of the support process determined
that the child had a learning disability. He was not classified, however, because the
teacher and the CST recognized that he was making adequate progress with the new
supports already in place as a result of SWPBS.
Not only has SWPBS been shown to be effective in addressing the behavioral
needs of students, it has also been credited with improving skills among teachers and
support personnel. For example, in an urban school district in Southern California,
behavior support plans were demonstrated to be more technically sound in schools where
SWPBS was being implemented in comparison to schools that were not implementing
SWPBS (Medley, Little, & Akin-Little, 2007). In still another example, a behavioral
intervention plan for recess, which was embedded in SWPBS, not only resulted in
reduced problem behaviors among children, but also increased the level of teacher
supervision (Franzen & Kamps, 2008). SWPBS has also been shown to have favorable
outcomes in alternate settings, having been credited with increasing positive behaviors in
a correctional facility for male juvenile offenders (Feinstein, 2003).
24
Behavior Problems of Preschool Children
Aggression and other behavioral problems are escalating among preschool
children (Campbell, 2002). As many as 25% of preschool children engage in behaviors
that are considered disruptive, dangerous, aggressive, and/or sometimes disgusting
(Webster-Stratton, 1999). Children living in poverty are especially at-risk for exhibiting
challenging behaviors (Qi & Kaiser, 2003). Children in Head Start classrooms exhibited
externalizing problem behaviors once every six minutes, which translated to
approximately 36 episodes of problem behaviors per hour in each classroom (Webster-
Stratton & Hammond, 1997). This helps to explain why the rate of expulsion from
school among preschool children is higher than that of any other age group (Gilliam,
2005). Behaviors such as noncompliance, being overactive, and fighting with peers,
however, are very common among preschool children (Campbell, 2002), and do not
necessarily suggest psychological disturbance. Certain behaviors are a result of typical
child development processes and may increase or decrease with age. Nonetheless, within
the context of a preschool classroom, many of these behaviors can be quite disturbing to
adults, and may in fact pose threats to the safety and welfare of peers. Additionally,
students engaging in these behaviors may also find themselves rejected by peers (Wood,
Cowan, & Baker, 2002), which then may serve to exacerbate the behavior problems.
It is often difficult to distinguish between annoying behaviors that occur among
typically developing preschool children and behaviors that indicate a more serious
problem. One reason for this difficulty is that differing viewpoints of preschool behavior
problems may be attributed to the observer. A parent or teacher may view
noncompliance or fighting as serious behavior problems, while a psychologist may view
25
the same behaviors as developmentally appropriate within a given context. Campbell
(2002) organized preschool behaviors into three categories: annoying behaviors, age-
specific problems, and symptomatic problem behaviors. Annoying behaviors refer to
behaviors typical for a specific age group that are a concern to some individuals. Age-
specific problem behaviors refer to behaviors that are an exaggeration in the frequency
and/or intensity of typical behaviors. These may or may not indicate a more serious
problem. Symptomatic problem behaviors refer to behaviors that are most likely
indicative of problems of clinical significance.
Assessing the severity or implications for future pathological disturbance of
problem behaviors among preschool students is dependent on several factors (Campbell,
2002). First, behaviors must be considered within the context of what is known about
child development. Most preschool age children will exhibit externalizing behaviors at
one time or another. Observers must consider the frequency and intensity of these
behaviors in comparison to same age peers in the same context. Second, the perceptions
and interpretations that the observer brings to the situation will also determine how
problematic the behavior is considered. These perceptions will also determine how a
teacher will respond to the behavior. If a teacher views aggression as typical behavior for
a three-year-old then it will not be cause for alarm, and it is more likely to be addressed
in the present moment. The third factor that contributes to how behaviors of preschool
children should be assessed is dependent on the level of family supports for child
development (Campbell). Unrealistic parental expectations and demands may contribute
to the development of inappropriate behaviors.
26
Although it is difficult to determine whether or not a particular behavior is
indicative of future pathology, it is increasingly understood that persistent, intense
challenging behaviors in preschool are associated with problems relating to peer
acceptance, school adjustment, and general school success in later years (Campbell,
2002). Dunlap et al. (2006) synthesized the growing body of evidence pertaining to the
presence, impact, prevention, and intervention of challenging behaviors in young
children. They developed ten summary statements that reflect the state of current
knowledge of preschool behavior problems based on a consensus of peer-reviewed
descriptive, experimental, and quasi-experimental research. The ten summary statements
correspond to three main categories: (1) Presence and impact of challenging behaviors;
(2) Prevention of challenging behaviors; and (3) Intervention with challenging behaviors.
The first two summary statements refer to the presence and impact of challenging
behaviors among young children:
1. When children with significant problems are neither identified in a
timely way nor given appropriate education and treatment, their
problems tend to be long lasting, requiring more intensive services and
resources over time. Moreover, when the challenging behavior of
young children is not addressed in an appropriate and timely way, the
future likelihood increases for poor academic outcomes, peer rejection,
adult mental health concerns, and adverse effects on their families,
service providers, and their communities (Dunlap, et al., 2006, p. 32-
33)
27
It is important to address behavior problems in preschool since longitudinal studies
revealed that students with a history of severe behavior problems had the lowest grade
point average and the highest high school drop-out rate (Tremblay, 2000). Moreover,
behavior problems in early childhood are identified as the single best predictor of future
serious behavior problems in adult life (Campbell, 2002).
2. “Although some systems and tools for early identification of children with
challenging behaviors are available, the actual identification of these children
and the provision of appropriate services are very low” (Dunlap et al., 2006,
p. 34)
As noted earlier, this can be attributed to a variety of factors that confound the
identification of children with problem behaviors (i.e. perceptions, developmental
ranges). However, Dunlap et al. present several explanations for under-identification of
young children with problem behaviors including lack of early behavior screening,
inadequate behavioral health services, and possible biases against identifying children
with behavioral challenges.
The next three summary statements presented by Dunlap et al. (2006) refer to the
prevention of challenging behaviors.
3. “Children and their families who access mental health and physical care are
less likely to have behavioral and social problems” (Dunlap et al., 2006, p.
35).
4. “Children who experience nurturing and positive parenting are more likely to
have healthy relationships and reduced problem behavior” (Dunlap et al.,
2006, p. 36).
28
Effective early intervention programs promote parenting skills and prevent abuse, thus
contributing to mental and physical health care, which can aid in preventing future
problem behaviors. The next summary statement regarding prevention of challenging
behaviors strongly relates to the dissertation.
5. “Children who experience high quality early education environments and
caregiver interactions are more likely to have better social competence outcomes
and fewer behavioral problems” (Dunlap et al., 2006, p. 36).
As will be discussed in more detail later in this chapter, one foundational element to the
PBS approach is aimed at building positive relationships between teachers and students.
The need for positive relationships is grounded in longitudinal research that indicates that
teacher-child closeness in early years has a positive impact on prosocial skills and peer
interactions in later elementary grades (Dunlap et al., 2006).
Each of the five remaining summary statements put forth by Dunlap et al. (2006)
concern interventions with challenging behaviors.
6. “Interventions based on a functional assessment of the relation between the
challenging behaviors and the child’s environment are effective for reducing
challenging behaviors of young children” (Dunlap et al., 2006, p. 37).
In the PBS approach, the use of functional assessments is indicated for students who
continue to exhibit challenging behaviors even after all other supports are in place. This
will be discussed in more detail later in this chapter.
7. “Teaching procedures have been demonstrated to be effective in developing
children’s skills and reducing challenging behaviors” (Dunlap et al., 2006, p. 37).
29
Young children often lack the skills necessary to function within a classroom setting.
They need direct instruction in social skills and language development in order to
navigate the rules and expectations of a classroom environment. The PBS approach
aligns with current research by advocating for teaching skills that encourage the
replacement of inappropriate behaviors with prosocial behaviors.
8. “Interventions involving alterations to features of the child’s activities and the
child’s social and physical environment have been demonstrated to reduce
challenging behaviors” (Dunlap et al., 2006, p. 37).
This involves identifying the antecedents to the child’s problem behaviors and altering
the environment in such a way that the behavior is less likely to occur. These alterations
can involve offering choices, embedding preferred activities into difficult ones, and
considering the arrangement of the physical environment and scheduling of activities
(Fox et al., 2003). The PBS framework addresses these alterations at the universal level
of instruction.
9. “Multicomponent interventions implemented over time and across multiple
relevant environments can produce durable, generalized increases in prosocial
behavior and reductions in challenging behavior” (Dunlap et al., 2006, p. 38).
This summarizing statement regarding interventions for challenging behaviors among
young children refers to the need to develop behavioral interventions that not only
address antecedents and consequences, but also environmental arrangement, scheduling,
and instructional modification throughout a variety of contexts such as classrooms,
school hallways, playgrounds, and school cafeterias. The statement also reflects the need
30
for behavioral interventions to be persistent over time in order to produce sustainable
changes in behavior.
10. “Family involvement in the planning and implementation of interventions
facilitates durable reductions in challenging behaviors of young children” (Dunlap
et al., 2006, p. 38).
Dunlap and colleagues (2006) indicated that the consensus of current research reflects the
need to include the child’s family. Numerous initiatives are in place to encourage and
promote family involvement in preschool programs. As noted in the previous chapter, the
NJDOE allocates funding for one parent community involvement specialist in each local
office of early childhood education. Further, PBS has been adapted for use in training
parents of toddlers and preschoolers. HOT DOCS (Armstrong, 2006) is based on the
PBS framework and identifies parents as the main providers of intervention with trainers
of the HOT DOCS program as helping partners.
The preponderance of the literature makes it clear that if left untreated, serious
behavior problems in preschool are almost sure to get worse. Early intervention is the
best hope for preventing escalation of behavior problems into older grades and adulthood.
There also appears to be a variety of evidence-based practices that can be utilized to
assist with the prevention and intervention of challenging behaviors among preschool
children. These practices and interventions are most likely to occur in preschool
programs. However, preschool teachers have reported that addressing challenging
behaviors is the single greatest obstacle to providing quality preschool programs (Micklo,
1992 as cited in Arnold, McWilliams, & Arnold, 1998). Helping teachers to set and
31
reinforce rules and routines may be critical for preventing challenging behaviors in
preschool settings (Arnold et al.).
Since Dunlap et al. (2006) aptly point out that little research has focused on
program procedures and systems that promote improved behaviors among young
children, it seems imperative that proper program evaluations of school-based programs
for addressing challenging behaviors occur with increased frequency. Given that
problem behaviors among preschool children are so common, and that these behaviors
may or may not be indicative of future pathology, it seems important that teachers and
other adults responsible for the care of these students be prepared to appropriately
address and manage the behaviors. A program evaluation that is practical, useful, proper,
and technically defensible (Maher, 2000) will help identify components of school-based
behavior intervention programs that are worthwhile and those that are in need of
improvement, thus providing teachers and other stakeholders with guidance on how to
best support the social and emotional development of young children.
Positive Behavior Support Model for Preschool Programs
Over the past two decades there has been an emphasis on getting young children
ready for kindergarten via preschool programming. It is generally accepted that this
includes basic academic readiness skills such as letter and number recognition.
Unfortunately, most people fail to recognize the need to prepare young children for the
social and emotional demands of school settings. As already noted, preschool children
are expelled from school at a rate that is more than three times the rate of any other grade
level (Gilliam, 2005). Moreover, there is a growing body of evidence indicating that
32
serious, persistent problem behaviors in preschool lead to future school failure and
serious problem behaviors later in life (Campbell, 2002; Dunlap et al. 2006; Pierce,
Ewing, & Campbell, 1999). Together, these facts underscore the need to address
challenging behaviors among preschool children with the same intensity and purpose as
addressing academic readiness skills.
The NJDOE Preschool Program Implementation Guidelines (2008) cites the PBS
Teaching Pyramid model presented by Fox et al. (2005) to address the social emotional
needs of preschool students. PIRT members throughout the state of New Jersey received
14 days of training on the PBS Teaching Pyramid (Fox et al., 2003) during the 2003-2004
and 2004-2005 academic years. After the training, PIRT members were required to
provide ongoing professional development on the PBS Teaching Pyramid for all district
preschool staff and coordinate efforts for successful PBS implementation in the early
childhood programs.
The Teaching Pyramid (Fox et al., 2003) was developed based on the public
health model of promotion, prevention, and intervention and Sugai’s & Horner’s (2002a)
three tiered PBS model (Fox & Hemmeter, 2009). It “defines the classroom practices
needed to support the social emotional development of young children” (Fox &
Hemmeter, 2009, p. 185). The Teaching Pyramid includes universal promotion,
secondary prevention, and tertiary intervention levels with descriptions for teaching
practices at each level of the pyramid. There are four levels of support on the PBS
Teaching Pyramid. Each of the four levels of support is discussed below.
Building Positive Relationships. The first of two levels of support aimed at
universal promotion is Building Positive Relationships. This refers to the need
33
to develop positive, supportive relationships among all relevant stakeholders: the teacher
and the child, the teacher and the parents, the teacher and other teachers, and the teacher
to other staff. Building Positive Relationships is viewed as the foundation for all other
teaching practices (Fox & Hemmeter, 2009). Fox et al. (2003) point out that positive
relationships with children increase the teacher’s ability to positively influence the child’s
behavior. Even very young children notice when adults are responsive and caring. As a
result, children are more likely to pay attention to what that teacher says, and they are
more likely to behave in ways that increase the amount of positive attention from that
teacher. Another benefit to developing positive, supportive relationships with children is
that under these conditions children are more likely to develop a positive self-
image, confidence, and a sense of security (Fox et al.). Some methods for building
relationships with preschool children include engaging in play, greeting every child by
name, having a conversation during lunch, and sending home positive notes (Fox et al.).
Classroom Preventative Practices. Classroom Preventative Practices is the
second level of support aimed at universal promotion. At this level of intervention,
consideration is made for the physical environment of the classroom. Teachers and other
support personnel are encouraged to rearrange the physical environment with the aim of
reducing the amount of wide-open space while also allowing for visual monitoring of all
areas of the classroom. Also, at this level, teachers are encouraged to arrange the
classroom schedule so that high energy activities are balanced with low energy activities,
and quiet activities are balanced with noisier activities. The Classroom Preventative
Practices level of the Teaching Pyramid also addresses the appropriateness and amount
of materials in the classroom. Teachers should ensure that the quality and quantity of
34
materials in the classroom can meet the needs of the wide range of developmental stages
of early childhood. Finally, at this level of the pyramid, teachers are also encouraged to
clearly define and teach classroom rules and routines. Rules and routines that are
consistently taught and followed make the environment more predictable for the child
and are likely to result in a reduction of problem behaviors. Rules at the preschool
classroom level should be quite simple, such as we use walking feet, we take turns, and
we use soft touch. Other classroom preventative practices that teachers can implement
include developing a sign-in method for students, creating waiting lists at highly
desirable centers in the classroom (i.e. computer area), using visual support for line-up
time, and providing warnings for transitions from one activity to another (i.e. five more
minutes until clean-up).
Social Emotional Teaching Strategies. The secondary prevention level of the
Teaching Pyramid is Social Emotional Teaching Strategies. This level is in place to
address the needs of all preschool children for the development of appropriate social
emotional skills. While some children learn social emotional skills via observational
learning and adult guidance, many children require planned and intentional instruction to
develop competencies with emotional regulation, problem solving, and friendship skills
(Fox & Hemmeter, 2009).
The ability to regulate emotions is dependent upon first developing a vocabulary
to identify feelings. Once children have the vocabulary to name feelings, they can be
taught to recognize those feelings in themselves and in others. After feelings are
recognized, children can be introduced to strategies that help regulate those feelings in
appropriate ways. The goal is for young children to begin to control anger and impulses
35
as they are developmentally able. At this level of the Teaching Pyramid, young children
are also introduced to problem solving skills, which include recognizing that a problem
exists, determining some possible solutions, selecting the best solution, and carrying it
out. Developing competency in friendship skills includes the ability to take turns, enter a
play group, share, offer assistance, and give compliments. Effective methods to teach
these skills include introducing the concept, modeling, role-playing, rehearsing,
prompting, and providing feedback (Fox & Hemmeter, 2009).
Intensive Individualized Interventions. The tertiary intervention level of the
Teaching Pyramid is Intensive Individualized Interventions. This top level of the
pyramid is reserved for use with approximately 5% of children who will continue to
exhibit intense and persistent challenging behaviors even after all other supports are in
place. At this level, a FBA is conducted and a behavior support plan is developed, which
includes interventions designed to prevent the behavior from occurring, to teach new
skills, and to create changes in adult responses to behaviors. The interventions are
comprehensive, and for consistency, should be implemented across all settings. Behavior
specialists or consultants should be provided to assist the teacher and family during the
initial implementation (Fox & Hemmeter, 2009).
Evaluating Positive Behavior Supports In Preschool Programs
While there is generally little research on the application of PBS for preschool
populations in comparison to other grade levels, several case studies on the use of PBS
strategies with preschool age children are emerging. For example, individualized PBS
for preschool aged children with autism resulted in dramatic improvements in functional
36
behaviors for six children. Dunlap & Fox (1999) implemented individualized supports to
assist families with children with autism. The children, all of whom were nonverbal,
exhibited a variety of problem behaviors including severe tantrums, running away from
adults, kicking, and head banging. The families were taught skills to promote long-term
changes. Prevention strategies were implemented and replacement skills were taught to
the children. Postintervention outcomes revealed reductions in tantrums and other
problem behaviors for all six children. Moreover, the children developed the ability to
play with family members, enroll in preschool, and improve communication via gestures
and one-word utterances.
Another case study of a young child with autism was presented by Buschacher &
Fox (2003). In this example, a comprehensive intervention plan based on the PBS
approach was developed in collaboration with the child’s family, school, speech therapist,
and others involved in his daily routines to address tantrums that occurred in his home,
community, school, and during private speech therapy. The behavior support plan
included prevention strategies and new skills to be taught, as well as appropriate
consequences such as praise or redirection as needed. Six months after the initial
implementation of the behavior support plan, tantrums were described as minimal, the
child was able to participate in routine community activities (shopping, beach,
playground, etc.), and he participated in language therapy.
PBS was also applied to two students in a community preschool program (Duda et
al., 2004). Results indicated that the children exhibited a reduction in problem behaviors
and increased engagement in classroom activities. The fidelity of implementation of the
behavior support plan was also evaluated. Structural supports, such as using specific
37
seating arrangements and establishing predictable routines were implemented with a
higher degree of fidelity than interactional interventions such as child-directed praise.
One of the first program-wide implementations of the PBS approach in an early
childhood program was in Kansas. It was established and assessed by Hemmeter, et al.
(2007). In the first year of implementation, teachers received pretraining and support
from a leadership team for development of behavior support plans. Additionally, three
program-wide rules/expectations (we use walking feet, we take turns, we use gentle
touch) were developed collaboratively with all stakeholders. In the second year, each
teacher was provided with a PBS Tool Kit (a notebook of resources for easy access to
support successful PBS implementation). Teachers were also given reinforcement by
their directors, and they received ongoing support from their leadership teams. They
were then able to share success stories via a weekly newsletter.
Outcomes of this program offer promise for other program-wide PBS initiatives
in early childhood settings. By the end of the first year, teachers indicated that they had
more confidence in dealing with challenging behaviors, thus relying on reduced levels of
outside support. Additionally, a policy change was made that eliminated the use of time
out as a behavioral intervention. At the end of the second year, there was a significant
reduction in teacher requests for crisis intervention. This resulted in a shift in
expenditures for the use of mental health consultants, with the focus moving from
predominantly intervention-based efforts to predominantly prevention-based efforts. By
the third year of the implementation of the PBS approach, only three requests were made
by teachers for crisis intervention in comparison to nearly 50 similar requests prior to
initiating PBS approach (Hemmeter et al., 2007).
38
Hemmeter et al. (2007) identified five factors that likely contributed to the
success of the PBS approach in this Kansas program: (1) A strong leadership team; (2)
Acknowledgement that the development of program-wide PBS takes time; (3)
Recognition for teachers’ commitment to the PBS approach; (4) Consultants with
experience in behavior support; and (5) Including mental health consultants in
development of the PBS program.
SWPBS and PBS for preschool populations appear to be a promising approach for
reducing challenging behaviors and promoting positive outcomes for all students,
educational staff, and schools. However, Sugai & Horner (2006) emphasized that
additional research is required to determine which aspects of SWPBS account for
reductions in challenging behaviors and sustainability and which are in need of
improvement. Hemmeter et al. (2007) acknowledge that a more rigorous evaluation is
necessary in order to establish program-wide PBS as an evidence-based practice.
Hemmeter and Fox (2006) developed, and are in the process of field testing, the
Teaching Pyramid Observation Tool Kit (TPOT). It is intended to assess the fidelity of
implementation of the Teaching Pyramid in preschool classrooms. The TPOT consists of
items that serve as indicators for teaching practices at each level of the pyramid. Items
include measures for supporting children’s play, providing feedback, examining
adequacy of classroom materials, teaching social emotional skills, and developing
individualized interventions. Administration of the TPOT includes an observation of the
classroom and an interview with the teacher. While this tool is very promising for
measuring outcomes of the use of the Teaching Pyramid, it does not appear to provide
any information about which components of the implementation process of the Teaching
39
Pyramid were most valuable to teachers and which components may be in need of
improvement.
Horner, Benedict, & Todd (2005, as cited in Benedict, Horner, & Squires, 2007)
developed another instrument for evaluating PBS practices in preschool settings. The
Preschool-wide Evaluation Tool (Pre-SET) was based on the Schoolwide Evaluation
Tool (SET), which was developed to evaluate universal and systems level PBS
interventions in elementary and upper grades (Sugai, Lewis-Palmer, et al., 2001, as cited
in Horner et al., 2004). Universal and systems level items on the Pre-SET were adapted
from the SET to be more applicable to early childhood settings. Unlike the SET, the
Pre-SET also includes categories to measure secondary and tertiary interventions, as well
as family involvement.
Benedict et al. (2007) used the Pre-SET to assess the impact of consultation on
PBS implementation in four early childhood classrooms. Results indicated that there was
an increase in percentage of the PBS features used by teachers after consultation, with the
greatest changes occurring in acknowledging positive behaviors and classroom
management. There were no changes in family involvement, monitoring and decision
making, or county/state support. The authors made an attempt to measure changes in
student behavior, but the overall problem behavior rates were low, and no discernable
differences were evident between pre- and post-consultation. A measure of the teachers’
perceptions of the PBS consultation was taken via a questionnaire. Generally, teachers
indicated that the PBS consultation was “excellent” (Benedict et al., p. 186) and would
recommend it to colleagues.
40
In addition to these program-wide evaluation tools, there is a limited but emerging
body of literature that examines and discusses specific behavioral management strategies
and components of PBS that relate to early childhood settings. For example, Stormont,
Covington Smith, and Lewis (2007) found a positive relationship between the teachers’
use of precorrection and praise and student behavior. Hiralall and Martens (1998) found
that training preschool teachers on scripted instructional sequences had a positive effect
on teacher and student behavior; half of the teachers maintained the use of an
instructional sequence (obtaining eye contact, signaling, directions, modeling, praise, and
redirectives) over a two month period and improvements in student behavior were
maintained. Nordquist and Twardosz (1990) emphasized the use of environmental
organization to prevent challenging behaviors in early childhood settings, indicating that
physical and programmatic features of classrooms can influence the incidence of
behavior problems. Neilson and McEvoy (2004) discussed the implications of functional
behavior assessments for preschool children with an emphasis on the need to involve
families.
The only process evaluation of a PBS program in an early childhood setting found
in the literature was conducted by Frey, Faith, Elliott, and Royer (2006). They evaluated
the implementation of universal level supports of a PBS model in a large Head Start
program. Frey et al. presented two evaluation questions. First, they assessed the
importance of the goals, procedures, and outcomes attributable to classroom management
interventions from the perspective of classroom teachers and other key stakeholders.
Second, they compared the differences in classroom environment between seven
classrooms that received intervention and an equal number of comparison classrooms.
41
Teachers who received the intervention were provided with mental health consultants
who facilitated collaborative efforts to design and implement a written preventative
classroom management plan. Teachers were asked to complete a satisfaction survey and
participate in a focus group. Items on the satisfaction survey were intended to gain
teachers’ perspectives of their input into the classroom management planning process,
knowledge of the mental health consultants, the problem-solving planning process, and
behavioral outcomes for their students. Independent observers used the Interaction
subscale of the Early Childhood Environment Rating Scale-Revised (ECERS-R) to
evaluate the classroom environment. Results of the survey indicated that teachers were
very positive about the classroom management interventions, with collaboration rated the
highest. No significant differences were found in classroom environments between
classrooms that received interventions and those that did not. However, the sample size
was too small to formulate any conclusions about the effects of the intervention.
As for all human services programs, the successful delivery of PBS programs in
early childhood settings is dependent upon the extent to which they are planned, carried
out, and modified (Maher & Bennett, 1984). A meaningful evaluation should start with
an in-depth understanding of the program, an open discussion of the program quality, and
the underlying values of the organization (Lee & Walsh, 2004). In public schools,
consultants for program evaluation are encouraged to engage teachers and other
stakeholders in the evaluation process so that the evaluation can have a meaningful
impact on how programs are implemented and valued (Lee & Walsh). Evaluations that
are purely outcome focused do little to identify program quality and fail to identify which
components of the program contributed to outcomes.
42
The approach to the evaluation of human service programs presented by Maher
(2000) is excellent for developing capacities of both novice and experienced evaluators to
identify a problem and build relevant processes for program evaluation. Program
evaluation is best conducted via a set of planned activities to assist with the development
and improvement of services (Maher & Bennett, 1984). Evaluation of programs in
schools has traditionally been informal (Maher & Bennett), with almost no attention paid
to preschool programs until recently. Maher and Bennett (1984) advocate for a program
planning and evaluation approach that promotes program improvements by making
evaluation efforts more open, encouraging documentation so that recommendations for
improvement can be communicated, and facilitating the review of such
recommendations. The program planning and evaluation framework presented by Maher
(2000) is well suited to the evaluation of PBS approaches in early childhood settings.
Summary
This chapter reviewed the literature relevant to the dissertation task. The actions
to address behavior problems in schools have evolved from almost strictly punitive
efforts to positive efforts. This paradigm shift began in special education as educators
recognized the need to implement measures aimed at preventing challenging behaviors
rather than using reactive approaches. The PBS framework for addressing challenging
behaviors was placed at the center of behavioral change efforts after IDEA 1997
mandated the use of positive intervention strategies for supporting students with
behavioral needs.
43
PBS offers a continuum of supports designed to meet the behavioral needs of all
students. Use of the PBS framework is emerging at school-wide levels, where all staff
work together to teach behavioral expectations and social skills. A number of case
studies have indicated that the PBS approach not only aids in the reduction of challenging
behaviors in students, but also improves teachers’ skills for effectively addressing
challenging behaviors.
While there has generally been a good deal of attention paid to the behavior
problems of school-age students, the behavioral needs of preschool students have
historically been neglected. During the past 20 years, however, attention paid to the
behavioral needs of preschool students has increased. A consensus of literature has
indicated that persistent, intense, challenging behaviors in preschool are associated with
problems in later life (Campbell, 2002). In order to prevent these negative outcomes,
preschool students need to experience high-quality early education programs where
behavioral interventions are based on functional assessments and teaching practices
develop prosocial skills (Dunlap et al., 2006).
The PBS Teaching Pyramid (Fox et al., 2005) was developed specifically to
address the social-emotional and behavioral needs of young children. This model was
selected by the NJDOE to be implemented in all public preschool programs. The
Teaching Pyramid is aimed at reducing challenging behaviors and increasing prosocial
behaviors. It is comprised of four levels: Building Positive Relationships, Classroom
Preventative Practices, Social Emotional Teaching Practices, and Individualized Intensive
Interventions.
44
PBS for preschool programs is in its infancy. Thus far, it has emerged almost
exclusively from case-based literature that examines the outcomes of individual students.
More recently, program-wide PBS efforts have gained attention. If the PBS approach,
however, is to be successful and sustainable over time, and not simply another swing of
the educational pendulum, research must be conducted to help identify which
components of PBS are most useful for developing teachers’ skills for addressing the
social emotional and behavioral needs of preschool children with challenging behaviors.
The program planning and evaluation framework presented by Maher (2000) was used to
evaluate the PBS approach in the OECE, and is discussed in the next chapter.
45
CHAPTER III
APPROACH TO PROGRAM EVALUATION
Overview
This chapter presents a description of the program evaluation framework
delineated by Maher (2000), which was used to evaluate the PBS program in the OECE,
with a focus on the 12 major activities of the Program Evaluation Phase. It also includes
a description of the PBS program, including important elements of the design of the
program. The final section of this chapter describes the organizational context in which
the evaluation plan was implemented.
The Program Evaluation Framework
The program planning and evaluation framework presented by Maher (2000)
consist of four phases: Clarification, Design, Implementation, and Evaluation. Together,
these four interrelated phases focus on the process of program planning and evaluation.
The purpose of focusing on the process of program planning and evaluation is to be able
to implement a program so that goals and needs of target populations can be met and
judgments about the merit of a program can be made. This chapter will briefly review
each phase of Maher’s framework, with an emphasis on the evaluation phase, which was
46
the focus of the dissertation. For more information about Maher’s program planning and
evaluation approach please see The Resource Guide for Planning and Evaluation of
Human Services Programs (Maher, 2000).
Clarification Phase
The purpose of the Clarification Phase, the first of the four phases of the program
planning and evaluation process, is to develop a clear understanding of the current
circumstances that are of concern to the client. This is accomplished by a series of
sequential, interrelated set of activities that result in an understanding of the target
population, their needs, and the context in which those needs are embedded. According
to Maher (2000), a clear understanding of the present situation is necessary to obtain a
controlled, predictable program planning and evaluation process, which will result in a
program that has value for the target population. Once all activities of the Clarification
Phase are completed, a written Clarification Report is developed for use by the consultant
and relevant stakeholders.
Design Phase
The second phase of Maher’s (2000) program planning and evaluation framework
is the Design Phase. The purpose of the Design Phase is to provide a clear understanding
of the program to be provided to the target population. According to Maher, a clear
understanding of the design of the program is necessary so that sound judgments can later
be made about how the program was implemented and the extent to which it added value
to the target population. The Design Phase provides clarity about the purpose, goals, and
47
activities of the human services program. It also provides guidance to program
implementers and administrators regarding how to proceed in a timely and economically
responsible manner. Without a well-developed program design, there is a risk that
implementors and other stakeholders will become disinterested in the program, which
will result in a reduction of desired outcomes.
The Design Phase is based on the information obtained during the Clarification
Phase. The four major activities of the Design Phase are to describe the program purpose
and goals, consider program design alternatives, develop the program, and document the
program design. These activities are sequential, interrelated, and reflexive. Each activity
guides the next, and changes in one activity may result in changes to the next. A written
Program Design Document directs the Implementation Phase and the Evaluation Phase.
Implementation Phase
The third major phase of Maher’s (2000) program planning and evaluation
framework is the Implementation Phase. The purpose of this phase is to ensure that the
program is implemented as it was designed. Maher noted several reasons that the
Implementation Phase is important. First, it is expected that the program will result in
value for the target population if it is implemented as intended. Second, if the program is
not implemented as designed there is a risk that worthwhile outcomes for the target
population will be diminished. Third, when a program is implemented as designed there
is a greater ability to make informed decisions about how to improve the program as it
operates. Finally, proper documentation of the implementation of the program is
necessary to determine how the program added value to the target population.
48
The major activities of the Implementation Phase are to review the program
design, facilitate the program implementation, and monitor the program process. As in
the Design Phase, each of these activities is sequential, interrelated, and reflexive.
Program Evaluation Phase
The last of the four major phases of Maher’s (2000) program planning and
evaluation framework is the Program Evaluation Phase, which was the focus of the
dissertation. The purpose of the Program Evaluation Phase is to gather and analyze data
so that sound judgments about the value of the program can be made. The Evaluation
Phase is actually one of the elements of the Design Phase, and as such begins early in the
process of program planning and evaluation. There are several reasons why the
Evaluation Phase is very important (Maher). First, a sound program evaluation can
assure that utilization of resources adds value to the target population. Second, a sound
program evaluation can contribute to program development and improvement.
Additionally, other program planning decisions, such as whether or not to expand the
program, can only be made based on important information about the program’s worth
and how it was implemented. Moreover, a sound program evaluation that addresses
external concerns such as those of boards of education and other entities, can contribute
to continued funding for the program. Finally, a sound program evaluation can facilitate
the involvement of key stakeholders for continued program improvement. Taken as a
whole, a sound program evaluation needs to be developed and conducted so that the
program can continue to operate, if appropriate, and it can be modified as needed to
better meet the needs of the target population.
49
Maher (2000) emphasizes four qualities of a sound program evaluation:
practicality, usefulness, propriety, and technical defensibility. First, a program
evaluation plan is considered practical if it can be implemented without interfering with
the daily activities and routines of the organization. Second, the program evaluation is
considered useful if it assists the key stakeholders with making effective decisions about
the program and its improvement. Third, a program evaluation is considered proper
when it aligns with all ethical and legal standards. Finally, a program evaluation is
technically defensible when the procedures, methods, and instruments can be justified
and are reliable, valid, and accurate.
There are twelve major activities in the Program Evaluation Phase. As in the
Design Phase and Implementation Phase, these activities are sequential, interrelated, and
reflexive. Although they are intended to be conducted in order, it may be necessary to
return to earlier activities so that adjustments can be made as deemed appropriate. The
remainder of this section will briefly describe each of the twelve activities. For a
comprehensive review of the steps, please see The Resource Guide for Program Planning
and Evaluation of Human Service Programs (Maher, 2000).
1. Identify the Client
The first of the twelve activities is to identify the client for the evaluation of the
program. Maher (2000) presents the following questions to consider when identifying
the client. (1) Who is the individual within the organization that is directly responsible
for the program design and implementation? (2) Who is the individual within the
organization that is directly responsible for overseeing the program, and is also
50
functioning in an administrative capacity? (3) Who is the external individual, group, or
agency that is interested in the design, implementation, and outcomes of the program?
The answers to these questions will aid in the identification of the primary client and
determine if there are multiple clients.
2. Determine the Client’s Needs for Program Evaluation
Once the client is identified, a thorough discussion of the reasons for a program
evaluation is warranted. A determination must be made as to whether or not the needs of
the client can be met via a program evaluation. Maher (2000) indicated that the
following reasons that contribute toward the importance of identifying the client’s needs
for program evaluation. First, clients are more likely to be involved in assuring that the
evaluation is planned and conducted appropriately if they have clarified why they need a
program evaluation. Second, consultants and stakeholders are in a better position to
decide whether and to what extent the needs of the client can be met via a program
evaluation if the client has explicitly articulated those needs. Third, once the program
evaluation needs of the client have been identified, it is possible to determine the client’s
level of understanding of and expectations for the program evaluation. There may also
be other reasons for determining the client’s needs for the program evaluation.
There are three tasks that assist with determining the client’s needs for program
evaluation. First, specify what the client wants to know about the program so that areas
of concern regarding the program can be identified. The information that the client
provides can be categorized into the current state of affairs and the desired state of
affairs regarding who the target population is, how the program was implemented, and
51
what value was added to the target population. Next, pinpointing the reasons that the
client wants knowledge about the program assists the consultant with determining
whether the client can be assisted by a program evaluation. Last, it is important to assess
how the client expects program evaluation information to be obtained. Clients that have
prior experience with a sound program evaluation process are more likely to understand
that the program evaluation process is a systemic process. Clients without sound
program evaluation experience may have unrealistic expectations. In either case,
assessing the client’s program evaluation expectations will assist with working with the
client.
3. Place the Program to be Evaluated into an Evaluable Form
In order for a sound program evaluation to occur, a human services program must
be placed into an evaluable form. Often, a consultant will be asked to evaluate a program
that has not been placed into a sound program design. In such instances, the consultant
will need to work with the client to place the program into an evaluable form via the
activities of the Design Phase (Maher, 2000) prior to engaging in any of the program
evaluation activities.
An evaluable program reflects a program design that meets three criteria: clarity,
compatibility, and development status. Clarity exists only to the extent to which written
information regarding each element of the program design is understood by all relevant
stakeholders. Compatibility exists only to the degree to which each program design
element is consistent with other program design elements. Development Status refers to
52
the degree to which each program design element is developed for successful
implementation.
Placing the program into an evaluable form may be time consuming, but it is very
important for several reasons. A fundamental task of the program planning and
evaluation process is to facilitate the continuous development and improvement of human
service programs. This task cannot be accomplished without a program that is clearly
understood by all stakeholders. Additionally, outcomes must not be considered in
isolation, but rather in relation to the program. In order to made sound judgments about
the program and the target population prior to and during the time that the program was
implemented, the program must be in an evaluable form. Moreover, human service
programs must be fully understood so that a determination can be made as to whether or
not a program can and/or should be replicated. Finally, placing a program into an
evaluable form is important because the designing and development of a program utilizes
a range of resources. Given such an expenditure of resources, it is necessary to know
what value the program is expected to bring to the target population so that these
expectations can be used as a basis for the program evaluation.
4. Delineate Program Evaluation Questions
Program evaluation questions are questions about some element of the program’s
design, implementation, or results that will facilitate program planning and evaluation
actions to be taken (Maher, 2000). These actions include making judgments about the
following: the merit of the program for serving the needs of the target population; the
worth of the program in adding value to the target population; the ability of the program
53
to be implemented as designed; and the program’s contributions to the organization
(Maher). Similarly, these actions may also include making decisions about the following:
how to use the evaluation information to make revisions in the program design; whether
and to what extent the program can be replicated in other settings; whether elements of
the program should be eliminated; and whether the entire program should be terminated.
Several tasks can be carried out to delineate program evaluation questions. These tasks
are: specify what needs to be known about the program; generate a list of program
evaluation questions; and select the most important questions to be answered. Once the
final program evaluation questions are identified they should be placed into a SMART
program evaluation form. The acronym SMART (specific, measurable, answerable,
relevant, timeframed) refers to characteristics of human service program evaluation
questions that increase the likelihood that data will be gathered specifically relating to the
question that the stakeholders will use to take effective program planning and evaluation
actions (Maher). Each program evaluation question is placed into a Program Evaluation
Protocol Worksheet.
5. For Each Program Evaluation Question, Specify the Data Collection Variables
A data collection variable refers to some item or matter that needs to be measured
to assist with answering the program evaluation question (Maher, 2000). There are two
tasks that must be taken for each program evaluation question. First, a list of variables on
which data can be collected is generated. Second, each variable must be operationalized
so that clarity will be reached as to what types of data need to be collected. This in turn
54
will serve to guide decisions about methods, procedures, and instruments for data
collection.
6. Describe the Data Collection Methods, Instruments, and Procedures
The next step in the Program Evaluation Phase is to determine how data will be
collected on each program question variable so that each question can be answered.
Maher (2000) presents four tasks that need to be accomplished for each program
evaluation question.
First, data collection variables for each question must be reviewed to identify the
most important ones. Each program evaluation question must be considered separately,
and then each variable is considered in relation to that question. A determination is made
regarding the importance of each variable and whether or not data can be collected on
each variable. It is possible that certain data collection variables are eliminated, and that
further reflection would result in other data collections variables being added to the list. If
the decision is made to add data collections variables to the list, they must be
operationalized and placed on a Program Evaluation Protocol Worksheet.
Once the most important data collection variables are identified, decisions are
made about the method and sources for data collection. The methods refer to the way in
which data are collected and may include questionnaires, tests, permanent product
review, rating scales, interviews, and observations. Data sources refer to the individual
or group on which data will be generated and may include the target population, program
personnel, files, records, or data bases, and other people.
55
The third task to be accomplished for each program evaluation question is to
decide about the procedures for data collection for each variable. Procedures refer to
when data are collected and whether or not a control group will be used, which will
impact upon the data collection procedures selected. Once procedures for data collection
have been identified, the final task is to select and/or develop instruments. In either
circumstance, instruments should meet the following qualities: practicality, usefulness,
propriety, and technical defensibility.
7. Describe the Methods and Procedures for Data Analysis
This activity is aimed at determining how to analyze the data that have been
collected so that program evaluation questions can be answered. It is important to
analyze and interpret the data in a systematic manner so that program evaluation
questions can be answered in a way that informs the client and other relevant
stakeholders. There are several tasks that will allow data to be analyzed and interpreted
using practical and technically sound methods and procedures (Maher, 2000). These
tasks include: selecting the unit of analysis; organizing and displaying the data;
identifying frames of reference; and determining statistical procedures.
8. Specify Program Evaluation Personnel and Responsibilities
The eighth step in the Program Evaluation Phase is to identify the people who will
be involved in the program evaluation and to clarify their roles and responsibilities
relevant to the program evaluation. The purpose of the activity is to increase the
likelihood that the program evaluation will occur as planned. There are several tasks that
56
must be completed for each program evaluation question, which will facilitate completion
of this step. These tasks are: identify the evaluation responsibilities and timelines;
determine the people who will be responsible; and discuss the timelines and
responsibilities with the designated people.
9. Delineate Guidelines for Communication and Use of Program Evaluation Information
The ninth step in the Program Evaluation Phase is a major activity of the program
planning and evaluation process. During this activity, guidelines are developed regarding
how to communicate and use program evaluation information for program planning.
Delineating guidelines for communication and use of program evaluation information
increases the likelihood that program planning actions will contribute to the continuous
development and improvement of the program. Communication refers to conveying the
results of program evaluation to targeted audiences in an informative manner, which can
be through written or oral methods. The use of program evaluation information refers to
reviewing, interpreting, and making decisions about program planning actions. The
following tasks need to be completed for each program evaluation question: target the
audiences for receipt of evaluation information; specify the evaluation information to be
communicated; determine how to involve the audience in the use of evaluation
information; and pinpoint program planning actions (Maher, 2000).
10. Construct Program Evaluation Protocols
During this activity, program evaluation protocols are developed and placed into a
written form as a program evaluation plan document. This activity is readily
57
accomplished via successful completion of the first nine activities, during which
information was placed onto respective Program Evaluation Protocol Worksheets for
each question. The headings on the Program Evaluation Protocol Worksheet developed
by Maher (2000) are:
• Program Evaluation Question
• Data Collection Variables
• Data Collection Methods, Instruments, and Procedures
• Methods and Procedures for Data Analysis
• Guidelines for Communication and Use of Program Evaluation Information
The following program evaluation plan format is presented by Maher (2000). It can be
used as a reference when there are questions about the program evaluation and how it
relates to program planning.
I. Overview of the Program Evaluation
A. Client and Client Information Needs
B. Timeframe of the Evaluation
II. Description of the Program that was Evaluated
III. List of Program Evaluation Questions
IV. Program Evaluation Protocols
Appendix A – Copies of Instruments
Appendix B – Professional Biographical Sketch of Consultant/Program Planning
and Evaluation Team (optional)
58
11. Implement the Program Evaluation
During this step, the program evaluation is implemented based on the information
provided in the program evaluation protocols. The aim is to make sure that the process of
the evaluation is controlled as expected based on the following indicators presented by
Maher (2000) for each program evaluation question: data are collected on variables
specified in the protocol; methods, procedures, and instruments designated in the protocol
are used; data analysis and interpretation are based on the methods and procedures
articulated in the protocol; and evaluation results are communicated to the target
audiences and used by them for program planning. If it is necessary to modify the
evaluation process and revise one or more protocols, a rational for such changes must be
made clear.
12. Evaluate the Program Evaluation
Through this final step of the Program Evaluation Phase, the program evaluation
that has been implemented is itself evaluated. This step is very important so that relevant
stakeholders can determine how to improve future program evaluations as well as the
entire program planning and evaluation process. Maher (2000) poses four questions that
align with the four qualities of a sound human services program evaluation: practicality,
utility, propriety, and technical defensibility. These questions are:
1. To what extent was the program evaluation conducted in a way that
allowed for its successful accomplishment? (Practicality)
2. In what ways was the resulting program evaluation information helpful to
people? Which people? (Utility)
59
3. Did the program evaluation occur in a way that adhered to legal strictures
and ethical standards? (Propriety)
4. To what degree can the evaluation be justified with respect to matters of
reliability and validity? (Technical Defensibility)
60
Description of the Program Design
The PBS program was already being implemented when it was decided to conduct
an evaluation of the program. In accordance with Maher’s (2000) framework for
program planning and evaluation, it was important to place the program into an evaluable
form. The consultant constructed the program design based on recollections as a
participant-observer, review of records, interviews with PIRT members, and meetings
with one of the OECE supervisors. After developing an initial program design document,
the consultant provided a copy to one PIRT member to obtain feedback. The following is
a description of the PBS program as implemented during the 2004-2005 through 2008-
2009 school years. The program design follows the format that is presented in The
Resource Guide for Planning and Evaluating Human Service Programs (Maher, 2000).
Target Population
The PBS program was targeted toward the teachers of approximately 1300
preschool students enrolled in the early childhood education program in a medium sized
urban public school district. For the 2008-2009 academic year there were 87 teachers
participating in the early childhood program. See Figure 1 for a complete distribution of
classrooms in the Early Childhood Program. Teachers may be working in classrooms at
one of the following locations:
(1) privately owned early childhood centers contracted with the public school
district;
(2) early childhood centers overseen by the county regional educational services
commission; or
61
(3) preschool classrooms located within the public school district schools.
About half of the teachers have obtained teacher certification via the “alternate
route” within the last three years. The remaining teachers earned teaching certification
via traditional methods. Most of the teachers in the early childhood program are
considered novice teachers in that they have fewer than five years of teaching experience.
All but three teachers are female. About one third of the teachers are bilingual
English/Spanish speakers.
Statement of Purpose
The program was organized around the PBS pyramid model selected by the
NJDOE for all public early childhood education programs. The purpose of the PBS
program was to provide the teachers of the OECE with the skills and supports necessary
to reduce challenging behaviors and increase prosocial behaviors among preschool
students in the classroom. Preschool teachers working as part of the OECE will receive
training, coaching, and modeling in the PBS model. Through this program, teachers will
develop the knowledge and skills necessary to reduce challenging behaviors and promote
prosocial behaviors among preschool students.
62
Figure 1. Distribution of Classrooms in the Early Childhood Program During 2008-2009.
Office of Early Childhood Education
County Regional Educational Services
Commission
Public School
District
Individually Owned Private Early
Childhood Centers
School A: 15 Classes
School K: 12 Classes
School L: 6 Classes
School M: 6 Classes
School B: 9 Classes
School C: 5 Classes
School D: 4 Classes
School E: 4 Classes
School R: 1 Class
School Q: 1 Class
School P: 3 Classes
School O: 4 Classes
School N: 4 Classes
School F: 4 Classes
Public School District
School G: 3 Classes
School H: 3 Classes
School I: 2 Classes
School J: 1 Class
63
Program Goals 1. Early childhood teachers will engage in teaching behaviors and activities that are
designed to prevent or reduce the incidence of challenging behaviors among
preschool children.
2. Early childhood teachers will identify triggers that result in the occurrence of
challenging behaviors.
3. Early childhood teachers will explain the function of challenging behaviors in
preschool children.
4. Early childhood teachers will identify adult responses that perpetuate the occurrence
of challenging behaviors in preschool children.
5. Early childhood teachers will identify new skills that can be taught to preschool
children that would replace challenging behaviors.
Eligibility Standards
All preschool teachers who teach in classrooms that fall under the auspices of the
Office of Early Childhood Education were expected to participate in the PBS approach.
Phases
The PBS pyramid model was introduced to teachers according to the following
structure and timeframes:
Year 1 1. All preschool teachers participated in a full-day professional development
workshop on the PBS approach for reducing challenging behaviors and
64
increasing prosocial behaviors as a means of offering teachers a starting point
for implementing the approach in their classrooms.
2. PIRT members observed classroom teachers during the regular school day and
provided consultation in the form of written strategies, modeling, and
coaching of the PBS approach.
3. Technical assistance was provided for functional behavioral assessments and
subsequent behavioral support plans for individual students, as needed.
Year 2
1. New preschool teachers, who joined the early childhood program, participated
in a full-day professional development workshop on the PBS approach.
a. Any teacher who received training in the prior year, but wanted the
benefit of reviewing the PBS model, also participated in the full-day
workshop.
2. Mini lunch-and-learn workshops were provided to teachers in small groups at
individual early childhood centers. Topics included specific strategies that
would facilitate the PBS approach in the classrooms (e.g., emotional literacy,
self-regulation for preschoolers, making friends, etc.)
3. PIRT members observed classroom teachers during the regular school day and
provided consultation in the form of written strategies, modeling, and
coaching of the PBS approach.
4. Technical assistance was provided for functional behavioral assessments and
subsequent behavioral support plans for individual students, as needed.
65
Years 3 and 4
1. New preschool teachers, who joined the early childhood program,
participated in a full-day professional development workshop on the PBS
approach.
a. Any teacher who received training in the prior year, but wanted the
benefit of reviewing the PBS model, also participated in the full-day
workshop.
2. Mini lunch-and-learn workshops were provided to teachers in small groups at
individual early childhood centers. Topics included specific strategies that
would facilitate the PBS approach in the classrooms (e.g., emotional literacy,
self-regulation for preschoolers, making friends, etc.)
3. Each classroom teacher was assigned a PIRT Coordinator to act as primary
consultant.
Components
A detailed description of the components that were part of each phase of the PBS
program is presented below. First, components relevant to training are presented,
followed by the components relevant to consultation.
I. Training
A. Activities
66
1. Full-day professional development workshops were provided to
teachers, assistant teachers and administrators on the PBS framework
selected by the NJDOE.
2. Mini lunch-and-learn workshops were provided to teachers in small
groups at individual early childhood centers.
B. Method
1. The full-day professional development was scheduled during the
regular work day on several different dates in order to be able to limit
participant attendance to between 30 and 40 teachers at each workshop.
Didactic methods were used to introduce teachers to basic PBS concepts
for developing positive relationships, creating supportive environments,
teaching social emotional skills, and supporting individual students with
persistent challenging behaviors. Each workshop included lecture and
PowerPoint presentation, video clips, modeling, break-out groups for
activities, development of classroom-wide action plans, and question and
answer periods.
2. For lunch-and-learn workshops, teachers were presented with specific
strategies that they could use in their classrooms to facilitate
implementation of the PBS approach.
C. Materials
Materials used by workshop presenters for full-day and lunch-and-learn
trainings were based on the Preschool Training Modules (Center for Social
and Emotional Foundations for Early Learning, CSEFEL, 2009) which
67
included PowerPoint presentations, presenter guides, and video clips.
Trainers also developed additional materials for modeling, role play, and
small group activities. Participant handouts from the Preschool Training
Modules (CSEFEL, 2009) were selected and adapted based on perceived
target population needs.
C. Forms
At the end of each workshop, each participant was provided with an
evaluation form to rate the quality of the workshop relevant to their needs.
D. Equipment
1. Audiovisual equipment including television, VCR, computer, and
PowerPoint projector were used in full-day workshops. Other equipment
included flip charts, pens, markers, and large child-sized puppets for
modeling how to teach prosocial skills.
2. Equipment for lunch-and-learn workshops included pens, markers, and
puppets.
E. Facilities
1. Full-day workshops were conducted at a predetermined central location
in a public school district building.
2. Lunch-and-learn workshops were conducted in common areas at
individual early childhood centers.
F. Roles, Responsibilities, Relationships
1. Preschool teachers were required to attend full-day workshops as
scheduled. Attendance at lunch-and-learn workshops was voluntary.
68
2. PIRT members were responsible for training teachers on the PBS
approach; observing classroom teachers during the regular school day and
providing consultation in the form of written strategies, modeling, and
coaching of the PBS approach; providing technical assistance for
functional behavioral assessments and subsequent behavioral support
plans for individual students, as needed; facilitating Requests for
Assistance regarding specific students; and conducting collaborative
meetings with other key stakeholders (parents, family workers, master
teachers, center directors, building principles, school nurses, etc.) to
develop action plans to support such students.
3. The Early Childhood Supervisors were responsible for monitoring the
implementation of the PBS program.
II. Consultation
A. Activities
1. PIRT members were to consult with classroom teachers regarding
behavior problems in the classroom.
2. PIRT members were to conduct regular classroom visits to observe and
model the use of appropriate behavior support strategies.
3. PIRT members were to assist with the planning and implementation of
behavior support plans for children who continued to exhibit challenging
behaviors after all other supports were in place.
69
B. Method
A consultation model was used to build teachers’ skills for preventing
challenging behaviors and promoting prosocial behaviors in preschool
students.
C. Techniques
1. Intervention and support meetings were held with teachers, family
workers, parents, and administrators regarding students who exhibited
persistent challenging behaviors.
2. Observations, coaching, and modeling in classrooms were used to build
teachers’ abilities to successfully implement strategies.
3. Written strategies were provided to teachers outlining individualized
behavior support strategies for specific students.
D. Materials
Materials for consultation, modeling and coaching were adapted from the
Preschool Training Modules (CSEFEL, 2009). Consultants also
developed additional materials for modeling of specific strategies.
E. Facilities
Intervention and support meetings were held in a meeting room at the
early childhood center of the teacher and student. Observation, coaching,
and modeling occurred in the classroom of the teacher and student.
F. Roles, Responsibilities, and Relationships
1. Preschool teachers were to implement the PBS approach in their
classrooms, make referrals to PIRT for children who continued to exhibit
70
persistent severe challenging behaviors, participate in request for
assistance meetings, and allow PIRT members into their classroom for the
purpose of observing, modeling, and coaching.
2. PIRT members were to review request for assistance forms, facilitate
request for assistance meetings, observe students and teachers in
preschool classrooms, and model and coach appropriate implementation of
specific strategies.
3. Family workers were to schedule request for assistance meetings,
attend request for assistance meetings, and function as liaison between
families and early childhood staff.
The preceding sections of this chapter described the approach used to evaluate the
PBS program in the OECE, and was based upon the program evaluation framework
presented by Maher (2000). The following sections of this chapter will describe the
relevant organizational context.
71
Relevant Organizational Context
The purpose of presenting the relevant organizational context is to carefully
consider information that relates to the readiness of the target population, the client,
relevant stakeholders and the organization for the design of a human services program
that can address important needs (Maher, 2000). According to Maher, relevant context
refers to information about whether, how, and when to proceed with designing and
implementing a human service program to meet the needs of a target population. This is
essential because the target population does not exist in a vacuum, but rather they are
embedded in several existing contexts (Maher, 2000). Accordingly, the relevant contexts
such as social, cultural, and community, must be understood in order to develop an
effective human services program.
Maher (2000) cites several reasons for delineating the relevant context. First,
contextual factors that may facilitate the design and implementation of a human service
program can be identified and considered. Conversely, contextual factors that may
inhibit program design and implementation can also be identified and considered.
Another reason for delineating the relevant context of the organization is to be able to
determine the readiness of the organization for a human service program, specifically
providing information regarding the extent to which a program may be designed and
when and if it can be implemented. Finally, understanding the relevant contextual factors
allows stakeholders to make judgments about the merit of the program and subsequently
allows for effective decision making about implementation of the program in other
settings.
72
There are several steps that a consultant and other relevant stakeholders can take
to understand the relevant context of the organization. Maher (2000) presented the A
VICTORY framework to facilitate the consideration of important contextual factors.
Each letter of the A VICTORY acronym represents a contextual factor, which should be
examined and considered with stakeholders in a progressive step-by-step manner. The
factors assessed with the A VICTORY framework are:
• Ability of the organization to commit resources for design, implementation, and
evaluation of a human services program
• Values that people within the organization and other relevant stakeholders ascribe
to the target population, their needs, and evaluation of the program
• Ideas that people have about the current situation with respect to the target
population, their needs, and evaluation of the program
• Circumstances within the organization that relate to its structure and direction
• Timing of the design, implementation, and evaluation of the human services
program
• Obligation of organizational members and other stakeholders to address the needs
of the target population in a programmatic manner
• Resistance that might be encountered with respect to the design, implementation,
and evaluation the human services program
• Yield or benefit that may result for the target population as a result of the
program and its evaluation
73
There are a number of approaches that can be used to obtain information about
each of these contextual factors. Interviews and questionnaires can be utilized to obtain
information from key stakeholders about the contextual factors on the A VICTORY
framework to identify which may serve to facilitate or inhibit the design and
implementation of a human service program. Permanent product reviews may be
conducted to make judgments about the contextual factors. Finally, participant
observation may allow for judgments to be made about the contextual factors based on
involvement with the organization.
Program planning and evaluation consultants may enter an organization at any
phase of the human services program. If a client seeks the services of a consultant prior
to the program implementation, the consultant can assist the client with working through
all four phases of the program planning and evaluation process. In this case, the
consultant will conduct an assessment of the relevant contextual factors during the
Clarification Phase (Maher, 2000). If, however, the client seeks to involve a consultant
during the Evaluation Phase, the consultant will need to place the program into an
evaluable form, which includes delineating the relevant contextual factors of the
organization (Maher, 2000). The current investigator was asked to be involved in
evaluating the PBS program after the program was already in the Implementation Phase.
Accordingly, the consultant used the A VICTORY framework to determine the readiness
of the organization for a program evaluation.
In order to delineate relevant contextual factors, the investigator interviewed one
of the supervisors of the OECE. The investigator also functioned as a participant
74
observer, which provided first-hand knowledge and insight into the organization. The
organizational context for the OECE is presented below.
Ability of the OECE to Commit Resources
The human resources of the OECE consisted of two supervisors, five PIRT
members, six master teachers, three school nurses, one community parent involvement
specialist, and three secretaries. The organization was able to dedicate staff time to the
development, implementation, and evaluation of a human services program. The OECE
was also able to influence the Early Childhood Center directors toward designating time
for teachers to be available for professional development. All technological resources
possessed by the OECE (computer, software, projectors, copiers, etc.) were made
available for the development, implementation, and evaluation of the PBS program.
Any informational resources possessed by the OECE were made readily available
toward the design, implementation, and assessment of the PBS model. Available
financial resources were made available, but available funds differed from year to year.
Temporal resources did not offer any restrictions toward the design, implementation, and
assessment of a human services program for the target population, as long as no
additional costs were incurred.
Values of the Organizational Members
Members of the OECE have traditionally been concerned with the needs of the
teachers, and have valued professional development. The desire to support teachers
working with children at risk for future school failure and children with special needs is
75
very strong. To that end, the OECE has traditionally been very responsive to providing
trainings for teachers on a variety of pedagogical and behavioral practices.
Ideas People Have About the Current Situation
The current supervisors were not the supervisors of the OECE at the outset of the
implementation of the positive behavior support approach. However, the previous
supervisor was clear that the OECE needed an approach for teachers to reduce
challenging behavior and increase prosocial behaviors in their classrooms. The current
OECE supervisors are also clear about the continued need for a human services program
for teachers on reducing challenging behaviors and increasing prosocial ones.
Accordingly, the supervisors are clear about the tasks to be accomplished. Building upon
teachers’ skills is embraced by all members of the organization.
Circumstances in the OECE with Respect to its Structure and Direction
Key administrators are expected to remain in current positions for the foreseeable
future. In the past, there have been frequent changes in OECE administration, but since
the missions and strategic plan are not expected to change over the next two to three
years, it is not likely that any potential changes in administration will impact the design,
implementation, and assessment of a human services program for the target population.
Timing of Using a Programmatic Approach in the Organization
Administrators and other key stakeholders were prepared to allow time for the
design, implementation, and assessment of a human services program. There were no
76
current events that impacted this program or the organization during the past four to five
years, and no current events are expected to have an impact on the program or the
organization in the coming two to three years. Funding sources were expected to remain
stable.
Obligation of Individuals and Groups
There were and are many active supporters for this program within the OECE
such as administrators, PIRT members, OECE nurses, and OECE master teachers. No
groups have been identified that may have opposed a program initiative for the target
population. Only the current investigator and one supervisor were originally aware of the
intent to evaluate the PBS program. PIRT members, center directors, teachers, and other
stakeholders were informed about the plan to evaluate the program when the supervisor
and the current investigator were ready to implement the evaluation.
Resistance Expected by Individuals and Groups
Minimal to no resistance was anticipated for the implementation of the PBS
program. Some center directors, however, may have been resistant to releasing teachers
for professional development because of the costs related to hiring substitute teachers on
professional development training days. Nonetheless, all center directors complied with
requests for allowing teachers to attend trainings. Resistance toward the evaluation of the
PBS program was anticipated on the part of some PIRT members because they might
have felt that an evaluation of the PBS program would reflect upon them poorly. In order
to reduce this resistance, the evaluation was designed to be completely anonymous.
77
Yield, or Value, of the Information
Perceived benefits for the target population included personal growth and
development in the area of teaching skills and classroom management. Potential
drawbacks of the program and its evaluation included uniformity of implementation
across all early childhood centers.
Summary
The context information indicated that the organization was ready to proceed with
the evaluation of the PBS program for the target population. The OECE was able to
commit resources and the timing was deemed appropriate. There were, and continue to
be, many active supporters within the organization for this program and its evaluation. It
was determined that resistance could be minimized via careful design of the evaluation.
Many benefits for the target population were identified.
78
CHAPTER IV
PROGRAM EVALUATION PLAN
Overview of the Chapter
This chapter describes and reviews the program evaluation plan for the PBS
program in the OECE. It follows the program evaluation framework presented by Maher
(2000), which includes determining the client’s needs for a program evaluation,
evaluation questions, methods and procedures for answering and analyzing responses to
each question, guidelines for communication and use of program evaluation information,
and a plan for evaluating the program evaluation plan. Approval from the Rutgers
University Institutional Review Board for the Protection of Human Subjects was obtained
prior to implementing the program evaluation plan.
Overview of the Program Evaluation Plan
Client Needs
One supervisor of the OECE was interested in obtaining information about the
PBS program that could be used for its continuous development and improvement.
Specifically, the supervisor wanted to know if the program was being implemented as
designed and what value, if any, it was providing for the target population. She also
79
wanted to know who participated in the PBS program, whether the classroom teachers’
knowledge, skills, and abilities for supporting students with challenging behaviors
improved, and whether the staff was satisfied with the program.
Time Frame
The program evaluation was designed to be implemented at the end of the fifth
year of implementation of the PBS program. This pilot program evaluation was
conducted during April 2009 through June 2009.
Description of the Program
The description of the PBS program was provided in the previous chapter.
List of Program Evaluation Questions
1. Who participated in the PBS approach?
2. How has the PBS approach been implemented?
3. What were the reactions of preschool teachers to the PBS approach in terms of
strengths, adequacies, and areas in need of improvement?
4. To what extent were the goals of the PBS program attained?
80
Program Evaluation Protocols
Protocol No. 1
Program Evaluation Question 1. Who participated in the PBS approach?
Data collection variables. The data collection variables included relevant
characteristics about the preschool teachers and students. For the teachers, these
variables included gender, type of teacher certification, number of years of teaching
experience, and the number of students referred to PIRT. For the students, these
variables included age, gender, classroom placement, why they were referred to PIRT,
and whether or not there was a CST referral.
Data collection methods, instruments, and procedures. Data were collected by
several methods. An annual permanent product maintained by the OECE was reviewed
to collect data on gender, the type of teacher certification, and the number of referrals to
PIRT. The data for the number of years of teaching experience were collected on
Instrument 1 (Appendix A). Data on relevant students’ characteristics were also
collected via the review of an annual permanent product maintained by the OECE. The
permanent product contains information on students referred by teachers to the PIRT
which included date of birth, school attended, name of teacher, date of referral, reason for
referral, whether or not the student was also referred to the CST, and the CST
determination. The current investigator conducted the permanent product review, as well
as distributed, collected and reviewed Instrument 1, Positive Behavior Support Teacher’s
Questionnaire (Appendix A). Data on preschool teachers were recorded on Instrument
81
1.1, Preschool Teacher Statistics. Data on students were recorded on Instrument 1.2,
Preschool Student Statistics.
Methods and Procedures for Data Analysis. Data analysis units included the
statistics regarding the relevant characteristics of participants. Means and percentages
were calculated for each variable. Data were placed in a table to display the frequency of
distribution for each variable. Means and percentages were also calculated for each
variable.
Personnel and Responsibilities. The current investigator was responsible for
collecting, reviewing, and analyzing data, as well as completing Instrument 1.1,
Preschool Teacher Statistics and Instrument 1.2, Preschool Student Statistics. The
investigator was also responsible for organizing and displaying data in tables.
82
Protocol No. 2
Program Evaluation Question 2. How has the PBS approach been implemented?
Data Collection Variables. Data collection variables included the description of
adherence to the program as well as judgments about the adherence to the program.
Variables also included the manner in which program activities, methods, and procedures
for the PBS approach were executed.
Data Collection Methods, Instruments, and Procedures. The methods of data
collection were permanent product review and interview of key stakeholders. Data were
collected on teacher trainings via permanent product review. Types and frequency of
teacher support provided were obtained via interview of key stakeholders. Data on
teacher trainings were recorded on Instrument 2.1, Professional Development on Positive
Behavior Supports. Data on types and frequency of teacher support provided were
recorded on Instrument 2.2, Provision of Support to Teachers.
Methods and Procedures for Data Analysis. Units of analysis were descriptive
statistics for teacher trainings and teacher support strategies. Frequency and percentages
were calculated and placed in tables.
Personnel and Responsibilities. The current investigator was responsible for
permanent product review and interview of key stakeholders. The investigator was also
responsible for collecting data, completing Instruments 2.1 and 2.2, and organizing and
displaying data.
83
Protocol No. 3
Program Evaluation Question 3. What were the reactions of preschool teachers
to the PBS approach in terms of strengths, adequacies, and areas in need of
improvement?
Data Collection Variables. Data collection variables were preschool teachers’
thoughts, judgments, and opinions of the implementation of the PBS approach for
reducing challenging behaviors among preschool students. Teachers were lead classroom
teachers in the OECE program. All lead classroom teachers in the program were
surveyed.
Data Collection Methods, Instruments, and Procedures. The data collection
method included the distribution, completion, and collection of Instrument 1, Positive
Behavior Support Teacher’s Questionnaire. Lead classroom teachers were asked to
complete Instrument 1 during regularly scheduled staff meetings.
Methods and Procedures for Data Analysis. The units of analysis were the
responses of lead teachers to the items on the questionnaire. There were a variety of
items on the questionnaire. Teachers were asked to respond to some questions on a 3-
point scale, some questions on a 5-point scale, and to place items on a list in rank order of
importance. The questionnaire also included open-ended items. Descriptive statistics
were used for data analysis and interpretation. The data were displayed in tables.
Personnel and Responsibilities. The investigator was responsible for distributing
and collecting Instrument 1. Lead teachers were responsible for completing and
returning Instrument 1 to the investigator. The investigator was responsible for
organizing and displaying data.
84
Protocol No. 4
Program Evaluation Question 4. To what extent were the goals of the PBS
program attained?
Data Collection Variables. Data collection variables were preschool teachers’
perceptions of their abilities to engage in the following activities:
Implement activities and routines that prevent challenging behaviors
Identify triggers that may result in challenging behaviors
Identify functions of challenging behaviors
Identify adult responses to challenging behaviors that may decrease challenging
behaviors
Teach new skills that would replace challenging behaviors
Data Collection Methods, Instruments, and Procedures. The data collection
method included the distribution, completion, and collection of Instrument 1, Positive
Behavior Support Teacher’s Questionnaire. Lead classroom teachers were asked to
complete Instrument 1 during regularly scheduled staff meetings.
Methods and Procedures for Data Analysis. The units of analysis were the
responses of lead teachers to the items on Instrument 1. There were a variety of items on
the questionnaire. Teachers were asked to respond to some questions on a 3-point scale,
some questions on a 5-point scale, and to place items on a list in rank order of
importance. The questionnaire also included open-ended items. Descriptive statistics
were used for data analysis and interpretation. The data were displayed in tables.
85
Personnel and Responsibilities. The investigator was responsible for distributing
and collecting Instrument 1. Lead teachers were responsible for completing and
returning Instrument 1 to the investigator. The investigator was responsible for
organizing and displaying data.
Communication of Program Evaluation Information
After the program evaluation data was collected, the evaluation consultant
analyzed the data and produced a report. The report included tables, graphs, and
narrative information. It was presented to the supervisors of the OECE at a face-to-face
meeting held during the fall of the next program year. During the meeting, the evaluation
consultant reviewed the report, discussed the findings from the evaluation, and made
initial recommendations. The supervisors of the OECE reviewed the evaluation
information provided and made determinations on how to address what changes and
improvements needed to be made to the PBS program. The supervisors of the OECE
were to meet with other key stakeholders such as PIRT members and other staff in the
OECE, early childhood center directors, and teachers to discuss evaluation information
and the modifications that will be made to the program as a result of the evaluation.
Evaluation of the Program Evaluation
After implementing the program evaluation and analyzing the data, the evaluation
consultant also reviewed the process of implementing the evaluation plan. The
evaluation consultant tallied the number of people who participated in the evaluation to
determine the response rate. The evaluation consultant interviewed the supervisors of the
86
OECE and other key stakeholders to elicit feedback about the process of the program
evaluation and reactions about its usefulness. The following four questions, which are
based on Maher’s (2000) four qualities of a sound human services program evaluation,
were addressed:
1. To what extent was the program evaluation conducted in a way that
allowed for its successful accomplishment? (Practicality)
2. In what ways was the resulting program evaluation information helpful to
people? Which people? (Utility)
3. Did the program evaluation occur in a way that adhered to legal strictures
and ethical standards? (Propriety)
4. To what degree can the evaluation be justified with respect to matters of
reliability and validity? (Technical Defensibility)
87
CHAPTER V
RESULTS OF THE PROGRAM EVALUATION
Overview
This chapter reviews and reports the results of the evaluation of the PBS program
in the OECE. Four program evaluation questions were addressed. The methods,
procedures, and instrumentation described in Chapter IV were used to answer each
programmatic question, and are discussed in the chapter. Copies of all instruments used
in the program evaluation are presented in Appendix A. This chapter also reviews the
results of the evaluation of the program evaluation.
Results of Program Evaluation Question 1
Program Evaluation Question 1: Who participated in the PBS approach?
The first program evaluation question sought to determine the relevant
characteristics of the teachers and students who were involved in the PBS program. It is
important to develop a thorough understanding of the participants of the program because
this information may help to better serve their needs. In order to answer this question,
data were collected via permanent product review and Instrument 1, Positive Behavior
Support Teacher’s Questionnaire. Data were organized and recorded by the program
88
evaluation consultant on Instrument 1.1, Preschool Teacher Statistics and Instrument 1.2,
Preschool Student Statistics. Data were collected on teachers’ gender, years of teaching
experience, and teacher certification. For students, data were collected on gender, age,
classroom placement, reason for referral to PIRT, whether or not there was a referral to
the CST, and the CST determination for eligibility for special education and related
services.
In order to answer the question about relevant characteristics of the preschool
teachers for the 2008-2009 academic year, the program evaluation consultant reviewed a
permanent product maintained by administrative staff in the OECE which consisted of a
spreadsheet of teacher credentials. An electronic version of the spreadsheet was provided
to the program evaluation consultant. The spreadsheet contained information about the
status of teacher certification. This data was recorded on Instrument 1.1, Preschool
Teacher Statistics. The program evaluation consultant also obtained information about
relevant teacher characteristics using Instrument 1, Positive Behavior Support Teacher’s
Questionnaire, which inquired about years of teaching experience. This data was also
recorded on Instrument 1.1, Preschool Teacher Statistics. Data on teachers were only
included for the 2008-2009 academic year because these were the teachers that
participated in the program evaluation. Data regarding the relevant characteristics of
preschool teachers are presented in Table 1.
In order to answer the question about relevant preschool student characteristics,
the program evaluation consultant reviewed a permanent product maintained by the
administrative staff of the OECE. This spreadsheet contained information about the
student’s age, classroom placement (in-district or contracted early childhood center), the
89
reason for the referral to the PIRT, whether or not the child was referred to the CST, and
the determination of eligibility for special education and related services made by the
CST. Relevant characteristics of preschool students for the four years of the PBS
program were reviewed. The data on the total number of students in the OECE program,
referrals to the PIRT, and referrals to the CST were recorded on Instrument 1.2,
Preschool Student Statistics and are presented in Table 2. Other relevant student
characteristics were also recorded on Instrument 1.2, Preschool Student Statistics and are
presented in Table 2.
Relevant Characteristics of Teachers
During the 2008-2009 academic year, the early childhood education program had
87 classroom teachers. The teachers were predominantly female (96.6%); only three
teachers were male. Data on teaching certification were available for 75 of the 87
teachers in the OECE. Of these teachers, 42 (48.2%) earned their teaching certification
via traditional means and 33 (37.9%) earned their teaching certification through the
alternate route program. The alternate route program is a non-traditional teacher
preparation program designed for those individuals who have not completed a formal
teacher preparation program at an accredited college or university, but wish to obtain the
necessary training to become a NJ certified teacher (NJDOE, 2009). To obtain teacher
certification via the alternate route, prospective teachers must apply for and obtain a
certificate of eligibility through the Office of Licensure and Academic Credentials and
secure employment in a public or private school requiring certification. The hiring
school district must enroll the prospective teacher in the Provisional Teacher Program,
90
which is a school-based training program. A provisional teaching certificate is issued for
two years. Alternate route teachers are also mentored by colleagues in the district and are
evaluated by district personnel. Once the Provisional Teacher Program is successfully
completed, a standard teaching certificate is issued.
Data on level of teaching experience were only obtained for teachers who
completed the questionnaire. The level of experience among the teachers varied. There
were nine teachers (12.3%) that had 1-2 years of teaching experience and nine teachers
(12.3%) that had 3-4 years of teaching experience. Twenty-one teachers (28.8%) had 5-6
years of teaching experience. Almost half of the teachers (46.6%) had seven or more
years of teaching experience. Data on relevant characteristics of preschool teachers are
presented in Table 1.
Table 1 Characteristics of Preschool Classroom Teachers Variable Frequency Percent Gender Female 84 96.6% Male 3 3.4% Teacher Certification Traditional Certification 42 48.2% Alternate Route 33 37.9% Unknown 12 13.7% Years of Teaching Experience (for teachers completing the questionnaire)
1-2 years 9 12.3% 3-4 years 9 12.3% 5-6 years 21 28.8% 7 or more years 34 46.6%
91
Relevant Characteristics of Preschool Students
2005-2006 School Year. During the 2005-2006 school year, there were 1154
students enrolled in the preschool program. All but 30 students attended preschool in
early childhood centers that were contracted with the OECE; the district had only two
regular education preschool classrooms within the public school buildings. Of the 247
students that were referred by teachers to the PIRT for assistance, 37.2% were female and
62.8% were male. Although data on age for the 2005-2006 year were missing, available
data suggested that there were approximately twice as many three-year-olds as four-year-
olds referred for assistance. The most common reason for referrals to the PIRT was for
language delays, which comprised 46.2% of referrals. Behavioral difficulties were the
second most common reason for referrals (26.7%), and students who presented with both
language and behavioral difficulties (22.7%) was the third most prominent reason for
referrals to the PIRT. There were eight students (3.2%) referred to the PIRT for other
reasons such as global delays, cognitive delays, motor impairments, hearing impairments,
and health concerns. Of the 247 children referred to the PIRT, 74 were referred to the
CST for evaluation for eligibility for special education and related services. This
reflected a CST referral rate of 6.4% of the total preschool enrollment for the 2005-2006
school year (see Table 2 and Figure 1).
2006-2007 School Year. During the 2006-2007 school year, there were 1147
students in the early childhood program, and 254 (70.9% male and 29.1% female) were
referred by teachers to the PIRT. As in 2005-2006, all but 30 students attended preschool
in early childhood centers that were contracted with the OECE; the district had only two
regular education preschool classrooms within the public school buildings. The
92
Table 2 Characteristics of Preschool Students Referred to the PIRT Variable 2005-2006 2006-2007 2007-2008 2008-2009 Total in OECE 1154 1147 1205 1350
Total referred to PIRT 247 (21.4) 254 (22.1) 258 (21.4) 219 (16.2)
Gender
Female 92 (37.2) 78 (29.1) 97 (37.2) 81 (37.0)
Male 155 (62.8) 180 (70.9) 162 (62.8) 138 (63.0)
Age
3-years-old 82a 151 (59.4) 96 (37.2) 148 (67.6)
4-years-old 41a 90 (35.4) 122 (47.3) 68 (31.1)
5-years-old 0 6 (2.4) 7 (2.7) 3 (1.4)
Unavailable - 7 (2.8) 33 (12.8) -
Classroom Placement
Contracted Center 247 254 243 (94.2) 209 (95.4)
In-district Classroom - - 15 (5.8) 10 (4.6)
Reason for Referral to PIRT
Language 114 (46.2) 127 (50.0) 152 (58.9) 99 (45.2)
Behavior 66 (26.7) 46 (18.1) 37 (14.3) 25 (11.4)
Language and Behavior 56 (22.7) 27 (10.6) 29 (11.2) 15 (6.8)
ESI-R Screening - - 28 (10.9) 73 (33.3)
Other 8 (3.2) 24 (9.4) 12 (4.7) 7 (3.2)
Unidentified 3 (1.2) 30 (11.8) - -
Referred to CST 74 (6.4) 51 (4.4) 32 (2.7) 36 (2.7)
Note. ESI-R screenings were not conducted for 2005-2006 and 2006-2007. aData on age of students referred to PIRT was only available for 123 of 247 students.
majority (59.4%) of the children referred to the PIRT were 3-years-old at the time of
referral, 35.4% were 4-years-old, and 2.4% were 5-years-old. Data on age were missing
for seven students (2.8%). The three most common reasons for referral to the PIRT were
language delays (50.0%), behavior difficulties (18.1%), and comorbid language and
behavior problems (10.6%). Twenty-four students (9.4%) were referred for other
93
reasons. Data on the reason for referral to the PIRT was missing for 30 students (11.8%).
Of the 254 students that were referred to the PIRT, 51 were referred to the CST for
evaluation for eligibility for special education and related services. This reflected a CST
referral rate of 4.4% of the total preschool enrollment for the 2006-2007 school year (see
Table 2 and Figure 2).
11541205
1350
247
74
1147
254
51 32
258 219
36
0
200
400
600
800
1000
1200
1400
1600
Total in Program Referrals to PIRT Referrals to CST
Num
ber o
f Pre
scho
ol S
tude
nts
2005-2006
2006-2007
2007-2008
2008-2009
20.0% 22.5%
6.7%4.4%
21.5%
2.7%
16.2%
2.7%
Figure 2. Referrals to PIRT and CST for four consecutive years.
94
2007-2008 School Year. During the 2007-2008 school year, there were 1205
students in the early childhood program, and 258 (37.2% female; 62.8% male) were
referred by teachers to the PIRT. The district opened 5 more preschool classrooms
within the public school buildings resulting in a total of 7 preschool classrooms within
the public school buildings. Accordingly, 105 (8.7%) of the 1205 students attended
school in district operated classrooms. Almost all of students (94.2%) referred to the
PIRT attended school in one of the centers contracted with the OECE to provide
preschool programming. Only 15 (5.8%) students attended school in one of the in-
district preschool classrooms that opened during the 2007-2008 school year. Thirty-
seven percent of students referred to the PIRT were 3-years-old, 47.3% were 4-years-old,
and 2.7% were 5-years-old. As in prior years, the most common reason for referral to the
PIRT was language (58.9%). The second and third most common reasons for referral
were behavior (14.3%) and comorbid behavior and language difficulties (11.2%). In
2007-2008, students were also referred to the PIRT based on performance on the Early
Screening Inventory-Revised (ESI-R), which is a standardized, norm-referenced
instrument used to identify young children at-risk for future school difficulties. There
were 28 students (10.9%) referred to the PIRT as a result of performance on the ESI-R.
Twelve students (4.7%) were referred to the PIRT for other reasons. Of the 258 students
referred to the PIRT for assistance, 32 were subsequently referred to the CST for
evaluation for eligibility for special education and related services. This reflected a CST
referral rate of 2.7% of the total preschool enrollment for the 2007-2008 school year (see
Table 2 and Figure 2).
95
2008-2009 School Year. During the 2008-2009 school year, there were 1350
students in the early childhood program. The district opened an additional three
classrooms, for a total of 10 district operated preschool classrooms, which resulted in 150
(11.1%) students attending preschool in district operated buildings. Of the total
enrollment, 219 (37.0% female; 63.0% male) were referred by preschool teachers to the
PIRT. More than 67% were 3-year-old students, 31% were 4-years-old, and 1.4 % were
5-years-old. All but 10 students (4.6%) were in classrooms in early childhood centers
that were contracted with the OECE to provide preschool programming. More than 45%
of students were referred to the PIRT for language difficulties. Results of ESI-R
screenings accounted for 33.3% of referrals to the PIRT. Behavioral difficulties
constituted 11.4% of referrals, and comorbid behavior and language issues comprised
6.8% of referrals. Seven students (3.2%) were referred to the PIRT for other reasons. Of
the 219 students referred to the PIRT for assistance, 36 were subsequently referred to the
CST for evaluation for eligibility for special education and related services. This
reflected a CST referral rate of 2.7% of the total preschool enrollment for the 2008-2009
school year (see Table 2 and Figure 2).
Commonalities and trends across the four years. Examination of data on
characteristics of preschool students across the four academic years revealed
commonalities and trends. In each year in which data were collected, there were
significantly more boys than girls referred by teachers to the PIRT for assistance.
Similarly, many more three-year-old students were referred than four-year-old students in
each year except in the 2007-2008 school year. Almost all students that were referred to
the PIRT attended school in contracted early childhood centers, which was expected
96
since there are relatively few in-district preschool classrooms. Language concerns were
the most common reason for referral in each of the four years. Behavior was the second
most common reason for referral to the PIRT in each year except 2008-2009, when
results of the ESI-R screenings were the second most common reason for referral.
Referrals to the CST declined in years 2006-2007 and 2007-2008, and stabilized in 2008-
2009 (see Figure 2).
97
Results of Program Evaluation Question 2
Program Evaluation Question 2: How has the PBS approach been implemented?
The second program evaluation question sought to provide a description of how
the PBS program was executed and the extent to which the implementation adhered to the
program design. This information is valuable toward making sound judgments about the
ability of the program to be implemented as designed, thus contributing to continuous
program development and improvement. Answers to this question may also assist with
making judgments about the program’s ability to contribute toward the OECE. In order
to answer this question, data were collected about trainings provided to teachers on the
PBS approach, as well as on the types and frequency of additional support provided by
PIRT to preschool classroom teachers. These data were collected via permanent product
review and interview of key stakeholders. Data on teacher trainings were recorded on
Instrument 2.1, Review of Professional Development on Positive Behavior Supports.
Data on additional supports provided to teachers were recorded on Instrument 2.2,
Provision of Support to Teachers. The program evaluation consultant was responsible
for reviewing data and recording it on Instruments 2.1 and 2.2.
Data were also collected about the extent to which preschool teachers
implemented specific classroom strategies that were introduced during professional
development workshops on the PBS approach. Data about the implementation of these
specific strategies were collected via classroom observations and were recorded by the
consultant to the program evaluation and other PIRT members during regular classroom
visits on Instrument 2, Preschool Positive Behavior Support Classroom Implementation
Checklist.
98
Professional Development
Table 3 presents a list of professional development workshops that were provided
to teachers and other staff on PBS and related topics during the 2004-2005 school year,
which was the first full year in which the PBS framework was introduced to teachers in
the early childhood program. Teachers were presented with a mandatory 2-day workshop
that provided an overview of PBS, as well as selected strategies that could be
implemented in the classroom. Teachers were also offered one-day workshops (provided
on Saturdays and attendance was voluntary) on developing play skills and enhancing self-
regulation. Several one-hour lunch-and-learn workshops were also offered. The lunch-
and-learn workshops were provided to small groups of teachers during their lunch hour
on a voluntary basis and often targeted specific strategies that could be implemented in
the classroom immediately. The lunch-and-learn workshops were repeated at multiple
early childhood centers so that they would be available to all interested teachers.
Professional development related to PBS that was provided during the 2005-2006
school year is presented in Table 4. At the beginning of the school year a mandatory
2-day workshop was provided to teachers and other staff. This workshop provided an
overview of how to embed language and social-emotional skill development throughout
the school day within the curriculum. Several lunch-and-learn workshops were provided
to support the implementation of specific strategies. Training on the PBS framework was
expanded to special education teachers via half day workshops.
99
Table 3 Professional Development Trainings Provided During 2004-2005
Title of Workshop
Type of Workshop
Purpose
Positive Behavior Supports
2-day workshop
Overview of PBS for all preschool teachers
Developing Play Skills in the Classroom
Full day Present importance of developing play skills to teachers with specific strategies to implement in the classroom
Enhancing Self-regulation in the Classroom
Full day Present the importance of developing self-regulation in preschool children with specific strategies to implement in the classroom
Promoting Emotional Literacy Lunch and Learn Present strategies for teaching preschoolers about feelings
Teaching Emotional Regulation for Special Education Teachers
Lunch and Learn Present strategies for teaching self-regulation to special education preschoolers
Strategies to Promote Self- Regulation
Lunch and Learn Present strategies for teaching self-regulation to regular education preschoolers
Reading Books for Functional Language
Lunch and Learn Present strategies to promote pragmatic language using literature
How to be a Good Friend Lunch and Learn Present strategies for developing friendship skills to teachers, parents, and family workers
Developing Language in the Classroom
Lunch and Learn Present strategies to increase language development
Professional development that was provided during the 2006-2007 school year is
presented in Table 5. A review of the PBS framework was provided to directors of early
childhood centers. New teachers were provided with a full-day introduction to the PBS
pyramid. Family workers and teaching assistants were also provided with a full-day
introduction to the PBS framework. The lunch-and-learn workshop on self-regulation
was repeated at teacher and director requests.
100
Table 4 Professional Development Trainings Provided During 2005-2006 Title of Workshop
Type of Workshop
Activity/Purpose
Integrative Language and Promoting Social-emotional Competence within the Preschool Classroom
2-Day Workshop
Overview of how to integrate language and social-emotional skill development throughout the school day
Mirror, Mirror on the Wall You Know Inclusion After All
Lunch and Learn
Present strategies to support the inclusion of special education preschoolers in regular education classrooms.
Building Positive Behavior Supports: Purposeful Strategies for Every Preschool Classroom
Lunch and Learn Make-and-take strategies to promote positive behaviors in the classroom
Teaching Emotional Regulation: Strategies to Enhance Self- Regulation in the Preschool Classroom
Lunch and Learn Make-and-take strategies to promote self-regulation
PBS for Preschool Children with Special Needs Part I
Half Day Overview of PBS for special education teachers
PBS for Preschool Disabled Teaching Assistants
Half Day Overview of PBS for special education teaching assistants
Table 5 Professional Development Trainings Provided During 2006-2007
Title of Workshop
Type of Workshop
Activity/Purpose
PBS for Directors
Full Day
Overview and refresher of PBS for early childhood center directors
Promoting Social-emotional Competence Within the Preschool Classroom for New Teachers
Full Day Introduction to PBS for new teachers
PBS for Family Workers Full Day Overview and refresher of PBS for family workers
PBS for Teaching Assistants Full Day Introduction to PBS for new teaching assistants
Teaching Emotional Regulation: Strategies to Enhance Self- Regulation in the Preschool Classroom
Lunch and Learn Make-and-take strategies to promote self-regulation
101
As in prior years, new teachers were provided with a full-day introduction to the
PBS framework during the 2007-2008 school year (see Table 6). However, other
professional development was not as extensive as in prior years. New family workers
and teaching assistants were provided with a full-day introduction to the PBS framework.
At director requests, a lunch-and-learn workshop was provided to parents to promote
positive behaviors at home.
Professional development appeared to decline even further during the 2008-2009
school year. No full-day workshops were provided to teachers. A few lunch-and-learn
workshops may have been conducted, but records were not available.
Table 6 Professional Development Trainings Provided by PIRT Members During 2007-2008 Title of Workshop
Type of Workshop
Activity/Purpose
Promoting Social-emotional Competence Within the Preschool Classroom for New Teachers
Full Day
Introduction to PBS for new teachers
PBS for New Family Workers and Teaching Assistants
Full Day Introduction to PBS for new family workers and teaching assistants
Establishing Home Routines Lunch and Learn Present strategies to parents to promote positive behaviors at home
Provision of Supports to Teachers
PIRT members were asked to refer to permanent product records to determine the
frequency of the provision of direct supports to preschool teachers. However, PIRT
members indicated that such a task was cumbersome and time-consuming. Alternatively,
102
data were collected via interview of select PIRT members. Aggregated estimates of the
frequency of direct support to teachers are presented in Table 7.
Table 7 Aggregated Estimates of Frequency of Direct Supports to Preschool Teachers Type of Support
Frequency
Written Strategies
6-7 times per week
Modeling of approaches and strategies in the classroom
4-5 times per week
Coaching/Verbal Instruction 15-18 times per week
PIRT members indicated that they provide written strategies to several teachers
each week for a cumulative average of approximately 6-7 times per week. They also
indicated that they model various approaches and strategies for preschool teachers at a
rate of approximately 4-5 times per week. Modeling refers to times that PIRT members
demonstrate the implementation of specific strategies in the classroom. Teachers are
expected to observe the implementation of the strategy by the PIRT member to gain an
increased understanding of how to use the strategy successfully with students. This type
of modeling usually occurs after written strategies are developed and discussed with the
classroom teacher.
PIRT members indicated that they provide coaching/verbal instruction to teachers
at a rate of approximately 15-18 times per week. Coaching and verbal instruction occurs
during routine classroom visits by PIRT members to preschool classrooms. During these
103
visits, PIRT members may provide coaching in the form of verbal instruction to teachers
in an effort to improve teacher performance in addressing the needs of the students.
Implementation of Specific Strategies
PIRT members conducted observations in 27 preschool classrooms to collect data
about each of 25 items that are indicative of use of the PBS approach. During regular
visits to classrooms, PIRT members evaluated each indicator on the degree of visibility
and evidence of use in the classroom. A score for the visibility of each indicator was
assigned as follows: (3) clearly visible, (2) moderately visible, or (1) not at all visible.
Scores for evidence of use were assigned as follows: (3) clear evidence of use, (2)
moderate evidence of use, or (1) no evidence of use. Scores were recorded by the
program evaluation consultant and other PIRT members on Instrument 2, Preschool
Positive Behavior Support Classroom Implementation Checklist. Results are presented in
Table 8.
The indicator that was most visible was the daily visual schedule, which was
clearly visible or moderately visible in 96.3% of classrooms. The daily visual schedule is
comprised of a series of pictures or other images that represent the plan of activities for
the school day. It is used to help children understand the classroom daily schedule so that
they will be able to determine the sequence of planned activities. The indicator that was
second most visible was the job chart, which was clearly visible or moderately visible in
88.9% of classrooms. Jobs create a sense of responsibility for each child toward the
classroom community. It also gives the teacher the opportunity to assign tasks
104
Table 8 Results of Positive Behavior Support Implementation Checklista
Degree of Visibility Evidence of Use Indicator Clear Moderate Not Clear Moderate Not
Daily Visual Schedule
19 (70.4)
7 (25.9)
1 (3.7)
3 (11.1)
10 (37.0)
14 (51.9)
Morning arrival schedule 3 (11.1) 4 (14.8) 20 (74.1) 2 (7.4) 3 (11.1) 22 (81.5)
Rules with visual cues 6 (22.2) 11 (40.7) 10 (37.0) 4 (14.9) 11 (40.7) 12 (44.4)
Lunch visual schedule 0 0 27 (100.0) 0 0 27 (100.0)
Rest visual schedule 0 1 (3.7) 26 (96.3) 0 1 (3.7) 26 (96.3)
Stop signs posted on doors 2 (7.4) 4 (14.8) 21 (77.8) 1 (3.7) 3 (11.1) 23 (85.2)
Hand washing schedule near sink
5 (18.5) 2 (7.4) 20 (74.1) 0 6 (22.2) 21 (77.8)
Brushing teeth schedule near sink
1 (3.7) 0 26 (96.3) 0 0 27 (100.0)
Visual schedule for bathroom 8 (29.6) 1 (3.7) 18 (66.7) 0 3 (11.1) 24 (88.9)
Transition cues 6 (22.2) 11 (40.7) 10 (37.0) 5 (18.5) 11 (40.7) 11 (40.7)
Magic Mouth 1 (3.7) 0 26 (96.3) 1 (3.7) 0 26 (96.3)
Board games 13 (48.1) 10 (37.0) 4 (14.8) 8 (29.6) 8 (29.6) 11 (40.7)
Songs basket 1 (3.7) 0 26 (96.3) 0 1 (3.7) 26 (96.3)
Job chart 17 (63.0) 7 (25.9) 3 (11.1) 16 (59.3) 6 (22.2) 5 (18.5)
Turn taking schedules 6 (22.2) 4 (14.8) 17 (63.0) 2 (7.4) 7 (25.9) 18 (66.7)
Visual support for line-up 1 (3.7) 0 26 (96.3) 1 (3.7) 0 26 (96.3)
Relaxation activities 0 0 27 (100.0) 0 0 27 (100.0)
Choice boards 0 0 27 (100.0) 0 0 27 (100.0)
Solution Kit 2 (7.4) 0 25 (92.6) 1 (3.7) 1 (3.7) 25 (92.6)
Feelings faces 2 (7.4) 5 (18.5) 20 (74.1) 0 3 (11.1) 24 (88.9)
Feelings books 1 (3.7) 9 (33.3) 17 (63.0) 1 (3.7) 4 (14.8) 22 (81.5)
Tucker Turtle prompts 0 0 27 (100.0) 0 0 27 (100.0)
Super Turtle Wall 0 0 27 (100.0) 0 0 27 (100.0)
Teasing Shields 0 0 27 (100.0) 0 0 27 (100.0)
Super Friend Wall 2 (7.4) 0 25 (92.6) 2 (7.4) 0 25 (92.6)
an = 27 classrooms
105
to children so that they can build upon a variety of skills. The third most visible indicator
was board games, which was clearly visible or moderately visible in 85.1% of
classrooms. Board games are used to teach and reinforce academic readiness skills as
well as teach prosocial skills such as turn-taking, sharing, and handling disappointment.
There were several indicators that were not at all visible in the classrooms that
were observed. These included the lunch schedule, relaxation activities, choice boards,
Tucker Turtle prompts, super turtle wall, and teasing shields. The lunch schedule is
intended to provide a visual sequence of what is expected during lunch routines.
Relaxation activities are intended to be used to offer strategies to students when feeling
upset. Choice boards should be presented to students who need assistance in making
choices for activities. Tucker Turtle, the super turtle wall, and teasing shields are
strategies that teach children how to respond appropriately when confronted with difficult
situations.
There were also several indicators that were not at all visible in all but one or two
classrooms that were observed. These included the rest schedule, brushing teeth
schedule, magic mouth, songs basket, visual support for line-up, solution kit, and super
friend wall. Like the lunch schedule discussed above, the rest schedule is intended to
provide a visual sequence of what is expected during rest time. The brushing teeth
schedule provides a visual description of the sequence of steps involved in brushing teeth,
which is encouraged after lunch. Magic mouth refers to a strategy that is used to signify
whose turn it is to speak during whole class morning meetings. The songs basket is used
to empower students so that they feel that they have a role in deciding which songs to
sing during various parts of the school day. It consists of placing two-inch square pieces
106
of paper, each with a visual representation of a preschool song, into a basket. A student
is invited by the teacher to select a paper from the basket and lead the class in singing.
Visual support for line-up is provided by placing markers on the floor in the location in
which the teacher would like the students to line up. The solution kit is intended to be
used to encourage independent problem solving. It is made up of pictorial representations
of solutions to common problems faced by preschool students that are placed in a binder
or other easily accessible container. The super friend wall is a specially designated wall
in the classroom to recognize students who have engaged in behaviors that are consistent
with being a good friend.
Evidence of use was poor across all indicators. The job chart had the highest
evidence of use; it was scored as clear evidence of use in 16 (59.3%) of the 27 classrooms
that were observed. All but three of the remaining indicators were scored as no evidence
of use in the majority of classrooms. Several of the indicators (brushing teeth visual
schedule, relaxation techniques, choice boards, Tucker Turtle prompts, super turtle wall,
and teasing shields) had no evidence of use in all classrooms.
Results of Program Evaluation Question 3
Program Evaluation Question 3: What were the reactions of preschool teachers to the
PBS approach in terms of strengths, adequacies, and areas in need of improvement?
The third program evaluation question sought to elicit the thoughts, opinions, and
judgments of teachers about the PBS program. Obtaining information about perceptions
of the target population is important so that decisions can be made about the worth of the
program in adding value to the target population. Answers to this question may also
107
provide information that will allow decisions to be made about how to use evaluation
information to make revisions in the design of the program. Preschool teachers were lead
classroom teachers at contracted early childhood centers and public school buildings.
The method for data collection was the distribution, completion, and collection of
Instrument 1, Positive Behavior Support Teacher’s Questionnaire. The questionnaires
were distributed, completed, and collected at regularly scheduled staff meetings. The
questionnaire required respondents to provide ratings on either a 5-point scale, a 3-point
scale, or an open-ended format. Respondents were also asked to rank the importance of
activities that were relevant to the PBS implementation in the OECE program.
The questionnaires were administered during the second half of the 2008-2009
school year. The evaluation consultant was responsible for distribution and collection of
the questionnaires. Accordingly, the evaluation consultant arranged with directors of
early childhood centers to attend a regularly scheduled monthly staff meeting at each
contracted center. At each staff meeting, the evaluation consultant explained the purpose
of the program evaluation to the preschool classroom teachers, obtained informed
consent, and distributed the questionnaires. The teachers were provided with sufficient
time to complete the questionnaires during the staff meeting. The evaluation consultant
remained at each staff meeting until all teachers completed and returned the
questionnaires.
Teacher Reactions
There were 87 teachers in the early childhood program during the 2008-2009
school year. Of the 87 teachers, 73 teachers completed Instrument 1, Positive Behavior
108
Support Teacher’s Questionnaire (see Appendix A), which reflected 83.9% of all the
teachers in the early childhood program. The Positive Behavior Support Teacher’s
Questionnaire consisted of seven parts with a total of 29 items. Part I consisted of three
questions requiring respondents to check a box indicating the response that best describes
the type of classroom teacher they are, the number of years of teaching experience they
have, and when they received a full day of PBS training. These data were presented and
reviewed earlier in this chapter. Part II consisted of five questions that were answered on
a 5-point scale pertaining to the level to which teachers felt competent about their
abilities to engage in activities related to the goals of the PBS program. Part III consisted
of six open-ended questions that also sought to obtain information about teachers’
abilities to engage in activities related to the goals of the PBS program. Part IV consisted
of five questions answered on a 5-point scale. Respondents were asked to indicate the
level to which they agreed or disagreed with statements pertaining to the PBS program.
Part V consisted of eight questions. Participants were asked to categorize activities of the
PBS program as strengths, adequacies, or areas in need of improvement on a 3-point
scale. Part VI asked that participants rank the three most important activities of the PBS
program for developing skills for reducing challenging behaviors and increasing positive
behaviors in the classroom. The results of Parts IV-VI of the Positive Behavior Support
Questionnaire are presented in Tables 9-15. Part VII invited teachers to provide any
additional comments or suggestions about the PBS program. Data obtained in Parts IV-
VII of the questionnaire are presented in the remainder of this section of the chapter.
Data obtained from Parts II and III of the questionnaire will be presented in the next
section of this chapter, Response to Program Evaluation Question 4.
109
Approximately one-fifth of teachers (20.5%) strongly agreed that their ability to
reduce challenging behaviors and increase positive behaviors among students improved
as a direct result of the PBS program. The highest percentage of teachers (45.2%)
somewhat agreed that their ability to reduce challenging behaviors and increase positive
behaviors among students improved as a direct result of the PBS program. The mean
score for this item was 3.81, where 5 represents strongly agree and 1 represents strongly
disagree. Responses to Part IV Item 1 are presented in Table 9.
Table 9 Results of Positive Behavior Support Teacher’s Questionnaire: Part IV, Item 1a My ability to reduce challenging behavior and increase positive behavior among my students has improved as a direct result of the PBS program.
1
Strongly disagree
2
Somewhat disagree
3
Neither agree nor disagree
4
Somewhat agree
5
Strongly agree
1 (1.4%) 2 (2.7%) 22 (30.1%) 33 (45.2%) 15 (20.5%) an = 73 teachers
Approximately half of the teachers (50.7%) somewhat agreed that the PBS
program was highly important to their directors. Twenty-six percent indicated that they
strongly agree with this statement, and nearly 22% neither agreed nor disagreed. One
respondent strongly disagreed with this statement. The mean score for this item was 4.0.
Responses to Part IV Item 2 are presented in Table 10.
More than half of the teachers (53.4%) felt that the PBS program was highly
important to the Early Childhood Office. Approximately 31% somewhat agreed with this
110
statement, and approximately 15% neither agreed nor disagreed. The mean score for this
item was 4.38. Responses to Part IV Item 3 are presented in Table 11.
Table 10 Results of Positive Behavior Support Teacher’s Questionnaire: Part IV, Item 2a I can tell that the implementation of the PBS program is highly important to my director or principal.
1
Strongly disagree
2
Somewhat disagree
3
Neither agree nor disagree
4
Somewhat agree
5
Strongly agree
1 (1.4%) 0 16 (21.9%) 37 (50.7%) 19 (26.0%) an = 73 teachers
Table 11 Results of Positive Behavior Support Teacher’s Questionnaire: Part IV, Item 3a I can tell that the implementation of the PBS program is highly important to the Early Childhood Office.
1
Strongly disagree
2
Somewhat disagree
3
Neither agree nor disagree
4
Somewhat agree
5
Strongly agree
0 0 11 (15.1%) 23 (31.5%) 39 (53.4%) an = 73 teachers
The highest percentage of the teachers (43.8%) somewhat agreed that they get
excellent support for implementing strategies from the PBS program. Opinions of the
remaining teachers were varied. Fifteen percent strongly agreed, 20.5% neither agreed
nor disagreed, and 13.7% somewhat disagreed that they get excellent support. Two
teachers (2.7%) strongly disagreed that they get excellent support for implementing PBS
strategies. The mean score for this item was 3.57. Responses to Part IV Item 4 are
presented in Table 12.
111
Table 12 Results of Positive Behavior Support Teacher’s Questionnaire: Part IV, Item 4a I receive excellent support for implementing strategies from the PBS program.
1
Strongly disagree
2
Somewhat disagree
3
Neither agree nor disagree
4
Somewhat agree
5
Strongly agree
2 (2.7%) 10 (13.7%) 15 (20.5%) 32 (43.8%) 11 (15.1%)
Note: Three teachers (4.1%) did not respond to this question. an = 73 teachers
The highest percentage of the teachers (47.9%) somewhat agreed that teaching
social-emotional skills is the most important component of educating preschool students.
Opinions of the remaining teachers were varied. More than 38% of teachers strongly
agreed that teaching social-emotional skills is the most important component of early
childhood education, and 8.2% neither agreed nor disagreed. One teacher (1.4%)
somewhat disagreed. Three teachers (4.1%) strongly disagreed. The mean score for this
item was 4.15. Responses to Part IV Item 5 are presented in Table 13.
Table 13 Results of Positive Behavior Support Teacher’s Questionnaire: Part IV, Item 5a Teaching social-emotional skills is the most important component of educating preschool students.
1
Strongly disagree
2
Somewhat disagree
3
Neither agree nor disagree
4
Somewhat agree
5
Strongly agree
3 (3.1%) 1 (1.4%) 6 (8.2%) 35 (47.9%) 28 (38.4%) an = 73 teachers
112
The next set of questions also contributed toward providing answers to Program
Evaluation Question 3 (What were the reactions of preschool teachers to the PBS
approach in terms of strengths, adequacies, and areas in need of improvement?).
Responses to items 1-8 of Part V of the questionnaire are presented in Table 14. As
indicated earlier, these items were scored on a 3-point scale (1 = needs improvement; 2 =
adequate; and 3 = strength). Each item is discussed below.
Participants were asked to categorize full-day professional development on the
PBS approach as a strength, adequacy or area in need of improvement. Full-day
professional development refers to trainings that were provided by PIRT members to
teachers. The goals of these trainings were to provide teachers with an overview of the
PBS framework, and to provide a basic repertoire of strategies that could be implemented
in the classroom to reduce challenging behaviors and promote pro-social behaviors.
Almost 55% of teachers indicated that the full day professional development trainings
were either adequate (27.4%) or a strength (27.4%). All but three of the remaining
respondents (41.4%) felt that these trainings were in need of improvement. After
completion of the questionnaires, informal discussion revealed that some teachers
indicated that full-day professional development was in need of improvement because
they had not received the training. The mean score for this item was 1.86. Responses to
Item 1 on Part V are presented in Table 14.
In the second question of this section, participants were asked to categorize mini
lunch-and-learn workshops. Mini lunch-and-learn workshops are one-hour workshops
provided to teachers by PIRT members at contracted early childhood centers. The
content of the workshops relates to a variety of components of the PBS program. The
113
Table 14 Results of Positive Behavior Support Teacher’s Questionnaire: Part V, Items 1-8a Respondents were asked to categorize each of the following activities/services related to PBS as a strength, adequacy, or area in need of improvement.
Score
Item
Item left
blank
1 Needs
Improvement
2 Adequate
3 Strength
1. Full day professional development on the PBS program
3 (4.1%)
30 (41.1%)
20 (27.4%)
20 (27.4%)
2. Mini Lunch-and-Learn workshops addressing specific aspects of PBS
6 (8.2%)
38 (52.1%)
19 (26.0%)
10 (13.7%)
3. Receiving written strategies from PIRT members
1 (1.4%)
13 (17.8%)
35 (47.9%)
24 (32.9%)
4. Having strategies modeled and/or coached for me by PIRT members
1 (1.4%)
19 (26.0%)
30 (41.1%)
23 (31.5%)
5. Obtaining action plans developed at Request for Assistance (RFA) meetings
2 (2.7%)
13 (17.8%)
38 (52.1%)
20 (27.4%)
6. PIRT assisted development of Behavior Support Plans based on Functional Behavior Assessments
2 (2.7%)
20 (27.4%)
32 (43.8%)
19 (26.0%)
7. Support from my director/principal
-
9 (12.3%)
23 (31.5%)
41 (56.2%)
8. Support from my PIRT Coordinator
1 (1.4%)
9 (12.3%)
35 (47.9%)
28 (38.4%)
an = 73 teachers
workshops are given during the teachers’ lunch periods and attendance is voluntary.
More than half of the teachers (52.1%) indicated that lunch-and-learn workshops are an
area that is in need of improvement. Informal discussion revealed that teachers felt that
these workshops were in need of improvement because they were not conducted with
regularity. More than one quarter of teachers (26.0%) felt that these workshops were
adequate and 13.7% felt that they were a strength. The mean score for this item was
1.58. Responses to Item 2 on Part V are presented in Table 14.
114
Teachers were asked to categorize written strategies received from PIRT
members. Written strategies are often provided to teachers after a request has been made
for assistance regarding a specific student. After consultation with the teacher regarding
the student or direct observation of the student in the classroom, teachers are provided
with strategies and ideas in writing that may serve to reduce the problem. The problems
may be related to language, behavior, or a variety of other reasons. Almost half of the
teachers (47.9%) indicated that they felt that written strategies were adequate. Almost
one third of teachers (32.9%) felt that written strategies were a strength of the PBS
program. Thirteen teachers (17.8%) felt that written strategies were an area that was in
need of improvement. One teacher did not respond to this item. The mean score for this
item was 2.15. Responses to Item 3 on Part V are presented in Table 14.
Teachers were also asked to categorize modeling and coaching of strategies.
After written strategies are developed and discussed with the classroom teacher, PIRT
members may be asked by teachers to model the implementation of certain strategies in
the classroom for clarity. PIRT members may also approach classroom teachers and
offer to model strategies in the classroom for the teacher. Coaching of strategies may
occur in the classroom or after a PIRT member has observed the teacher implement the
strategy. The highest percentage of teachers (41.1%) felt that coaching and modeling of
strategies was adequate. Nearly one third of teachers (31.5%) felt that coaching and
modeling of strategies was a strength. More than one quarter of teachers (26.0%) felt that
coaching and modeling was an area in need of improvement. One teacher (1.4%) did not
respond to this item. The mean score for this item was 2.06. Responses to Item 4 on Part
V are presented in Table 14.
115
Action plans are developed at a collaborative meeting that includes the teacher,
the parent(s), the PIRT coordinator, and other relevant stakeholders, as needed, after a
Request for Assistance (RFA) has been made by a teacher to the PIRT. Action plans are
written documents that delineate the concerns that the teacher has regarding the student,
strategies to address the concerns, who is responsible for implementing the strategies, and
a timeline for implementation and follow-up. More than half of the teachers (52.1%) felt
that action plans were adequate. Approximately 27% of teachers felt that action plans
were a strength, and 17.8% of teachers felt that action plans were in need of
improvement. The mean score for this item was 2.10. Responses to Item 5 on Part V are
presented in Table 14.
Behavior Support Plans are developed for specific students after a teacher has
made a RFA to the PIRT. The development of a behavior support plan occurs for
students whose behavior continues to be intense and persistent even after all other
supports are in place. The development of the behavior support plan is a complex
process that occurs after a FBA is conducted. Teachers were asked to categorize
behavior support plans. The highest percentage of teachers (43.8%) felt that behavior
support plans were adequate. More than a quarter of teachers (26%) felt that behavior
support plans were a strength. A similar number of teachers (27.4%) felt that behavior
support plans were in need of improvement. Two teachers did not respond to this item.
The mean score for this item was 1.99. Responses to Item 6 on Part V are presented in
Table 14.
Teachers were asked to categorize the support received from their directors.
More than half of the teachers (56.2%) felt that support from their directors was a
116
strength. Nearly one third of teachers (31.5%) felt that director support was adequate,
and 12.3% of teachers felt that this was an area in need of improvement. The mean score
for this item was 2.44. Responses to Item 7 on Part V are presented in Table 14.
Teachers were also asked to categorize the support they received from their PIRT
coordinator. Nearly half of teachers (47.9%) indicated that the support from PIRT
coordinators was adequate. More than one third of teachers (38.4%) felt that support
from PIRT coordinators was a strength, and 12.3% of teachers felt that this was an area in
need of improvement. One teacher left (1.4%) this item blank. The mean score for this
item was 2.26. Responses to Item 8 on Part V are presented in Table 14.
The next section of the questionnaire asked teachers to rank activities/services
that they felt were or would be most helpful in developing skills for reducing challenging
behaviors and increasing positive behaviors in their classrooms. The teachers were
presented with a list of eight items (full-day professional development workshop, lunch-
and-learn workshops, written strategies from PIRT members, coaching and/or modeling
of strategies, action plans developed at RFA meetings, behavior support plans,
director/principal support, PIRT coordinator support) and were directed to place a 1 in
front of the item that they felt was or would be most helpful, a 2 in front of the item that
was or would be second most helpful, and a 3 in front of the item that was or would be
third most helpful. Weighted rankings indicated that teachers overwhelmingly felt that
full-day professional development was or would be the most helpful. Coaching and
modeling of strategies by PIRT members was ranked as the second most helpful. Lunch-
and-learn workshops were ranked third most helpful (see Table 15).
117
Table 15 Results of Positive Behavior Support Teacher’s Questionnaire: Part VI, Weighted Rankings Please indicate the first, second, and third most important activities/services that you believe were or would be most helpful in developing your skills for reducing challenging behaviors and increasing positive behaviors in your classroom. Place a 1 in front of the item that you deem most helpful, a 2 for the second most helpful, and a 3 for the third most helpful. You may leave the remaining items blank.
Activity/Service
Weighted Score
Weighted Rank
Full-day professional development
144
1
Coaching and/or Modeling of strategies
74
2
Lunch-and-learn workshops
61
3
Director/principal support
21
4
PIRT coordinator support
20
5
Action plans developed at RFA meetings
17
6
Written strategies from PIRT members
17
6
Behavior Support Plans
13
8
Part VII, which was the final section of the Positive Behavior Support Teachers
Questionnaire, invited teachers to provide any additional comments or suggestions
concerning the PBS program. Of the 73 teachers that completed the questionnaire, 27
teachers (36.9%) provided comments. Careful review of the comments by the program
evaluation consultant revealed the emergence of 5 common themes. Almost half of the
teachers (48.1%) who provided comments indicated that they were interested in more
training on the PBS approach. Some teachers (18.5%) indicated that it was difficult to
implement PBS strategies because they were unrealistic or there were too many other
classroom demands. Other teachers (14.8%) indicated that they were satisfied with and
interested in more support from PIRT coordinators. Requests included the desire for
118
more modeling of strategies and scheduled time to meet with PIRT coordinators. Still
other teachers (11.1%) indicated that they were happy with and regularly used PBS
related strategies. Two teachers (7.4%) were interested in additional parental
involvement.
Results of Program Evaluation Question 4
Program Evaluation Question 4: To what extent were the goals of the PBS program
attained?
The last program evaluation question sought to determine the extent to which the
preschool teachers’ abilities to decrease challenging behaviors and increase prosocial
behaviors among preschool students improved in relation to the PBS program. It is
important to be able to answer this question so that sound judgments can be made about
the merit and worth of the program. Parts II and III of Instrument 1, Positive Behavior
Support Teacher’s Questionnaire aimed to elicit data to answer this question.
In Part II, lead classroom teachers were asked to rate on a 5-point scale how
competent they felt about their ability to engage in each of five activities that relate to the
goals of the program. Responses to Items 1-5 in Part II are presented in Tables 16, and
18-21.
In Part III, teachers were presented with open-ended questions in which they were
asked to provide specific examples of their knowledge of the same goal-related activities
that they were asked to self-evaluate in Part II. Teachers were asked to provide two
examples for each open-ended question that was presented. Responses were read and
scored by the evaluation consultant. A score of 1 indicated that the response was judged
119
to be in need of improvement, and was given if neither of the responses provided by the
teacher was accurate. A score of 2 indicated that the response was judged to be adequate,
and was given if one of the two responses was accurate. A score of 3 indicated that the
response was a strength, and was given if both examples provided by the teacher were
accurate. See Table 17 for ratings of teachers’ responses to Part III.
The first item in Part II asked teachers to rate how they felt about their ability to
implement activities and routines in their classrooms that would prevent or reduce
challenging behaviors. The majority of respondents indicated that they felt moderately
competent (54.8%) or extremely competent (28.8%) about their ability to reduce or
prevent challenging behaviors. Nine teachers (12.3%) were neutral and only three
teachers (4.1%) indicated that they felt minimally competent. Responses to Part II, Item 1
are presented in Table 16. Teachers were then asked to list two ways they build positive
relationships with students and families because building positive relationships is the
foundation to the PBS approach. More than half of teachers (57.5%) provided responses
that were a strength and 31.5% of teachers provided responses that were adequate. Only
8 teachers (11.0%) provided examples that needed improvement (See Table 17).
Teachers’ self-assessment of their ability to implement activities that would prevent or
reduce challenging behaviors was generally consistent with their ability to list two ways
to build positive relationships.
The next item in Part II asked teachers to rate how they felt about their ability to
identify triggers that may result in children engaging in challenging behaviors. The
majority of respondents indicated that they felt moderately competent (53.4%) or
extremely competent (31.5%) about their ability to identify triggers. Nine teachers
120
Table 16 Results of Positive Behavior Support Teacher’s Questionnaire: Part II, Item 1a How competent do you feel about your ability to implement activities and routines in your classroom that will prevent or reduce challenging behaviors?
1
Not at all competent
2
Minimally competent
3
Neutral
4
Moderately competent
5
Extremely competent
0 3 (4.1%) 9 (12.3%) 40 (54.8%) 21 (28.8%)
Note: One respondent left this item blank. an = 73 teachers Table 17 Ratings of Teacher Responses to Positive Behavior Support Teacher’s Questionnaire: Part IIIa
Item
1 Needs
Improvement
2 Adequate
3 Strength
Please list two ways in which you build positive relationships with your students and their families.
8 (11.0%)
23 (31.5%)
42 (57.5%)
Please list two functions of challenging behaviors that are seen in preschool children.
67 (91.8%)
4 (5.5%)
2 (2.7%)
Please provide two examples of triggers that may result in the occurrence of challenging behaviors in preschool children.
36 (49.3%)
15 (20.5%)
22 (30.1%)
Please list two things that you could do to prevent the triggers that you listed in #3 from occurring.
30 (41.1%)
25 (34.2%)
18 (24.7%)
Please list two new skills that can be taught to a preschool child that could replace challenging behaviors.
38 (52.1%)
20 (27.4%)
15 (20.5%)
Please list two strategies that, as a result of PBS training(s) you have attended, you are implementing in your classroom to promote social-emotional learning.
34 (46.6%)
12 (16.4%)
27 (37.0%)
Note: A score of 3 was given if both examples provided by the teacher were accurate. A score of 2 was given if only one of the examples provided was accurate, and a score of 1 was given if neither of the responses was accurate. an = 73 teachers
121
(12.3%) were neutral and only two teachers (2.7%) indicated that they felt minimally
competent. Responses to Part II, Item 2 are presented in Table 18. When teachers were
asked to provide two examples of triggers in an open-ended format, about half of the
teachers (49.3%) provided examples that were in need of improvement. Almost one-third
of teachers (30.1%) provided examples that indicated a strength in this area and 20.5%
provided examples that were judged to be adequate (see Table 17). Teachers’ self-
assessment of their ability to identify triggers to challenging behaviors was generally
inconsistent with their ability to list two examples of triggers.
Table 18 Results of Positive Behavior Support Teacher’s Questionnaire: Part II, Item 2a How competent do you feel about your ability to identify triggers that may result in children engaging in challenging behaviors?
1
Not at all competent
2
Minimally competent
3
Neutral
4
Moderately competent
5
Extremely competent
0 2 (2.7%) 9 (12.3%) 39 (53.4%) 23 (31.5%) an = 73 teachers
The third item in Part II asked teachers to rate how they felt about their ability to
identify the functions of challenging behaviors. The majority of respondents indicated
that they felt moderately competent (53.4%) about their ability to identify functions of
challenging behaviors. An equal number of teachers felt extremely competent (21.9%) or
neutral (21.9%). Only two teachers (2.7%) felt minimally competent to identify triggers
to challenging behaviors. Responses to Part II, Item 3 are presented in Table 19. In
122
contrast, an overwhelming majority of teachers (91.8%) were unable to list two functions
of challenging behaviors that are seen in preschool children. Only 2 teachers (2.7%)
provided a response that was judged to be a strength, and 4 teachers (5.5%) provided
responses that were judged to be adequate (See Table 17). Teachers’ self-assessment of
their ability to identify functions of behavior was not consistent with their ability to list
two functions of challenging behaviors seen in preschool children.
Table 19 Results of Positive Behavior Support Teacher’s Questionnaire: Part II, Item 3a How competent do you feel about your ability to identify the functions of challenging behaviors exhibited by the children in your classroom?
1
Not at all competent
2
Minimally competent
3
Neutral
4
Moderately competent
5
Extremely competent
0 2 (2.7%) 16 (21.9%) 39 (53.4%) 16 (21.9%) an = 73 teachers
The fourth item in Part II asked teachers to rate how they felt about their ability to
identify adult responses to challenging behaviors that might serve to decrease challenging
behaviors. As in the previous items, the highest percentage of respondents indicated that
they felt moderately competent (49.3%) about their ability to identify adult responses that
might reduce challenging behaviors. Twenty-six percent of teachers felt extremely
competent and 21.9% were neutral. Only two teachers (2.7%) felt minimally competent
to identify adult responses to reduce challenging behaviors. Responses to Part II Item 4
are presented in Table 20. When teachers were asked to list two things they could do to
prevent behavioral triggers from occurring in an open-ended format, 41.1% of teachers
123
listed preventions that were judged to be in need of improvement, 34.2% listed
preventions that were judged as adequate and only 24.7% listed preventions that were
judged to be a strength (See Table 17). Teachers’ self-assessment of their ability to
identify adult responses to challenging behaviors that may serve to decrease challenging
behaviors was generally not consistent with their ability to list things they could do to
prevent triggers from occurring.
Table 20 Results of Positive Behavior Support Teacher’s Questionnaire: Part II, Item 4a How competent do you feel about your ability to identify adult responses to challenging behaviors that may serve to decrease challenging behaviors?
1
Not at all competent
2
Minimally competent
3
Neutral
4
Moderately competent
5
Extremely competent
0 2 (2.7%) 16 (21.9%) 36 (49.3%) 19 (26.0%) an = 73 teachers
The last item in Part II asked teachers to rate how they felt about their ability to
teach new skills that would replace challenging behaviors. The majority of teachers
indicated that they felt moderately competent (53.4%) about their ability to teach new
skills to replace challenging behaviors. Almost one quarter of teachers (24.7%) felt
extremely competent and 16.4% were neutral. Three teachers (4.1%) felt minimally
competent to teach new skills to replace challenging behaviors. Responses to Part II Item
5 are presented in Table 21. There was a contrast between teachers’ self-assessment
regarding teaching new replacement skills and their ability to list two new replacement
skills to be taught to preschool children. In an open-ended format, 52.1% of teachers
124
listed replacement skills to be taught to preschool children that were judged to be in need
of improvement, 27.4% of responses were judged to be adequate and only 20.5% of
responses were judged to be a strength (See Table 17).
Table 21 Results of Positive Behavior Support Teacher’s Questionnaire: Part II, Item 5a How competent do you feel about your ability to teach new skills to the children in your class that would replace challenging behaviors?
1
Not at all competent
2
Minimally competent
3
Neutral
4
Moderately competent
5
Extremely competent
1 (1.4%) 3 (4.1%) 12 (16.4%) 39 (53.4%) 18 (24.7%) an = 73 teachers
As a final item in Part III, teachers were asked to list two strategies which
promote social-emotional learning that they are implementing in their classrooms as a
result of PBS trainings. Ratings of their responses are presented in Table 17. Almost
half of the teachers (46.6%) provided responses that were judged to be in need of
improvement, 16.4% provided responses that were judged as adequate, and 37.0%
provided responses that were judged to be a strength.
Summary
Relevant characteristics of teachers were examined for the 2008-2009 school
year. There were 87 preschool teachers in the early childhood program; all but three
were female. Almost half of the teachers earned their teaching certification via the
traditional route. Nearly 40% of teachers earned their certification via the alternate route.
125
Data on teaching credentials was unavailable for the remaining teachers. Relevant
characteristics of students were examined for 4 academic years. The number of referrals
to the PIRT ranged from 219-258. For all years, the most common reason for referral
was language concerns. For most years, the second most common reason for referral was
behavior concerns.
Professional development was provided to teachers to facilitate the
implementation of the PBS program. It appeared that trainings were most robust in the
earlier years of the PBS program, and declined steadily. Other supports that were
provided to teachers to facilitate the implementation of the PBS program included written
strategies, modeling of approaches and strategies, and the provision of coaching and
verbal instruction. The implementation of specific indicators related to the PBS program
was evaluated by classroom observation. The indicators that were most clearly visible in
the classrooms were the daily visual schedule, the job chart, and board games. Many
indicators were not at all visible. Generally, there was poor evidence of use across all
indicators.
Reactions of teachers revealed that the majority of teachers somewhat to strongly
agreed that the PBS program helped improve their skills for reducing challenging
behaviors and increasing prosocial skills among their students. The majority of teachers
rated all the components of the PBS program as either adequate or a strength except for
lunch-and-learn workshops, which were identified as in need of improvement because
they were not offered with regular frequency. Teachers were asked to rank which
components of the PBS program were or would be most helpful for developing their
skills for reducing challenging behaviors and increasing prosocial skills among their
126
students. Weighted ranking revealed that teachers felt that full-day professional
development on PBS, coaching/modeling of strategies, and lunch-and-learn workshops
were the three components that were or would be most helpful to them.
In order to determine if the goals of the program were met, teachers were asked to
rate how competent they felt about their abilities to engage in each of five activities
related to the PBS program. They were then asked to demonstrate their competencies by
providing specific examples of their knowledge in each of these areas in an open-ended
format. There was a discrepancy between teachers’ self-assessment of their
competencies and their ability to list examples of these skills in all areas examined except
for one, building positive relationships. Discrepancies between self-assessment of
competency and ability to demonstrate skills were evident in identifying triggers to
challenging behaviors, identifying the functions of challenging behaviors, identifying
adult responses to challenging behaviors that would likely reduce problems, and teaching
new skills to replace challenging behaviors.
Communication of Program Evaluation Information
Upon completion of data analyses, the program evaluation information was
communicated to both supervisors of the OECE, one of whom was the client of the
program evaluation. All four program evaluation questions were able to be answered.
Information obtained was analyzed as described in the protocols for each question. A
final report was compiled and presented to both supervisors at a face-to-face meeting that
was held in the fall of the school year after the data were collected. The information in
the report was formally reviewed in detail with the supervisors.
127
CHAPTER VI
EVALUATION OF THE PROGRAM EVALUATION
Overview
The final program planning and evaluation activity is to evaluate the program
evaluation. According to Maher (2000), evaluating the program evaluation allows key
stakeholders to use evaluation information for judging the program’s worth and to make
subsequent program planning decisions. Evaluating the program evaluation provides
information so that decisions can be made as to how future program evaluations and the
entire program planning and evaluation process can be improved. The evaluation of the
program evaluation can be facilitated by using the four qualities of a sound human
services program evaluation delineated by Maher, which are practicality, utility,
propriety, and technical defensibility. The following four questions, which are based on
these qualities, are:
1. To what extent was the program evaluation conducted in a way that
allowed for its successful accomplishment? (Practicality)
2. In what ways was the resulting program evaluation information helpful to
people? Which people? (Utility)
3. Did the program evaluation occur in a way that adhered to legal strictures
128
and ethical standards? (Propriety)
4. To what degree can the evaluation be justified with respect to matters of
reliability and validity? (Technical Defensibility)
Responses to these questions can be obtained from a variety of people who have
been involved in the evaluation and through sources of data. In order to evaluate the PBS
program evaluation, these four questions were addressed during a meeting with the
supervisors of the OECE, through observations of the program, and through review of
data obtained from the program evaluation. This chapter discusses each of the four
questions.
Practicality
This question examines whether or not the program evaluation was conducted in a
manner that allowed for its successful accomplishment. Concerns about practicality were
an issue for the program evaluation consultant and the client throughout the program
evaluation process. Two program evaluation instruments were used for this program
evaluation. The first was Instrument 1, Positive Behavior Support Teachers
Questionnaire. The teachers were asked to respond to a questionnaire that would take
approximately 20 minutes to complete. It was only practical to engage the teachers for
this amount of time during regularly scheduled staff meetings, which required the
cooperation from the directors of the early childhood centers. The supervisors and
directors indicated that it was a worthy use of time and practical to ask teachers to
complete the questionnaire at the staff meetings. Directors expressed their support for the
129
program evaluation verbally, and provided teachers with ample time during staff
meetings to complete the questionnaire. All of the teachers that were asked to complete
the questionnaire at staff meetings participated, resulting in a 100% response rate. This
unusually high response rate can be attributed to the support received from supervisors
and directors for this program evaluation. Another successful feature of this approach to
asking teachers to complete the questionnaire is that that teachers were not asked to
expend any personal time to complete the questionnaire. This procedure for
questionnaire completion did not present any disruptions to the daily operation of the
early childhood program, and was embraced by all key stakeholders.
The second program evaluation instrument, Preschool Positive Behavior Support
Classroom Implementation Checklist, was completed by PIRT members during regular
visits to preschool classrooms. Only 27 implementation checklists were completed by
PIRT members, a much lower response rate than for the teacher questionnaire. It may be
hypothesized that the time required to make the observations and complete the checklist
was cumbersome for PIRT members. It may not have been practical to ask PIRT
members to set aside time during regularly scheduled classroom visits for the
accomplishment of this program evaluation task.
Although response rates differed between the two instruments, the supervisors of
the early childhood program and other key stakeholders were satisfied with the clarity of
the questionnaire and the implementation checklist. The overall design of the program
evaluation was practical for the public school setting in which it was executed.
130
Utility
The purpose of asking and answering this question is to provide key stakeholders
with information that will allow them to make effective decisions about the program and
how to improve it. Interviews with the supervisors of the early childhood program
revealed that the information obtained from the program evaluation will be helpful
toward the future development of the PBS program. The supervisors will be able to refer
to the program evaluation information as a tool for planning future professional
development for teachers and early childhood staff. They will also be able to use the
information to guide functions of PIRT members in relation to the PBS program. The
supervisors also indicated that the information obtained highlights the need for ongoing
program evaluation. The supervisors plan to refer to the program evaluation information
to modify the PBS program implementation and to inform future program evaluations.
The program evaluation information is also useful to teachers, PIRT members,
and other key stakeholders. The objective information provided by the program
evaluation can be utilized by all stakeholders to improve service delivery to students.
Students at greatest risk can be more easily targeted for support and the process of
implementing the PBS program can be revised to meet teachers’ needs.
Propriety
The purpose of asking and answering this question is to ensure that the program
evaluation occurs in a way that adheres to ethical and legal standards. Legal and ethical
concerns were openly discussed prior to implementing the program evaluation process.
The program evaluation plan was thoroughly reviewed and approved by one supervisor
131
of the early childhood office and the chairperson of this dissertation. The program
evaluation process was also approved by the superintendent of schools. Data for the
program evaluation was collected and reported in a manner that protected the
confidentiality of all participants.
Technical Defensibility
The final question for the evaluation of the program evaluation addresses the
reliability, validity, and accuracy of the methods, procedures, and instruments of the
program evaluation. The methods and procedures used for evaluation of the PBS
program evaluation were deemed to be justifiable based on the context within which the
program evaluation occurred. Two instruments were designed and used to evaluate the
PBS program in the OECE. Caution needs to be taken when considering the reliability
and validity of each of the instruments.
Instrument 1, Positive Behavior Support Teacher’s Questionnaire was completed
during regularly scheduled staff meetings at individual early childhood centers. Results
obtained from this instrument must be considered in light of the context in which they
were administered. Although every precaution was used to ensure confidentiality, and
these precautions were thoroughly explained to teachers prior to completing the
questionnaire, it is possible that teachers felt pressured to respond to questions in a
manner that would be most flattering to themselves and the center in which they worked.
Also, the presence of their directors during completion of the questionnaires may have
influenced responses even further.
132
Instrument 2, Preschool Positive Behavior Support Classroom Implementation
Checklist was intended to be completed by PIRT members during regularly scheduled
classroom visits. Pressure to conduct the observation and complete the checklist may
have influenced the reliability of the information obtained. Data obtained as a result if
this instrument must be interpreted cautiously.
Each instrument was examined by select PIRT members and one supervisor from
the OECE prior to use in the program evaluation process, and deemed to have adequate
levels of content and face validity. It is important to emphasize, however, that this
evaluation was formative in nature and not a random sample controlled study. Results
are only valid within the context of the PBS program and the specific population served.
Summary
The final program planning and evaluation activity is to evaluate the program
evaluation so that subsequent program planning and evaluation decisions can be made.
Four questions were developed based upon the four qualities of a sound human services
program that are delineated by Maher (2000). These qualities are practicality, utility,
propriety, and technical defensibility. Responses to these questions were obtained from
interviewing key stakeholders, observations of the program, and through review of data
obtained from the program evaluation.
First, the evaluation of the program evaluation determined that the
implementation of the evaluation was conducted in a practical manner. Second, key
stakeholders indicated that the program evaluation was useful in that it provided objective
information that will allow effective decisions to be made about PBS program
133
development and implementation, as well as future program evaluations. Third, it was
determined that adherence to legal and ethical concerns was strictly maintained. Finally,
it was determined that results of the program evaluation were reliable and valid only
within the context of the program and the specific population served. Results cannot be
generalized to other PBS programs and settings.
134
CHAPTER VII
CONCLUSIONS AND RECOMMENDATIONS
Overview
This chapter presents conclusions and recommendations drawn after the
completion of a systematic program evaluation of a PBS program in the OECE. The
program evaluation presented in this dissertation captured a snapshot of the PBS program
during the 2008-2009 school year. It also provided an overview of how the PBS program
was implemented during the preceding four academic years. Conclusions and
recommendations are based on the information obtained as a result of the program
evaluation implemented during this timeframe. Conclusions are presented for the
findings of the program evaluation as well as for the findings of the dissertation.
Recommendations for the continued implementation of the evaluation plan are offered
followed by recommendations for improvements to the PBS program.
135
Conclusions
Findings of the Program Evaluation
Upon review of information obtained from the completion of the program
evaluation, four main conclusions were drawn. First, the majority of teachers were
generally satisfied with the PBS program. Second, the PBS program was implemented
most robustly in the earliest years of program implementation. Third, there was a
discrepancy between teachers’ self-assessment of skills obtained as a result of
participation in the PBS program and their ability to demonstrate acquired skills via
open-ended questions. Fourth, teachers indicated that professional development on PBS
and coaching from PIRT members were most helpful for building skills aimed at
reducing challenging behaviors and promoting prosocial behaviors among preschool
students. The remainder of this section of the chapter discusses each of these conclusions
in greater detail.
The majority of teachers were generally satisfied with the PBS program. This
conclusion was drawn from teachers’ responses to select questions on Instrument 1,
Positive Behavior Support Teacher’s Questionnaire. More than 65% of teachers
indicated that their skills for reducing challenging behaviors and promoting prosocial
behaviors improved as a direct result of the PBS program. Similarly, the majority of
teachers recognized that the PBS program was highly important to directors and the
OECE. The majority of teachers also indicated that they receive excellent support for
implementing strategies related to the PBS program.
The PBS program was implemented most robustly in the earliest years of program
implementation. This conclusion was drawn from permanent product review and
136
interview of key stakeholders. In the first full year of implementation of the PBS
program, nine different workshops were offered to teachers. During the second and third
years, five different workshops were offered to teachers each year. By the fourth year of
implementation, only three workshops were offered. Professional development declined
even further during the fifth year of implementation. A few workshops may have been
offered, but records were not available. However, aggregated estimates of direct supports
to teachers in the form of written strategies, modeling of approaches, and coaching were
reported to remain robust in the fifth year of the PBS program.
There was a discrepancy between teachers’ self-assessment of skills obtained as a
result of participation in the PBS program and their ability to demonstrate acquired skills
via open-ended questions. This conclusion was based upon information obtained from
select questions on Instrument 1, Positive Behavior Support Teacher’s Questionnaire.
Teachers were asked to rate on a 5-point scale how competent they felt about their ability
to engage in each of five activities that relate to the goals of the PBS program. Teachers
were also presented with open-ended questions in which they were asked to provide
specific examples of their knowledge of the same goal-related activities. While
approximately three-fourths of teachers rated themselves as moderately to extremely
competent across each of the five activities, generally fewer than half of their open-ended
responses were judged to be a strength. This discrepancy suggests that teachers
overestimated their abilities to engage in activities related to the goals of the PBS
program.
Teachers indicated that professional development on PBS and coaching from
PIRT members were most helpful for building skills aimed at reducing challenging
137
behaviors and promoting prosocial behaviors among preschool students. Teachers were
asked to rank activities and services that they felt were or would be most helpful in
advancing their skills for addressing the needs of preschool students with challenging
behaviors. Weighted rankings indicated that teachers believed that full-day professional
development, coaching and modeling of strategies, and lunch-and-learn workshops were
the most helpful.
Findings of the Dissertation
It was concluded from conducting this evaluation of the PBS program that (1) the
evaluation plan is feasible; (2) key stakeholders found the evaluation useful; and (3) there
is a desire to continue to use the plan for ongoing program evaluations of the PBS
program. Most significantly, it was clear that the evaluation can be implemented as part
of regular programming within the Office of Early Childhood Education with only minor
adjustments to the program evaluation plan. Feedback from teachers indicated that they
were interested in results of the program evaluation, in contributing toward improving the
PBS program, and in future evaluations of the PBS program. Feedback from the
supervisors of the OECE and other key stakeholders indicated that the evaluation was
clear, practical, and not disruptive of daily routines. The supervisors of the OECE felt
that the program evaluation was conducted successfully. They were very satisfied with
the information obtained from the evaluation. Moreover, the supervisors would like to
continue to improve the evaluation plan for the PBS program.
138
Recommendations
Recommendations for Improvement to the PBS Program
Although findings from the evaluation of the PBS program found that teachers
were generally satisfied with the program, the overall information derived from the
program evaluation suggests that improvements can be made to the PBS program that
will serve to further enhance the skills of classroom teachers for addressing the needs of
students with challenging behaviors. It is important to improve the PBS program because
intense, persistent, challenging behaviors in preschool are associated with school
adjustment and success in later years; behavior in preschool is the best predictor of
serious behavior problems in adulthood (Campbell, 2002). High quality early childhood
education environments lead to better social competencies and decreased behavior
problems in preschool students (Dunlap et al., 2006). It has also been shown that
appropriate teaching procedures are effective in developing children’s skills and reducing
challenging behaviors (Dunlap et al.). Collectively, these facts underscore the need to
improve teachers’ skills for addressing the needs of young children exhibiting
challenging behaviors. Improvements to the PBS program are a promising approach for
increasing teachers’ skills in this area. Therefore, the following recommendations are
made based upon the information obtained from the completed evaluation of the PBS
program.
First, it is recommended that the level of professional development for classroom
teachers remain strong each year of program implementation. In the first full year of the
PBS program, a variety of professional development sessions were offered, which
included full-day trainings, half-day trainings, and lunch-and-learn sessions. By the
139
fourth year of the PBS program, professional development decreased markedly. By the
fifth year of the PBS program, professional development was so sparse that systematic
documentation did not occur. The current program evaluation did not examine the
reasons for the decline in professional development. Teachers, however, indicated that
professional development was among the most helpful components of the PBS program
for improving skills for addressing the needs of children with challenging behaviors. It
is, therefore, recommended that professional development on the PBS program increase
to a level substantially similar to that of the first two years of program implementation. It
is further recommended that the professional development sessions be planned in
advance by PIRT members in concert with the supervisors of the OECE. It is also
recommended that a structured, systematic process for documenting professional
development be developed and maintained by the OECE.
In order to better achieve the goals of the PBS program, planners of professional
development can examine information obtained from the program evaluation to
determine areas of greatest need. Program evaluation information revealed that teachers
were able to demonstrate appropriate methods and strategies for building positive
relationships with students and families. However, teachers were generally not as
successful at demonstrating their abilities in the following areas: (1) identifying triggers
to challenging behaviors; (2) identifying functions of challenging behaviors; (3)
identifying adult responses to challenging behaviors that would likely reduce problem
behaviors; and (4) teaching new skills to replace challenging behaviors. PIRT member
and other support personnel should focus on developing these skills in classroom
teachers.
140
Further improvements to the PBS program would contribute toward teachers’
abilities for addressing challenging behaviors with the same intensity and purpose as
addressing academic readiness skills. The best way to improve the PBS program is to
develop a multisystems school-wide approach that involves classroom, non-classroom,
and individual student perspectives. Sugai & Horner (2002a) delineated five steps for
developing and maintaining a school-wide PBS program that are recommended for
incorporation into the PBS program of the OECE. These steps were reviewed in Chapter
II, and are to (1) establish a PBS leadership team; (2) secure school-wide supports from
staff; (3) develop data-based action plans; (4) arrange for high quality fidelity of
implementation; and (5) conduct formative data-based monitoring.
Hemmeter et al. (2007) emphasized the need for a program-wide PBS approach in
early childhood programs that are comprised of multiple schools. The OECE is one such
program. Accordingly, it is recommended that improvements to the PBS program
incorporate five factors that Hemmeter and colleagues identified as mandatory for
successful program-wide PBS implementation. These factors, which were also discussed
in Chapter II, are (1) development of a strong leadership team; (2) acknowledgment that
program-wide PBS takes time to develop; (3) recognition for teachers embracing the PBS
approach; (4) availability of consultants with experience in behavior support; and (5)
involvement of mental health consultants in the development of the program-wide PBS
approach.
141
Recommendations for Implementation of the Evaluation Plan
The following recommendations are made based on the evaluation of the
evaluation plan of the PBS program. In order to continue to successfully evaluate the
PBS program, the following recommendations are made. First, supervisors in the OECE
should designate an evaluation consultant to be responsible for ongoing evaluation of the
PBS program. The current investigator agreed to commit significant amounts of personal
time toward the successful completion of this evaluation because obtained data was made
available for dissertation purposes. The supervisors of the OECE will need to provide
future evaluation consultants with the appropriate resources necessary, particularly
temporal and human resources, to execute a successful program evaluation. The
evaluation consultant is responsible for ensuring that all aspects of the program
evaluation are conducted appropriately, including data collection, data analysis, and
communication of the results of the evaluation. A designated evaluation consultant, in
conjunction with access to all necessary resources, will increase the likelihood that future
program evaluations will be conducted successfully.
Second, consideration should be made for the improvement of the data collection
instruments used during the evaluation process. Instrument 2, Preschool Positive
Behavior Support Classroom Implementation Checklist can be a valuable tool for
identifying the extent to which specific strategies related to the PBS program are being
implemented in the classroom. For this program evaluation, however, only 27
implementation checklists were completed and the reliability of the instrument was
questionable. In order to be able to use the instrument to draw meaningful conclusions,
future evaluation consultants should collaborate with other key stakeholders to
142
investigate methods for improving how this instrument can be used in its current form, or
to determine how to improve the structure and use of the instrument. Further, PIRT
members, or other staff in the OECE that will be responsible for conducting observations
to complete this instrument must be provided with the opportunity to prioritize classroom
visits for the purpose of completing this instrument.
Improvements can be made to Instrument 1, Positive Behavior Support Teacher’s
Questionnaire as well. Consideration should be made to revise this instrument so that
open-ended questions align more accurately with questions in which teachers are asked to
self-assess specific skills related to the goals of the PBS program. This will increase the
ability of the instrument to identify improvements in teachers’ skills that are achieved as
a result of the PBS program. Accordingly, the data obtained from the evaluation plan
will be more likely to provide the supervisors of the OECE and other key stakeholders
with more meaningful and valuable information about the merit and worth of the PBS
program.
Third, one of the findings of the evaluation indicated that professional
development on the PBS framework declined over time. The current evaluation plan did
not address the reasons for this decline. Consideration should be made to include
examination of this phenomenon in future program evaluation plans so that meaningful
information can be obtained about how to improve the delivery and frequency of
professional development.
Fourth, improvements for communicating results of the program evaluation must
be considered. Original guidelines for communicating the results of the evaluation
included disseminating results to the supervisors of the OECE during a face-to-face
143
meeting. This meeting occurred as outlined in the evaluation plan; a written report was
presented to the supervisors, findings were discussed, and initial recommendations were
made. The program evaluation guidelines also included plans for the supervisors of the
OECE to meet with staff in the OECE, early childhood center directors, and classroom
teachers to discuss evaluation information and the modifications that will be made to the
program as a result of the evaluation. To date, however, communication of program
evaluation information to these key stakeholders has not occurred. It is recommended
that a plan to share evaluation information with all key stakeholders be revised to ensure
that all interested parties are informed about the results of the program evaluation. This
revision may include a change in designation of which stakeholders will participate in the
initial face-to-face meeting with the evaluation consultant and the supervisors of the
OECE. Additional participants in this meeting may serve to alleviate the supervisors in
the OECE from having the sole responsibility of disseminating evaluation information to
other key stakeholders. Another possible change for communicating results of the
program evaluation is to have the evaluation consultant be responsible for disseminating
evaluation information to all key stakeholders, rather than to the supervisors of the OECE
only. Regardless of how the plan to communicate evaluation information is revised, it is
important to maintain the integrity of the plan to communicate evaluation results.
Sharing evaluation information properly and thoroughly contributes to the abilities of all
key stakeholders to make decisions about (1) development and improvement of the PBS
program, and (2) improvements to the evaluation plan.
Finally, it must not be overlooked that program evaluation is an ongoing process
of gathering, analyzing, interpreting, and using information so that judgments can be
144
made about the worth of the program (Maher, 2000). Therefore, it is important to
emphasize that program evaluation plans are not intended to be fixed or constant. Rather,
program evaluation plans must remain fluid. There are principles and procedures that
structure the process of program evaluation, but program evaluation is intended to be an
ongoing process. As the PBS program continues to develop, it should be continuously
evaluated to that sound judgments can be made as to whether or not it is adding value to
the target population.
Summary
The purpose of this dissertation was to evaluate the PBS approach for reducing
challenging behaviors and increasing prosocial behaviors among preschool children in a
public urban preschool program. This chapter presented findings of the program
evaluation and findings of the dissertation. Findings of the program evaluation revealed
that teachers were generally satisfied with the PBS program and that the program was
implemented most robustly in the earliest years of program implementation. Findings of
the program also revealed that there was a discrepancy between teachers’ self-assessment
of skills and their ability to demonstrate the skills via open-ended questions. A final
major finding of the evaluation was that teachers indicated that professional development
on the PBS approach and coaching from PIRT members were most helpful for increasing
their skills. Findings of the dissertation were that the evaluation plan is feasible, key
stakeholders found the evaluation information useful, and there was a desire to continue
to use the evaluation plan for ongoing evaluations of the PBS program.
145
This chapter also presented recommendations for improvements to the PBS
program as well as recommendations for improvements to ongoing and future evaluations
of the PBS program. Recommendations for improvements to the PBS program included
robust professional development on the PBS approach to classroom teachers that is based
upon information obtained from the program evaluation about teachers’ skills. Another
recommendation was to utilize a systems level approach to implementing PBS that aligns
with best practices found in current literature. This includes incorporating a school-wide
approach (Sugai & Horner, 2002a), and a program-wide approach (Hemmeter et al.,
2007) to PBS implementation.
Recommendations for improvements to the evaluation plan included designating
an evaluation consultant, developing improvements to data collection instruments, and
considering improvements for communicating results of the program evaluation. Most
importantly, it was recommended that all stakeholders be mindful that program
evaluation plans are not intended to be fixed or constant. Program evaluation is an
ongoing process that provides information about the merit and worth of the program. It is
only through continuous development of evaluation plans such as those delineated by
Maher (2000) that improvement to programs such as the PBS program can be made.
146
REFERENCES
Armstrong, K. (2006, December). HOT DOCS: Helping Parents to Promote School
Readiness. Presented at the Winter Conference of the New Jersey Association of
School Psychology.
Arnold, D. H., McWilliams, L., & Arnold, E. H. (1998). Teacher discipline and child
misbehavior in day care: Untangling causality with correlational data.
Developmental Psychology, 34(2), 276-287.
Benedict E. A., Horner, R. H., & Squires, J. K. (2007). Assessment and implementation
of positive behavior support in preschools. Topics in Early childhood Education,
27(3), 174-192.
Buschbacher, P. W., & Fox, L. (2003). Understanding and intervening with the
challenging behavior of young children with autism spectrum disorder.
Language, Speech, and Hearing Services in Schools, 34, 217-227.
Campbell, S. B. (2002). Behavior Problems in Preschool Children: 2nd Edition: Clinical
and Developmental Issues. New York: Guilford.
Carr, E. G. (2007). The expanding vision of positive behavior support: Research
perspectives on happiness, helpfulness, hopefulness. Journal of Positive Behavior
Interventions, 9(1), p. 3-14.
Carr, E. G., Dunlap, G., Horner, R. H., Koegel, R. L., Turnbull, A. P., & Sailor, W., et al.
(2002). Positive behavior support: Evolution of an applied science. Journal of
Positive Behavior Interventions, 4(1), p. 4-16, 20.
Center on the Social and Emotional Foundations for Early Learning (CSEFEL, 2009).
Preschool Training Modules. Retrieved on May 18, 2009 from
147
www.vanderbilt.edu/csefel/preschool.html.
Duda, M. A., Dunlap, G., Fox, L., Lentini, R., & Clarke, S. (2004). An experimental
evaluation of positive behavior support in a community preschool program.
Topics in Early childhood Special Education, 24(3), 143-155.
Dunlap, G. & Fox, L. (1999). A demonstration of behavioral support for young children
with autism. Journal of Positive Behavior Interventions, 1(2), 77-87.
Dunlap, G., Sailor, W., Horner, R. H., & Sugai, G. (2009). Overview and history of
positive behavior support. In W. Sailor, G. Dunlap, G. Sugai, & R. H. Horner
(Eds.), Handbook of positive behavior support (pp. 3-16), New York: Springer.
Dunlap, G., Strain, P. S., Fox, L., Carta, J. J., Conroy, M., & Smith, B. J., et al. (2006).
Prevention and intervention with young children’s challenging behavior:
Perspectives regarding current knowledge. Behavioral Disorders, 32(1), 29-45.
Feinstein, S. (2003). School-wide positive behavior supports. The Journal of
Correctional Education, 54(4), 163-173.
Fox, L., Dunlap, G., Hemmeter, M. L., Joseph, G. E., & Strain, P. S. (2003). The
teaching pyramid: A model for supporting social competence and preventing
challenging behavior in young children. Young Children, 58(4), 48-52.
Fox, L. & Hemmeter, M. L., (2009). A program-wide model for supporting social
emotional development and addressing challenging behavior in early childhood
settings. In W. Sailor, G. Dunlap, G. Sugai, & R. Horner (Eds.), Handbook of
Positive Behavior Support (pp. 177-202). New York: Springer.
Fox, L., Jack, S., & Broyles, L. (2005). Program-wide positive behavior support:
Supporting young children’s social-emotional development and addressing
148
challenging behavior. Tampa, Florida: University of South Florida, Louis de la
Parte Florida Mental Health Institute.
Franzen, K. & Kamps, D. (2008). The utilization and effects of positive behaviors
support strategies on an urban school playground. Journal of Positive Behavior
Interventions, 10(3), 150-161.
Freeman, R., Eber, L., Anderson, C., Irvin, L., Horner, R., Bounds, M., & Dunlap, G.
(2006). Building inclusive school cultures using school-wide PBS: Designing
effective individual support systems for students with significant disabilities.
Research & Practice for Persons with Severe Disabilities, 31(1), 4-17.
Frey, A. J., Faith. T., Elliott, A., & Royer, B. (2006). A pilot study examining the social
validity and effectiveness of a positive behavior support model in head start.
School Social Work Journal, 30(2), 22-44.
Frey, A. J., Lingo, A., & Nelson, C. M. (2008). Positive Behavior Support: A call for
leadership. Children & Schools, 30(1), 5-14.
Gilliam, W. (2005). Prekindergarteners left behind: Expulsion rates in state
prekindergarten programs. Foundation for Child Development Policy Brief Series
No. 3.
Hemmeter, M. L. & Fox, L. (2006). Teaching pyramid observation tool for preschool
classrooms (TPOT): Research edition. Retrieved August 15, 2009 from
www.cde.state.co,us/early/downloads/PBS/TPOT_revised_02-08.pdf.
Hemmeter, M. L., Fox, L., Jack, S., & Broyles L. (2007). A program-wide model of
positive behavior support in early childhood settings. Journal of Early
Interventions, 29(4), 337-355.
149
Hiralall, A. S. & Marten, B. K. (1998). Teaching classroom management skills to
preschool staff: The effects of scripted instructional sequences on teacher and
student behavior. School Psychology Quarterly, 13(2), 94-115.
Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. B.
(2004). The school-wide evaluation tool (SET). Journal of Positive Behavior
Interventions, 6(1), 3-12.
Individuals with Disabilities Education Act (1997). Office of Special Education and
Rehabilitative Services, United States Department of Education. Retrieved April
22, 2009 from http://www.ed.gov/offices/OSERS/Policy/IDEA/the_law.html
Lee, J. & Walsh, D. J. (2004). Quality in early childhood programs: Reflections from
program evaluation practices. American Journal of Evaluation, 25(3), 351-373.
Maher, C. A. (2000). Resource Guide: Planning and Evaluation of Human Resources
Programs. Unpublished manuscript.
Maher, C. A. & Bennett, R. E. (1984). Planning and Evaluating Special Education
Services. Englewood Cliffs, NJ: Prentice-Hall.
Medley, N. S., Little, S., & Akin-Little, A. (2008). Comparing individual behavior
support plans from schools with and without schoolwide positive behavior
support: A preliminary study. Journal of Behavior Education, 17, 93-110.
Neilsen, S. L. & McEvoy, M. A. (2004). Functional behavioral assessment in early
education settings. Journal of Early Interventions, 26(2), 115-131.
New Jersey Department of Education. (2001). Preschool Program Implementation
Guidelines. Trenton, NJ: New Jersey Department of Education, Division of Early
150
Childhood Education. Retrieved in September 15, 2002 from
www.nj.gov/education/ece/dap/.
New Jersey Department of Education. (2008a). Preschool Program Implementation
Guidelines. Trenton, NJ: New Jersey Department of Education, Division of Early
Childhood Education. Retrieved on May 2, 2009 from
www.nj.gov/education/ece/dap/.
New Jersey Department of Education. (2008b). Elements of High Quality Preschool
Programs, N.J.A.C. 6A:13A. Trenton, NJ: New Jersey Department of
Education, Division of Early Childhood Education. Retrieved on March
18, 2009 from www.nj.gov/education/ece/code.
Nordquist, V. M. & Twardosz, S. (1990). Preventing behavior problems in early
childhood special education classrooms through environmental organization.
Education & Treatment of Children, 13(4), 274-281.
Office of Special Education Programs. (2009). Positive Behavior Support and the Law.
Retrieved on May 2, 2009 from the Office of Special Education Programs,
Technical Assistance Center on Positive Behavior Interventions and Support.
www.pbis.org/school.pbis_and_the_law.aspx.
Pierce, E. W., Ewing, L. J., & Campbell, S. B. (1999). Diagnostic status and
symptomatic behavior of hard-to-manage preschool children in middle childhood
and early adolescence. Journal of Clinical Child Psychology, 28(1), 44-57.
Qi, C. H. & Kaiser, A. P. (2003). Behavior problems of preschool children from low-
income families: Review of the literature. Topics in Early Childhood Special
Education, 23(4), 188-216.
151
Stormont, M. A., Covington Smith, S., & Lewis, T. L. (2007). Teacher implementation
of precorrection and praise statements in head start classrooms as a component of
a program-wide system of positive behavior support. Journal of Behavioral
Education, 16, 280-290.
Sugai, G. & Horner, R. (2002a). The evolution of discipline practices: School-wide
positive behavior supports. Child & Family Behavior Therapy, 24:1(23), 23-50.
Sugai, G. & Horner, R. (2002b). Introduction to the special issue on positive behavior
support in schools. Journal of Emotional & Behavioral Disorders, 10(3), 130-
135.
Sugai, G. & Horner, R. (2006). A promising approach for expanding and sustaining
school-wide positive behavior support. School Psychology Review, 35(2), 245-
259.
Sugai, G. & Horner, R. (2007). School-wide positive behavior support and response to
intervention: Lessons being learned. Retrieved May 3, 2009 from
www.pbis.org/common/pbisresources/presentations/grstillinois2007.ppt
Tremblay, R. E. (2000). The development of aggressive behavior during childhood:
What have we learned in the past century? International Journal of Behavioral
Development, 24(2), 129-141.
Turnbull, III, H. R., Wilcox, B. L., Stowe, M., & Turnbull, A. P. (2001). IDEA
requirements of use of PBS: Guidelines for responsible agencies. Journal of
Positive Behavior Interventions, 3(1), 11-18.
Webster-Stratton, C. (1999). How to Promote Children’s Social and Emotional
Competence. London: Paul Chapman.
152
Webster-Stratton, C. & Hammond, M. (1997). Treating children with early-onset
conduct problems: A comparison of child and parent training interventions.
Journal of Consulting and Clinical Psychology, 65, 95-103.
Wood, J. J., Cowan, P. A., & Baker, B. L. (2002). Behavior problems and peer rejection
in preschool boys and girls. The Journal of Genetic Psychology, 163(1), 72-88.
154
Instrument 1. Positive Behavior Support Teacher’s Questionnaire
PART I 1. Please check one of the following that best describes you: Regular education classroom teacher employed by a contracted early childhood center Regular education classroom teacher employed directly by the district
Special education classroom teacher Regular education teacher’s assistant employed by a contracted early
childhood center Regular education teacher’s assistant employed directly by the district Special education teacher’s assistant Family worker Early Childhood Center Director/Assistant Director Head Teacher Other: 2. If you are a teacher or teacher’s assistant, please indicate the number of years
that you have been teaching: 1-2 3-4 5-6 7 or more 3. Have you attended at least one full-day professional development on the Positive Behavior Support (PBS) framework during the time that you have been employed as part of the school district’s early childhood program?
Yes……….When? 2002-2004 and/or 2005-2009 No
Directions: Please do not put your name or any other identifying information on this
paper. Please write legibly and answer the questions to the best of your ability.
155
Positive Behavior Support Teacher’s Questionnaire (cont.)
PART II
1 2 3 4 5 not at all minimally neutral moderately extremely competent competent competent competent How competent do you feel about your ability to: 1. Implement activities and routines in your classroom that will prevent or reduce challenging behaviors? 2. Identify triggers that may result in children engaging in challenging behaviors? 3. Identify the functions of challenging behaviors exhibited by the children in your class? 4. Identify adult responses to challenging behaviors that may serve to decrease challenging behaviors? 5. Teach new skills to the children in your class that would replace challenging behaviors?
156
Positive Behavior Support Teacher’s Questionnaire (cont.) PART III 1. Please list two ways in which you build positive relationships with your students and their families.
a.
b.
2. Please list two functions of challenging behaviors that are seen in preschool children.
a.
b.
3. Please provide two examples of triggers that may result in the occurrence of challenging behaviors in preschool children.
a.
b.
4. Please list two things that you could do to prevent the triggers that you listed in #3 from occurring.
a.
b.
5. Please list two new skills that can be taught to a preschool child that could replace challenging behaviors.
a.
b.
157
Positive Behavior Support Teacher’s Questionnaire (PART III Cont.) 6. Please list two strategies that, as a result of PBS training(s) you have attended, you are implementing in your classroom to promote social-emotional learning.
a.
b.
PART IV Please indicate how much you agree or disagree with each statement.
1 2 3 4 5 Strongly
disagree
Somewhat disagree
Neither agree or disagree
Somewhat agree
Strongly agree
1.
My ability to reduce challenging behavior and increase positive behavior among my students has improved as a direct result of the PBS program.
2.
I can tell that the implementation of the PBS program is highly important to my director or principal.
3.
I can tell that the implementation of the PBS program is highly important to the Early Childhood Office.
4.
I receive excellent support for implementing strategies from the PBS program.
5.
Teaching social-emotional skills is the most important component of educating preschool students.
158
Positive Behavior Support Teacher’s Questionnaire (cont.) PART V Please categorize each of the following activities/services related to PBS as either a strength, adequacy, or an area in need of improvement: Strength Adequate Needs
Improvement 1.
Full day Professional Development on the PBS program
2.
Mini Lunch-and-Learn workshops addressing specific aspects of PBS
3.
Receiving written strategies from PIRT members
4.
Having strategies modeled and/or coached for me by PIRT members
5.
Obtaining action plans developed at Request for Assistance (RFA) meetings
6.
PIRT assisted development of Behavior Support Plans based on Functional Behavior Assessments
7.
Support from my director/principal
8.
Support from my PIRT coordinator
PART VI Please indicate the first, second, and third most important activities/services that you believe were or would be most helpful in developing your skills for reducing challenging behaviors and increasing positive behaviors in your classroom. Place a 1 in front of the item that you deem most helpful, a 2 for the second most helpful, and a 3 for the third most helpful. You may leave the remaining items blank. Full day professional development
Lunch-and-learn workshops
Written strategies from PIRT members
Coaching and/or modeling of strategies
Action plans developed at RFA meetings
Behavior Support Plans
Director/principal support
PIRT coordinator support
159
PART VII
Please feel free to provide any additional comments or suggestions concerning the PBS
program.
160
Instrument 1.1 Preschool Teacher Statistics
Refer to permanent product records to obtain information on preschool teacher gender and teacher certification. Refer to Instrument 1 Positive Behavior Support Teacher’s Questionnaire for data on years of teaching experience and PBS training status. Enter the number and percentage that corresponds to each data element. Academic year:
Data Element Number Percent Gender Female Male Years of Teaching Experience 1-2 years 3-4 years 5-6 years 7 or more years Teacher Certification Certified Pending (Alternate Route)
161
Instrument 1.2. Preschool Student Statistics
Refer to permanent product records to obtain information on preschool student gender, age, reason for referral to PIRT, referral to CST, and special education eligibility status. Enter the number and percentage that corresponds to each data element. Academic year:
Data Element Number Percent Gender Female Male Age 3 years old 4 years old 5 years old Classroom Placement Contracted Early Childhood Center In-district Classroom Reason for Referral to PIRT ESI-R Screening Language Behavior Language and Behavior Other Referral to CST 3 years old 4 years old 5 years old Special Education Eligibility Status
Eligible Not Eligible Not Evaluated
163
Instrument 2.1 Review of Professional Development on Positive Behavior Supports Refer to permanent product records to obtain information on the name of the workshop, the date it was presented and the number in attendance. Enter the corresponding information.
Name of Workshop Date Number in Attendance
Audience
164
Instrument 2.2 Provision of Support to Teachers This form is to be completed by PIRT members. Academic year: 1. Refer to permanent product records to obtain data on the type of support provided to
teachers and the frequency to which that support is provided. Enter the corresponding
data into the table.
Type of Support Provided to Teachers Frequency per Week Modeling Coaching Written Strategies Verbal Instruction
2. Please indicate the number of Behavior Support Plans
that you developed this academic year: