CaliforniaCollaborativeontheSocialandEmotionalFoundationsforEarlyLearning:

ProfessionalDevelopmentEvaluationReport

EvaluationofTeachingPyramidProfessionalDevelopmenttoPromoteSocial-EmotionalDevelopmentinPreschoolChildreninCalifornia

FINALEVALUATIONREPORTSUBMITTEDTOTHECALIFORNIADEPARTMENTOFEDUCATIONBY

UNIVERSITYOFCALIFORNIA,LOSANGELESSRMEVALUATIONGROUP

JUNE2016

THISPAGEINTENTIONALLYLEFTBLANK

i

TableofContents

EXECUTIVESUMMARY.................................................................................................................................1

EVALUATIONPURPOSEANDQUESTIONS....................................................................................................9

PROGRAMBACKGROUNDANDDESIGN....................................................................................................11

TeachingPyramidBackground..............................................................................................................11

TeachingPyramidProfessionalDevelopmentComponents..................................................................12

SiteClassifications.................................................................................................................................13

EVALUATIONDESIGN,METHODS,ANDLIMITATIONS...............................................................................15

EvaluationDesign..................................................................................................................................15

SampleSelection...................................................................................................................................15

SurveyMethods.....................................................................................................................................15

ObservationandInterviewMethods.....................................................................................................18

AnalysisMethods...................................................................................................................................18

Limitations.............................................................................................................................................18

EVALUATIONFINDINGS.............................................................................................................................21

TeachingPyramidServicesandSupport................................................................................................21

PartnerSiteImplementationofTeachingPyramidComponentsandPolicies......................................26

ImpactonKnowledge,Practice,andAttitudes......................................................................................32

CONCLUSIONSANDRECOMMENDATIONS................................................................................................47

Recommendations.................................................................................................................................47

Conclusion.............................................................................................................................................48

REFERENCES...............................................................................................................................................49

APPENDICES...............................................................................................................................................51

ii

ListofTables

Table1:SurveySampleCharacteristics.....................................................................................................17

Table2:PartnerSiteStaffParticipationinTeachingPyramidTrainingsandCoaching.............................29

Table3:PartnerSiteClassroomStaffKnowledgeofStrategies,byAmountofTrainingReceived

(%Knowledgeable/ExtremelyKnowledgeable)..........................................................................34

Table4:TrainingAttendees’KnowledgeBeforeandAfterTeachingPyramidProf.Development...........35

Table5:FrequencyofPartnerSiteClassroomStaffUseofTeachingPyramidStrategies.........................38

Table6:PartnerSiteClassroomStaffUseofTeachingPyramidStrategiestoFosterSocial-Emotional

Development,byAmountofTraining(%SayingOncePerDayorMore)...................................39

Table7:PartnerSiteClassroomStaffUseofTeachingPyramidStrategiesforClassroomSet-Up,by

AmountofTraining(%SayingYes).............................................................................................39

Table8:PartnerSiteClassroomStaffConfidenceinSupportingChildrenwithChallengingBehaviors....43

iii

ListofFigures

Figure1:TeachingPyramidTieredApproach............................................................................................12

Figure2:BenchmarksofQualityDataforSelectedPartnerSites.............................................................27

Figure3:PartnerSiteParticipationinTeachingPyramidTrainingandCoaching......................................28

Figure4:PartnerSiteClassroomStaffKnowledgeofSocial-EmotionalDevelopmentStrategies.............32

Figure5:TeachingPyramidTrainingAttendees’PerceivedChangeinKnowledge...................................36

Figure6:FrequencyofPartnerSiteUseofStrategiestoFosterSocial-EmotionalDevelopment

(%Usually/Always)......................................................................................................................37

Figure7:TeachingPyramidTrainingAttendees’PerceivedChangeinStrategyUse(%Usually/Always).40

Figure8:Non-PartnerSiteClassroomStaff’sPerceivedChangeinStrategyUseFollowingTeaching

PyramidTraining.........................................................................................................................41

Figure9:PartnerSiteClassroomStaffConfidenceinStrategiestoSupportChildrenwithChallenging

Behaviors,byAmountofTrainingReceived(%Confident/ExtremelyConfident)......................44

Figure10:TrainingAttendees’ChangeinConfidenceFollowingTeachingPyramidTraining

(%Confident/ExtremelyConfident)............................................................................................45

1

EXECUTIVESUMMARY

Evaluation Purpose and Evaluation Questions In the springof2015, theCaliforniaDepartmentofEducation (CDE) contractedwith theUniversityofCalifornia, Los Angeles (UCLA) SRM Evaluation Group to conduct an evaluation of Teaching Pyramidprofessionaldevelopmentforearlychildhoodeducationproviders.TheTeachingPyramid,alsoknownasthe California Collaborative on the Social and Emotional Foundations for Early Learning (CA CSEFEL),providesasystematicframeworkforpromotinghealthysocial-emotionaldevelopmentinyoungchildrenandpreventingchallengingbehaviors.TheSanMarcosOfficeof theWestEdCenter forChild&FamilyStudies (referred to in this report as theWestEd Teaching Pyramid team) provides Teaching PyramidprofessionaldevelopmentforearlychildhoodeducationstaffinCalifornia.

The purpose of this evaluation was to better understand the impact of the Teaching Pyramidprofessional development on early education practitioners’ knowledge, practice, and attitudes. Theevaluationquestionsaddressedprocessandimplementationaswellasoutcomes.Thespecificquestionsthatguidedtheevaluationareshownbelow.

Process/Implementation Questions 1. WhatsystematicservicesdoesWestEd’sTeachingPyramidteamprovidetoadministratorsand

earlychildhoodeducationteachingteams,andwhatisthequalityofthoseservices?2. TowhatextenthasWestEd’sTeachingPyramidteamprovidedguidanceandsupporttoensure

continuationofTeachingPyramidpracticeswhenitssupportisdiscontinued?3. TowhatextenthavepartnersitesimplementedtheTeachingPyramidprofessionaldevelopment

components?4. Towhatextenthavepartnersitesdevelopedspecificpoliciesandproceduresforworkingwith

childrenwhohavechallengingbehaviors?

Outcome Questions 1. What is the impact of the Teaching Pyramid professional development on program/site and

teachingstaff?2. To what extent has Teaching Pyramid professional development prepared staff to more

effectivelysupportchildren,includingthosewithchallengingbehaviors?

Teaching Pyramid Background TheTeachingPyramidisbasedupontheprinciplesoftheNationalCenterontheSocialandEmotionalFoundations for Early Learning (National CSEFEL). It provides a systematic, tiered framework thatpromotes social and emotional development, prevents challenging behaviors, and addressesproblematic behaviors. This tiered approach is grounded in the Early Childhood Positive BehaviorSupport (EC-PBS)model.WestEd’s Teaching Pyramid team adapted the National CSEFEL approach toinclude fourprofessionaldevelopmentcomponents,withagreateremphasison implementationwithfidelityandsustainabilityovertime.

Early childhood education providers contractwith theWestEd Teaching Pyramid team to receive thefourcomponentsasapackageofservices.Theseservicesaresite-based,meaningmultiplestaffacrossthe early childhood education sites receive training. Sites include public school districts, Head Start

2

programs,TitleV-fundedprograms,community-basedorganizations,privatecenters,andcountyofficesofeducation.Thefourcomponentsare:

• TrainingModules:o Module 1: Promoting Children's Success by Building Relationships and Creating

SupportiveEnvironmentso Module2:Social-EmotionalTeachingStrategieso Module3a:DeterminingtheMeaningofChallengingBehavioro Module3b:IndividualizedIntensiveInterventions

• Coaching• Leadershipteamdevelopmentandfacilitation• Follow-upforsustainability(trainingandtechnicalassistance)

Thefirstthreecomponentstypicallyoccurwithinthefirstyear,overthecourseofsixtoninemonths;thefollow-upforsustainabilitycomponentisoptionalinsubsequentyears.

Evaluation Design and Methods The evaluation team used a mixed-methods approach to answer the evaluation questions, includingsurveys,observations, interviews,andsecondarydata.Theapproachwasaconvergentparalleldesign(Creswell & Plano Clark, 2011), where the survey, observation, and interview data were collectedconcurrently and triangulated in the analysis. Interviews and observations were conducted betweenspring2015andspring2016.Surveyswereconductedinfall2015andwinter2016.

Becausethegoaloftheevaluationwastoassessprofessionaldevelopment,UCLA,CDE,andtheWestEdTeaching Pyramid team decided to focus on sites that had received most or all of the professionaldevelopment components (i.e., “partner sites”). TheWestEd Teaching Pyramid team identified eightpartnersitesthathadbeenconsistentlyimplementingtheTeachingPyramidcomponents.Teachersandadministrators from these eight sites were surveyed, interviewed and/or observed. In an attempt tobalanceanypotential selectionbias, theevaluationalso includedonenew“entry” site. Supplementalsurveys were also given to non-partner “practicing” site staff who had attended Teaching Pyramidtrainingwithinthelasttwoyears.

Inall,fivesurveyswereadministeredaspartofthisevaluation:

• Surveyofselectedpartnersiteclassroomstaff;• Surveyofselectedpartnersiteadministrators;• Surveyoftrainingattendeesfrompracticing,non-partnersites;and• Pre-andpost-trainingsurveysoftrainingattendeesfromanentrysite.

It is important to note that organizations became partner sites because they were committed toimplementingtheTeachingPyramid.Moreover,thesiteschosenfortheevaluationwereselectedduetotheirorganizationalabilitytoimplement.Inaddition,datafrommultipleyearsshowedthatthesiteshadvery low teacher turnover,1 and site visits suggest there was strong administrator support, includingfinancialsupport,andmanagementbuy-infortheTeachingPyramid.TheseconditionslikelycontributedtotheuseandcontinuationofTeachingPyramidpractices.Assuch,theoutcomesdescribedmaynotbe1Althoughtheintentwastoanalyzechangesinteacherturnoverovertime,theturnoverwastoolow(i.e.zerooroneperson/year)toreliablyanalyzetrendchangesduetotheTeachingPyramid.

3

the same for sites that lack these favorable conditions. The findings should be interpreted in thiscontext.

Evaluation Findings The evaluation findings provide an overview of Teaching Pyramid services and follow-up support,partner site implementation of these components, and the impact of the Teaching Pyramid on earlychildhoodeducators.Eachoftheseaspectsisdiscussedinturn.

Teaching Pyramid Services ThefirstcomponentoftheTeachingPyramidissitestafftraining.Accordingtoobservations,interviews,and the post-training entry site survey, the training modules are perceived as being of high quality.Interview participants were uniformly positive and appreciative of the training, and reported leavingeach module with concrete ideas for how to implement pyramid strategies. Likewise, surveyrespondents considered their instructors to be extremely knowledgeable. Both survey and interviewparticipantsreportedverypositiveopinionsoftheirtrainers,describingthemas“veryknowledgeable”and“awesome.”Themajorityofparticipantsfelttheirtimeintrainingwaswellspentandusefultotheireverydaywork, though it should alsobenoted that some indicated the eight-hourdaywas too long,“overwhelming,” and/or “intense.” As such, they suggested shorter,more frequent trainings instead.Further, interview data and observations of the training modules suggested that some participantswouldbenefitfromhavingthetrainingmaterialsinSpanishaswellasEnglish.

Thesecondcomponent,coachingsessions,areavaluablepartoftheTeachingPyramidandarecrucialto successful application of classroom strategies. The supportive approach reaffirms strengths,acknowledges teachers’ concerns regarding application, and gives them the tools to improve. Thecoaches were highly knowledgeable and demonstrated emotional intelligence; teachers praised theirability to put them at ease, and appreciated their positive, nonjudgmental, constructive tone.Throughouttheobservedsession,thecoachmodeledeffectivepractice,highlightedmaterialstoreview,and continually pushed strategies for sustaining Teaching Pyramid implementation once coachingended.

TheWestEdTeachingPyramidteamalsooverseesthedevelopmentofasite-wideleadershipteam—thethird componentof theTeachingPyramid. They facilitate fivemeetingsduring the six- tonine-monthcontract period. According to online Teaching Pyramid materials, the establishment of a site-basedleadershipteamis“thefirststep”inimplementation.2Theultimatepurposeoftheleadershipteamistoact as a guiding force for Teaching Pyramid support and site-wide impact and to oversee TeachingPyramidgrowthandimplementationbeyondandalongsidethetrainingmodules.

Guidance and Support to Ensure Continuation of Teaching Pyramid Practices The Teaching Pyramid’s fourth component—follow-up for sustainability—is designed to promoteenduring application of the Teaching Pyramid.Within this component, theWestEd Teaching Pyramidteam provides training sessions on advanced topics, training refreshers, technical assistance forleadership teams, and additional coaching. As the name suggests, this component is not part of theinitialpackage.Eventhoughsitesareencouragedto invest intheseadditionaltrainingstorefreshandadvance their practice, all follow-up or additional trainings are contingent on site budgets. The

2Seehttp://www.cainclusion.org/teachingpyramid/

4

interviews suggest that follow-up coaching is seldom utilized due to budget constraints, and surveyresultsindicatethatadministratorsaremorelikelythanclassroomstafftoattendadvancedtrainings.

Sustainability ispromotedbyothermeans.Per thesurveys, theWestEdTeachingPyramid teammostcommonly recommends thecontinuationof the site leadership team.Aspartof the leadership team,sitesareencouragedtodocumentandmonitorimplementationusingaBenchmarksofQualitytool.TheTeaching Pyramid team also facilitate involvement with countywide leadership teams, whichadministrators said they valued for the opportunities they provided to network with other TeachingPyramid sites and organizations. Thesemeetings built and reinforced regional buy-in, which led to astrongbaseofeducatorsadvocating to localdecisionmakers forcontinuedTeachingPyramid trainingsupport,mainlyintermsoffunding.

Other resources and tools also are provided. The WestEd Teaching Pyramid team administers theTeaching PyramidObservation Tool (TPOT) as ameasure of howwell teachers are implementing theTeaching Pyramid. This tool involves classroom observations and a teacher interview. The resultssuggest, however, that there is a need formore frequent andwidespreadadministrationof the tool.Additionally, there is annual symposia where teachers and administrators present on their sites’Teaching Pyramid practices. Thismeeting is highly regarded as anothermuch appreciated chance tonetworkwith colleagues. ThroughWiggio—anonline forumopen to partner sites—members can askothermembersquestions,accessanddownloadmaterialsandresources,participate inwebinars,andobtain informationaboutupcomingevents.Wiggiohas thepotential tobeofgreatbenefit,butsomeevaluation participants reported not knowing how best to navigate the site. Finally, the TeachingPyramidwebsiteprovidesbackground,family,andclassroommaterials,aswellasotherresources.Thissiteisopentoanyone,notjustpartnersites.

Partner Site Implementation of Teaching Pyramid Components Whiletheselectedpartnersiteswerechosenduetotheirperceivedconsistentimplementation,itisstillimportanttodocumenttheiractualimplementationbecausetheTeachingPyramidrecommendsallfouroftheabovecomponentsbeimplementedtoachieve“fidelityandsustainabilityovertime.”3Alloftheselectedpartnersiteshadparticipatedinthestafftrainingseries.Eachhadatleastoneadministrator,aswellasclassroomstaff,attendthethreecoremodules(Modules1–3a).AllsiteshadreceivedaTeachingPyramid coach as part of the package of services in the initial year. Sites were also encouraged toidentify an internal coach to support teachers indeployingTeachingPyramid strategies, although thesurveyresultscallintoquestionthelevelofimplementationofthiscomponent.

Accordingtothepartnersiteadministratorsurveysandinterviews,sevenoutoftheeightsitesreportedhavingaprogramleadershipteam.However,teacherinterviewssuggestthattheteamanditsrolemaynot have been prominent or always visible to staff members who were not actually on the team.Evaluators found that leadership regarding the Teaching Pyramid often took place in the context ofregular staff meetings and staff development efforts rather than in the context of formal TeachingPyramidleadershipteam.

The site leadership team is also charged with conducting the Benchmarks of Quality assessment tomonitorsite-wideimplementation.Theassessmentprovidesinformationregardingtheimplementationof10criticalelementsof theTeachingPyramid.Datawereonlyavailable for sixof theeight selected3Source:http://www.cainclusion.org/teachingpyramid/components.html

5

partner sites. Additionally, for those six sites, the data showed that the least implemented areawasmonitoringimplementationdataandoutcomes.

All sites had practices in place, to varying degrees, to ensure continuation of the Teaching Pyramid.ThesesustainablepracticesrangedfromutilizingadvancedtrainingsandrefreshersorongoingTeachingPyramidcoaching,toincludingTeachingPyramidtopicsinstaffmeetings,tosimplypracticingstrategieswithco-workers.SitesalsoutilizedleadershipteammeetingstofurthersupporttheirTeachingPyramidwork.

Partner Site Policies and Procedures for Working with Children with Challenging Behaviors All but one of the selected partner sites had a policy for working with children with challengingbehaviors. A few sites changed aspects of their existing policies and procedures as a result of theirparticipation inTeachingPyramid training. These changes included rewriting referral forms to includemoreTeachingPyramidlanguageandstrategies,changingformscompletely,suchas lessonplans,andoverlayingTeachingPyramidstrategiesonexistingpoliciesandprocedures.However,mostsitesalreadyhadclearpoliciesinplacethatwereagency-specificandcouldnotbechanged.

Impact of Teaching Pyramid Professional Development TheresultsoftheevaluationsuggestthatTeachingPyramidprofessionaldevelopmentledtoincreasedknowledge and use of classroom strategies to foster social-emotional development. Additionally, theprofessionaldevelopmentinfluencedbothindividualattitudesandorganizationalculture.

Knowledge Most classroom staff in selected partner sites felt they were knowledgeable or extremelyknowledgeableabout strategies topromotehealthy social-emotionaldevelopment.Whenaskedwhattheyattributedthisknowledgeto,themostcommonanswerwasprofessionaldevelopment, includingTeachingPyramidtraining.

Furthermore, the results of the other surveys—i.e., the non-partner practicing site training attendeesurvey and pre- and post-training surveys at the new entry site—indicated that staff increased theirknowledgeafterreceivingprofessionaldevelopment.Trainingattendeesperceivedthelargestincreasesintheirknowledgeofpositivedescriptiveacknowledgement(PDA),specificstrategiestoteachconflictresolutionskills,andspecificstrategiestoteachchildrenhowtorecognizetheirownfeelings.

Classroom Practice Theevaluation evidence suggests that the Teaching Pyramidpositively influenced classroompractice.Thevastmajorityofteachersandteachers’assistantssaidtheyusuallyoralwaysusedspecificTeachingPyramid strategies such as giving specific positive acknowledgement when children demonstratedappropriate behaviors, intentionally planning for transitions, and referring to classroom expectationswhentalkingtochildrenabouttheirbehavior.

Mostclassroomstaffhadpostedadailyschedulewithpicturesandprogram-wideexpectationsonthewall, and had arranged the classroom to prevent challenging behaviors. Likewise, the post-trainingsurveyattheentrysiteshowedasignificantincreaseinfrequencyofgivingPDA,intentionallyplanningfortransitions,andreferringtoclassroomexpectations.

6

Individual and Organizational Attitudes TeachersfeltthattheirapplicationofTeachingPyramidpractices increasedtheirsenseofprofessionalself-efficacy, resulting in more fulfilling and satisfying interactions with children; staff members alsolinkedthesefeelingstotheirownjobsatisfaction.Teachersnotedthattheyfeltmore“calm,”“relaxed,”and “in control.” Inmany cases, interviewees reported that the training affirmed pre-existing beliefsaboutearlychildhoodeducation.However,beinggiventoolstomoreactivelyreflectonthemeaningofchildren’sbehavior,respondconstructively,andwitnesschangesinchildren’sbehaviorofferedtangibleevidenceoftheirownabilitytopositivelyimpactthechildrenintheircare.

Aspreviouslynoted,languagewasoneofthebiggestpracticalshiftsthatoccurredacrosspartnersites,and this shift extended not only to children but also to adults. Adults gave each other positivedescriptive acknowledgement, and this enhanced interpersonal relationships as well as feelings ofrespect and recognition. Many teachers reported feeling more appreciated and having increasedenjoymentatwork.Giventhe importanceofcollaborationandteamteaching inthiscontext,strongercollegialrelationshipsbodewellformoraleandthesustainabilityoftheTeachingPyramidframework.

More Effective Support of Children with Challenging Behaviors TheTeachingPyramidaimstopreventchallengingbehaviorsbefore intensive intervention isrequired,andmostteachersfocusedonthisaspect.Nevertheless,challengingbehaviorscanarise,andtheresultsof the surveys and interviews suggest that approximatelyhalf of the classroomstaff felt confidentorextremelyconfidentthattheycouldsupportchildrenwhoexhibitedchallengingbehaviors.Oneteacherexplained,“Ihavemoretools….Thestrategiesaremoreclearandorganizedinmymind.”Yet,teachersalsoreportedwantingevenmoresupportwithstrategiestoaddressthesechallengingbehaviors,suchassupportusingBehaviorObservationReports–a formthathelpsteachersanalyzeand interpretthemeaningofachallengingbehavior.

Conclusions and Recommendations ThegoalofthisevaluationwastoassesstheTeachingPyramidprofessionaldevelopmentanditsimpactonearly childhoodeducationpractitioners’ knowledge, practice, andattitudes. Theevaluation resultsshow that theWestEd Teaching Pyramid teamprovides highquality trainings and coaching. TeachingPyramid professional development positively contributes to increased understanding and use ofstrategies to promote healthy social-emotional development, and gives educators more confidencewhendealingwith challenging behaviors. Furthermore, site staff reported that applying the TeachingPyramidresultedinmorefulfillingandsatisfyinginteractionswithbothchildrenandco-workers.

Strengths of Teaching Pyramid Professional Development • Promotes the use of classroom strategies throughout California that foster healthy social-

emotional development in children and appropriately support children with challengingbehaviors.

• Provides high-quality trainingmodules designed to engage early childhood education staff inmeaningfullearningexperiences.

• Providesstrength-based follow-upcoachingthat facilitates teachersapplying their learnings intheclassroom.

• Engagesstaffsite-widetopromotesustainablechangeinclassroomlanguage,environment,andpractice.

• ProvideswonderfulnetworkingandlearningopportunitiesthroughitsannualSymposium.

7

Recommendations The findings from this study were overwhelmingly positive, confirming the work and efforts of theTeaching Pyramid staff and partner sites. The recommendations below are intended to furtherstrengthentheprogram.

Recommendation 1: Follow-up for Sustainability Eventhoughsitesareencouragedtoinvestinfollow-uptrainingsandcoachingtoadvanceandrefreshtheir practice, these trainings are contingent upon site budgets and many site staff are unable toparticipate.Asdemonstratedbythesurveyresults(Table2),someclassroomstaffhaveneverreceivedformaltrainingorreceivedanycoaching.Follow-uptraining isanimportantpartof implementationatfidelity since staff may have attended Teaching Pyramid training years ago or there may be newteachers.Administratorsandteachers identifiedcoachingasanessentialpieceof implementationandsustained growth. CDE should explore additional ways to encourage and financially support sites inTeachingPyramidtrainingsandcoachingaftertheirinitialfirstyear.

Additionally, the leadership team, combinedwith strong administrative presence and support for theTeaching Pyramid at the various sites, is essential for continuing site-wide practice. The WestEdTeachingPyramidteamshouldcontinuetoprovideguidanceandsupport to thesite leadership team,evenafter the first year. This continued guidance for partner sites should specifically addresshow toeffectivelyusedatatoimproveTeachingPyramidimplementation.

Recommendation 2: Data Collection, Monitoring, and Sharing CDEandtheWestEdTeachingPyramidteamshouldexplorehowtopromotemoreconsistentcollection,monitoring,andsharingofdatabypartnersites.NotalloftheselectedpartnersiteshadBenchmarksofQuality (BOQ)dataavailable.Sincethesedata fosterquality implementationof theTeachingPyramid,theTeachingPyramidteamcouldworkwithsitestoensuremoretimelycollectionandsharingofthesedata. Likewise, theBOQdata showed thatmonitoring implementationandoutcomedatawasnot “inplace”foranyofthesites.TheTeachingPyramidteamshouldexplorehowtosupportthepartnersites,notonlyinthecollectionandsharing,butalsointheuseofthedata.

There is also a need for more timely and widespread administration of the Teaching PyramidObservationTool (TPOT).As this tool isadministeredbytheWestEdTeachingPyramidTeam,andnotthesites,theyshouldconsidertrainingmorepersonneltoadministerthetoolmorefrequently.

Recommendation 3: Manageable and Accessible Training and Coaching The trainings are highly regarded for their content and delivery, yet they are intense and requireengagementatahighlevel.TheWestEdTeachingPyramidteamshouldconsidertheoptionofofferingshorter,more frequent sessions. In addition, someparticipants’ first language is not English and theyhave varying degrees of proficiency. As such, materials and training in Spanish could provide evengreater access for Spanish speaking teachers. Further,much of the Teaching Pyramid strategy use islanguage dependent. Many materials have been translated already for use with families, but theTeachingPyramidshouldcontinuetoexpandthispracticeandmakeallmaterialsavailabletoteachersandfamiliesinmultiplelanguages.Similarly,WestEdTeachingPyramidteamshouldcontinueitseffortstorecruitandtrainbilingualcoaches.

8

Recommendation 4: Supporting Children with Challenging Behaviors There is a need for more professional development around challenging behaviors. While the initialtraining and coaching supports these needs, there is the opportunity for more classroom staff tobecomeconfident in their ability to support these children. Some teachersalsoexpressedadesire tolearnmoreaboutaddressing“topofthePyramid”children.

Recommendation 5: Wiggio Support AccesstoWiggioismadeavailabletopartnersites;however,aspartoffollow-upsupport,theTeachingPyramidteamshouldconsideraddingamini-unitonusingandnavigatingWiggiotogetthemostoutofthisextensiveresource.

Recommendation 6: Expanded Funding and Incentives for Data Collection A common theme throughout the findings and recommendations is the need for increased funds forfurthertraining,coaching,andteacherreleasetime.TheTeachingPyramidisextremelywell-received,andadditional fundingwouldhelpsustaintheprogramandexpand its reach.Withthephasingoutofthe Race to the Top Early Learning Challenge grants, CDE could explore different funding models.Additionally,CDEcouldconsiderhowtosetasidefundstoincentiveconsistentdatacollectionandusebythepartnersites.Thus,CDEshouldconsiderhowtosecureadditionalfundingfortraining,coaching,anddatasupport,whilerecognizingconstraintswithintheECEfield.

9

EVALUATIONPURPOSEANDQUESTIONSIn the springof2015, theCaliforniaDepartmentofEducation (CDE) contractedwith theUniversityofCalifornia, Los Angeles (UCLA) SRM Evaluation Group to conduct an evaluation of Teaching Pyramidprofessionaldevelopmentforearlychildhoodeducationproviders.TheTeachingPyramid,alsoknownasthe California Collaborative on the Social and Emotional Foundations for Early Learning (CA CSEFEL),providesasystematicframeworkforpromotinghealthysocial-emotionaldevelopmentinyoungchildrenandpreventingchallengingbehaviors.TheWestEdCenterforChild&FamilyStudiesprovidesTeachingPyramidprofessionaldevelopmentforearlychildhoodeducationstaffinCalifornia.

The purpose of this evaluation was to better understand the impact of the Teaching Pyramidprofessional development on early childhood education practitioners’ knowledge, practice, andattitudes. Theevaluationquestions addressedprocess and implementation, aswell asoutcomes. Thespecificquestionsthatguidedtheevaluationareshownbelow.

Process/Implementation Questions 1. WhatsystematicservicesdoesWestEd’sTeachingPyramidteamprovidetoadministratorsand

earlychildhoodeducationteachingteams,andwhatisthequalityofthoseservices?2. TowhatextenthasWestEd’sTeachingPyramidteamprovidedguidanceandsupporttoensure

continuationofTeachingPyramidpracticeswhenitssupportisdiscontinued?3. TowhatextenthavepartnersitesimplementedtheTeachingPyramidprofessionaldevelopment

components?4. Towhatextenthavepartnersitesdevelopedspecificpoliciesandproceduresforworkingwith

childrenwhohavechallengingbehaviors?

Outcome Questions 1. What is the impact of the Teaching Pyramid professional development on program/site and

teachingstaff?2. To what extent has Teaching Pyramid professional development prepared staff to more

effectivelysupportchildren,includingthosewithchallengingbehaviors?

Theevaluationteamgatheredinformationthroughsurveys,interviews,classroomobservations,andtrainingobservationstoanswerthesequestions.SiteswerepurposivelyselectedbyCDE,theWestEdTeachingPyramidTeam,andtheUCLAteam.MoredetailontheselectionprocessandthesitescanbefoundintheEvaluationDesign,Methods,andLimitationssection.

11

Inadditiontothefourprofessionaldevelopmentcomponents,theCACSEFELTeachingPyramidhasbeenadaptedandenhancedinseveralways,includingbutnotlimitedto:

• AligningmaterialswiththeCAEarlyLearning&DevelopmentSystem

• Increasedfocusonteacher’sself-reflectionandemotionalexperience

• Additionofearlychildhoodmentalhealthconcepts

• Useofpositivedescriptiveacknowledgement(PDA)topromoteintrinsicmotivation

• Additionofeducationonbraindevelopment

• Streamlinedformsandmaterials

• Removaloftime-outandphysicalrestraintreferences

• Presenceofasystematicconflictresolutionprocess

• Focusonteachers’andadministrators’under-standingofconceptsbehindthestrategies

CALIFORNIA ENHANCEMENTS

PROGRAMBACKGROUNDANDDESIGN

T e a c h i n g P y r a m i d B a c k g r o u n d TheTeachingPyramidisbasedupontheprinciplesoftheCenterontheSocial andEmotional Foundations forEarly Learning (NationalCSEFEL).NationalCSEFELishousedatVanderbiltUniversityandisaproject designed to strengthen the capacity of Head Start andchildcare programs in order to improve the social and emotionaloutcomesofyoungchildren. Itwas initially funded in2003bytheOfficeofHeadStartandtheChildCareBureau,withsupportfromthe Administration on Children, Youth, and Families at the U.S.DepartmentofHealthandHumanServicesinorderto(a)promotethe social-emotional development and school readiness of youngchildren, ages birth to five, and (b) disseminate research andevidence-based practices to early childhood programs across thecountry.

In California, theNational CSEFEL frameworkwas adapted by theSanMarcosOfficeoftheWestEdCenterforChild&FamilyStudies(referredtointhisreportastheWestEdTeachingPyramidteam)toinclude four professional development components. The fourcomponents—training modules, coaching, leadership team, andfollow-up for sustainability—provide a greater emphasis onimplementationwithfidelityandsustainability.California’sadaptedprogram is known as the Teaching Pyramid. It is part of theCalifornia Collaborative on Social Emotional Foundations for EarlyLearning(CACSEFEL),astate-widemulti-agencygroupfocusedonspreading theTeachingPyramid framework throughoutCalifornia.Please see the sidebar for more information regarding theCaliforniaadaptations.4

Initially, First 5 Los Angeles (First 5 LA) funded the San MarcosOfficeof theWestEdCenter forChild&FamilyStudies toprovidetrainingandtechnicalassistancetosixschoolreadinesssitesinLosAngelesCounty. TheEarlyEducation&SupportDivision (EESD)oftheCaliforniaDepartmentofEducation(CDE)laterunitedwiththeWestEd Teaching Pyramid team to have California become anofficialpartner.CDEfundedtheprogramwithaportionoftheRaceto the Top Early Learning Challenge grant funds. The WestEdTeachingPyramidteamandEESDcurrentlyco-leadCACSEFEL.

4Source:http://cainclusion.org/teachingpyramid/materials/family/CACSEFEL_Enhancements.pdf

12

T e a c h i n g P y r a m i d P r o f e s s i o n a l D e v e l o p m e n t C o m p o n e n t s

The Teaching Pyramid approachprovidesasystematic,tieredframeworkthat promotes social and emotionaldevelopment, prevents challengingbehavior, and addresses problematicbehavior (Figure 1). This approach isgroundedintheEarlyChildhoodPositiveBehavior Support (EC-PBS) model. Thebase of the pyramid fosters nurturingand responsive relationships for allchildren, along with building supportiveclassroom environments, with the goalof preventing challenging behaviors. Asyou move towards the top of thepyramid, it targets those children whoneed more emotional supports and/orintensiveinterventions

TheTeachingPyramidprofessionaldevelopmentisapackageofservicesandearlychildhoodeducationproviders contractwith theWestEd Teaching Pyramid team to receive them. These services are site-based,meaningmultiplestaffacrossearlychildhoodeducationsitesreceivetraining.Sitesincludethosein public school districts, Head Start programs, Title V-funded programs, community-basedorganizations,privatecenters,andcountyofficesofeducation.Theprofessionaldevelopmentpackageincludesfourcomponents:(1)stafftraininginfourmodules;(2)coaching;(3)aleadershipteam;and(4)follow-upforsustainability(trainingandtechnicalassistance).Thefirstthreecomponentstypicallyoccurwithin the first year, over the course of six to nine months, and the follow-up for sustainabilitycomponent is optional in subsequent years. The sections that followprovide anoverviewof the fourcomponents;theEvaluationFindingssectionprovidesmoredetail.

Training Modules Thetrainingseriesiscomposedoffour8-hourmodules(Modules1,2,3a,and3b).ThemodulesfollowthetieredTeachingPyramidframework,progressingfromthebaseofthepyramid,focusingonbuildingstrong relationships and supportive environments, through more targeted social-emotionaldevelopmentstrategiesand,ultimately,individualizedinterventionsthataddresspersistentchallengingbehavior.Themoduletopicsareasfollows:

• Module 1: Promoting Children's Success: Building Relationships and Creating SupportiveEnvironments

• Module2:Social-EmotionalTeachingStrategies• Module3a:DeterminingtheMeaningofChallengingBehavior• Module3b:IndividualizedIntensiveInterventions

Figure1:TeachingPyramidTieredApproach

13

Technicalassistanceisalsoprovidedtosupportclassroomteachers in implementingTeachingPyramidstrategiesintheirclassrooms.

Coaching In between each training module, a WestEd Teaching Pyramid team coach schedules follow-upobservationsandmeetingstoassesshowparticipantsare integratingTeachingPyramidstrategiesandpractices in theclassroom.Thecoachobservesandvideotapesaportionof theday, thenmeetswithparticipants to watch the video and debrief. The teachers and coach develop an action plan thatincludesstrategiesforteacherstopractice,feedbackforthecoach,andnextsteps.

Typically, a session includes two to three teachers and/or assistantswho are in the same classroom.Thesesessionsaremeanttoparalleltheapproachteacherstakewiththeirstudents,andthereforethecoach and teachers focus on what is going well and how to build on those strengths—for example,throughtheuseofpositivedescriptiveacknowledgements(PDAs).

Additionally,eachsiteisaskedtoidentifyatleastoneindividualtoreceivetrainingtoactasaninternalcoach to work with classroom teachers, build site capacity, and provide ongoing support forimplementationoftheTeachingPyramidapproach.

Leadership Team TheWestEd Teaching Pyramid team assists each site in developing a leadership team, composed ofadministrators, teachers, and specialists, which acts as a steering committee for program-wideimplementationoftheTeachingPyramid.Theyfacilitateteammeetingsoverthefirstyear,whichoccurat least five times during the training cycle. The leadership team monitors technical assistance andtraining needs, as well as problem-solves for classroom- or site-specific issues. Leadership teammembersattendtrainingsandprovidesupportfortraining,coaching,andclassroompractices.

Follow-up for Sustainabil ity Programs that have successfully completed the minimum training series and coaching sessions areprovided access to advanced training and additional technical assistance to strengthen the staffknowledgebaseandensuresustainability.ThisaspectoftheTeachingPyramidtrainingisoptional,andimplementationisdependentonthesite’sabilitytosecurethenecessaryfunding.

S i t e C l a s s i f i c a t i o n s Aspreviouslymentioned,sites implementingtheTeachingPyramid includeschooldistricts,HeadStartprograms,TitleV-fundedprograms,community-basedorganizations,privatecenters,andcountyofficesofeducation.Thesesitesareclassifiedintofourtypes:

• Entrysite:EntrysiteshavebegunadoptingtheTeachingPyramidapproachandhavesomebutnotallofthecomponentsinplace.Thesesitesareinterestedingrowingtothenextlevel.

• Practicing site: Practicing sites are using the approach and implementing some or all of thecomponents.

• Partnersite:Partnersiteshavecompletedthethreeprimarycomponents,andareinterestedinsustainability and fidelity. These sites collaboratewith theWestEd Teaching Pyramid team tocollectdata,sharepractices,andsupportsite-wideimplementation.

14

• Partnersitewithmentorclassroom:Withinsomepartnersites,individualclassroomsundergoadministrationof theTeachingPyramidObservationTool (TPOT) and receive adesignationof“implementingwithfidelity.”Theseclassroomsareidentifiedasmentorclassrooms,andthesesitesarecommittedtosharingtheirpracticesbymentoringotherprograms.

As of April 2015, there were 21 partner sites from 12 counties (Alameda, Fresno, Glenn, Merced,Monterey, Orange, Sacramento, San Diego, Santa Clara, San Francisco, Ventura, Yolo). Within thesesites,sevenclassroomshadbeendesignatedasmentorclassrooms.

15

EVALUATIONDESIGN,METHODS,ANDLIMITATIONS

E v a l u a t i o n D e s i g n Theevaluationteamusedamixed-methodsapproachtoanswertheevaluationquestions,drawingfromsurveys,observations, interviews,andsecondarydata.Theapproachwasaconvergentparalleldesign(Creswell & Plano Clark, 2011) where the survey, observation, and interview data were collectedconcurrentlyandtriangulatedintheanalysis.

S a m p l e S e l e c t i o n As noted in the previous section, early childhood education sites that receive Teaching Pyramidprofessionaldevelopmentare classified intooneof four types: entry,practicing,partner, andpartnerwithmentorclassroom.Sincethegoaloftheevaluationwastoassessprofessionaldevelopment,UCLA,CDE, and the WestEd Teaching Pyramid team decided to focus on partner sites because they hadreceivedmostoralloftheprofessionaldevelopmentcomponents.

EightpartnersiteswereidentifiedasconsistentlyimplementingtheTeachingPyramidcomponentsandincludedinthestudy.Whilethisapproachofferstheabilitytounderstandoutcomesforsitesthathadimplemented (or received the “dosage” of) the professional development components, it alsointroduces the potential for selection bias. To balance this potential bias, one entry site was alsoincluded. Supplemental surveys were also given to staff from practicing sites who had attendedTeachingPyramid trainingwithin the last two years. The selected sites representedamixof typesofagencies served, including school districts, private non-profits, Head Start programs, and communitycolleges.

S u r v e y M e t h o d s Five surveys were conducted as part of this evaluation. All of the following surveys collected dataregardingtheknowledgeanduseofTeachingPyramidstrategies:

• Surveyofselectedpartnersiteclassroomstaff• Surveyofselectedpartnersiteadministrators• Surveyoftrainingattendeesfromnon-partnerpracticingsites• Pre-andpost-trainingsurveysoftrainingattendeesfromanentrysite

Sincethefocusoftheevaluationwasonthepartnersites,thesurveyanalysisprimarilyfocusedontheresults of the partner site classroom staff and administrator surveys. The surveys from other sitesprovidesupplementalinformationtobetterunderstandtheprofessionaldevelopment’simpactoutsideofthepartnersitecontext.

Partner Site Surveys ThesurveyofselectedpartnersiteclassroomstaffoccurredbetweenOctober2015andJanuary2016.Sincethenumberofstaffateachpartnersitewassmall(30orfewer),theevaluationteamdistributedand collected hard copies in order to increase response rates. During site visits, the surveys weredistributed to teachers, teachers’ assistants, and any other classroom staff. Eighty-four percent ofeligibleclassroomstaffcompletedthesurvey.

16

Partner site administrators were surveyed between October 2015 and January 2016 to gatherinformation about the organization and implementation of the Teaching Pyramid components. Thissurveywasconductedonline,viaSurveyMonkey.Atleastoneadministratorfromeachsitereceivedanemail with a survey link. Thirteen administrators were invited to participate and 11 surveys werecompleted,withoneormoreadministratorsateachsitecompletingthesurvey.

Practicing Site Surveys The survey of training attendees from non-partner practicing sites was conducted in October andNovember 2015. This survey gathered feedback from those who had attended a Teaching Pyramidtrainingwithinthelasttwoyears(2013–2014and2014–2015).Thesamplingframewascompiledfromtheemailaddressesof thosewhoattended training,withpartnersiteattendees removed.Theonlinesurveywassenttoallemailaddressesonthesamplingframe.Atotalof901surveysweresentand297werereturned,fora33%responserate.

Entry Site Surveys The pre- and post-training surveys at the entry site were designed to better understand changes inteacherknowledgeandpracticeand togatherperceptiondataabout thequalityof the trainings.Theevaluationteamsurveyedonenewsitewhosestaffhadreceivedthemoduletraining.Thepre-trainingsurveywasgivenatthebeginningofthefirstmodule inthefallof2015.Thepost-trainingsurveywasgiven inthespringof2016afterthe lastmodule intheseries.Thesurveyswereadministered inhardcopyatthetrainings,distributedandcollectedbyevaluationstaff.Atotalof42surveyswerecollectedatthefirsttraining(pre-training),and40werecollectedatthelast(post-training).Onehundredpercentofthosewhoreceivedasurveycompletedit.Thepost-trainingsurveyaskedrespondentswhethertheyhadcompletedapre-trainingsurvey,sothatthepre-andpost-trainingsurveyscouldbecompared.

17

Table1:SurveySampleCharacteristics

Characteristic

PartnerSiteSurvey

(ClassroomStaff)

PracticingSiteSurvey(TrainingAttendees)

EntrySiteSurvey

(Post-Training)# % # % # %

PositionType(MultipleResponsesAllowed) ProgramDirector,SiteAdministrator/Supervisor 9 9% 58 25% 6 15%

Teacher 62 60% 119 50% 16 40%Teacher'sAssistant 33 32% 25 11% 21 53%

Other 2 2% 55 23% 3 8%Total 106 103%a 237 109%a 46 115%a

YearsofWorkinginEarlyChildhoodEducationLessthan5years 31 30% 33 11% 7 18%

5to9years 28 27% 48 17% 14 35%10to19years 25 24% 116 40% 15 38%

20ormoreyears 19 18% 92 32% 4 10%Total 103 100% 289 100% 40 100%

Missing 2 8 0 HighestLevelofEducation

HighSchoolDiploma/GED 32 32% 13 6% 10 26%Associate’sDegree 26 26% 48 21% 8 21%Bachelor’sDegree 29 29% 84 36% 13 33%

Master’sDegreeorHigher 10 10% 67 29% 4 10%Other 3 3% 21 9% 4 10%Total 100 100% 233 100% 39 100%

Missing 5 64 1 Ethnicity(MultipleResponsesAllowed)

AfricanAmerican 4 4% 10 4% 5 13%Asian 15 15% 26 11% 0 0%

NativeAmerican/AlaskaNative 0 0 5 2% 2 5%HawaiianorPacificIslander 0 0 3 1% 2 5%

Hispanic/Latino 56 57% 93 40% 20 51%White 20 20% 87 38% 20 51%Other 2 2% 7 3% 0 0%

DeclinedtoState 3 3% 14 6% 0 0%Total 100 101%a 245 106%a 49 126%a

Gender Male 4 4% 8 3% notasked notasked

Female 95 96% 223 97% notasked notaskedTotal 99 100% 231 100% notasked notasked

Missing 6 66 a Responsestotalmorethan100%duetorespondentschoosingmorethanoneresponse.

18

O b s e r v a t i o n a n d I n t e r v i e w M e t h o d s Inordertorepresentthescope,characteristics,andqualityofsupportofferedbytheTeachingPyramidprogram, theevaluation teamconductedobservationsand interviewsover12monthsat theselectedpartnersites.Similarly tothesurvey,pre-andpost-trainingobservationswereconductedattheentrysite.

Observation Data Collection ObservationsofTeachingPyramidtrainings(n=4),classroomteaching(n=24),astatewideleadershipteammeeting(n=1),atraineroftrainerssession(n=1),andacoachingsession(n=1)wereconductedat the various agencies. Observations lasted 1–8 hours, depending on the event being observed.Specifically,trainingslasted8hours,thestatewideleadershipteammeetingobservationlasted6hours,and classroomobservations and the coachingmeeting lasted approximately 1 hour. Field notesweretakenateachevent.

Interview Data Collection Interviewswereconductedateachofthepartnersiteswithteachers (n=20)andadministrators (n=15).OneWestEdTeachingPyramid teamcoachwasalso interviewed.The researchersalsohadmanyless formal conversations, during which WestEd Teaching Pyramid team staff, teachers, siteadministrators, and coaches shared information about their history at each site, their processes, andother details relevant to the context of their work. Interviewswere audio recorded and transcribed.Fieldnotesweretakenfollowingtheinformalconversations.

A n a l y s i s M e t h o d s Thesurveydatawereanalyzedusingdescriptiveand,whereappropriate,inferentialstatisticalmethods.RawdatawereeitherdownloadedfromSurveyMonkeyorenteredfromhardcopiesandimportedintoSPSS statistical analysis software. The data were cleaned and prepared for analysis. The analysisincluded those who worked with preschool children within the last 3 years. The observation andinterview data were reviewed, manually coded, and then interpreted, looking at the themes andpatternsthatemerged.

The evaluation team assessed patterns and themes from all datasets (surveys, interviews, andobservations),searchingforcommonalitiesanddissimilarities.Thesefindingswereusedtoanswertheevaluation questions and to form conclusions. While the analysis investigated site variation due toimplementationdifferences,theresultsarenotreportedbysitetoprotectconfidentiality.

L i m i t a t i o n s Althoughgeneralizabilityof the findingswasnotagoalof thisevaluation, it shouldbenoted that theselected partner siteswere identified as thosewith consistent implementation.Moreover, data frommultipleyearsshowedthatthesesiteshadverylowteacherturnover.5Sitevisitsalsosuggestedstrongadministrator support, including financial support, andmanagementbuy-in for theTeachingPyramid.These conditions likely contributed to the use and continuation of Teaching Pyramid practices. Thus,when considering the findings of this evaluation, contextmust be recognized. Specifically, these sites5Althoughtheintentwastoanalyzechangesinteacherturnoverovertime,theturnoverwastoolow(i.e.,zerooronepersonperyear)toreliablyanalyzetrendchangesduetotheTeachingPyramid.

19

maydiffer in importantways fromsites thatdonot implementconsistently,particularly in relationtoorganizationalsupport.Assuch,theoutcomesmaynotbethesameforsitesthat lackthesefavorableconditions.

Anotherlimitationisthelackofstudentdata.WhilethisevaluationintendedtoanalyzeDesiredResultsDevelopmentProfileData (DRDP)over time, someof the siteseitherdidnothaveorwereunable toaccessdataforpreviousyears.Furthermore,withthechangefromDRDP2010toDRDP2015,thedatawerenotdirectlycomparableacrossyears.

21

EVALUATIONFINDINGSTeaching Pyramid professional development was evaluated using a framework similar to ThomasGuskey’s (2000) 5 Levels of Professional Development, which identified the features of high qualityprofessionaldevelopmentandastructureforgatheringrelevantevaluationevidence.Throughsurveys,interviews, observations, and other data sources, informationwas gathered on the trainingmodules,coaching,administratorandclassroomstafflearninganduseofnewknowledgeandskills,andpartnersiteorganizationalcontext.

T e a c h i n g P y r a m i d S e r v i c e s a n d S u p p o r t

What systematic services does WestEd’s Teaching Pyramid team provide to administrators and early chi ldhood education teaching teams, and what is the quality of those services?

This section sheds light on the professional development services offered by the WestEd TeachingPyramid teamand thequality of those services.As described in theProgramBackground andDesignsection, the Teaching Pyramid professional development is a package of services that includes stafftraining in four modules; coaching; facilitation of a leadership team; and follow-up for sustainabilitythrough training and technical assistance. The first three components typically occur within the firstyear,andthefollow-upcomponentisoptionalinsubsequentyears.Eachisdiscussedinturn.

Training Modules As previously noted, training and technical assistance typically occurs over the course of six to ninemonths. TheWestEd Teaching Pyramid team leads fourmodules to introduce key concepts, providepracticalstrategies,andengageinmeaningful learningexperiencesthatsupporttheintegrationoftheTeachingPyramidapproach intoeveryday teachingpractices.Modules include interactivediscussions,videopresentations,andhands-onactivities.Theycenteronthefollowingtopics:

o Module 1: Promoting Children's Success by Building Relationships and Creating SupportiveEnvironments

o Module2:Social-EmotionalTeachingStrategieso Module3a:DeterminingtheMeaningofChallengingBehavioro Module3b:IndividualizedIntensiveInterventions

To emphasize the importance of a strong foundation, portions of Modules 2, 3a, and 3b are alsodevotedtoreviewingtheprinciplesfromModule1.Specifically,throughoutthefourmodules,trainerscontinuallystresstheimportanceofrelationshipsandenvironmentasthekeytosuccessoftheTeachingPyramid. All of the observed trainingmodules were led byWestEd Teaching Pyramid team certifiedtrainers.Atoneof theobserved trainings, two“trainers in training”directedportions in fulfillmentoftheircertificationrequirements.6

6The“train-the-trainers”programisofferedbytheWestEdTeachingPyramidteamtocertifythosewhoareinterestedinteachingtheTeachingPyramidmodules.Theyalsooffersasimilarcoachingtrainingprogram.

22

Site staff attend the training modules (and receive coaching) as part of a site-based team. As theinterviews revealed, this format is beneficial because it provides the ability to focus small groupdiscussions on site-specific issues, an increased sense of accountability to colleagues, and shareddevelopment of language and instructional habits necessary for implementing the Teaching Pyramidconsistentlyatthesite.Someintervieweesattributedbetterclassroomimplementationandoutcomestobeingpartof thesametrainingcohortasothermembersof theirclassroomteam;theynotedthathavinganentireteamtrainedwasessentialtosuccessattheclassroomlevel.

Aspartofthetrainings,participantsreceiveabindercontainingprintedcopiesofallofthePowerPointpresentations and resources covered in the four modules. This binder serves as the main writtenreference for participants throughout the trainingmodules, mirroring the training in its content andorder of presentation of topics. Not only was the binder heavily relied upon during the trainings,teachersatpartner sitesconsistentlycited itasoneof theirmain resourcesas they implemented theTeachingPyramidmonthsoryearsafterthetraining.Inadditiontothebinder,trainingparticipantsareencouragedtoaccessonlineresourcesthroughtheTeachingPyramidsectionoftheWorkingTogetherforInclusion&Belongingwebsite,aswellastheNationalCSEFELwebsite.7

Overall, the trainingmoduleswere high quality. Interview participantswere uniformly positive aboutandappreciativeof theiroverallexperiencewith themoduleseries. It shouldbenotedthat,althoughthemajorityof respondents felt the timewaswell spent, some felt that theeight-hourdaywas “toolong,”“overwhelming,”and/or“intense,”andtheysuggestedshorter,morefrequenttrainingswouldbepreferable.Nevertheless,eventhosewhonoteddifficultyinlosingtheirSaturdaysormissingworkfeltitwasaworthwhileexperience.ParticipantsreportedleavingeachmodulewithconcreteideasforhowtoimplementTeachingPyramidstrategies.

Thepost-trainingsurveyresultsattheentrysitesupportthispositiveassessment.Forallfourmodules,thesurveyaskedabouttheknowledgelevelsoftheinstructors,whetherthetimewaswellspent,andhowuseful the trainingswere toparticipants’everydaywork.Almostofallof the respondents (97%–100%) rated their instructors as “knowledgeable” or “extremely knowledgeable.” For each module,more than nine out of 10 felt like their time in the trainingswaswell spent. Likewise, almost of theattendees(95%–97%)saidthetrainingswereusefulorextremelyusefultotheireverydaywork.Inopen-endedcommentsinthesurveys,aswellasininterviews,participantsreportedverypositiveopinionsoftheirtrainers,oftennamingspecifictrainersanddescribingthemintermssuchas“veryknowledgeable”and“awesome.”

Coaching AsdescribedintheProgramBackgroundandDesignsection,fourcoachingsessionsareprovidedduringthe training period. The WestEd Teaching Pyramid team coach meets with site teachers after eachtraining module and conducts classroom observations and provides feedback. Coaches develop anactionplanthatoutlineshowtoincorporateTeachingPyramidconceptsandtoolsintheclassroom.

The observed coaching session highlighted the remarkable quality of the coaching services. Thestrengths-based approach appeared to validate teachers’ concerns of how difficult it was to applystrategies in themoment, and togive them the tools to improve. The coachwalked througha tapedexperience,pinpointingwherestaffhadenactedconceptsandmodelingwhattheycoulddonexttime.7Seehttp://csefel.vanderbilt.edu/andhttp://www.cainclusion.org/teachingpyramid/

23

Throughoutthesession,thecoachhighlightedsectionsinparticipants’binderstoreviewandcontinuallypushedstrategiesforsustainingTeachingPyramidimplementationoncecoachingended.

At the partner sites, interviewees had nothing but praise for the WestEd Teaching Pyramid teamcoacheswithwhom theyworked. Their comments affirmed that coacheswere highly knowledgeableeducators and experts on the Teaching Pyramid model. Moreover, they felt coaches were skilled atmaking participants feel at ease, and successful in cultivating a positive, nonjudgmental, andconstructivetonewhenofferingfeedback.Finally,theparticipantsviewedthecoachesasaccessibleforadditionalhelpifandwhenitwasneeded.

Importantly, although all participants had positive things to say about coaching in general, not allpartnersiteparticipantswereabletospeakabouttheircoachingexperiences ingreatdetail.Thiswasespecially true for thosewhohad completed the initial training a year ormorebefore the interview.SomeparticipantsalsohaddifficultydifferentiatingbetweentheWestEdTeachingPyramidcoachandotherinstructionalcoacheswhohadvisitedtheirclassrooms—coacheswhomayhavebeentrainedontheTeachingPyramidbutwerenotassociatedwiththecoachingcomponentof theTeachingPyramidprogram. For example, one participant noted that in addition to theWestEd Teaching Pyramid teamcoachvisits,herFirst5CoachwasCSEFEL-trainedandhelpedwithsomeTeachingPyramidstrategies.Likewise, some partner sites had an administrator or other person on staffwho acted as an internalTeachingPyramidcoachinadditiontoothercoachingandprofessionallearningsupportduties.

Leadership Team The establishment of a site-based leadership team is considered the first step in implementing theTeaching Pyramidmodel and, as part of thepackageof services, theWestEd Teaching Pyramid teamoverseesitsdevelopment.ProgramdocumentsstatetheWestEdTeachingPyramidteamwillworkwithsite contacts to facilitate the formation of an initial leadership team, composed of administrators,supervisors,teachers,andspecialiststaff.Thisteamisexpectedtomeetoncepriortotheinitialtrainingmodulesandfivetimesduringthetrainingcycle.TheultimatepurposeoftheleadershipteamistoactasaguidingforceforTeachingPyramidsupportandsite-wideimpact,andtooverseeTeachingPyramidgrowthandimplementationalongsideandbeyondthetrainingmodules.

TheWestEdTeachingPyramidteamfacilitatesandencouragesinvolvementwithcountywideleadershipteams.Administratorsatsevenoftheeightsurveyedpartnersitesreportedthatatleastonepersonattheirsiteparticipatedinacounty-levelleadershipteam.Administratorintervieweesvaluedtheseteamsfor the opportunities they provided to networkwith other Teaching Pyramid sites and organizations.Thesemeetingsalsoservedtobuildandreinforceregionalbuy-inamongparticipatingearlychildhoodeducationproviders,whichinturnledtoastrongbaseofeducatorsadvocatingforcontinuedTeachingPyramidtrainingsupport(mainlyintermsoffundingpriorities)tolocaldecisionmakers.

To what extent has WestEd’s Teaching Pyramid team provided guidance and support to ensure continuation of Teaching Pyramid practices when its support is discontinued?

Akeyelementof anyprofessional developmentprogram is theabilityof service recipients to sustainpractices learned beyond the point at which direct training is offered. The Teaching Pyramid hasmechanisms for sustainability built into the program,most explicitly through the fourth professionaldevelopment component—follow-up for sustainability. Additionally, the leadership team at each

24

participating site is designed to promote Teaching Pyramid practices after initial WestEd TeachingPyramid team services end. The Teaching Pyramid also provides a plethora of resources that earlychildhoodeducationproviderscanaccess.Thissectiondescribestheseservicesandresources.

Follow-up for Sustainabil ity TheWestEdTeachingPyramidteamprovidestrainingsessionsonadvancedtopics,trainingrefreshers,technicalassistanceforleadershipteams,andadditionalcoaching.Advancedtrainingsessionsinclude:8

• DiversePerspectivesonBehavior:HonoringCultureandFamilyThroughDialogue• Acknowledge,Ask,Adapt:TrainingStrategiestoImproveCommunicationwithFamilyMembers

inCollaborativeRelationships• SpunkyorSquirrely?TheRelationshipbetweenTemperamentandBehavior• UsingMusic,Art,andMovementtoSupportSocial-EmotionalDevelopment• SkillsforLife:GoingDeeperwithConflictResolutionandProblem-Solving• ClosetoMagic:Positive,DescriptiveAcknowledgementCanChangeEverything• Practice Makes Permanent! Increasing Language Models while Building the Brain and

DevelopingSocial-EmotionalSkills

Sitesareencouragedtoinvestinthesetrainingstoadvanceandrefreshthepracticeofteacherswhomayhavebeentrainedinitiallyoverayearago.However,alladditionaltrainingsarecontingentonthesitehavingthebudgettoincorporatethem.Ininterviews,alladministratorsexpressedconcernsabouthowtodeepentheknowledgeofthosewhohavebeentrainedaswellastobringnewpeopleintothetraining;theyexpressedadesiretoprovidesupportandtrainingforcurrentstaffandfornewstaffandsubstitutes.Atthesametime,someexpressedregretatnothavingthefundstodoeverythingtheywouldliketodo.Forexample,fundingcoachingbeyondthefirstyearisidealasfarasimplementingwithfidelity;however,manysitessawitasmorecosteffectivetohandlethisin-house,withsupervisors,administrators,andteachersdrivingthecontinuedconversation.

Benchmarks of Quality Assessment ThepartnersiteadministratorsurveyaskedwhethertheWestEdTeachingPyramidteamhadprovidedguidancetoensurecontinuationofTeachingPyramidpracticesonceinitialsupporthadended;allsitesresponded“yes.”Specifically,engagementwithaleadershipteam—whethersite-basedorotherwise—wasthemostcommonexampleofguidancegiven.Aspartofitsworkwithsite-basedleadershipteams,the WestEd Teaching Pyramid team encourages the documentation of implementation using theBenchmarks of Quality (BOQ) tool. Typically, the BOQ occurs annually each spring. The assessmentcollectsdataonthefollowing10criticalelements:

A. EstablishleadershipteamB. Staffbuy-inC. FamilyinvolvementD. Program-wideexpectationsE. Strategiesforteachingandacknowledgingtheprogram-wideexpectationsF. AllclassroomsdemonstratetheadoptionoftheTeachingPyramidG. Proceduresforrespondingtochallengingbehavior

8Source:http://www.cainclusion.org/teachingpyramid/sustainability.html

25

H. StaffsupportplanI. CoachingandtechnicalassistanceJ. Monitoringimplementationandoutcomes

ThisBOQtoolallowssitestosetTeachingPyramidgoalsandmonitorprogress.Theaboveitemscanbedesignatedas“notinplace,”“inprocess,”or“inplace.”

TheWestEdTeachingPyramidteamprovidedBOQdataforsixoftheeightselectedpartnersites.ThelackofBOQdatafortheremainingtwositescouldbeanindicationthattheirleadershipteamsdidnotconduct these self-assessments. The potential low implementation of this assessment tool itself wasfurthersupportedbytheinterviews.OnlyoneintervieweeexplicitlymentionedtheBOQ;itcameupinthe context of discussing the countywide leadership team, where it was the focus of the work thathappenedthere.

Other Resources and Tools Teaching Pyramid Observation Tool The Teaching Pyramid Observation Tool (TPOT) is considered ameasure of program implementationfidelity. It is a copyrightedassessment instrumentdevelopedby researchers andusednationally. It isintendedtobegivenbyatrainedadministratorwhoobservesaclassroomandinterviewstheteacherswithinit.ImplementationoftheTPOTisthecriterionforbeingdesignatedasapartnersitewithmentorclassroom,whichisagreatpointofpride.Forexample,onesitepostedthedesignationontheirpublicwebsite,accompaniedbyacongratulatorylettertotheirteachers.

WhileuseoftheTPOTfosterssustainability,implementingithassomechallenges.First,logistically,theWestEdTeachingPyramidteamdoesnothaveenoughpersonneltrainedtoaccommodatepartnersites’requeststohavetheTPOTadministered.Second,thementordesignationstayswiththeclassroomforthree years, even if one ormore of the observed teachersmoves to another classroom and/or newuntrained teachers join the classroom. Indeed, as evidenced by conversationswith site staff, teacherteamsfrequentlychangedasteachersshiftedtodifferentclassrooms.

Symposia TheWestEd TeachingPyramid teamholds annual symposia atwhich teachers and administrators areinvitedtoshowcasewhattheyaredoingattheirsitestosustain,expand,andstrengthentheiruseoftheTeachingPyramid. Topics included, engaging families, techniques for addressing challengingbehavior,and how to use data to advocate for your program. Interviewed administrators and teachersacknowledgedthebenefitsofattendingtheseeventsandspokeenthusiasticallyaboutpastandfuturesymposia.

Online Resources TheWestEdTeachingPyramid teamprovidesbackground, family, and classroommaterials, aswell asother resources, on the Teaching Pyramid pages of theWorking Together for Inclusion & Belongingwebsite,whichisopentoanyone,notjustpartnersites.

Wiggio is an online forum open to partner sites through which teachers and administrators can askquestionsofothermembers.Theycanalsoaccessanddownloadmaterialsandresources,participateinTeachingPyramidwebinars,andobtain informationaboutupcomingevents.MostadministratorswhowereinterviewedhadaccessedWiggio,andsomereportedusingittofindresources,suchasIn-Service

26

Spotlights.However,most also expressed some confusion about howbest to use it.One intervieweenotedthat ithadmanygreatforumsandresourcesbutalsocommented,“It isnotvery intuitive. Iamnewtoit,butIhaveahardtimefindingwhatIwant….IneedaWiggiolesson.”

P a r t n e r S i t e I m p l e m e n t a t i o n o f T e a c h i n g P y r a m i d C o m p o n e n t s a n d P o l i c i e s

To what extent have partner sites implemented the Teaching Pyramid professional development components?

Whiletheselectedpartnersiteswerechosenduetotheirperceivedconsistentimplementation,itwasstillimportanttodocumenttheiractualimplementationoftheprofessionaldevelopmentcomponents.TheTeachingPyramidteamrecommendsthatsitesimplementallfourcomponentsinordertoachievefidelityandsustainability.Inadditiontothesurveysandinterviews,theBOQassessmentalsoprovidesdataregardingsite-wideimplementation.

Benchmarks of Quality As mentioned previously, Benchmarks of Quality is a site-wide assessment tool intended to beimplementedby leadership teams. Itprovides informationregarding the implementationof10criticalelementsoftheTeachingPyramid.Withineachelement,thereanumberofitemstoberated.Aspartofthisevaluation, theWestEdTeachingPyramid teamprovidedBOQdata fromsixof theeight selectedpartnersites.

At the six sites that completed the BOQ, there was distinction in the implementation of the criticalelements. Figure 2 shows thepercentageof each element’s total item responses thatwere rated “inplace.” It does not reflect the percentage of sites, but rather provides a comparison of the criticalelement implementation across the sites. Of the ten critical elements, the adoption of the TeachingPyramidintheclassroomwasrankedhighest,with70percentofitemsratedas“inplace.”Thiselementincluded items like“teachersandprogramstaffhavestrategiestopromotepositiverelationshipswithchildren,eachother,andfamiliesinplace,andtheyusethosestrategiesonadailybasis”and“teachersandprogramstaffhavearrangedenvironments,materials,andcurriculum inamanner thatpromotessocial-emotional development and guides appropriate behavior.” The least implemented area wasmonitoringimplementationandoutcomes,withnoneoftheitemresponsesreceiving“inplace.”

Therewasvariationamongthesites.Twositesindicated“inplace”implementationinmostareas;twowere in themiddle,withamixtureof“inplace”and“inprocess”; theremaining twoweremostly“inprocess,”orreportedacombinationof“notinplace”and“inprocess.”

27

Figure2:BenchmarksofQualityDataforSelectedPartnerSites

Source:WestEdinternaldata,July2015.

0%

29%

33%

33%

56%

56%

56%

57%

57%

70%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

MonitoringImplementauonandOutcomes

StaffSupportPlan

FamilyInvolvement

CoachingandTechnicalAssistance

EstablishLeadershipTeam

StaffBuy-in

StrategiesforTeachingandAcknowledgingtheProgram-wideExpectauons

Program-wideExpectauons

ProceduresforRespondingtoChallengingBehavior

AllClassroomsDemonstratetheAdopuonoftheTeachingPyramid

PercentofTotalElementItemsRated"InPlace"

28

Site Implementation of Training AsshowninFigure3,alloftheselectedpartnersiteshadparticipatedinthetrainingseries.Eachhadatleastoneadministrator,aswellasclassroomstaff,attendthethreecoremodules(Modules1–3a).Thefourth module (Module 3b) is intended for those supporting challenging behaviors and intensiveinterventions.

SomeagenciesthathadadoptedtheTeachingPyramidagency-widehadinvestedinhavingtheWestEdTeachingPyramidteamtraintheirownstafftoleadthemodules.ThismakestrainingnewstafforthosewhomissedtheTeachingPyramidtrainingsmorecost-effectiveandefficient.AsshowninFigure3,sixoftheselectedpartnersiteshadinvestedin“trainingthetrainers”andfourhadparticipatedintrainingforcoaches’professionaldevelopment.

Figure3:PartnerSiteParticipationinTeachingPyramidTrainingandCoaching

Source:Partnersiteadministratorsurvey,partnersiteadministratorinterviews,andpartnersiteclassroomstaffsurvey(October2015–January2016).

Table2showsthepercentageofsitestaffthatreceivedthevarioustrainings.Modules1and2werethemostcommonlyattended;100%and91%ofadministratorsattendedthesetrainings,respectively,andapproximatelythreeoutoffourclassroomstaffmembers(74%and76%,respectively)didso.Althoughstillamajority,fewerpartnersitestaffattendedModules3aand3b.Eightoutofthe11administrators(73%) also participated in the “Going Deeper” training, and nine out of the 11 (82%) attended thewebinars.

8 8 8

6

4

0

1

2

3

4

5

6

7

8

Modules1-3a Module3b CoachingfromWestEdTeachingPyramidTeam

TrainingtheTrainers TrainingforCoaches

#ofSelectedPartne

rSite

s

29

Table2:PartnerSiteStaffParticipationinTeachingPyramidTrainingsandCoaching

TeachingPyramidTrainingComponentAdministrators

(n=11)aClassroomStaff

(n=101)Module1:BuildingRelationshipsanCreatingSupportiveEnvironments 100% 74%Module2:Social&EmotionalTeachingStrategies 91% 76%Module3a:IndividualizedIntensiveIntervention:DeterminingtheMeaningofChallengingBehavior 82% 67%Module3b:IndividualizedIntervention 82% 58%GoingDeepertraining 73% 31%One-dayoverviewtraining 18% 17%3-houronlineoverviewtraining na 2%Webinars 82% 7%In-servicespotlights 27% 15%CoachingbyWestEdTeachingPyramidTeam 45% 24%Coachingbyanotherprogramstaffmember 18% 13%Other 18% 4%None 0% 20%

aThetotalsamplesizeis11,andonetotwopeoplecanresultinlargepercentagedifferences.Assuch,thepercentagesshouldbeinterpretedwithcaution.Source:Partnersiteclassroomstaffsurvey,partnersiteadministratorsurvey(October2015–January2016).

Site Implementation of Coaching CoachingfromtheWestEdTeachingPyramidteamispartofthepackageofservicesintheinitialyear.Sitesalsoareencouragedtoidentifyaninternalcoachwhowillsupportclassroomteachersindeployingthe Teaching Pyramid strategies. The internal coach receivesmentoring and support from aWestEdTeachingPyramidteamcoach.

PartnersitesurveyssuggestedthatfewerthanhalfoftheclassroomstaffhadreceivedcoachingfromaWestEd Teaching Pyramid team coach (24%) (Table 2).Most interviewed teachers reported receivingsomecoaching.Asmentionedearlier,someinterviewparticipantswereunabletodifferentiatebetweenthe WestEd Teaching Pyramid team coach and other instructional coaches who had visited theirclassrooms.Manyofthesesiteshadgonethroughtheirtrainingandcoachingcyclestwotothreeyearsago, and respondents may have had difficulty recalling their coach specifics. Thus, it was unclearwhethersomesitesfullyimplementedthecoachingcomponent.

Additionally,asmallpercentageofclassroomstaff(13%)recalledbeingcoachedbyaninternalprogramstaff member, calling into question the level of implementation around the internal coachingcomponent(Table2). Interviewsalsosuggestedthe inconsistentdeploymentof internalcoaching.Thelackof timeon thepart of a potential coachandof teacherswas cited as apotential reason for lowimplementation.

Site Implementation of Leadership Team According to thepartner siteadministrator surveysand interviews, sevenoutof theeight siteshadaprogram leadership team. Interviewed teachers who were part of the team saw his or her role asprovidingtheteacherperspective.Interviewswithotherteachersrevealed,however,thattheteamandits rolemay not be prominent or visible to staffmemberswho are not actually on it. The interviewssuggested that some teachers were either unaware of their Teaching Pyramid leadership team or

30

uninvolvedwithit.Otherteacherswhowereawareoftheirleadershipteam,butnotonit,thoughtitsactivitiesextendedbeyondthescopeoftheTeachingPyramidandincludedbroaderstrategicplanningandprofessionaldevelopmentfunctions.

ThisdoesnotsignifythatsiteleadershipwasuninvolvedorlackingcommitmenttosupportingTeachingPyramidpracticesandstrategies;onthecontrary,siteadministrationsweregenerallyveryenthusiasticabout the Teaching Pyramid as a whole. Evaluators found that leadership regarding the TeachingPyramidoftentookplaceinthecontextofregularstaffmeetingsandstaffdevelopmentefforts,ratherthan in the context of a formal leadership team; however, these professional development activitiesmayhavebeenplannedattheteammeeting.

Site Implementation of Fol low-up and Sustainabil ity Activit ies Accordingtotheadministratorsurveyandinterviews,allsiteshadpracticesinplace,tovaryingdegrees,in an effort to ensure continuation of the Teaching Pyramid. These practices ranged from utilizingTeaching Pyramid resources (such as advanced trainings and refreshers and ongoing coaching), toincludingTeachingPyramidtopicsinstaffmeetings,tosimplypracticingthestrategieswithco-workers.AdministratorsalsomentionedutilizingleadershipteammeetingstofurthersupportTeachingPyramidwork.

As shown above in Table 2, advanced trainings and webinars were more commonly utilized byadministratorsthanclassroomstaff;thismaybeduetothetimingofthetrainingsandthechallengeofsecuring classroom coverage. Almost three out of four administrators (73%) and approximately onethirdofclassroomstaff(31%)atpartnersitesattendedtheadvancedtraining,“GoingDeeper.”Whilealargeshareofadministrators(82%)attendedTeachingPyramidwebinars,only7%ofclassroomstaffdidso. Nevertheless, attending follow-up trainings and any kind of refresher was spoken of highly ininterviews.

Participantshadaccesstotwoonlineresources.Allreportedaccessingtheseonlineresourcestosomedegree.TheTeachingPyramidWiggiowebsitewas intendedtobeanaccesspointtoonlinewebinars,discussionforums,andresources.However,interviewsindicatethattherewasinconsistentuseofthisplatform and not all administrators and coaches knew how to use it as intended. In addition, theTeaching Pyramid website pages were cited by both teachers and administrators as an easy-to-usesourcefordownloadingmaterials.Teachersappreciatedwhencoachesandadministratorsprintedthematerialsforthem,however,asmanydidnothaveaccesstoacomputertodosointheclassroomanddidnothaveasmuchtimeastheywouldliketoexplorethesiteandfindthematerials.

Perhaps the resource employed the most at selected partner sites was the binder that individualsreceivedatTeachingPyramidtrainings.Individualsusedthematerialstorefreshtheirmemoriesaboutparticularstrategiesandastemplatesforuseintheirclassrooms.Thisheldtrueforparticipantswhohadbinders from 3 years ago as well as those who were recently trained. The investment in continuedrefinementanddevelopmentof thebinders thusappears tobepayingoff inboth the shortand longterm.

31

To what extent have partner sites developed specif ic policies and procedures for working with children who have challenging behaviors?

Thesurveygiventoadministratorsshowedthatallbutoneofthepartnersiteshadapolicyinplaceforworkingwithchildrenwithchallengingbehaviors.Theinterviewdataalsosupportedthisfinding.Whenaskedduringinterviews,staffatsomeagenciesreportedchangingaspectsoftheirexistingpoliciesandprocedures followingparticipation in theTeachingPyramid training.However,mostalreadyhadclearpoliciesinplacethatwereagency-specificandnotabletobechanged.

A few sites did rewrite certain policies andprocedures in response to the Teaching Pyramid training.These changes included rewriting referral forms to include more Teaching Pyramid language andstrategies,completelychangingforms,suchaslessonplans,andoverlayingTeachingPyramidstrategiesontoexistingpoliciesandprocedures.OnesitereportedincludingtheTeachingPyramidtrainingasanexpectationfornewhires,adoptingthetrainingagency-wide,andcommittingtotrainingallnewhires,includingsubstitutesandteachingassistants.

Most interviewparticipants reportedusing the forms fromtheTeachingPyramidbinder tohelp themdevelop individual interventions for students. Themost common formmentionedwas theBehavioralObservationReport(BOR),whichisusedtodocumentstudentbehavioranddescribetheconditionsinwhich the behavior occurs. BORs are collected and analyzed to determinewhat the child is trying tocommunicateandtodeterminepatternssothattheteacherbecomesmoreawareofwhatistriggeringthebehaviorandcanmitigateit.Despiteparticipants’commonmentionsofBORs,manyadministratorsstillreportedpushingforincreaseduse;teachersadmittedtoknowingaboutBORsbutnotusingthemasmuchastheywantedto,citingtimeandattentionawayfromtheotherchildrenasanobstacleaswellaswantingmoresupportandpracticewiththisform.

32

I m p a c t o n K n o w l e d g e , P r a c t i c e , a n d A t t i t u d e s

What is the impact of the Teaching Pyramid professional development on program/site and teaching staff?

TheresultsoftheevaluationsuggestthatTeachingPyramidprofessionaldevelopmentledtoincreasedknowledgeofanduseofclassroomstrategiestofostersocial-emotionaldevelopment.Additionally,theprofessionaldevelopment influencedboth individualattitudesandorganizational culture.This sectiondescribesthesefindings.

Knowledge of Strategies to Promote Healthy Social-Emotional Development The surveys and interviews indicate that Teaching Pyramid professional development resulted inincreasedknowledgearoundstrategiestopromotechildren’ssocial-emotionaldevelopment.Asshownin

Figure 4, most classroom staff at partner sites felt they were knowledgeable or extremelyknowledgeableinstrategiesrelatedtosocial-emotionaldevelopment.Whenaskedwhattheyattributedtheir knowledge to, the most common answer was professional development, including TeachingPyramidtraining(69%;notshown).Practiceand/orexperiencewasthenextmostprevalentanswer,at26%(notshown).

Figure4:PartnerSiteClassroomStaffKnowledgeofSocial-EmotionalDevelopmentStrategies

9%

3%

1%

3%

2%

1%

0%

1%

1%

1%

1%

35%

31%

25%

19%

18%

18%

18%

16%

15%

13%

13%

56%

66%

74%

78%

80%

81%

82%

83%

84%

86%

86%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

DevelopingaTeacherSupportPlanorPosiuveBehaviorSupport(PBS)plansforindividualchildren

Collecungdatatodeterminewhyachildhaspersistentlychallengingbehaviorsintheclassroom

Specificstrategiestoteachconflictresoluuonskillstopreschoolchildren

PosiuveDescripuveAcknowledgement(PDA)

Strategiestoarrangetheclassroomtopreventchallengingbehavior

Strategiestokeepchildrenengagedwhentransiuoningbetweenacuviues

Specificstrategiestoteachpreschoolchildrenhowtorecognizesomeoneelse'sfeelings

Strategiestobuildstrongposiuverelauonshipswithfamilies

Classroomstrategiestodevelopposiuverelauonshipswithpreschoolchildren

Specificstrategiestoteachfriendshipskillstopreschoolchildren

Specificstrategiestoteachchildrenhowtorecognizetheirownfeelings

NonetoLizleKnowledgeable SomewhatKnowledgeable KnowledgeabletoExtremelyKnowledgeable

33

Source:Partnersiteclassroomstaffsurvey(October2015–January2016).n=101–104.

34

Partnersitesurveyparticipantswerebrokendownintothreegroups—thosewhoreceivednoneofthemoduletrainings,thosewhoreceivedsomeofthemoduletrainings,andthosewhoreceivedallofthemoduletrainings.ResultsshowthatagreaterproportionofrespondentswhoattendedmoreTeachingPyramidtrainingtendedtoreportbeingknowledgeableorextremelyknowledgeable(Table3).Althoughthis connection does not prove a causal relationship, it does lend support to the conclusion that theprofessional development contributed to increased knowledge. In a few cases there was a higherproportion of those with no modules than those with some modules who indicated they wereknowledgeable or extremely knowledgeable. One possibility for this phenomenonmay be that thosewith no training may believe they are already knowledgeable because they have not received thetrainingdetailingTeachingPyramidstrategies;thus,“theydon’tknowwhattheydon’tknow.”Thisideawas supported by informal conversationswith theWestEd Teaching Pyramid team and the coachingobservation. It is further described on pages 35 – 36 regarding perceived use of strategies beforereceivingtraining.

Table3:PartnerSiteClassroomStaffKnowledgeofStrategies,byAmountofTrainingReceived(%Knowledgeable/ExtremelyKnowledgeable)

SurveyItems:Howknowledgeableareyouabout:

TrainingReceived

TotalNo

ModulesSome

ModulesAll

ModulesClassroomstrategiestodeveloppositiverelationshipswithpreschoolchildren 70% 75% 96% 84%

PositiveDescriptiveAcknowledgement(PDA) 50% 78% 91% 78%Strategiestoarrangetheclassroomtopreventchallengingbehavior

61% 70% 95% 80%

Strategiestokeepchildrenengagedwhentransitioningbetweenactivities

74% 65% 93% 81%

Strategiestobuildstrongpositiverelationshipswithfamilies 61% 83% 95% 83%Specificstrategiestoteachfriendshipskillstopreschoolchildren 75% 88% 91% 85%Specificstrategiestoteachchildrenhowtorecognizetheirownfeelings 83% 79% 93% 86%

Specificstrategiestoteachpreschoolchildrenhowtorecognizesomeoneelse'sfeelings 67% 79% 91% 82%

Specificstrategiestoteachconflictresolutionskillstopreschoolchildren 71% 63% 82% 74%

Collectingdatatodeterminewhyachildhaspersistentlychallengingbehaviorsintheclassroom

63% 50% 76% 66%

DevelopingaTeacherSupportPlanorPositiveBehaviorSupport(PBS)plansforindividualchildren

45% 46% 65% 56%

Source:Partnersiteclassroomstaffsurvey(October2015–January2016).n=101–103.

Thepre-andpost-trainingsurveysat theentrysitealsoshowan increase inknowledgeafterthefourtrainingmodules.Table4displaysthemeanvaluesbeforeandafterthetrainings.Thescalerangedfrom0to8,where“8”was“extremelyknowledgeable.”Forallitems,theperceivedincreasewasstatistically

35

significant (MannWhitneyU,p<.05).9 Interviewdata confirm that teachersandadministratorsgainedknowledgeofspecificstrategiestheycouldusetofosterhealthysocialandemotionaldevelopment intheirclassroomsaswellastoolstopreventnegativebehaviorsfromoccurring inthefirstplace.Manyintervieweesalsoexpressedwantingevenmoreknowledgeonhowtoaddresstheneedsofthe“topofthePyramid”children.

Table4:TrainingAttendees’KnowledgeBeforeandAfterTeachingPyramidProfessionalDevelopment

SurveyItems:Onascaleof0–8,howknowledgeableareyouabout: BeforeModule1 AfterModule3bClassroomstrategiestodeveloppositiverelationshipswithpreschoolchildren 5.75 6.82

Positivedescriptiveacknowledgement(PDA) 4.18 6.88Strategiestoarrangetheclassroomtopreventchallengingbehavior 5.40 6.67Strategiestokeepchildrenengagedwhentransitioningbetweenactivities 5.67 6.76

Strategiestobuildstrongpositiverelationshipswithfamilies 6.00 6.82Specificstrategiestoteachfriendshipskillstopreschoolchildren 5.80 7.00Specificstrategiestoteachchildrenhowtorecognizetheirownfeelings

5.83 6.94

Specificstrategiestoteachpreschoolchildrenhowtorecognizesomeoneelse'sfeelings

5.69 6.94

Specificstrategiestoteachconflictresolutionskillstopreschoolchildren 5.67 6.88

Collectingdatatodeterminewhyachildhaspersistentlychallengingbehaviorsintheclassroom 5.21 6.70

DevelopingaTeacherSupportPlanorPositiveBehaviorSupport(PBS)plansforindividualchildren 4.50 6.52

Note:Reflectsmeanknowledgeratingscoresforthewholegroup.Individualsratedthemselvesonascalefrom0to8(0=notatall;2=alittleknowledgeable;4=somewhatknowledgeable;6=knowledgeable;and8=extremelyknowledgeable).Source:Pre-andpost-trainingsurveyofentrysitetrainingmoduleattendees(September2015andApril2016);n=33–42.

Training participants from practicing non-partner sites also perceivedmoderate to large increases intheirknowledgeofchildren’ssocial-emotionaldevelopment.Thesurveyaskediftheystayedthesameorincreasedintheirknowledgeofstrategiescomparedtotwoyearsago,beforetheyreceivedTeachingPyramid training. As shown in Figure 5, they perceived the largest increases in their knowledge ofpositivedescriptiveacknowledgement,orPDA(61%),specificstrategiestoteachconflictresolutionskills(52%),andspecificstrategiestoteachchildrenhowtorecognizetheirownfeelings(51%).Again,whenaskedwhattheythoughtcausedtheirincreasesinknowledge,themajority(81%)saidTeachingPyramidprofessionaldevelopment(notshown).

Interviewed teachers felt that Teaching Pyramid training further confirmedwhat they believed aboutchildren’ssocial-emotionaldevelopment. Italsocausedthemtobemoreawareand intentionalabout9Although the twosamples (pre/post)weredependent, theywerenotexactly the sameatboth timepoints.Asmallnumberofrespondentstookthepre-trainingsurveybutnotthepost-trainingsurvey,andindividualpre/postscores were not linked. The decision was made to use the MannWhitney U for independent samples, whichshowedsignificanceevenwithamoreconservativeestimateoftheteststatistic.

36

theapproachtheyweretakingwithchildren,andgavethemmorespecificstrategiestoteachsocialandemotionalskills.ImplicitinTeachingPyramidtrainingisthatteachersdoknoweffectivesocial-emotionalstrategies,but that,asModule1 teaches, “knowingsomethinganddoing itare twodifferent things.”Teaching Pyramid training hopes to impart knowledge and tools to increase the use of effectivestrategies.

Figure5:TeachingPyramidTrainingAttendees’PerceivedChangeinKnowledge

Note:Includesadministrators,teachers,teachers’assistants,andotherswhoattendedTeachingPyramidtrainingin2013–2014and2014–2015;n=238–245.Source:Practicingsitetrainingparticipantsurvey(October–November2015).

Classroom Practice TheevidencesuggeststhatTeachingPyramidprofessionaldevelopmentpositivelyinfluencesclassroompracticeaswell.Interestingly,theresultsofpre-andpost-trainingsurveysattheentrysiteindicatethatteachers and teachers’ assistantsmay have initially thought theywere engaging in Teaching Practicestrategies before they received professional development, but their assessment of their individualpractice changed after they became more knowledgeable. For example, before training, a largepercentage of respondents (79%–91%) said they usually or always engaged in the three activities—giving positive acknowledgement, intentionally planning for transitions, and referring to classroomexpectations. However, the post-training survey asked them to once again rate their use of thesestrategiesbeforeattendingthefirstmodule,andtheresultswerenoticeablydifferent.Uponreflection,

5%

2%

3%

3%

3%

3%

2%

1%

2%

2%

1%

22%

21%

18%

14%

16%

17%

16%

12%

13%

14%

8%

37%

35%

37%

39%

36%

33%

33%

38%

34%

31%

29%

36%

42%

42%

44%

46%

47%

49%

49%

51%

52%

61%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

DevelopingaTeacherSupportPlanorPosiuveBehaviorSupport(PBS)plansforindividualchildren

Collecungdatatodeterminewhyachildhaspersistentlychallengingbehaviorsintheclassroom

Strategiestokeepchildrenengagedwhentransiuoningbetweenacuviues

Specificstrategiestoteachpreschoolchildrenhowtorecognizesomeoneelse’sfeelings

Strategiestoarrangetheclassroomtopreventchallengingbehavior

Strategiestobuildstrongposiuverelauonshipswithfamilies

Specificstrategiestoteachfriendshipskillstopreschoolchildren

Classroomstrategiestodevelopposiuverelauonshipswithpreschoolchildren

Specificstrategiestoteachchildrenhowtorecognizetheirownfeelings

Specificstrategiestoteachconflictresoluuonskillstopreschoolchildren

PosiuveDescripuveAcknowledgement(PDA)

Noincrease Smallincrease Moderateincrease Largeincrease

37

amuchsmallerpercentageofrespondents(47%–59%)ratedtheirprevioususeas“usually”or“always”(notshown).

This previous lack of understanding regarding Teaching Pyramid principles was highlighted in theobservedcoachingsessionandsupportedbyinformalconversationswiththeWestEdTeachingPyramidteam. Teachers often felt they already knew and applied strategies before receiving trainings and/orcoaching.Aspartoftheprofessionaldevelopment,“ahamoments”emergedwheretheyrealizedtheirprior activities had been managing children instead of supporting healthy social-emotionaldevelopment. Thus, their new knowledge and mindset shift helped them reflect upon their use ofclassroomstrategies.

Ahighproportionofteachersandteachers’assistantsatpartnersitesusedTeachingPyramidstrategies.Asshown inFigure6,morethaneightoutof10teachersandteachers’assistantssaidtheyusuallyoralways give specific positive acknowledgement when children demonstrate appropriate behaviors(93%), intentionally plan for transitions (83%), and refer to classroom expectations when talking tochildren about their behavior (88%). Indeed, evaluators were able to observe PDA at all sites andevidenceofteacherstalkingaboutclassroomexpectations,aswellasotherTeachingPyramidstrategiessuch as working with an emotions chart and managing transitions efficiently and with appropriatebehaviors.

Figure6:FrequencyofPartnerSiteUseofStrategiestoFosterSocial-EmotionalDevelopment(%Usually/Always)

Note:Reflectsteachersandteachers’assistantsatselectedpartnersites;n=93–94.Source:Partnersiteclassroomstaffsurvey(October2015–January2016).

93%83% 88%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Giveposiuveacknowledgementtochildrenthatspecificallydescribestheposiuvebehaviorthatyou

observewhentheydemonstrateappropriatebehaviors

Intenuonallyplanfortransiuonsbetweenacuviuestokeepthechildrenengagedinappropriate

behaviors

Refertoclassroomexpectauonswhentalkingtoachildabouthis‎/

herbehavior

%Respo

nding"U

sually"o

r"Always"

38

Interviews and observations generally support the picture painted by partner site survey resultsregarding classroom practice. Although PDA was widely observed, a common theme across allinterviews was the challenge surrounding the new language required to provide this feedback. Thisstrategyrequiresthattheteacherandadministratordropphrases like,“goodjob,”and“I likethewayyou _____,” in favor of simply acknowledging what the child is doing, thereby promoting intrinsicmotivation.Oneteacherexplained:

Thehardest thingwas the language—gettingaway from just the “good job.”Doing the PDAstooktime.SometimesIcatchmyself.Ipractice—Iconstantlyhavetohavethatinthebackofmymind,butIseethebenefitof it.Evenwithstaff—thatisthehardesttodo,toremindmyselftogivePDAstothem.ThatissomethingIamstillworkingon.

ManysitesadoptedPDAsite-wide,encouragingstafftopracticeoneachother.Severalparticipantssaidthishadcreatedamorepositiveclimateforbothadultsandchildren,andmostteachersnotedthattheysawpositiveimpactsonchildren’sbehavior.

Anaddedchallengenotedby some teacherswas that this language shiftwasparticularlydifficult if ateacher’sfirstlanguagewasnotEnglish:

The PDAs [were the most difficult] because of my language. I have to think before I speak.EnglishismysecondlanguageandIwantedtosayit inafluidmanner—notlikearobot,[but]naturally.Ihadtopractice.Thatwasmybiggestchallengebecauseofmysecondlanguage.

Thesurveyalsoaskedhowmanytimesduringatypicalweekteachersandteachers’assistantsengagedinspecificstrategiessuchasimplementingactivitiesinsmallgroups,reviewingclassroomexpectationswiththewholeclass,orconductingactivitiestopracticefriendshipskills,expressfeelings,orrecognizesomeone’selsefeelings.Sincethisquestionaskedaboutfrequency,theanalysisincludedonlyteachersandassistantswhotaughtorassistedwithoneclasseachdaysothatresponseswouldbecomparablebetweenstaff.Withtheexceptionofreviewingclassroomexpectations,morethanhalfoftheclassroomstaffsaidthat,onaverage,theyimplementedthesestrategiesatleastonceaday(Table5).RespondentswhohadreceivedmoreTeachingPyramidtrainingreportedahigheruseofstrategies(Table6).

Table5:FrequencyofPartnerSiteClassroomStaffUseofTeachingPyramidStrategies

SurveyItems:Duringatypicalweek,howmanytimesdoyou…

Frequency

TotalNoneLessthanonceaday Onceaday

Morethanonceaday

Implementactivitiesinsmallgroupsof3to5children 5% 28% 34% 34% 100%Reviewclassroomexpectationswiththewholeclass 3% 53% 17% 27% 100%Conductactivitiesthathelpchildrenpracticefriendshipskills 0% 34% 25% 42% 100%Conductactivitiesthathelpchildrenpracticeexpressingtheirownfeelings 2% 43% 15% 40% 100%Conductactivitiesthathelpchildrenpracticerecognizingsomeoneelse'sfeelings 2% 45% 19% 34% 100%

Note:Reflectsteachersandteachers’assistantsatselectedpartnersiteswhotaughtoneclassaday;surveyresponseswererecodedfromnumberranges;n=64–65.Source:Partnersiteclassroomstaffsurvey(October2015–January2016).

39

Table6:PartnerSiteClassroomStaffUseofTeachingPyramidStrategiestoFosterSocial-EmotionalDevelopment,byAmountofTraining(%SayingOncePerDayorMore)

SurveyItems:Duringatypicalweek,howmanytimesdoyou…

#ofModulesReceived

TotalNo

ModulesSome

ModulesAll

Modules

Implementactivitiesinsmallgroupsof3to5children 44% 65% 83% 68%

Reviewclassroomexpectationswiththewholeclass 24% 47% 53% 44%Conductactivitiesthathelpchildrenpracticefriendshipskills 61% 71% 67% 66%Conductactivitiesthathelpchildrenpracticeexpressingtheirownfeelings 33% 47% 73% 55%Conductactivitiesthathelpchildrenpracticerecognizingsomeoneelse'sfeelings 42% 47% 64% 53%

Note:Reflectsteachersandteachers’assistantsatselectedpartnersiteswhotaughtoneclassaday;n=64–65.Source:Partnersiteclassroomstaffsurvey(October2015–January2016).

MostpartnersiteclassroomstaffarrangedtheirclassroomsasrecommendedbytheTeachingPyramid.Observations and survey results support this finding. For example, approximately eightoutof 10hadposted a daily schedulewith pictures (82%) or program-wide expectations (78%); just asmany hadarrangedtheclassroomtopreventchallengingbehaviors(79%).TheevaluationteamfoundevidenceofTeachingPyramidexpectationsinallobservedclassrooms.Table7showsthebreakdownbytheamountoftrainingreceived.Thesamplesizewastoosmalltoconductachi-squaretestofindependence.Thus,the percentages reflect sample differences and may or may not be characteristic of all partner sitedifferences.

Table7:PartnerSiteClassroomStaffUseofTeachingPyramidStrategiesforClassroomSet-Up,byAmountofTraining(%SayingYes)

SurveyItems

#ofModulesReceived

TotalNo

ModulesSome

ModulesAll

ModulesDoesyourclassroomhaveadailyschedulehungonthewallwithpictureswherechildrencanseeit? 48% 90% 96% 82%Doesyourclassroomhaveprogram-wideexpectationshungonthewallwithpictureswherechildrencanseethem? 61% 84% 83% 78%Isyourclassroomenvironmentarrangedtopreventchallengingbehavior? 65% 70% 90% 79%

Note:Reflectsteachersandteachers’assistantsatselectedpartnersites;n=88–91.Source:Partnersiteclassroomstaffsurvey(October2015–January2016).

The results from the other surveys also support the idea that Teaching Pyramid professionaldevelopmentcontributestopositivechanges inteacherpractice.Thepost-trainingsurveyattheentrysite, for example, shows a significant perceived increase in frequency of giving positive

40

acknowledgement, intentionally planning for transitions, and referring to classroom expectations. Asdiscussedingreaterdetailelsewhere,observationsalsoconfirmsomeuseofPDAandevidenceofclearclassroom expectations. Since earlier analysis demonstrated that respondents tend to overestimatetheiruseofstrategiespriortotraining,thisanalysiscomparesaretrospectivequestionandpost-trainingquestionthatwerebothgivenafterthetraining(WilcoxonSigned-Ranktest,p<.05).Figure7illustratestheperceivedchange.

Figure7:TeachingPyramidTrainingAttendees’PerceivedChangeinStrategyUse(%Usually/Always)

Note:Retrospectivepre-questionwasgivenonpost-trainingsurvey;Includesteachersandteachers’assistants.n=34.Source:Post-trainingsurveyofentrysitetrainingmoduleattendees(April2016).

Participantsatnon-partnerpracticingsiteswhoattendedTeachingPyramidprofessionaldevelopmentalsoshowedincreasesinsocial-emotionaldevelopmentclassroompracticesaftertraining.AsshowninFigure8,theyreportedthebiggestincreasesinstrategiesrelatedtoPDA(88%reportedamoderateorlarge increase), specific strategies to teach friendship skills (85%), and specific strategies to teachchildren how to recognize their own feelings (81%). When asked what they thought caused theirincreaseinstrategyuse,70%attributedittoTeachingPyramidprofessionaldevelopment,and19%saidprofessionaldevelopmentingeneral(notshown).

53%59%

47%

94% 91% 88%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Giveposiuveacknowledgementtochildrenthatspecificallydescribestheposiuvebehaviorthatyou

observewhentheydemonstrateappropriatebehaviors

Intenuonallyplanfortransiuonsbetweenacuviuestokeepthechildrenengagedinappropriate

behaviors

Refertoclassroomexpectauonswhentalkingtoachildabouthis‎/

herbehavior

%Respo

nding"U

sually"o

r"Always"

RetrospecuvePre Post

41

Figure8:Non-PartnerSiteClassroomStaff’sPerceivedChangeinStrategyUseFollowingTeachingPyramidTraining

Note:Includesteachersandteachers’assistantswhoattendedaTeachingPyramidtrainingin2013–2014and2014–2015,andwhohadworkedinearlychildhoodeducationformorethan2years;n=129–133.Source:Practicingsitetrainingparticipantsurvey(October–November2015).

Another important aspectof theTeachingPyramid strategieshas todowith thebeliefs thatunderliethem.Teachersareencouragedtoframedecisionsandclassroompracticestosupportthenotionsthatsocial and emotional skills can and should be taught, and that behavior is a formof communication.Ratherthanadheringtoascriptorcurriculum,implementationatfidelityallowsforsomeflexibilityandadjustment. The emphasis is on understanding the Teaching Pyramid reasoning and embracing thismindset,usingitasaguideforclassroomdecisions. InkeepingwiththeTeachingPyramidphilosophy,the evaluation team saw this as a strength of Teaching Pyramid implementation at partner sites, asunderlying beliefswere being operationalized in context-specificways that are responsive to the siteanditschildren.

Shifts in Organizational Attitudes As mentioned previously, indicators of healthy organizational cultures were already present in themajority of selected partner sites. Namely, these sites have historically enjoyed low turnover, highmorale, and strong support for professional development at the leadership and administrative levels.That said, interviewsoffered insight intoadditionalholisticbenefitsofTeachingPyramidparticipationforpartnersites.

Onesuchthemeinvolvedteachers’reflectionsthattheirimplementationofTeachingPyramidpracticesincreasedtheirsenseofprofessionalself-efficacy,resultinginmorefulfillingandsatisfyinginteractionswithchildren.Beyondfeelingpreparedandconfidenttomeettheneedsofthechildren,staffmembers

2%

4%

3%

2%

3%

2%

9%

10%

18%

12%

17%

17%

18%

17%

29%

40%

43%

32%

36%

37%

36%

59%

38%

42%

49%

45%

44%

38%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Givingposiuveacknowledgementtochildrenwhentheydemonstrateappropriatebehaviors

Plannedtransiuonsbetweenacuviuesthatkeepchildrenengagedinappropriatebehaviors

Specificstrategiestoteachfriendshipskillstopreschoolchildren

Specificstrategiestoteachchildrenhowtorecognizetheirownfeelings

Specificstrategiestoteachpreschoolchildrentorecognizesomeoneelse’sfeelings

Specificstrategiestoteachconflictresoluuonskillstopreschoolchildren

Acuviuesinsmallgroupsof3to5children

Noincrease Smallincrease Moderateincrease Largeincrease

42

linkedthesefeelingstotheirownjobsatisfaction.Teachersnotedthattheyfeltmore“calm,”“relaxed,”and“incontrol.”Oneteachersummeditupbysaying,“Istartedusingmoreofthosewordsand…Ifeelbetter. I feel like I knowmore.” Inmany cases, interviewparticipants reported that the training onlyaffirmed pre-existing beliefs about early childhood education. However, being given tools to moreactively reflecton themeaningof children’sbehavior, respondconstructively,andwitnesschanges inchildren’s behavior offered tangible evidence of their own ability to positively impact the children intheircare.

As alreadynoted, languagewasoneof themost evident practical shifts that occurred across partnersites,andthisshiftextendednotonlytochildrenbuttoadultsaswell.Whenaskedabouttheimpactofthe Teaching Pyramid training at a staff level, interviewees singled out greater appreciation for theimportanceofthefirstlevelofthepyramid(NurturingandResponsiveRelationships)asafoundationalprinciple for working with children and adult peers. Examples of adults giving each other “positivedeposits” and PDA enhanced interpersonal relationships, feelings of respect, and recognition. Manyteachers reported feeling more appreciated and experiencing increased enjoyment at work. Forexample,oneteacherexplained:

Ithasmadeitareallymorepositiveexperience.Beforethereweredayswhereyoujustcouldn'tdo it. Now it is much more positive. The whole culture of the school has just done a real180.Therewasa lotofblaming, ‘Whyareyoudoing it thisway?!’andself-reflectionwasnotabletohappenbefore.-acomplete180,itisamazing!

Given the importance of collaboration and team teaching in these contexts, stronger collegialrelationshipsbodewellforthesustainabilityoftheTeachingPyramidframeworkandsitemorale.

To what extent has Teaching Pyramid professional development prepared staff to more effectively support chi ldren, including those with challenging behaviors?

Asshownintheprevioussection,TeachingPyramidtrainingcontributedtoknowledgeanduseofsocial-emotionaldevelopmentstrategiestosupportchildren.Datacollectedfromthesurvey,interviews,andobservations provide evidence that not only are partner site staff learning and engaging in TeachingPyramid activities, but they also feel more prepared to support all of their preschool students. Oneteacherdescribedthisshift:

It has made me feel more confident. I have more tools. When parents come to me and askquestions,likeabouthowIhandleaconflictorwhat'smyapproach,IfeellikeIhavemoreverbaltools.Thestrategiesaremoreclearandorganizedinmymind.

The Teaching Pyramid is a tiered framework that aims to promote appropriate social-emotionaldevelopment andprevent challengingbehaviors before intensive intervention is required. Thus,mostteachers focused on how the training helped them support children in the prevention of challengingbehaviors. One Teaching Pyramid strategy for preventing challenging behaviors is to start with theenvironment.Specifically,howtheroomissetupandwhatispostedonthewallscanpreventbehaviorchallengesbeforetheystart.Themajorityofteachersandteachers’assistants(79%)reportedthattheyarrangedtheirclassroomswiththis inmind.Somenotedthattheymadechangesintheirroomset-up

43

basedonobservingtheir“hotspots”(whereconflictstendtohappenregularly)and“coolspots”(wherethings regularly go well). Some classrooms had clearly defined areas where a child could go “to getaway,”suchasasmallmeshtent.Allclassroomshadclearlydefinedcentersandareasforsmallgroupstowork.

Inadditiontochangingthephysicalspace,TeachingPyramidtrainingsuggeststhatteachershaveclearvisual expectations postedwith pictures (ideally photos of the students themselves) to help childrenremember classroom expectations. Most classrooms had visual schedules to give students clearexpectations of where to be and when, and of what is coming next; all classrooms had postedexpectations,alsowithvisuals.

Despiteeffortstopreventthem,challengingbehaviorscanariseintheclassroom.Asshownin

Table8,thepartnersitesurveyaskedaboutconfidenceinthreeareas:identifyingthereasonachildhaschallenging behavior, matching an effective intervention with a child’s challenging behavior, anddevelopingasuccessfulpositivebehaviorsupportplan.Theevaluationdatasuggestthatapproximatelyhalfof theclassroomstaff feltconfidentorextremelyconfident that theycouldsupportchildrenwhoexhibitedchallengingbehaviors.A largeproportionofrespondentsalso feltsomewhatconfident inallthreeareas(38%,42%,and30%,respectively).

Table8:PartnerSiteClassroomStaffConfidenceinSupportingChildrenwithChallengingBehaviors

SurveyItems:Currentlyhowconfidentareyouthatyoucould…

LevelofConfidence

TotalNotatallconfident

Alittleconfident

Somewhatconfident Confident

Extremelyconfident

Identifythereasonwhyachildhasapersistentlychallengingbehavior 3% 9% 38% 48% 2% 100%Matchaneffectiveinterventionwithachild'spersistentlychallengingbehavior 2% 11% 42% 39% 5% 100%Developapositivebehaviorsupportplanthatresultsinimprovedbehavior 2% 21% 30% 42% 5% 100%

Source:Partnersiteclassroomstaffsurvey(October2015–January2016).n=98–100.

Figure9furtherbreaksdownconfidencelevelbytheamountoftrainingtheclassroomstaffreceived.Itillustratesthatahigherproportionofrespondentswhoreceivedallmoduleswereconfident.Whilethisdoes not signify a causal relationship, it does support the idea that professional developmentcontributestotheselectedpartnersitestafffeelingmorepreparedtodealwithchallengingbehaviors,especially since the lastmodule in the training series focuseson supporting childrenwith challengingbehaviors.

44

Figure9:PartnerSiteClassroomStaffConfidenceinStrategiestoSupportChildrenwithChallengingBehaviors,byAmountofTrainingReceived(%Confident/ExtremelyConfident)

Source:Partnersiteclassroomstaffsurvey(October2015–January2016).n=98–100.

Again,theresultsoftheentrysitepre-andpost-trainingsurveyssuggestthatprofessionaldevelopmentcontributes to increased confidence in dealing with challenging behaviors. Respondents significantlyincreasedtheirconfidencelevelsoverthetrainingperiod(MannWhitneyU,p<.05).10Figure10,onthenextpage,illustratesthisincrease.

10Althoughthetwosamples (pre/post)weredependent, theywerenotexactlythesameatbothtimepoints.Asmallnumberofrespondentstookthepre-trainingsurveybutnotthepost-trainingsurvey,andindividualpre/postscores were not linked. The decision was made to use the MannWhitney U for independent samples, whichshowedsignificanceevenwithamoreconservativeestimateoftheteststatistic.

32%38%

29%

39%

26%

43%

63%56% 57%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Idenufythereasonwhyachildhasapersistentlychallenging

behavior

Matchaneffecuveintervenuonwithachild'spersistentlychallengingbehavior

Developaposiuvebehaviorsupportplanthatresultsin

improvedbehavior

%ofStaffSaying"C

onfid

ent"or"Extrem

elyCo

nfide

nt"

NoModules SomeModules AllModules

45

Figure10:TrainingAttendees’ChangeinConfidenceFollowingTeachingPyramidTraining(%Confident/ExtremelyConfident)

Source:Pre-andpost-surveyofentrysitetrainingmoduleattendees(September2015andApril2016).n=31–41.

This relationship between training and confidence is further supported by the survey responses oftraining participants at non-partner practicing sites. Approximately 75% of these training attendeesstated thatwhen theywere identifying the reasons for a persistently challenging behavior,matchingeffective interventions, and developing a positive behavior support plan, they hadmoderate to largeincreases in their confidence levels compared to two years ago (not shown). Interview data andobservations underscore this finding. Teachers were able to cite specific strategies they used withchildrenwhohadchallengingbehaviors.

Teachersandadministratorsalsodescribedachange in their thinkingaboutchallengingbehavior.Forexample,manysaidtheyweremorereflectiveandreframedbehaviorasaformofcommunication.Oneinterviewee asked, “How can we support the children who are really struggling? [We] need to slowdownandlookateverythingbeforejumpingtoconclusions.Youhavetobethatdetective.”

Thischangeinmindsetwasalsoobservedinthecoachingsession.TheparticipatingteachersseemedtohaveanunderstandingoftheTeachingPyramidpracticesingeneralbutnotofhowtoputtheconceptsintopractice.Bywalkingthroughanactualtapedexperience,thecoachpinpointedwheretheyenactedtheconceptsandmodeledwhattheycoulddonexttime.Thisledtoan“ahamoment”forallofthem,when they realized they had previously been “managing children,” but they were now “developingchildren.”

49%

41% 41%

68%61%

58%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Idenufythereasonwhyachildhasapersistentlychallengingbehavior

Matchaneffecuveintervenuonwithachild'spersistentlychallengingbehavior

Developaposiuvebehaviorsupportplanthatresultsin

improvedbehavior

%Saying"Con

fiden

t"or"Extrem

elyCo

nfide

nt"

Pre Post

47

CONCLUSIONSANDRECOMMENDATIONSThegoalofthisevaluationwastoassessTeachingPyramidprofessionaldevelopmentanditsimpactonearlyeducationpractitioners’ knowledge,practice, andattitudes. The findings reveal that theWestEdTeachingPyramidteamprovideshighqualitytrainingsandcoaching.TheTeachingPyramidprofessionaldevelopment positively contributes to increased understanding and use of strategies to promotehealthysocial-emotionaldevelopment,alongwithgivingteachersmoreconfidencewhendealingwithchallenging behaviors. Furthermore, site staff found that applying the Teaching Pyramid resulted inmorefulfillingandsatisfyinginteractionswithbothchildrenandco-workers.

R e c o m m e n d a t i o n s The findings from this study are overwhelmingly positive, confirming the work and efforts of theTeaching Pyramid staff and local partners. The recommendations below are intended to furtherstrengthentheprogram.

Recommendation 1: Follow-up for Sustainability Eventhoughsitesareencouragedtoinvestinfollow-uptrainingsandcoachingtoadvanceandrefreshtheir practice, these trainings are contingent upon site budgets and many site staff are unable toparticipate.Asdemonstratedbythesurveyresults(Table2),someclassroomstaffhaveneverreceivedformaltrainingorreceivedanycoaching.Follow-uptraining isanimportantpartof implementationatfidelity since staff may have attended Teaching Pyramid training years ago or there may be newteachers.Administratorsandteachers identifiedcoachingasanessentialpieceof implementationandsustained growth. CDE should explore additional ways to encourage and financially support sites inTeachingPyramidtrainingsandcoachingaftertheirinitialfirstyear.

Additionally, the leadership team, combinedwith strong administrative presence and support for theTeaching Pyramid at the various sites, is essential for continuing site-wide practice. The WestEdTeachingPyramidteamshouldcontinuetoprovideguidanceandsupport to thesite leadership team,evenafter the first year. This continued guidance for partner sites should specifically addresshow toeffectivelyusedatatoimproveTeachingPyramidimplementation.

Recommendation 2: Data Collection, Monitoring, and Sharing CDEandtheWestEdTeachingPyramidteamshouldexplorehowtopromotemoreconsistentcollection,monitoring,andsharingofdatabypartnersites.NotalloftheselectedpartnersiteshadBenchmarksofQuality (BOQ)dataavailable.Sincethesedata fosterquality implementationof theTeachingPyramid,theTeachingPyramidteamcouldworkwithsitestoensuremoretimelycollectionandsharingofthesedata. Likewise, theBOQdata showed thatmonitoring implementationandoutcomedatawasnot “inplace”foranyofthesites.TheTeachingPyramidteamshouldexplorehowtosupportthepartnersites,notonlyinthecollectionandsharing,butalsointheuseofthedata.

There is also a need for more timely and widespread administration of the Teaching PyramidObservationTool (TPOT).As this tool isadministeredby theWestEdTeachingPyramidTeam,andnotthesites,theyshouldconsidertrainingmorepersonneltoadministerthetoolmorefrequently.

48

Recommendation 3: Manageable and Accessible Training and Coaching The trainings are highly regarded for their content and delivery, yet they are intense and requireengagementatahighlevel.TheWestEdTeachingPyramidteamshouldconsidertheoptionofofferingshorter,more frequent sessions. In addition, someparticipants’ first language is not English and theyhave varying degrees of proficiency. As such, materials and training in Spanish could provide evengreater access for Spanish speaking teachers. Further,much of the Teaching Pyramid strategy use islanguage dependent. Many materials have been translated already for use with families, but theTeachingPyramidshouldcontinuetoexpandthispracticeandmakeallmaterialsavailabletoteachersandfamiliesinmultiplelanguages.Similarly,WestEdTeachingPyramidteamshouldcontinueitseffortstorecruitandtrainbilingualcoaches.

Recommendation 4: Supporting Children with Challenging Behaviors There is a need for more professional development around challenging behaviors. While the initialtraining and coaching supports these needs, there is the opportunity for more classroom staff tobecomeconfident in their ability to support these children. Some teachersalsoexpressedadesire tolearnmoreaboutaddressing“topofthePyramid”children.

Recommendation 5: Wiggio Support AccesstoWiggioismadeavailabletopartnersites;however,aspartoffollow-upsupport,theTeachingPyramidteamshouldconsideraddingamini-unitonusingandnavigatingWiggiotogetthemostoutofthisextensiveresource.

Recommendation 6: Expanded Funding and Incentives for Data Collection A common theme throughout the findings and recommendations is the need for increased funds forfurthertraining,coaching,andteacherreleasetime.TheTeachingPyramidisextremelywell-received,andadditionalfundingwouldhelpsustaintheprogramandexpanditsreach.WiththephasingouttheRacetotheTopEarlyLearningChallengegrants,CDEexploredifferentfundingmodels.Additionally,CDEcould consider how to set aside funds to incentive consistent data collection and use by the partnersites. Thus, CDE should consider how to secure additional funding for training, coaching, and datasupport,whilerecognizingconstraintswithintheECEfield.

C o n c l u s i o n This report evaluates the Teaching Pyramid professional development, including its impact on earlyeducationpractitioners’knowledge,practice,andattitudes. Inconclusion, theTeachingPyramid teamprovides high quality trainings and coaching. The Teaching Pyramid programpositively contributes totheunderstandinganduseofstrategiesthatpromotehealthysocial-emotionaldevelopmentinchildren.

49

REFERENCES

Creswell,J.W.,&PlanoClark,V.L.(2011).Designingandconductingmixedmethodsresearch.ThousandOaks,CA:SAGEPublicationsInc.

Guskey,T.(2000).Evaluatingprofessionaldevelopment.ThousandOaks,CA:Corwin.

51

APPENDICESAppendixA:PartnerSurveyInstrument

AppendixB:PartnerSurveyTables

AppendixC:OtherSurveyInstruments

AppendixD:InterviewProtocol

AppendixE:BenchmarksofQualityTool