Promoting the Social-Emotional and Literacy … talk pt 1...Promoting the Social-Emotional and...
34
J. Lawrence Aber Professor of Applied Psychology and Public Policy New York University March 9, 2009 Promoting the Social-Emotional and Literacy Development of Low-Income Children: Results from a School Randomized Trial IES-PIRT Fellowship Proseminar Series New York University
Promoting the Social-Emotional and Literacy … talk pt 1...Promoting the Social-Emotional and Literacy ... Reading, Writing, Respect and ... Co-occurrence of social-emotional and
Morningside Center for Teaching Social Responsibility
Funders:Institute for Education Sciences, DOE
National Center for Injury Prevention and Control, CDCWilliam T. Grant Foundation
NIMH
Acknowledgements
Contents
4Rs: Background, Program, and Study DesignSample and MeasurementYear 1 Program Impacts: Classroom-LevelYear 1-2 Program Impacts: Child-LevelNext Steps and Future Directions
4Rs Program Implementation Studies, Years 1 and 2
Reading, Writing, Respect and Resolution (4Rs):Background, Program and Study Design
Background (1)
Co-occurrence of social-emotional and behavioral problems with low academic achievement.Theoretical and initial empirical links between self-regulation and reading/math.Emphases on standardized testing and instructional improvement have crowded out attention to social-emotional-character development (among other things).
Background (2)
Early efforts at whole school strategies to prevent behavior problems, violence, and substance use plagued by intervention design and implementation fidelity problems.Early research on whole school strategies plagued by low power, and inappropriate statistical analyses.Need to rigorously test promising but unproven approaches to SEL/SACD.
Background (3)
Birth of “Social and Character Development” Research Network.7 different interventions in 7 different sites.7 Local Evaluations and 1 National Evaluation (Mathematica Policy Research).
Background (4)
Prior Quasi-Experimental Study of earlier version of 4Rs: The Resolving Conflict Creatively Program.Results
Aber et al., 1996Aber et al, 1998Aber, Brown & Jones, 2003
Evidence suggestive that intervention is promising.
Background (5)
The 4Rs ProgramUniversal, school-based intervention in literacy development, conflict resolution, and intergroup understanding.3 Primary components:
7-unit literacy-based curriculum in conflict resolution and social-emotional learning.
Each unit organized around grade-appropriate book, includes 2 literacy activities, and 3-5 SEL lessons (21-35 total lessons).
Total possible activities per unit = 5-7Total possible activities per year = 35-49
Training and ongoing coaching of teachers in the delivery of the 4Rs curriculum.
Family Connections1 parent-child “homework” per unit
Presenter
Presentation Notes
The 4Rs is a universal, school-based intervention in literacy development, conflict resolution, and intergroup understanding that integrates conflict resolution into the language arts curriculum. The 4Rs uses high quality children’s literature as a springboard for helping students gain skills and understanding in the areas of community building, handling anger, listening, assertiveness, cooperation, negotiation, mediation, celebrating differences and countering bias. By highlighting universal themes of conflict, feelings, relationships, and community, the 4Rs curriculum adds meaning and depth to literacy instruction and is intended to enrich conflict resolution as well as social emotional learning. The basic idea behind the intervention is to target the social cognitive and interpersonal behavioral processes thought to lead to aggressive and violence behavior. At its most general level, it is designed to help children think, feel, and act differently in situations of interpersonal conflict. This notion stems from a developmental psychopathology perspective that emphasizes the identification of processes that mediate the influence of risk factors on maladaptation as targets of preventive interventions. The program has three primary components developed for use with children in grades K through 8: A comprehensive 7-unit literacy-based curriculum in conflict resolution and social-emotional learning. Each unit begins with a comprehensive book reading and discussion, ensuring students understand the primary themes of the story and allowing them to connect the themes to their own lives. This is followed by three to five conflict resolution skill lessons in which children are able to practice specific skills in the context of a larger discussion of the book. The lessons are designed to engage the children in the learning and practice of conflict resolution skills and community building relevant to each unit. Training and ongoing coaching of teachers in the delivery of the 4Rs curriculum. Intensive professional development for teachers consists of 25 hours of introductory training, and a minimum of 12 yearly classroom coaching sessions which encompass class lesson modeling and workshops led by the staff developer, co-planning and teaching of lessons by the teacher and staff developer, and finally, lessons observations and feedback. Family-Child Connections. Which is essentially a parent guide that includes a memo to the teacher introducing activities they can recommend for the parent and child; a model letter from the teacher to parents or guardians describing the 4Rs Program; and a homework assignment for each unit to be completed jointly by parent and child at home. Each take home unit includes an overview of the theme, a summary of the book used in the unit, a related activity for parents and children to do together, and suggestions for further activities that relate to the book and the theme. In addition, workshops that mirror the the themes of focus in the classrooms are offered to parents.
Multi-Level Theory of Change - 1At the level of individuals:
Children’s exposure to the curriculum (participating in read aloud/book talk, working through applied learning lessons) will directly influence their SEL and academic skills.
Changing children’s SEL skills as a result of 4Rs will indirectly influence their academic outcomes.
Teacher’s exposure to the training and ongoing coaching will directly influence (a) their social-emotional skills, beliefs in the value of SEL, levels of exhaustion and burnout, (b) their classroom management, infusion of core 4Rs messages throughout their day and in interaction with children, and (c) the quality of their relationships with the children in their classrooms.
Heuristic Model: 4Rs Child-Level Study
4RsExperimental (classroom
and parent) vs.Control
TeacherDevelopment
Extended Opportunities &
Supports
Social-EmotionalSkills &
Behaviors
Literacy Skills &Academic
Achievement
Presenter
Presentation Notes
As one of the seven sites involved in this national multi-site evaluation of social and character education, the complementary study of the 4Rs Program is unique. First, our study rigorously tests a program and developmental model that is based on integrating a social and emotional learning intervention within a literacy development curriculum. The program and developmental models incorporate key elements from theories of (a) children’s social-information processing (e.g., hostile attributional processing; Dodge, et al., 2002); (b) the stages of children’s development of social understanding (e.g., Selman, 2003); and (c) children’s literacy development in the elementary school years (e.g., Snow, Burns & Griffin, 1998). Second, in addition to testing intervention effects on trajectories of children’s social, emotional, and academic development, our study also examines direct intervention effects on changes in teacher’s own social and emotional skills, professional development, and provision of extended opportunities to engage children in social and emotional learning. Third, we are also focusing on whether and how (a) changes in teacher development and extended opportunities for social-emotional learning mediate the effect of 4Rs on children’s social-emotional and academic development, and (b) the degree to changes in children’s social-emotional development and changes in children’ academic achievement influence each other in a mediated uni-directional, or in a bi-directional manner. Note importance of policy-experiments that examine the question of whether or not changes in children’s social-emotional functioning is important for learning and achievement outcomes.
Multi-Level Theory of Change - 2
At the level of the setting:By changing (a) teacher beliefs, practices, and relationships, and (b) classroom norms for behavior, 4Rs indirectly influences the overall culture and climate of the classroom.By operating as a whole school intervention, the 4Rs indirectly influences the overall culture and climate of the school.
Across levels:By changing the classroom system and the overall school culture and climate, 4Rs indirectly influences children’s SEL and academic outcomes.
ChildDevelopmentalOutcomes: SEL &Academic Achievement
Overall Study Design3-year, 6 wave longitudinal experimental design18 NYC elementary schools matched and randomly assigned to 4Rs and control group (9 assigned to each group)Intervention is implemented school-wide, grades K-6 for 3 yearsAll 3rd grade children in each school followed over three years through 5th gradeSchools represent demographic character of NYC public elementary schools
Matching and Random AssignmentProcess:1. 41 schools identified2. 24 schools agreed to random assignment3. 24 schools were pairwise matched on 20 key characteristics 12
pairs4. 1 school in each pair was assigned to intervention, 1 to control5. 9 best matching pairs were kept, 3 pairs back-ups20 School Characteristics Include:
Size (total N, and numbers of target classrooms)Race/ethnic compositionSchool lunch receiptAttendanceReading achievement % within year mobility/two-year stabilityExpendituresOrganizational Readiness
Data Collection Overview
Classroom-based administration of surveys to children (2 50-minute classroom periods)Teacher reports on children and on teachersParent reports on childrenSchool recordsClassroom ObservationsStaff/Teacher rating of school climatePrincipal Interview4Rs Implementation
3rd Grade(2004-2005)
Fall Spring
4th Grade(2005-2006)
Fall Spring
5th Grade(2006-2007)
Fall Spring
Design Challenges: Estimating Power in Cluster-Randomized TrialsCRT means random assignment of clusters, or groups of individuals (e.g., schools, classrooms) to treatment and control conditions.Power is calculated at the level of random-assignment (schools).Need adequate statistical power to detect expected Treatment/Control differences.Power to detect differences between groups depends on:
cluster size (# kids) number of clusters (# schools) anticipated intra-class correlations (variation in the outcomes between clusters)expected effect size (magnitude of the difference between groups)blocking/matching
Additional factors that make a difference:Baseline covariates (help)Longitudinal data (helps)Treatment effect variability (heterogeneity of the treatment effect across blocks, can hurt…a lot)
Key Design IssuesThe biggest challenge is including enough schools to detect expected effects.
Need 18 (actually more) schoolsWith 10 schools have power of ~.5 to detect small effects (.3); and ~.75 to detect effect sizes of .4.With 18 schools have power of >.8 to detect effect sizes of .3 & .4 Dramatic influence of:
Expected intraclass correlations (differ by outcome)Pairwise matching
Consider tradeoffs carefully:The visibility of a bad draw, likely with small # of schools.Value and implications of matching
How many units per block?
Evaluating the results with 18 schoolsUse a 1-tailed test?Alpha level?Confidence intervalsSize of the effect
Strategic use of covariates (collect baseline data)Increasing the experimental contrast (techniques to account for low-dosage AND/OR take-up in the control schools)
Data Analytic Challenges
Nesting:time nested in children, children nested in classrooms, classrooms nested in schools How to track the longitudinal impact on children as they change classrooms across 3 years?
Mobility:How to account for within-and between-year year child and teacher mobility in such a longitudinal design?
Presenter
Presentation Notes
Question 1a-1d. In order to assess the direct effects of the 4Rs intervention on trajectories of both the teacher mediators and the child outcomes a series of increasingly complex analytic steps will be undertaken. First, repeated measures multivariate analyses of covariance (MANCOVA) will be used to evaluate the impact of the 4Rs on the constructs representing the four primary outcome domains outlined in Figure 1 (teacher development, extended opportunities and support for social-emotional learning, social-emotional skills and behaviors and academic achievement). 4Rs delivery will be operationalized first as intervention vs. control and will be included as the between subjects factor. These initial analyses will be conducted at the end of Year 2, at the end of Year 3, and for youth outcomes only, at the end of Year 4 using all five data collection points. Second, Hierarchical Linear Modeling (HLM) techniques using HLM5 (Bryk & Raudenbush, 1992; Willett & Sayer, 1994) will be employed to assess the impact of classroom context (i.e., treatment level) on child-level change. Because the intervention is delivered at the classroom level and specific hypotheses are made regarding the impact of the treatment conditions on both children and teachers, a set of multilevel classroom analyses will be conducted to estimate both within and between classroom effects of the treatment conditions. HLM is an alternative to more typical multivariate repeated-measures analysis of variance methods (described above as an initial analytic step), and can provide a more optimal test of patterns of individual development and factors related to that development (Willett, Singer & Martin, 1998). HLM allows for the simultaneous estimation of variance associated with individual (within-subjects) and population (between-subjects) growth curves based on the specification of fixed- and random-effect variables in the model (Bryk & Raudenbush, 1992; Burchinal, Bailey, & Snyder, 1994). HLM also allows for missing data at the time varying (within-subjects) level as estimation of individual growth curves are assumed to vary systematically around population (between-subjects) growth curves. As such, there may be one or more missing data points in any longitudinal design. In the proposed study the assessment of children who vary to some degree in age at a minimum of 4 different time points will enable the estimation of growth over time as well as across ages. With each subject’s growth represented at a minimum of 4 time points, it will be possible to (1) identify and test the appropriate average functional form of the developmental trajectories of the key outcomes of interest (e.g., with intercept, slope, and quadratic parameters which are defined in more detail below), (2) describe the degree to which there is significant variation around the average sample trajectories to be predicted, and (3) to assess the impact of both person-level (e.g., demographic and ecological characteristics of children and families), and classroom-level (e.g., treatment vs. control and treatment intensity, teacher characteristics, and overall levels of classroom disruptiveness) factors on children’s developmental trajectories. Question 2a-b. In order to more directly assess the mediated effects outlined in questions 2 and 3 (below), the interrelations among the constructs depicted in Figure 1 will be examined using Latent Growth Curve (LGC) modeling procedures (Willet & Sayer 1994; McArdle & Bell, 2000). Because each variable is measured at a minimum of four time points, growth trajectories of the primary constructs as well as their interrelations (i.e., the relations between independent and dependent variables) can be modeled. The four measurements of the primary domains depicted in Figure 1 will allow the estimation of at least two growth parameters that describe the pattern of change for each construct, namely the intercept and slope. The intercept represents individual differences in the level of a particular construct at a particular time (e.g., at the first assessment), and the slope represents the linear trend of an individual’s trajectory across repeated measures. The parameters of primary interest are the mean intercept and mean slope, which can be interpreted as the average level and slope of the trajectory across the sample. In addition, LGC modeling allows for the estimation of the variances of the intercept and slope. Significant variation in the intercept and/or slope of the growth curve indicates that there are significant individual differences to be predicted. Non-significant variation indicates that the growth curve is essentially the same for all teachers and/or children. Because LGC modeling is an extension of structural equation modeling procedures, the same goodness of fit criteria are applicable, and successively nested models can be evaluated against each other. The analyses will follow three basic steps and as such will include three types of models. First, unconditional models (i.e., models including no predictor variables) estimating latent growth curves for the teacher and child variables will be examined. These models will offer useful information about the average pattern of change across the four time points in the variables of interest, and will serve as a baseline against which the predictive models can be compared. Second a series of nested conditional models will examine the mediated effects of trajectories of teacher development and extended opportunities and support for children to engage in social-emotional learning. Following Baron & Kenny (1986) it is expected that when change in characteristics of teacher development and their provision of extended opportunities for youth to engage in social-emotional learning are considered as mediators of the relationship between 4Rs intervention and youth outcomes any direct effects of the 4Rs intervention will be significantly reduced or eliminated. Question 3. In order to understand the degree to which changes in social-emotional skills and behavior mediate changes in academic achievement, a sequence of increasingly complex mediated models will be tested. Building from the teacher mediational models described in 2a-b above, three alternate models will be compared. Specifically, a model in which changes in social-emotional learning is considered a mediator of changes is academic achievement will be contrasted to one in which changes in academic achievement is considered a mediator of changes in social-emotional learning. Finally these two unidirectional models will be compared to one in which bi-directional associations between social-emotional learning and academic achievement are estimated allowing them to simultaneously influence each other over time. Question 4. Outside consultation will be solicited to assist Brown and Jones in the conduct of a cost benefit analysis of the relative costs associated with and the benefits accruing from participation in the 4Rs program during the fourth year of the study. Toward this end we have recruited the advice and consultation of an expert in such analyses as part of school-based prevention efforts, Dr. Michael Foster of Penn State University. His participation ensures that we will be able to complete this goal. See attached letter of support. Question 5. To begin to answer question 5 in which the organizational and administrative characteristics necessary for schools to both reduce variability in and increase quantity of program delivery are examined, we will employ both school level administrative records and profiles publicly available from the NYC Department of Education as well as the management information system data collected by ESR Metro since the inception of the 4Rs program to look at preliminary associations between changes in characteristics of schools and the degree to which 4Rs schools meet their implementation benchmarks.
Sample and Measurement
Sample: Year 1 ChildrenTotal N = 942 children (in 18 schools)2 waves: Fall & Spring 3rd grade
Gender:Girls 51%Boys 49%
Race/Ethnicity:Non-Hispanic White 5%Hispanic & Hispanic Bi-Racial 45%Black or African American 41%Other 9%
Poverty:Poor 45%Not Poor 55%
Sample: Year 1 Teachers/Classrooms
Total N = ~85 Teachers/Classrooms (18 schools)2 waves: Fall & Spring 3rd grade
Gender:Female 94%Male 6%
Age 37.0(10.0) yrs.Race/Ethnicity:
Non-Hispanic White 56%Hispanic & Hispanic Bi-Racial 14%Black or African American 26%Other 4%
Note: 1. Underlined factors indicate slight differences in items between NYU and MS factor construction.2. Asterisks indicate factors with the lowest fit statistics.
Presenter
Presentation Notes
Note differences, but in general models at the first order level fit well.
Note: 1. Underlined factors indicate slight differences in items between NYU and MS factor construction.2. Asterisks indicate factors with the lowest fit statistics.
Measurement Modeling: First-Order CFA, 18 schoolsTeacher-Report on Child
Presenter
Presentation Notes
At this point note, what the 6 child outcomes of focus are: HAB INSA Agg and Prosocial Fantasies Internalizing Literacy Skills
Teacher BurnoutTeacher Role in Social-Emotional LearningTeacher Emotional AbilityClassroom Management StrategiesSocial Networks
Teacher-Child Relationships(Teacher-Report on each child)
Teacher-Child ClosenessTeacher-Child ConflictTeacher-Child DependencyStress Associated with Child Behavior
Presenter
Presentation Notes
Class scales: ES: pos and negative climate, teacher sensitivity, regard for stu perspec IS: concept development, quality of feedback, instructional learning formats OC: productivity, behavior management
CLASS: Classroom Observation Scoring System(Pianta, La Paro, & Hamre, 2005)
Three primary dimensions (low=1, high=7)Instructional Support, e.g.:
Concept DevelopmentQuality of FeedbackLanguage Modeling
Observations are 2 hours and include 4 20 min. segments.
Measures/Constructs: Classrooms
Baseline Subgroup Differences
Gender, Race/Ethnicity, Family Income, TreatmentMany gender differences in child, teacher, and parent report scales
(e.g., Competent and prosocial cognitions and behaviors, B<G; aggressive/externalizing cognitions and behaviors, B>G)
Many race/ethnic differences in child, teacher, and parent report scalesPoor children look worse than non-poor children Few (almost none) significant Treatment/Control differences in Fall 2004
Year 1 Program Impacts: Classroom-Level
Brown, J.L., Jones, S.M., & Aber, J.L. (under review). Improving Classroom Quality: Teacher Influences and Experimental Impacts of the 4Rs Program. Journal of Education Psychology.
Analytic Strategy: Classroom-LevelFocus on 3 classroom climate constructs (CLASS)
Emotional, Instructional, Organizational
2-Level Hierarchical Linear Model using HLM 6.02
Level 1: Classroom levelCovariates:
Class size, Special Ed., T. Age, T. Race/Ethnicity, Yrs. Teaching Exp.Baseline: T. Burnout Constructs, T. Emotional Ability Constructs
Level 2: School LevelTX/Control dummy; 8 school match dummies
Estimate TX impact on W2 (note, no baseline control)
0
0.2
0.4
0.6
0.8
1
OverallClassroom
Quality
Emotional Instructional Organizational
Classroom-Level Tx Impacts in Effect Sizes
Classrooms in the Tx group had higher mean Overall Classroom Quality scores, accounted for by higher mean Emotional Support and Instructional Support scores, than the control group
*
*
*
n.s.
Year 1-2 Program ImpactsChild-Level
Jones, S.M., Brown, J.L., & Aber, J.L (under review). The Longitudinal Impact of a Universal School-Based Social-Emotional and Literacy Intervention. Child Development.
