Academic Engagement Training 19/20

Hello, my name is Robert Brader, the WSC Performance Measurement Coordinator. In this session, I’ll be discussing Academic Engagement, but first a little bit about performance measures,

Performance measures are measurable indicators of a program's performance as it relates to member service activities. Performance measurement is the process of regularly measuring the services provided by your program and the effect your program has in communities, and in the lives of members or community beneficiaries. The main purpose of performance measurement is to strengthen your AmeriCorps program, to foster continuous improvement and to identify best practices and models that merit replication. Performance measurement will also help to identify programmatic weaknesses that need attention.

As a CNCS intermediary, WSC is required to ensure that all Service Sites Establish, track, and assess performance measures for their programs. Establish ambitious and realistic performance measures in consultation with WSC Ensure that programs under our oversight collect and organizes performance data on an

ongoing basis, at least quarterly Ensure that programs under our oversight track progress toward meeting Their

performance measures; Ensure that any program under our oversight corrects performance deficiencies

promptly; and fairly present the results in reports to WSC.

In this session, I’ll discuss the community problem, the evidence to support the intervention, the intervention itself, how to track the output & outcomes, Dosage level requirements and the definition of improvement. Feel free to use this training material to write your Data Collection Strategy.

We know that academic Engagement is associated with positive academic outcomes, including achievement and persistence in school; higher levels of engagement are found in classrooms with supportive teachers and peers, with challenging and authentic tasks, opportunities for choice, and sufficient structure.

Data on student engagement, as indicated by student behaviors such as attendance, disciplinary incidents and academic course failure, shows significant disparities for low-income students. Low income students are consistently failing 25% more than their non-low income peers.

These trends have major implications for the students, who, as a result, will have limited opportunities to find meaningful, well-paying work, particularly in today’s global economy, where a college degree and high proficiency in reading and math are critical to success.

The data here suggests that the primary goal or our AmeriCorps members will be a relentless focus on keeping students on track and connecting them to individualized support services such as tutoring, mentoring and leadership opportunities as soon as they show warning signs.

Since engagement is multifaceted, we know that there isn’t just one approach. However much of the research on the subject of engagement points toward relationship building. Our AmeriCorps members will build strong relationships with students, staff, families and the community to ensure all students have the skills and support to graduate and build a college going culture. By placing strong role models into our engagement programs everyday gives our students one more opportunity to form a relationship with a key adult who can help them stay on a positive track. Sinclair et al, 1998; Sinclair et al, 2005Fredricks , Blumenfeld, & Paris, 2004 Mandernach , 2009 Niemi, 2007Brewster & Bowen, 2004

WSC recommends project sites follow a Check & Connect informed intervention, or a similar evidence informed strategy as a way to help students stay in school, progress in school, and complete school.

For over 25 years, Check & Connect (C&C) mentors have helped keep thousands of students on track toward graduation. C&C is the only dropout prevention intervention listed on the IES What Works Clearinghouse found to have positive effects on staying in school. The Check component is designed to continually assess student engagement through close monitoring of student performance and progress indicators. The Connect component involves program staff giving individualized attention to students, in partnership with school personnel, family members, and community service providers.

Member strategies will focus on relationship building with peers and adults, problem solving, personalized tutoring, promoting participation/affiliation with school, conflict resolution with data driven feedback to students to instill self and social awareness. Other strategies include partnership with school personnel, parents and community service organizations. The AmeriCorps member will regularly review information on attendance, behavior, or academic challenges and will collaborate with school officials when problems are identified. The member will emphasize school completion, will promote college readiness, preparation for graduation and future aspirations. The AmeriCorps member will help coordinate services and provide ongoing feedback and encouragement to students, especially emphasizing the importance of staying on-track and in school.

Despite the breadth and diversity of programs supporting college- and career-readiness and success, careful analysis demonstrates that successful programs share a number of common programmatic and structural features. Programs should exemplify strategically designed approaches to providing rigorous, supportive, and relevant learning environments rich in positive relationships, applicable college knowledge, and youth-centered programming.

Check & Connect

Since our programs are targeting low-income schools, participation is open for all students. Eligible students must be enrolled in grades K-12 schools and/or school districts with a majority of Economically Disadvantaged Students (i.e. free and reduced lunch status of school or district) and/or a majority of students with special or exceptional needs. Generally, tier 2 level students within low-income schools.

To start, most students will be identified through an Early Warning tracking System (EWS) measuring attendance, behavior, and/or course performance. In some cases disengaged students are identified by a student engagement survey, such as the (SEI).

The Early Warning System (EWS) High School Tool was developed by the National High School Center at the American Institutes for Research, in collaboration with Matrix Knowledge Group, to allow users to identify students showing early warning signs of risk for dropping out of high school. The tool calculates research-based early warning indicators that are predictive of whether students graduate or drop out of high school. The early warning indicators include course failures, grade point average (GPA), credit accumulation, and behavioral incidents. Designed to be used throughout the school year on a regular basis – after the first 20 or 30 days of the school year and after each grading period – users can identify at-risk students, assign them to customized interventions, and monitor the response to these interventions.

The Student Engagement Instrument (SEI) is an Evidence-based student self-report survey designed to measure cognitive and affective engagement. The SEI is based on a model of engagement that grew out of work with Check & Connect.

Our minimum recommended dosage level is one 30-minute session each month for six months. The frequency and intensity of services may be longer and more often for students demonstrating increased risk of disengagement.

Completion can occur after a student receives the minimum allowable dosage or longer, depending on the program model…some programs support students for an entire year.

To ensure non-duplication of students, the project site will need to track the amount of time spent with each student to ensure the dosage requirement can be met. Additionally, each site should use an Educational Service Log showing the student’s name and/or ID number, grade level, and school; the start date and end date for services, the number of hours of services provided each week, month, and school year; the total number of service contacts, and a determination improvement.

In order to measure Attitude, each project site will use a student self-report pre and post program

survey containing the same questions in order to measure at least two points in time. WSC recommends site use the Check and Connect Student Engagement Instrument (SEI) or a comparable evidence-informed assessment capable of measuring both Cognitive and Affective indicators.

There are five subtypes included in the SEI that can be measured. For Cognitive (e.g., self-regulation, perceived relevance of schooling and future goals)

Affective engagement (e.g., belonging, relationships with teachers and peers) at school and with learning. Research has confirmed the factor structure of the SEI, measurement reliability across grades 6-12 (Betts et al., 2010) and downward to grades 3-5 (SEI-E; Carter, Reschly, Lovelace, Appleton, & Thompson, 2012).

The SEI Survey will measure multiple indicators of self-reported student engagement attitudes on a 4-point response scale. Pre-intervention scores will be compared with post-intervention scores, with “improvement” defined as an increase of 5% or greater increase on at least one of the indicators without the other indicators decreasing. Project sites have the option to choose to measure one, two or all the subtypes.

The pilot study of the SEI was conducted with a large, diverse group of 9th grade students. An initial set of items were drawn from the literature and then refined following focus groups with students; a total of 56 items were originally piloted on the survey. Exploratory and confirmatory factor analyses identified 35-items that loaded on 6 factors: 3 representing cognitive engagement and 3 representing affective engagement

With respect to validity, high correlations, in expected directions have been found between SEI scores and other measures of school performance such as achievement, attendance and disciplinary incidents

Finally, a recent study found, after controlling for demographic, behavior, and achievement variables, that SEI scores in the 10th, 11th, and 12th grades significantly predicted college attendance and persistence through the first year

The SEI has been extended downward to grades 3-5

Finally, Student engagement data may be used to identify those who are at-risk for disengagement and dropout as well as to inform and monitor the effects of interventions. For example, if you are tracking students by attendance, the SEI can be administer as a way to tailor an individualized intervention.

Here are the SEI assessment questions by subtype. If you use different questions, make sure they can be arranged by subtype.

Arrange each attitudinal question by a four scale option: Scores are calculated by summing a group of engagement questions associated with at least one, two or all five of the Academic Engagement subtypes compare the score between pre and post.

When you complete your data collection strategy, I’d like to know what subtype you are measuring. One final word of caution, service sites might want to consider administering the post assessment at mid-year since scores have been found to decline the closer students reach the summer.

Our Behavior indicator is Chronic Absenteeism. Programs will report the number of students with improved attendance after completing the program.

In our last grant cycle, we set the improvement rate at 90%. However we’ve expanding the definition include those students with an increase of 10% from the pre-program score or 90% or higher during the time spent while in the program.