Student Growth Measures in Teacher Evaluation

Facilitator’s Guide for Training Module 4:

Scoring an Individual SLO

Overview

Training Purpose and Goals

The student learning objectives (SLO) training is intended to assist teachers and administrators in implementing SLOs in their schools. At the conclusion of the four training modules, teachers should be able to

Explain what makes a high-quality SLO.

Select or develop appropriate assessments.

Develop growth targets.

Score an individual SLO.

Facilitators should be able to

Review the components of the Ohio Teacher Evaluation System Framework and Model.

Deconstruct Ohio’s model for developing SLOs.

Understand the tools and processes used in developing, organizing, and reviewing SLOs.

Use strategies to facilitate work on SLOs in your school district or school.

Audience

This facilitator’s guide is designed as a blueprint for the training that will be delivered to teachers within their schools to prepare them to develop and implement SLOs this year (2012–13), whether SLOs are being implemented on a pilot basis or a fully operational basis within the evaluation system. Facilitator notes, as well as suggestions of the direct language to use in conveying the material to teachers, are provided.

Timing

The training that you will deliver is split into four modules. Each module is designed to be delivered in 60 minutes, during time designated for professional development or common planning.

Resources

The following resources are provided for use in delivering the four training modules to staff within each school:

Facilitator’s guide (this document)

Slide presentation

Participant handouts

Tips and tools for presenting the module, which includes an annotated agenda and anticipated questions.

Facilitator’s Guide: Module 4—2

Why Student Learning Objectives?

Recent federal and state policies have led to new systems of educator evaluation. These new systems include multiple measures, one of which is often student learning or student growth. SLOs have recently come to the forefront of this work and have been highlighted as a promising approach for measuring student growth, particularly for the majority of teachers in subjects or grades not covered by a standardized assessment.

The Ohio Context

We know that a well-designed evaluation system can support professional growth and improve educator practice, which, in turn, leads to better student outcomes. In Ohio, as elsewhere, a major challenge in creating new evaluation systems is to figure out how to measure student growth for the majority of teachers and students for whom state-administered standardized tests or relevant and validated vendor assessments are not available. An additional challenge is incorporating measures of student growth within the evaluation system in such a way that directly ties this information to the instructional planning and goals of teachers and represents the unique needs and goals of the school district.

Advised by the Ohio Department of Education (ODE), many school districts will implement SLOs as a part of the “locally determined measure” portion of the student growth component, which is 50 percent of the overall evaluation per legislation. If value-added or approved vendor assessments are available, those options must be used, and locally determined measures such as SLOs will account for up to 40 percent of the overall evaluation. In grades or subjects where value-added or approved vendor assessments are not available, locally determined measures may account for up to 50 percent of the evaluation, which is the entire student growth component.

Some school districts will pilot SLOs in the 2012–13 school year, whereas other school districts will fully implement SLOs within their teacher evaluation system.

Facilitator’s Guide: Module 4—3

Slide 1

Slide 1 is the title slide for Module 4.

Slide 2

Review the training norms with the group.

Slide 3

Plan to spend 15-20 minutes on slides 1–11.

Say:

“Now we are proceeding to the fourth and final module of the training on student growth measures and student learning objectives.”

Slide 4

Say:

“The previous three modules have covered the first four steps in the SLO development process. This module will focus on the fifth step, which is to monitor and prepare for the SLO rating and discussion, or what we are referring to as “scoring.”

“In this final stage, the SLO has been set, reviewed, and approved, and now you are gathering and analyzing evidence of progress toward the goal, and organizing and presenting evidence of final attainment of the goal.”

Facilitator’s Guide: Module 4—4

Slide 5

Say:

“First, let’s review the timeline in which SLO scoring will take place. For 2012–13, the writing, approval, and revision time is shortened, so that SLO submission is occurring by December 14; reviews and revisions, if necessary, are occurring in December; and the SLO is final by January. In future years, the SLO should be set and final somewhat sooner, in November. Both this year and moving forward, the SLO calculation of attainment and scoring must be completed prior to the May 1 deadline. So, as we are discussing the processes of monitoring progress, gathering and presenting evidence, and scoring SLOs, this is the date to keep in mind from which we are working backwards. Also, note that ample time for committee or evaluator review, per your district’s requirements, must be built in prior to this date to ensure that the entire teacher evaluation process is complete by May 1.”

Slide 6

Say:

“In a moment, we will go through the steps leading up to SLO scoring, but so that we can all visualize what we’re heading toward, let’s define what we mean by an SLO score.”

“Simply put, the SLO score is the percentage of students that met their growth target. The percentage then falls into one of five categories: Most Effective (5), Above Average (4), Average (3), Approaching Average (2), and Least Effective (1).”

“These numerical ratings will then be combined with other student growth measures in the system to arrive at a final rating. We will review this process at the end of this hour, but first, let’s go through a few simple examples.”

Facilitator’s Guide: Module 4—5

Slide 7

Say:

“Consider, for example, a teacher who has 100 students in his or her SLO. If 90 of those students meet the growth target and 10 don’t quite reach it, this means 90 percent met the target, which is the threshold for the rating of Most Effective, or 5.”

Slide 8

Say:

“Consider an example where of 89 students in an SLO, 74 meet the growth target. Seventy-four out of 89 is 83 percent, which corresponds to a rating of Above Average. As you can see, each calculation is simply the percentage of students meeting the growth target.”

“In this example of 89 students, below are the thresholds for each performance level. As you can see, 27 of the 89 students would have to miss the growth target in order for the rating to be below the “Average” threshold. So, while it is important to set accurate growth targets, there is room in this process for some error and adjustment as everyone learns to gauge accurate growth in their content and with their assessments.”

“Now, let’s outline the process that you will go through to arrive at one of these five ratings for an SLO.”

Slide 9

Say:

“Backing up for a moment, after the SLO has been reviewed and approved, the final assessment that will be used to demonstrate growth and the standards and content that you are going after will be set. After review and any needed revisions, the teacher should walk away from the approval with an SLO checklist that has every applicable box checked and every criterion in the seven components satisfied. It is then up to the teacher or teacher team to decide the best ways to monitor progress toward that goal and attainment of those standards. Some type of formative or interim assessments should be used to monitor student progress. A midcourse check-in with an evaluator or appropriate committee, [depending on your district circumstance] is recommended so that the teacher has an opportunity to discuss student progress.”

Facilitator’s Guide: Module 4—6

“The teacher should bring any information to the check-in that will help the evaluator gauge progress toward the learning target. An example would be any student work products or outcomes of the formative or interim assessments that can show development of the skills or content knowledge necessary to reach the final goal.”

“If students seem to be off track, the teacher should identify those specific students and come prepared to discuss ways to change instruction to facilitate making up ground.”

Slide 10

Say:

“Now we’ll move into discussing how to prepare for the final scoring process. Again, this will occur at the end of the interval of instruction. In general, something to remember is that much of the onus of preparing the evidence of whether students met their targets or not does falls on the teacher. To make this review process possible, it is helpful to follow these steps exactly so that every time an evaluator or local committee is looking at this evidence from one teacher to another, they know exactly what they are looking at because it is organized in the same way. The more streamlined the process, the more likely it is both efficient and fair.”

Slide 11

Say:

“There are several documents that the teacher should have ready in order to present the evidence for review. The evidence should include:

A copy of the original, approved SLO template and checklist for reference.

The completed Individual SLO Scoring Template, which we’ll review in a moment.

Copies of the student assessments on which attainment of the growth targets was measured (not completed student versions, just the assessments themselves),

An explanation of how the assessment was scored. For example, if multiple assessments were used, detail how the scores of the assessments were combined to arrive at the final score. If any rubrics were used, provide copies. If team scoring was used, describe that process.”

Facilitator’s Guide: Module 4—7

Slide 12

Plan to spend 15-20 minutes on slides 12–16.

Say:

“After the final assessment has been administered and results are available, the teacher will complete the Individual SLO Scoring Template to present the evidence on growth target attainment for each student. The Ohio Department of Education (ODE) is in the process of making a final version of this spreadsheet, but the one we are showing you here gives you a sense of where they are headed in their thinking. The scoring template is formatted in an easy-to-use spreadsheet. A screenshot of the scoring template is provided on Handout 4.1, along with the SLO Scoring Matrix. This template will help you do the simple calculations involved in arriving at the final score, but its main purpose, really, is to present this information in a standardized way from one teacher to the next.”

Slide 13

Say:

“Handout 4.2 provides instructions for completing the Individual SLO Scoring Template in eight steps.”

This slide details the steps in completing the Individual SLO Scoring Template.

Review each step, referring to the slide and pointing to the applicable columns on the scoring template as you speak, if possible.

Say:

“Once you have completed the Individual SLO Scoring Template, it is useful for presentation of the data to sort the scoring template by growth target tiers, then by students who have met or not met the target. This information will be useful during the conversation about how the SLO results can be used to think about changes in practice and instructional strategies that might be used in the future to reach any identified groups of students who did not meet the growth targets.”

Ask the group if there are questions on the process, including what data should be entered into each cell.

Facilitator’s Guide: Module 4—8

Slide 14

Say:

“Clearly, as with any assessment, the more students included in the SLO, the less any one student contributes to the teacher’s overall rating. With any SLO, multiple measures of student performance or assessments are helpful to increase the reliability of the overall measure, but they can become especially important as fewer students are included in an SLO. For example, you would want to avoid setting an SLO that included 10 students and only one multiple-choice assessment taken at one point in time. In that scenario, much of the final SLO score would hinge on each student’s performance at that one point in time. There may be situations where it is appropriate for a teacher to only have 10 students in an SLO because he or she is responsible for facilitating the learning of those 10 students in that context. In that case, the evaluator or local committee would want to ensure that the SLO was capturing multiple measures of each student’s learning.”

“Consider a scenario where 10 students are all in two different courses—one course with 120 students and one course with 40 students. Suppose that the assessments in each of the courses are centered around writing skills, and these 10 students have writing deficiencies. Suppose both teachers did not set realistic growth targets for these students. While it is important to carefully set growth targets for every student, the teacher with 40 students is more affected by the SLO rating by setting those 10 inaccurate growth targets. So, in cases where the number of students covered by an SLO is comparatively low, multiple measures and special attention paid to the accuracy of growth targets are important.”

Slide 15

“Before we move on, are there any questions? Now we’re going to discuss an SLO scoring example. As we read this example, consider, with the information you’re given, what aspects of the SLO appear to be strong. Also consider any improvements that might be suggested to Mr. Ramirez when setting the SLO growth targets next time.”

Note: If enough time in the session remains, the facilitator can choose to do this activity as a “fish bowl,” and have three participants come to the front of the room and have their conversation around this example. One of the participants should record the key thoughts or themes that come up as these two questions are answered.

Otherwise, read the SLO example aloud to the group, and facilitate a discussion around the provided questions.

Example 1 Background: Expect groups to indicate that all components of the SLO seem to be high-quality multiple measures within the assessment,

Facilitator’s Guide: Module 4—9

and a large majority of students are meeting their growth target. Because the six students who did not meet their growth target are in bottom of the middle tier, participants may note that in future years, the teacher could further refine the growth targets by potentially setting a fourth growth target tier or providing targeted assistance to students in this section of the achievement spectrum. The teacher may want to examine the assessments of these students to determine if any trends in content misunderstanding or skill deficiencies exist within the group.

Slide 16

Say:

“Now, in groups of three or four, read through each of the remaining examples describing SLO scoring outcomes. Discuss your answers to the provided questions. Designate one person within your group to record your responses and another person to share during our large group discussion. We’ll reconvene the large group in about 15 minutes.”

Example 2 Background: Expect groups to voice that this SLO was based on one assessment that was administered without direct pretest data to gauge growth. Expect participants to talk about increasing the ways in which calculus students are assessed in order to gauge growth in multiple ways or find more reliable ways to gauge growth by incorporating multiple sources of pretest data. Participants might also mention monitoring student growth more closely during the course to track progress.

Example 3 Background: Expect groups to voice that in subsequent years, the assessments for these two SLOs would want to continue to be vetted to make sure they are rigorous and are including grading practices that reduce subjectivity. Also, expect participants to note that with smaller numbers of students, even one student missing the growth target can quickly reduce the overall percentage attaining the goal. This is an issue to be aware of in terms of stability of the measure, hence making the multiple measures important.

Examples continue in the next column.

Spend approximately 15 minutes on the large group share-out if rich discussion is occurring. When the discussion concludes and all concerns have been addressed, move on to the final section on student growth measure scoring.

Example 4 Background: Expect the groups to voice concern that of those not meeting their growth targets, a majority were higher achieving students from the top tier of expected growth. High-achieving students may be already meeting a level of proficiency that is acceptable, but are not able to stretch to meet targets for growth as easily as students in the middle of the achievement spectrum. With this particular teacher, the issue may be that (1) the growth targets were too ambitious or (2) the students were not receiving enough enrichment to propel them past a certain level of proficiency. This issue could be dealt with in subsequent years by either setting growth targets more conservatively on the high end of achievement or ensuring that instruction throughout the course is assisting already high-achieving students.

Facilitator’s Guide: Module 4—10

Slide 17

Plan to spend 15 minutes on slides 17–21.

Say:

“Now we’ll move on to the final section, which describes the process through which SLO scores will be combined with other student growth measures, if applicable, and entered into the online system.”

“In order to provide a consistent process for student growth measure scoring and minimize the burden on local education agencies (LEAs), each LEA will enter teacher scores within the Ohio Teacher and Principal Evaluation System (eTPES), which will then automatically record and calculate overall scores. The district plan will already be submitted in eTPES, which will have provided the default percentages for how much of the final score will be attributed to different types of student growth measures based on the three categories of teachers: A, B, and C.”

Slide 18

Say:

“Now, we return to which student growth measures each category of teacher has available to them. The following chart is an example that shows a district that has decided that, when they are available, value-added scores will account for 40 percent of a teacher’s overall score, while LEA measures, specifically SLOs, will account for 10 percent. For category B teachers, or those without value-added but with vendor assessments, those vendor assessments will account for 30%, and SLOs and Shared Attribution will account for 10% each. Then for category C teachers, SLOs will count for 40%, and shared attribution 10%. Note that this is just an example of how one district could choose to develop their plan.”

“ODE will have entered a teacher roster as well as value-added reporting into eTPES.”

Facilitator’s Guide: Module 4—11

Slide 19

Say:

“In some cases, districts may want to make changes to the percentage attributed to each student growth measure per teacher. Ideally, any variations would be made at the beginning of the year when the district plan is submitted in the system.

For example, it may be the case that a teacher only has one year of value-added data, and the evaluator or committee feels it would be appropriate to shift more of a percentage of the evaluation onto LEA measures. In this scenario, consistency in variations should be maintained based on circumstance.”

“For example, if teachers with one vendor assessment are allocated 20 percent to the measure, but teachers with two or more vendor assessments are allocated 30 percent to the measures, this allocation should be maintained for all teachers in that circumstance.”

“Also, a percentage shouldn’t be changed for one teacher without considering if it creates incomparability between the measures used in other grades and subjects.”

Slide 20

Say:

“This worksheet represents the scores that will be entered for each teacher. Let’s look at Handout 4.4 for an easier-to-read version of this worksheet.

Value-added data and percentages allocated to each measure from the district plan will already be populated in the system.

The greyed-out boxes - the “value added” score and percentage, the “vendor assessment” score and percentage, the “SLO/Other score and percentage” and the “shared attribution” score and percentage - will be calculated and populated by the system, so only the individual scores, 1-5 in each yellow-highlighted section, will be entered by the teacher or LEA representative.

Note that each teacher can incorporate up to five vendor assessments and six LEA measures (five SLOs and a shared attribution measure) if those measures are available. So, a teacher who did not have value-added or vendor assessments available could submit up to five SLOs in the system.

However, all measures within each of these three categories would be weighted equally in the system. For example, all five SLOs would receive the same weight.”

Facilitator’s Guide: Module 4—12

Slide 21

Say:

“The final student growth measure score will be calculated by the eTPES system. The calculation used to arrive at this score is still being finalized and will be approved by the student growth measures advisory committee soon. The final SGM score will be converted into a rating of Above, Expected, or Below. We know that this still leaves some ambiguity about how the final SGM score will be calculated, but there will be additional materials available online that detail this process once it is approved.”

Slide 22

Plan to spend 5 minutes on slides 22–23.

This slide is a reflection, asking participants to anticipate the stumbling blocks or concerns that LEAs and teachers will have around the SLO scoring process and combined student growth measure scoring. Be sure to ask for ideas about how this training can better address those foreseen issues or concerns.

Slide 23

Say:

“This concludes Module 4. Thank you for participating.”

Facilitator’s Guide: Module 4—13