Smarter Balanced ELA Interim Assessment ScoringTEACHER HAND SCORING SYSTEM (THSS) MATERIALS
Presenter
Presentation Notes
Welcome….
Interims: Purpose and IntentPurpose:
• Support learning and teaching
Intended uses:• Check student progress • Inform classroom instruction• Practice with accessibility tools
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
2
Presenter
Presentation Notes
Smarter Balanced designed the interim assessments for instructional use to support learning and teaching throughout the school year in order to -allow teachers to check student progress -provide information to improve classroom instruction -provide practice with accessibility tools used on the summative
Benefits of Hand ScoringCalibration:• Teachers calibrate scoring • Student responses calibrated to fixed rubric• Scoring aligned to Smarter Balanced assessments• Materials consistent across grade levels, content areas, item types
Interpretation of student learning:
• Allows teachers to identify areas of strength and weakness
2/3/2017
WASHINGTON STATE OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
Presenter
Presentation Notes
The greatest benefit of hand scoring is that it allows teachers to interpret strengths and weaknesses for their students in the particular skill being assessed. Student responses can be one piece of evidence (of multiple measures). Similar to formative assessment processes used in the classroom, teachers can evaluate the constructed responses to determine next instructional steps. On a larger scale, these items allow for calibration to a fixed rubric, to provide a consistent ruler among students and teachers, and they help teachers and students better understand expectations of the summative assessment. Across grade levels, content areas and item types, THSS materials are formatted in similar styles. Each item has a training guide and exemplars.
Interim Assessment ProcessStep 1: Planning and PreparationUse Assessment Viewing Application (AVA), the interim blueprints (inside Test Administration Resources), andTHSS Materials (inside TIDE)
Step 2: Administration and Scoring
Use Test Administration interface (TA) to manage student testsTrain and score constructed response items in Interim Teaching Hand Scoring System (THSS)
Step 3: Reflection
View summary reports in the Online Reporting System and/or more detailed item-level and student-level reports in AIR Ways Reporting
Presenter
Presentation Notes
The Teacher Hand Scoring materials are used in the midst of a set of assessment events. To help them determine which Interim assessment to give to students, educators should access the Assessment Viewing Application (AVA), the teacher hand-scoring (THSS) materials located in TIDE, and the interim blueprints posted in the General Information folder on the WCAP Portal. The Test Administration interface and the Washington Secure browser are used to administer the Interim Comprehensive Assessments (ICA) and the Interim Assessment Blocks (IAB) to students. Student responses to constructed response items are then scored at the local level via the Teacher Hand Scoring System by educators, using the THSS materials. After hand-scoring is completed, data is available in both the Online Reporting System (ORS) and in the AIR Ways reporting system. If no hand-scoring is needed, which is true for many of the IABs, scores are available in ORS and AIR Ways soon after the completion of testing. Each of these systems has their own User Guide and Training Module available on the WCAP Portal. Please refer to those materials for more specific questions about ORS and AIR Ways.
Step 1: Planning and PreparationASSESSMENT VIEWING APPLICATION (AVA)INTERIM BLUEPRINTSTEACHER HAND SCORING SYSTEM (THSS) MATERIALS IN TIDE
Planning: Using AVAPurpose:• Allows educators to view interim assessments for
planning and preparation
• Provides an understanding of content being assessed and time needed to complete interims
What is available to view:• Actual interim assessment items presented online in
*IAB is new for 2016–17 ** IAB includes new items in 2016–17 *** The Edit/Revise IAB for grade 8 will be separated as soon as possible when items are available in the item bank
Presenter
Presentation Notes
This is a list of ELA Blocks for 2016-17. The check marks indicate IABs that have some hand-scored items. Most of these blocks contain 1-2 hand-scored items, with the exception of brief writes which are all hand-scored. Specific writing types for Performance tasks can be seen in AVA. G3: Opinion, G4-5: Narrative, G6: Argumentative, G7: Explanatory, G8: Explanatory, HS: Explanatory For more information about the interims: https://portal.smarterbalanced.org/library/en/interim-assessments-overview.pdf
Planning: Using Interim BlueprintsAccess blueprints: Test Coordinator Resources > General Information > Blueprint ELA/Literacy Interim Assessment
Sample:
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
8
Presenter
Presentation Notes
The Blueprints show the break down for each block: total number of items, targets assessed with accompanying DOK, detailed information about the items, such as how many are machine scored and how many are short text (meaning they would need hand scoring). Note: Some are labeled as human scored rather than short text.
Planning: Time for Training and ScoringTraining:
• Allow time for training PRIOR to scoring in the system
• 1–2 hours per training guide and exemplars set
• Training time should decrease with familiarity
Scoring:
• Time varies by item and number of student responses
2/3/2017
WASHINGTON STATE OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
11
Preparation: Materials in TIDE
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
10
Presenter
Presentation Notes
The complete set of THSS materials is located in TIDE. Click on the arrow next to the “General Resources” and then choose THSS Materials. A pop up window with a downloadable list of materials will appear which includes: this presentation “Overview of Scoring ELA Interims.ppx” grade level zip files.
Preparation: Using THSS materialsDownload materials from TIDE:
• Grade-level zip files • Reference document (labeled “InterimItems2016”)• Training Guide (labeled “SG”) • Exemplars (labeled “TM”)
Note: Materials also available in the THSS while scoring but only for the particular item being scored
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
11
Presenter
Presentation Notes
The THSS materials help prepare educators to score student responses from the Smarter Balanced Interim Assessments (IABs, and ICAs). The materials are located in zip files by grade level. They include a reference guide for the grade level and Training Guide (labeled “SG”) and Exemplars (labeled “TM”) for each hand-scored item. These materials (Training Guide and Exemplars) are also accessible in the THSS after students have completed the interim assessment, but only for the particular item being scored.
Download materials from TIDE
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
12
Presenter
Presentation Notes
In TIDE: located in zip files by grade level Reference document (labeled “InterimItems2016”) Training Guide (labeled “SG”) Exemplars (labeled “TM”)
Reference Guide—Zip folder
2/3/2017 13
Presenter
Presentation Notes
The purpose of the Reference Guide is to provide a summary of the various items with the following: item number, title, claim, target, item stem, and test selection name. This can be used for planning purposes.
Training Guide—Zip folder
2/3/2017 14
Presenter
Presentation Notes
There are also Training Guides (labeled “SG”)
Exemplars—Zip folder
2/3/2017 15
Presenter
Presentation Notes
And Exemplars (labeled “TM”) for each hand-scored item
WASHINGTON STATE OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
Training Guides for ELA• Instructions• Stimulus• Item stem• Target specific rubric• Condition codes• Target chart
2/3/2017
Presenter
Presentation Notes
Each training guide contains item specific stimulus, item stem, target specific rubric, condition codes and a target chart. The condition codes are the only part that is the same for reading, research, and brief writes; the full write codes are a more extensive list. Instructions: Provide step by step directions for using the materials Stimulus: The passage or text to which students are asked to respond Item stem: Task students are asked to complete Target specific rubric: Use to score student responses Condition codes: Use during scoring to identify reasons that responses are not scorable Target chart: Table that shows claim, targets and standards Note: Due to the animation of this slide, all pages of the document above may not appear when this presentation is printed.
Steps for Using Training Guides (SG)1. Start by reading the preamble (brief writes), stimuli, and
accompanying item stem.
2. Examine the target-specific rubric for each score point.
3. Read through the supporting documents:
• Condition code document―to better understand how to score unusual responses;
• Target chart―to support the understanding of all targets in each Claim.
4. Next, move on to the Exemplars Packet.
2/3/2017
WASHINGTON STATE OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
Presenter
Presentation Notes
Reading the Training Guide is the first step in the process of training yourself (or a group of educators) to score an interim assessment item.
Exemplars for ELA
2/3/2017
WASHINGTON STATE OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
• Instructions• Metadata:
• Claim• Target• Standards• Writing/Stimulus Type
• Anchor set• Check set• Key
Presenter
Presentation Notes
The exemplars contain many examples of scored student responses. Reading and understanding the exemplar is the most important part of training yourself (or a group of educators) to score interim assessment items. The Anchor Set of responses are arranged from highest scoring to lowest scoring and are used to help explain how the rubric is applied to student work. The Check Set of responses are randomly ordered and are used to help check your understanding of the rubric. Exemplars Packet A document containing an anchor set, a check set, and the key for scoring the check set is located in the Exemplar tab. The anchor set contains approximately 10 examples of student responses and, when available, scoring comments. The anchor set is provided to help the scorer understand how the rubric and scored papers align. Responses are ordered high-to-low for score points 2, 1, and 0. The check set contains approximately 10 randomly ordered example student responses without scores or scoring comments. The check set is an opportunity for the scorer to practice accurate scoring. The score points and scoring comments for the check set can be found in the key. By viewing the Training Guide and practicing using the Exemplars Packet, educators are ready to begin scoring assigned student responses. Note: Due to the animation of this slide, all pages of the document above may not appear when this presentation is printed.
Steps for Using Exemplars (TM)1. Read through the anchor set of student responses and scores to see
how rubric and scored papers align.
2. Read through the check set of random student responses.
• Practice assigning scores using rubric.
• Compare your scores to the key.
3. Next, you are ready to score your assigned student responses.
2/3/2017
WASHINGTON STATE OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
Step 2: Administration and ScoringTEST ADMINISTRATION
INTERIM TEACHER HAND SCORING SYSTEM (THSS)
202/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
Test Administration
Use the Test Administration application to administer interims
For complete directions see the TDS Interface Module
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
21
Presenter
Presentation Notes
Use the Test Administration interface to open Interim test sessions. Students will use the secure browser to take the test. Please view the TDS Interface Module located in the Modules folder of the Test Administration Resources section of the WCAP Portal for complete information. After student tests have been completed and submitted you can move on to scoring their responses.
Scoring Student Responses1. Log in to the Interim Teacher Hand Scoring System
2. Filter to organize responses
• Dropdown menus: Test, session, grades, subjects, or scorers
• Green heading: Student name, item, session, status, or assigned to
22
Student Response Page
Rubric
Student’s Response
Enter the score
Exemplar and Training Guide
Task or item
Submit score
Presenter
Presentation Notes
Select “Score” to view the individual student response page. The Exemplar and Training Guides are located here (the same ones found in TIDE), a rubric, a drop down to enter the score, a condition code (information about the condition code can be found in the Training Guide), the item stem, and the student’s response. You enter the score in the middle of the screen, and then submit the score at the bottom of the screen.
Response List: Completing Item Responses
24
Presenter
Presentation Notes
When you have finished scoring, you must check the box next to the student's name on the Score Response page and then click the button at the bottom of the page called Mark Selected as Complete. Once an item is marked complete it is submitted and can no longer be edited or viewed. It will no longer appear on this list. Once all of a student’s items for a specific test have been scored and marked complete the student’s score for the entire test will be available in the Online Reporting System and in AIR Ways. If you get into ORS or AIR Ways and don’t see data for a student, go back to THSS and make sure none of there responses are sitting on this list.
Appropriate use of materialsThis presentation may be shared with any audience
• Contains no secure information• Also available in the WCAP Portal
Training Guides and Exemplars may be used for these audiences:• Individuals – teachers, parents, and/or students• Groups – during a PLC, department, or school-wide meeting
Contains non-secure, but non-public information: Do not distribute
2/3/2017
WASHINGTON STATE OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
Presenter
Presentation Notes
Use this presentation with interested stakeholders. It is also available in the General Information folder of the Test Administration Resources portion of the WCAP Portal. Training Guides and Exemplars may be used with individuals and/or with groups, such as PLCs, departments, or at a school-wide meetings. These materials are considered non-secure but also non-public. These materials are intended for training and scoring purposes only and should not be distributed for other purposes. Teachers could use the materials with students to help them understand expectations; however, students should not receive copies or distribute any of these materials. Important to know: Designated as “non-secure* and non-public*” *Non-secure means access to the items, answers, and student responses – for educators and students (also not intended for accountability purposes) *Non-public means they are not be posted publicly (such as a district or teacher website) Optional to allow Local districts to establish timeframe, policies around usage, and scoring practices Important that districts/schools/teachers use discretion to check progress of specific concepts at strategic points during the year
Step 4: ReflectionONLINE REPORTING SYSTEM (ORS)
AIR WAYS REPORTING
262/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
Reflection“Formative Assessment is not a test but a process—a planned process involving a number of different activities. One of those activities is the use of assessments, both formal and informal, to elicit evidence regarding students’ status: the degree to which a particular student has mastered a particular skill or body of knowledge.”
–Jim Popham, Transformative Assessment
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
27
Use data from ORS and AIR Ways to reflect on student learning in order to improve teaching and learning
AIR Ways Reporting:Purpose: Use data to determine strategies to improve teaching and learning for classes, groups, and/or individual studentsPerformance data:• District• School• Class• Student
Information presented:• Item, answer key/rubrics• Actual student responses (including hand scored items)• Item difficulty and frequency distribution• Content alignment (target)
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
28
Presenter
Presentation Notes
Formerly known as Interim Reports Viewer (IRV) that was beta tested last year, AIR Ways reporting is used for viewing interim results. Keep in mind: Students need to take interims first If any hand scoring is required, that scoring must be completed prior to reports populating in AIR Ways Reporting
AIR Ways Reporting: Features
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
29
Expand
Presenter
Presentation Notes
Data can be viewed by clicking the plus to expand the columns to show information for the items on which students performed the best, the worst, and all items.
AIR Ways Reporting: Features
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
30
Item numberPoints possibleAverage
Student scores:Click the underlined numbers to show the interim items
Presenter
Presentation Notes
AIR Ways Reporting: Features
• Displays 5 items for which student performance was the highest
• Displays 5 items for which student performance was the lowest
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
31
Presenter
Presentation Notes
Identifies 5 items within an interim assessment for which student performance was the lowest Identifies 5 items within an interim assessment for which student performance was the highest
Details for an item: Frequency distribution of scores (how many 0, how many 1) Difficulty estimate (based on summative field test/operational data): e.g., Easy, Moderate, Difficult Content alignment: target language Item with answer choices Item with student response (both MC and constructed)
AIR Ways Reporting: Details displayedDetails for a student:
• Comparison to “Everyone”• Item with student’s response• Student performance descriptor
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
33
Student’s information
Presenter
Presentation Notes
Details for a student: Comparison to “everyone” in the class Item with student response Student performance descriptor: e.g. Below, At/Near, Above
Comparing ORS to AIR Ways:Both systems…
• Show overall performance for classroom, school, district on interim assessments, including average score and percent in each proficiency category
• Provide the point value for each interim item
• Indicate the skill being assessed–by assessment target
• Show all student opportunities
• Have export and print capabilities
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
34
Presenter
Presentation Notes
For exporting and printing note that CSV files = Excel spreadsheet
Comparing ORS to AIR WaysSPECIFIC TO ORS
• Can view reports for ALL state assessments, all subjects
• Provides combined classroom, school, district performance for each interim item and/or targets
• Has trend reports
• Has family and student friendly reports
SPECIFIC TO AIR WAYS• Interim item data only
• Shows each original item and alsoindividual student responses
• Includes scoring rubrics and correct responses
• Sorts the items by 5 best/5 worst for a group of students
2/3/2017
OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION
35
Presenter
Presentation Notes
Shows: -each item as presented to students (blank) and as submitted by students (with their answer included) -scoring rubric -breakdown of ICA items by target or claim Sorts the items on which students/rosters did best and worst Allows the educator to organize tests by “test reason” for multiple opportunites (COMING SOON)
For Further InformationShelley O’Dell– ELA Assessment [email protected]
THSS Materials 11-19 Expanded information about THSS materials that are in TIDE
2/3/2017
AIR Ways 28-35 Added information about AIR Ways
Updates to the 2016-2017 are noted below.
Presenter
Presentation Notes
Throughout the year, if updates are needed to this module, the changes will be detailed in the revision log on the PowerPoint. Any changes will be communicated in the Washington Assessment Weekly newsletter as well
