Smarter Balanced Member Portal - Smarter Balanced Assessment

January 7, 2015

Smarter Balanced Assessment Consortium:

Achievement Level Setting Final Report

ALS Final Report

1

Contents Executive Summary ................................................................................................................................ 5

Overview of Tasks and Processes ........................................................................................................... 5 Online Panel Results ................................................................................................................................ 5 In-Person Workshop Results ................................................................................................................... 7 Results of Cross-Grade Review ............................................................................................................ 10 Impact by Subgroup .............................................................................................................................. 12 Action by Chiefs ..................................................................................................................................... 16

Section 1. Introduction ........................................................................................................................ 18 Smarter Balanced Background ............................................................................................................ 18 Purpose and Organization of this Report ............................................................................................ 19

Section 2. Communications Plan ........................................................................................................ 20 Purpose and goals ................................................................................................................................ 20 Targets ................................................................................................................................................... 20 Research and Planning ......................................................................................................................... 21

Environmental scan. ......................................................................................................................... 22 Focus group study. ............................................................................................................................ 23

Recruitment and Outreach ................................................................................................................... 23 Social media campaign/paid promotion. ........................................................................................ 24

Media Strategy/Communications Planning ........................................................................................ 24 Communications planning for key activities and state approval. .................................................. 24

Contingency planning ........................................................................................................................... 25 Intake interviews: Development of contingency plan, messaging and protocols. ........................ 25 Ebola contingency response. ........................................................................................................... 26

Digital and Creative Support ................................................................................................................ 26 Expansion of video production scope. ............................................................................................. 27

Achievement Level Naming Research ................................................................................................. 27 Interviews with assessment experts. ............................................................................................... 28 Focus groups among teachers. ........................................................................................................ 29 Online surveys among parents and students. ................................................................................. 29

Section 3. Logistics ............................................................................................................................. 31 Schedule ................................................................................................................................................ 31 Facilities ................................................................................................................................................. 32 Hardware and Software Support.......................................................................................................... 33 Meetings ................................................................................................................................................ 33

ALS Final Report

2

In-person workshop........................................................................................................................... 33 Cross-grade review. ........................................................................................................................... 36

Technical Advisory Committee Meetings ............................................................................................. 36 Collaboration Conferences ................................................................................................................... 36 Advisory Panel Meetings ....................................................................................................................... 37 National Conference on Student Assessment .................................................................................... 37 Other Meetings ...................................................................................................................................... 37 Security .................................................................................................................................................. 38

Section 4. Audit ................................................................................................................................... 40 Purpose of the Audits ........................................................................................................................... 40 Selection of the Auditors ...................................................................................................................... 40 Major Audit Functions ........................................................................................................................... 41

Review of tools and materials. ......................................................................................................... 41 On-site review of procedures. ........................................................................................................... 41 Review of documentation. ................................................................................................................ 42

Legal review ........................................................................................................................................... 42 Section 5. Panelist Recruitment .......................................................................................................... 43

Online Panel for Achievement Level Setting ....................................................................................... 43 Implementation. ................................................................................................................................ 43

Findings and Refinements ................................................................................................................... 46 In-Person Panel for Achievement Level Setting .................................................................................. 46

Implementation. ................................................................................................................................ 47 Selection Process .................................................................................................................................. 48 Findings and Refinements ................................................................................................................... 50 Cross-Grade Review Panel ................................................................................................................... 54

Implementation. ................................................................................................................................ 54 Findings and Refinements ................................................................................................................... 55

Section 6. Standard Setting Design and Implementation ................................................................... 57 The Bookmark Procedure ..................................................................................................................... 57

Creating ordered item booklets ....................................................................................................... 57 Item mapping. ................................................................................................................................... 59 Typical application of the bookmark procedure. ............................................................................. 59

Design and Implementation of the Online Panel ................................................................................ 60 Software development. ...................................