Smarter Balanced Assessment Consortium Smarter Balanced 2016-17 Technical Report Introduction and Overview

Smarter Balanced Assessment Consortium:

2016-17 Technical Report  Validity

 Reliability, Precision and Errors of Measurement  Test Fairness  Test Design

 Scores, Scales, and Norms and  Administration

 Reporting and Interpretation

Smarter Balanced 2016-17 Technical Report Table of Contents

ii

Table of Contents

Introduction and Overview .........................................................................................................................vi

Technical Report Approach ................................................................................................................................................... vi

Peer Review Guidelines and Established Standards........................................................................................................... vi

Overview and Background of the Smarter Balanced Theory of Action .......................................................................... vii

Six Principles of Smarter Balanced Underlying the Theory of Action............................................................................. viii

Purposes for the Smarter Balanced Assessment System................................................................................................... ix

Overview of Report Chapters: ................................................................................................................................................ x Chapter 1: Validity ............................................................................................................................................................. xi Chapter 2: Reliability/Precision and Errors of Measurement...................................................................................... xi Chapter 3: Test Fairness.................................................................................................................................................... xi Chapter 4: Test Design ...................................................................................................................................................... xi Chapter: 5 Scores, Scales and Norms ............................................................................................................................. xii Chapter 6: Test Administration ....................................................................................................................................... xii Chapter 7: Reporting and Interpretation ...................................................................................................................... xii

Acknowledgments ................................................................................................................................................................ xiii

References ............................................................................................................................................................................. xvi

Chapter 1 : Validity ................................................................................................................................. 1-1

Introduction .......................................................................................................................................................................... 1-2

A Note on the Validity Evidence Presented in Technical Report.................................................................................... 1-2

The Validity Argument ........................................................................................................................................................ 1-3 Intended Purposes of the Smarter Balanced System for Summative Assessments ............................................... 1-3 Types of Validity Evidence.............................................................................................................................................. 1-3 Overview of the Validity Argument .............................................................................................................................. 1-5 Evidentiary Framework .................................................................................................................................................. 1-5

Conclusion for Summative Test Validity Results ............................................................................................................ 1-17

References .......................................................................................................................................................................... 1-18

Chapter 2 : Reliability, Precision and Errors of Measurement ...................................................................... 2-1

Introduction .......................................................................................................................................................................... 2-2

Simulation Studies for 2016-17 Operational Summative Tests ..................................................................................... 2-2 Tests for Special Populations ...................................................................................................................................... 2-13 Item exposure ............................................................................................................................................................... 2-30

Internal Reliability Estimates ........................................................................................................................................... 2-31 Paper/Pencil Test Reliability........................................................................................................................................ 2-39 Classification Accuracy ................................................................................................................................................. 2-40 Standard Errors of Measurement (SEMs).................................................................................................................. 2-57

Smarter Balanced 2016-17 Technical Report Table of Contents

iii

References .......................................................................................................................................................................... 2-63

Chapter 3 : Test Fairness ......................................................................................................................... 3-1

Introduction .......................................................................................................................................................................... 3-2

Definitions for Validity, Bias, Sensitivity, and Fairness ................................................................................................... 3-3 Validity.............................................................................................................................................................................. 3-3 Attention to bias, sensitivity and fairness in test development............................................................................... 3-3

Smarter Balanced Item Development ............................................................................................................................... 3-4 Guidelines for General Accessibility ............................................................................................................................. 3-5 Smarter Balanced Accessibility and Accommodations Framework ......................................................................... 3-7 Meeting the Needs of Traditionally Underrepresented Populations ...................................................................... 3-8 How the Framework Meets Needs of Students Who Are ELs .................................................................................. 3-8 How the Framework Meets Needs of Students with Disabilities............................................................................. 3-9 The Individual Student Assessment Accessibility Profile (ISAAP)............................................................................. 3-9 Usability, Accessibility, and Accommodations Guidelines ...................................................................................... 3-11

Provision of Specialized Tests or Pools............................................................................................................................ 3-12

Differential Item Functioning (DIF).................................................................................................................................. 3-13 Method of Assessing DIF.............................................................................................................................................. 3-14 DIF Results for the Summative Pools ......................................................................................................................... 3-18

Test Fairness and Implications for Ongoing Research .................................................................................................. 3-21

References .......................................................................................................................................................................... 3-22

Chapter 4 : Test Design....................................................................................................................