Smarter Balanced Assessment Consortium Smarter Balanced Assessment Consortium: 2015–16 Summative Assessment

  • View
    6

  • Download
    0

Embed Size (px)

Text of Smarter Balanced Assessment Consortium Smarter Balanced Assessment Consortium: 2015–16...

  • June 2, 2017

    Developed by CRESST for Smarter Balanced

    Smarter Balanced

    Assessment Consortium: 2015–16 Summative Assessment

    Technical Report

    ▪ Validity ▪ Reliability, Precision and Errors of Measurement

    ▪ Test Fairness ▪ Test Design

    ▪ Scores, Scales, and Norms and ▪ Administration

    ▪ Reporting and Interpretation

  • Smarter Balanced 2015–16 Summative Assessment Technical Report

    Introduction

    i

    Table of Contents Introduction and Overview ........................................................................................................................... v

    Overview .................................................................................................................................................... v

    Technical Report Approach ..................................................................................................................... vi

    Peer Review Guidelines and Established Standards ........................................................................... vii

    Overview and Background of the Smarter Balanced Theory of Action .............................................. viii

    Six Principles of Smarter Balanced Underlying the Theory of Action ................................................. viii

    Purposes for the Smarter Balanced Assessment System ..................................................................... x

    Overview of Report Chapters: ................................................................................................................. xi

    Chapter 1: Validity ............................................................................................................................... xii

    Chapter 2: Reliability/Precision and Errors of Measurement .......................................................... xii

    Chapter 3: Test Fairness .................................................................................................................... xii

    Chapter 4: Test Design ....................................................................................................................... xii

    Chapter: 5 Scores, Scales and Norms ............................................................................................. xiii

    Chapter 6: Test Administration ......................................................................................................... xiii

    Chapter 7: Reporting and Interpretation .......................................................................................... xiii

    Acknowledgments .................................................................................................................................. xiv

    Outside Groups and Organizations that Collaborated with the Smarter Balanced Assessment

    Consortium ......................................................................................................................................... xiv

    Contributors to the Accessibility Accommodations Framework. .................................................... xiv

    Other Acknowledgments. ................................................................................................................... xv

    References ............................................................................................................................................. xvi

    Chapter 1 : Validity .................................................................................................................................... 1-1

    Introduction ........................................................................................................................................... 1-2

    Purposes of the Smarter Balanced System for Summative Assessments ....................................... 1-2

    Summary of Validity Argument ............................................................................................................. 1-3

    Validity Framework for Summative Assessments............................................................................... 1-3

    Essential Validity Evidence Derived from the Standards ................................................................. 1-13

    The 2014 Standards’ Five Primary Sources of Validity Evidence ................................................... 1-14

    Conclusion for Summative Test Validity Results .............................................................................. 1-16

    References .......................................................................................................................................... 1-17

    Chapter 2 : Reliability, Precision and Errors of Measurement............................................................... 2-1

  • Smarter Balanced 2015–16 Summative Assessment Technical Report

    Introduction

    ii

    Introduction ........................................................................................................................................... 2-2

    Simulations Studies for 2015-16 Operational Summative Tests ..................................................... 2-2

    Tests for Special Populations......................................................................................................... 2-10

    Item exposure ................................................................................................................................. 2-26

    Observed Reliability ............................................................................................................................ 2-27

    Paper/Pencil Test Reliability .......................................................................................................... 2-35

    Classification Accuracy ................................................................................................................... 2-35

    Standard Errors of Measurement .................................................................................................. 2-51

    Online Platform Effects ....................................................................................................................... 2-57

    References .......................................................................................................................................... 2-61

    Chapter 3 : Test Fairness ......................................................................................................................... 3-1

    Introduction ........................................................................................................................................... 3-2

    Definitions for Validity, Bias, Sensitivity, and Fairness. ................................................................. 3-3

    The Smarter Balanced Accessibility and Accommodations Framework ........................................... 3-4

    Meeting the Needs of Traditionally Underrepresented Populations. ............................................ 3-7

    How the Framework Meets Needs of Students Who Are ELLs. ..................................................... 3-7

    How the Framework Meets Needs of Students with Disabilities................................................... 3-7

    The Individual Student Assessment Accessibility Profile (ISAAP). ................................................. 3-8

    Usability, Accessibility, and Accommodations Guidelines: Intended Audience and Recommended

    Applications. ...................................................................................................................................... 3-9

    Guidelines for Accessibility for English Language Learners. ....................................................... 3-11

    Provision of Specialized Tests or Pools ............................................................................................. 3-13

    Fairness as a Lack of Measurement Bias: Differential Item Functioning Analyses ....................... 3-14

    Differential Item Functioning (DIF) Analyses for the Calibration Item Pool .................................... 3-14

    Test Fairness and Implications for Ongoing Research ..................................................................... 3-21

    References .......................................................................................................................................... 3-22

    Chapter 4 : Test Design ............................................................................................................................ 4-1

    Introduction ........................................................................................................................................... 4-2

    A Brief Description of Smarter Balanced Content Structure.............................................................. 4-2

    Synopsis of Assessment System Components ................................................................................... 4-4

    Evidence-Centered Design in Constructing Smarter Balanced Assessments .................................. 4-4

    Test Blueprints ...................................................................................................................................... 4-5

    Operational Summative Assessment Blueprints and Specifications. ........................................... 4-7

  • Smarter Balanced 2015–16 Summative Assessment Technical Repor