Smarter Balanced Assessment Consortium Smarter Balanced 2016-17 Technical Report Introduction and Overview

  • View
    0

  • Download
    0

Embed Size (px)

Text of Smarter Balanced Assessment Consortium Smarter Balanced 2016-17 Technical Report Introduction and...

  • Smarter Balanced Assessment Consortium:

    2016-17 Technical Report  Validity

     Reliability, Precision and Errors of Measurement  Test Fairness  Test Design

     Scores, Scales, and Norms and  Administration

     Reporting and Interpretation

  • Smarter Balanced 2016-17 Technical Report Table of Contents

    ii

    Table of Contents

    Introduction and Overview .........................................................................................................................vi

    Technical Report Approach ................................................................................................................................................... vi

    Peer Review Guidelines and Established Standards........................................................................................................... vi

    Overview and Background of the Smarter Balanced Theory of Action .......................................................................... vii

    Six Principles of Smarter Balanced Underlying the Theory of Action............................................................................. viii

    Purposes for the Smarter Balanced Assessment System................................................................................................... ix

    Overview of Report Chapters: ................................................................................................................................................ x Chapter 1: Validity ............................................................................................................................................................. xi Chapter 2: Reliability/Precision and Errors of Measurement...................................................................................... xi Chapter 3: Test Fairness.................................................................................................................................................... xi Chapter 4: Test Design ...................................................................................................................................................... xi Chapter: 5 Scores, Scales and Norms ............................................................................................................................. xii Chapter 6: Test Administration ....................................................................................................................................... xii Chapter 7: Reporting and Interpretation ...................................................................................................................... xii

    Acknowledgments ................................................................................................................................................................ xiii

    References ............................................................................................................................................................................. xvi

    Chapter 1 : Validity ................................................................................................................................. 1-1

    Introduction .......................................................................................................................................................................... 1-2

    A Note on the Validity Evidence Presented in Technical Report.................................................................................... 1-2

    The Validity Argument ........................................................................................................................................................ 1-3 Intended Purposes of the Smarter Balanced System for Summative Assessments ............................................... 1-3 Types of Validity Evidence.............................................................................................................................................. 1-3 Overview of the Validity Argument .............................................................................................................................. 1-5 Evidentiary Framework .................................................................................................................................................. 1-5

    Conclusion for Summative Test Validity Results ............................................................................................................ 1-17

    References .......................................................................................................................................................................... 1-18

    Chapter 2 : Reliability, Precision and Errors of Measurement ...................................................................... 2-1

    Introduction .......................................................................................................................................................................... 2-2

    Simulation Studies for 2016-17 Operational Summative Tests ..................................................................................... 2-2 Tests for Special Populations ...................................................................................................................................... 2-13 Item exposure ............................................................................................................................................................... 2-30

    Internal Reliability Estimates ........................................................................................................................................... 2-31 Paper/Pencil Test Reliability........................................................................................................................................ 2-39 Classification Accuracy ................................................................................................................................................. 2-40 Standard Errors of Measurement (SEMs).................................................................................................................. 2-57

  • Smarter Balanced 2016-17 Technical Report Table of Contents

    iii

    References .......................................................................................................................................................................... 2-63

    Chapter 3 : Test Fairness ......................................................................................................................... 3-1

    Introduction .......................................................................................................................................................................... 3-2

    Definitions for Validity, Bias, Sensitivity, and Fairness ................................................................................................... 3-3 Validity.............................................................................................................................................................................. 3-3 Attention to bias, sensitivity and fairness in test development............................................................................... 3-3

    Smarter Balanced Item Development ............................................................................................................................... 3-4 Guidelines for General Accessibility ............................................................................................................................. 3-5 Smarter Balanced Accessibility and Accommodations Framework ......................................................................... 3-7 Meeting the Needs of Traditionally Underrepresented Populations ...................................................................... 3-8 How the Framework Meets Needs of Students Who Are ELs .................................................................................. 3-8 How the Framework Meets Needs of Students with Disabilities............................................................................. 3-9 The Individual Student Assessment Accessibility Profile (ISAAP)............................................................................. 3-9 Usability, Accessibility, and Accommodations Guidelines ...................................................................................... 3-11

    Provision of Specialized Tests or Pools............................................................................................................................ 3-12

    Differential Item Functioning (DIF).................................................................................................................................. 3-13 Method of Assessing DIF.............................................................................................................................................. 3-14 DIF Results for the Summative Pools ......................................................................................................................... 3-18

    Test Fairness and Implications for Ongoing Research .................................................................................................. 3-21

    References .......................................................................................................................................................................... 3-22

    Chapter 4 : Test Design....................................................................................................................