Plans for Reporting Results and Performance Level Setting

The Second Annual New Jersey Leadership SummitJanuary 21, 2015

Jeffrey NellhausChief of Assessment, PARCC, Inc.

1. PrioritiesProvide a brief review of the priorities that drive PARCC

2. Reports of ResultsYou will become knowledgeable about the kinds of metrics and reports that will be used to report the results of the PARCC assessments next fall

3. Performance Level Setting (aka standard setting)You will be knowledgeable about the process PARCC intends to conduct this summer to identify the threshold scores for each of its five performance levels

Objectives for this morning …

2

PARCC Consortium in 2015

3

AR

COIL

LA

MA

MS

NM

NY

OH

TN

MD

NJ

RI

Governing Board States

DC

• Administering the test in 12 members

• 5 million students

• Nearly 15% of students in grades 3-11 nationally

• About 3 in 4 will take the tests on computer

Develop tests worth taking

Texts worth readingEssays worth writing

Problems worth solvingAccessible to all students

Goals …

Report resultsworth using

ValidReliable Timely

Actionable

4

Determine whether students are college- and career-ready or “on track”

Provide tools to assess student learning and support instruction

during the school year

Report comparable results across schools, districts and member states

Generate data, including growth metrics, for accountability uses

Priority Purposes

Use technology to increase student access and engagement, and

create efficiencies

5

PARCC Assessment System

6

Draft Score Report Designs“Results Worth Using”

7

PARCC Score Report Design Study

• PARCC conducted a 5-month, multi-phase Score Report Design Study to establish priorities for the kinds of data the reports should include and how they should be presented.

• Stakeholders providing input included:Teachers and Administrators (Central Office and Building)ParentsHigher Education RepresentativesPARCC state education leadershipPARCC Technical Advisory Committee (TAC)

8

Data Management & Reporting System

• PARCC score reports will be available to local educators online through its interactive web-based Data Management and Reporting System, and to parents via a printed student reports

• PARCC’ web-based system will provide authorized users to customize reports using the following “filters”

9

StudentSchool DistrictState PARCC

YearSubject

Type of TestGrade

Student Group

Overall ScoresStatus Scores

Growth ScoresSub-Scores

Item Level Scores

Performance Levels

•Math•ELA/L

Scale Scores

• Mathematics• ELA/L

• Reading• Writing

Sub-Scores

•Whether the student’s performance in the sub-category was At/Above, Near, or Below the performance of students who scored “just Level 4” overall in the content area

Metrics for Reporting Results

• At the level, student the number of points earned

• At the school level, the average number of points earned by all the students in the school who took the item.

Item Scores

Draft of Individual Student Report

11

Draft of Student Roster - Sub-Scores

12

Draft of Student Roster – Item Scores

13

Draft School by Grade Summary Report

14

Draft District by School Summary Report

15

Draft State by District Summary Report

16

Draft PARCC by State Summary Report

17

Performance Level Setting

18

• PARCC will report results using five performance levels (names under discussion)

• Students performing at Levels 4 or 5 on Algebra II, Integrated Math 3, or ELA/L grade 11 assessments will earn a College- and Career-Ready Determination

• Public colleges and universities in PARCC states have agreed

to allow students who have been accepted and have earned CCR Determinations to enter directly into credit-bearing courses without remediation

PARCC Performance Levels

19

Performance Level Setting: What is it?

20

Use Performance Level Descriptors & Data from Research

Identify Threshold Scores

Five PARCC Performance Levels

Who will participate?

• High School Panels (6)• Grade 3-4 Panel (2)• Grade 5-6 Panel (2) • Grade 7-8 Panel (2)

Performance Level Setting

21

K-12 educators

Postsecondary faculty

Business people

Grade-Span Panels

Each Panel will Include 20

Participants

Schedule of Performance Level Setting Events

22

PARCC Performance Level Setting Schedule

Panel Start Date End Date

Subject/Grade

Mathematics PanelsELA/Literacy

Panels

Dry Run 5/21/2015 5/28/2015 Algebra II and Integrated 3 Grade 11

High School 7/27/2015 7/31/2015

Algebra I and Integrated 1

Geometry and Integrated 2

Algebra II and Integrated 3

Grade 9

Grade 10

Grade 11

Grades 7–8 8/17/2015 8/21/2015 Grades 7 and 8 Grades 7 and 8

Grades 3–6 8/24/2015 8/28/2015Grades 5 and 6

Grades 3 and 4

Grades 5 and 6

Grades 3 and 4

Performance Level Setting:What Process Will be Used?

PARCC will Use an Evidence-Based Process• The process combines expert judgment with data from

research

• The process will generate threshold scores that – Reflect PARCC’s performance level descriptors (PLDs)– Are consistent with the results reported by other well-regarded

state, national and international assessment programs

• Because comparability is a priority, every state will adopt the same threshold scores

23

Performance Level Setting: Training

Panelists will:• Practice with online tutorial • Take the computer-based assessment• Review the performance level descriptors (PLDs)• Write “borderline” performance level descriptors

(PLDs)• Practice with the process

24

Proposed Performance Level Setting Method

25

Yes/No Method (proposed)

Review test items

• Answer the question:

• Based on the PLDs, would a student performing at Level X be likely to answer the item correctly?

Impact data

• Impact data includes:• Empirical

data from research

• Judgments of other panelists

• % of students performing at each level

• % who got item correct

Grade 3 Sample Item and PLDs

Grade 3 Math : Sub-Claim AThe student solves problems involving the Major Content for the grade/course with connections to the Standards for

Mathematical Practice.Level 5: Distinguished

Command Level 4: Strong Command Level 3: Moderate Command Level 2: Partial Command

Time 3.MD.1-13.MD.1-2

Tells, writes and measures time to the nearest minute. Creates two-step real world problems involving addition and subtraction of time intervals in minutes.

Tells, writes and measures time to the nearest minute. Solves two–step word problems involving addition and subtraction of time intervals in minutes.

Tells, writes and measures time to the nearest minute. Solves one-step word problems involving addition or subtraction of time intervals in minutes.

Tells, writes and measures time to the nearest minute. Solves one-step word problems involving addition or subtraction of time intervals in minutes, with scaffolding, such as a number line diagram.

26

Patricia needs to read for 120 minutes each week.• She read for 26 minutes on Monday.• She read for 39 minutes on Tuesday.• She read for 38 minutes on Thursday.

How many more minutes does Patricia need to read this week?

minutes

Click on the answer box and type in a number. Backspace to erase.

Research to Inform the Performance Level Setting Process

• Research Questions– What percentage of students are likely to be college- and career-ready,

according to PARCC’s definition?– What percentage of students are likely to be on track to college and career

readiness at earlier grades?

• Two studies to inform performance level setting

27

Postsecondary Educators

Judgment Study

Benchmark StudySATACT

NAEPTIMSSPISA

Performance Level Setting Timeline

28

Sign up for the PARCC Updates

newsletter at www.parcconline.org

• E-mail me jnellhaus@parcconline.org

• Follow me on Twitter• @jmnellhaus

• Follow us on Twitter @parccplace

• Visit our website www.parcconline.org

• http://parcc.pearson.com/

• Sign up for the PARCC Updates newsletter on our website

Updates and more information

29