What Does Smarter Balanced Want Students to Know and Show? Heather Dallas, The Curtis Center at UCLA...

What Does Smarter Balanced Want Students to Know and Show?

Heather Dallas, The Curtis Center at UCLAShelbi Cole, Student Achievement PartnersJudy Hickman, Smarter Balanced at UCLA

California Math Council South ConferenceNovember 6, 2015

Why New Assessments?

Why are we changing what students need to know and be able to do?

Me: My daughter’s fingernails appear to be detaching at the base. Could that be related to the illness she had four weeks ago?

Doc: [Gets on his computer to examine current research.]

Do you want a pediatrician who relies on the most recent information or what he/she learned 20 years ago?

What does his future look like?

What skill set do his future employers value?

A Balanced Assessment System

Common Core State Standards

specify K-12

expectations for college and career readiness

All students leave

high school college

and career ready

Teachers and schools have

information and tools they need

to improve teaching and

learningInterim assessments

Flexible, open, used for actionable

feedback

Summative assessments

Benchmarked to college and career

readiness

Teacher resources for formative

assessment practices

to improve instruction

Building the Math Item Pool for an Adaptive Test

7

The CCSSM Requires Three Shifts in Mathematics

• Focus strongly where the standards focus

• Coherence: Think across grades and link to major topics within grades

• Rigor: In major topics, pursue conceptual understanding, procedural skill and fluency, and application with equal intensity

8

Mathematics topics

intended at each grade by

at least two-thirds of A+

countries

Mathematics topics intended at each grade by at least two-thirds of 21 U.S. states

Shift #1: Focus Strongly where the Standards FocusThe shape of math in A+ countries

1 Schmidt, Houang, & Cogan, “A Coherent Curriculum: The Case of Mathematics.” (2002). 9

10

GradeFocus Areas in Support of Rich Instruction and Expectations of Fluency and Conceptual Understanding

K–2Addition and subtraction - concepts, skills, and problem solving and place value

3–5Multiplication and division of whole numbers and fractions – concepts, skills, and problem solving

6Ratios and proportional reasoning; early expressions and equations

7Ratios and proportional reasoning; arithmetic of rational numbers

8 Linear algebra and linear functions

Shift #1: Focus Key Areas of Focus in Mathematics

11

Shift #2: CoherenceThink Across Grades

4.NF.4. Apply and extend previous understandings of multiplication to multiply a fraction by a whole number.

5.NF.4. Apply and extend previous understandings of multiplication to multiply a fraction or whole number by a fraction.

5.NF.7. Apply and extend previous understandings of division to divide unit fractions by whole numbers and whole numbers by unit fractions.

6.NS. Apply and extend previous understandings of multiplication and division to divide fractions by fractions.

6.NS.1. Interpret and compute quotients of fractions, and solve word problems involving division of fractions by fractions, e.g., by using visual fraction models and equations to represent the problem.

Grade 4

Grade 5

Grade 6

Shift #3: RigorRequired Fluencies for Grades K-6

12

Grade Standard Required Fluency

K K.OA.5 Add/subtract within 5

1 1.OA.6 Add/subtract within 10

2 2.OA.22.NBT.5

Add/subtract within 20 (know single-digit sums from memory)Add/subtract within 100

3 3.OA.73.NBT.2

Multiply/divide within 100 (know single-digit products from memory)Add/subtract within 1000

4 4.NBT.4 Add/subtract within 1,000,000

5 5.NBT.5 Multi-digit multiplication

6 6.NS.2,3 Multi-digit divisionMulti-digit decimal operations

Using Computer Adaptive Technology for Summative and Interim Assessments

• Provides accurate measurements of student growth over timeIncreased precision

• Item difficulty based on student responsesTailored for Each

Student

• Larger item banks mean that not all students receive the same questionsIncreased Security

• Fewer questions compared to fixed form testsShorter Test Length

• Turnaround time is significantly reducedFaster Results

• GMAT, GRE, COMPASS (ACT), Measures of Academic Progress (MAP)Mature Technology

How CAT Works (Binet’s Test)

Rating Item Difficulty

A

B

C

D

E

Percentage of 4th graders getting problems like these correct (Based on

spring 2014 data)

Enter a fraction equal to 1/3.

65%

60%

45% 30%

10%

35%

35%

40%

50%

70%

75%

17

Implementing at Scale: Item Specifications

Difficulty vs. Complexity

• The adaptive test will select items of appropriate difficulty level, but all students will see items across the full range of available complexity.

• Difficulty is about how hard the item is (i.e., how many students got it right); while complexity is about the kind of thinking a student needs to do to solve the problem.

18

How do we ensure that “difficult” items are difficult for the right reasons?

A set of item quality criteria is being applied to the evaluation of items in the Smarter Balanced bank, and one is about the time vs. information tradeoff from an item:

4a. Is the time spent on the item due to the required mathematics or due to the complexity of the item itself?

19

Will the CAT select items for a student aligned to standards in other grade levels?

• Once 2/3 of items required by the blueprint have been delivered in a student’s test, the pool may be expanded to include items with primary alignment to other grade levels for students performing at the highest and lowest performance levels– The goal is to increase precision of

measurement for very high and very low achieving students (minimize error in student’s score)

20

Why Update Item Specifications?

• Increase transparency for stakeholders. Enhancements include:– More items to illustrate each assessment target– Enhanced descriptions for how to adjust the

difficulty of items within a target– Clear labeling of which items allow calculators or

other tools– Reveal new item types planned for future pilot

testing– Remove any task models that were unsuccessful

during pilot testing phase

21

Did you know?

There are over 1,000 example items for mathematics across the Smarter Balanced grades 3-8 item specifications!

What are we waiting for? Let’s take a look!

22

23

Digging Into the Item Specifications

Did you know over 1000 math items can be found in the Mathematics Item Specifications?

• Mathematics Content Specifications

• CAT Item Specs, Version 3 (9/28/2015)

• Mathematics PT Item Specs All Grades (PDF) (5/27/14)

• Scoring Guide for Selected Short-Text Mathematics Items

http://www.smarterbalanced.org/smarter-balanced-assessments

Eliciting Problem Solving: Claim 2

Problem solving sits at the core of doing mathematics.

Proficiency at problem solving requires students to choose to use concepts and procedures from across the content domains and check their work using alternative methods.

25

Distinctives of Claim 2 Items:

• Multiple approaches are feasible or a range of responses is expected (e.g., if a student can solve a word problem by identifying a key word or words and selecting operations, then it is Claim 1).

• The use of tools in Claim 2 is intended to support the problem solving process. In some cases, students may be asked to display their answer on the tool (e.g., by clicking the appropriate

point or interval on a number line or ruler).

• Assessing the reasonableness of answers to problems is a Claim 2 skill.

26

Claim 2 Exemplars

27

Claim 2 Exemplars

28

Claim 2 Exemplars

29

Claim 2 Exemplars

30

Claim 2 Exemplars

31

Claim 2 Exemplars

32

Claim 2 Exemplars

33

Eliciting Reasoning: Claim 3

This claim refers to a recurring theme in the CCSSM content and practice standards: the ability to

construct and present a clear, logical, convincing argument.

Assessment tasks that address this claim will typically present a claim or a proposed solution to a problem and will ask students to provide, for example, a justification, and explanation, or counter-example.

34

Distinctives of Claim 3 Items:

• Items and task assessing Claim 3 may involve application of more than one [content] standard. The focus is on communicating reasoning rather than demonstrating mathematical concepts or simple applications of mathematical procedures.

• Targeted content standards for Claim 3 should belong to the major work of the grade.

35

Claim 3 Exemplars

36

Claim 3 Exemplars

37

Claim 3 Exemplars

38

Claim 3 Exemplars

39

Eliciting Modeling: Claim 4

Modeling is the process of choosing and using appropriate mathematics and statistics to analyze empirical situations,

to understand them better, and to improve decision-making

40

Distinctives of Claim 4 Items:

• Claim 4 differ from those in Claim 2, because while the goal is clear, the problems themselves are not yet fully formulated (well-posed) in mathematical terms.

• Claim 4 items and tasks should sample across the content domains, with many of these involving more than one domain.

• Because of the high strategic demand that substantial non-routine tasks present, the technical demand will be lower—normally met by content first taught in earlier grades.

41

Claim 4 Exemplars

42

Claim 4 Exemplars

43

Claim 4

Claim 4 Exemplars

45

Development Processes

Item Development Considerations

• Engaging for all students• Range of items of high and low difficulty• New item types for computer response

capture and real-time scoring• Robust tagging for interoperability

Operational Blueprint

Requirements

Field Test Item Development

Plan

Content and Item

Specifications

Tagging requirements

Claim, Target,

Task ModelItem Type

Scoring method

Cognitive Complexity

Difficulty

Accessibility, Bias/Sensitivity, other Guidelines

Specifications/Archetypes

● Item Specifications● Reflect lessons from pilot● Content reviews● Accessibility, bias, sensitivity

reviews● Archetypes

● Models● Content reviews● Accessibility, bias, sensitivity

reviews● Language Complexity Rubric

Foundation of Development Approach

Item Types

• Multiple Choice• Multiple Select• Hot Spot• Equation/Numeric• Graphing• Fill In Tables• Matching Tables• Drag and Drop• Short text

Item Quality - Supporting Activities

• Item Reviews– Occur in advance of audits– Independent but managed by CTB– Items housed in DAS and ITS systems– Comments evaluated for item revision, approvals

• Item Audits– Final quality control activity before release– Managed by Smarter Balanced– Items housed in ITS, feedback collected on

Google

Quality Criteria

Purpose

•Define characteristics of Smarter Balanced items/tasks to ensure high quality items

•Provide item authors with quality guidelines to apply during development

•Provide item reviewers with quality guidelines for item/task review

Role

•Expand traditional item development quality guidelines

•Ensure focus on important elements of evidence and alignment

•Provide gatekeeper criteria to focus authoring and review

•Guide overall quality training and processes

Language Complexity

Cook, H. G. & MacDonald, R. (2013). Tool to Evaluate Language Complexity of Test Items (WCER Working Paper No. 2013-5). Retrieved from University of Wisconsin– Madison, Wisconsin Center for Education Research website: http://www.wcer.wisc.edu/publications/workingPapers/papers.php

Questions?

Heather Dallas: dallas@math.ucla.eduShelbi Cole: scole@studentsachieve.net

Judy Hickman: judy.hickman@smarterbalanced.org

California Math Council South ConferenceNovember 6, 2015