The Application of ECD to the Redesign of Advanced Placement Exams CCSSO June 2012 Maureen Ewing,...

The Application of ECD to the Redesign of Advanced Placement Exams

CCSSOJune 2012

Maureen Ewing, Kristen Huff, Amy Hendrickson, Pamela Kaliski

Why ECD for AP?

• Need targets of measurement and curricular requirements

• No guesswork on part of the test user, test maker or test taker about what is valued in the domain

• Lack of ambiguity and use of task models helps ensure stable measurement

• Enhances validity argument

Three Critical ECD Phases

Experts with Disciplinary Knowledge Unpacked the Content omains

•BIG IDEAS(LEVEL 1)

•ENDURING UNDERSTANDINGS (LEVEL 2)

•SUPPORTING UNDERSTANDINGS(LEVEL 3)

1

2 2 2

3

Experts with Disciplinary Knowledge Unpacked the Domains

•IDENTIFY SKILLS OR REASONING PROCESSES

•ARTICULATE DEFINITION OF SKILL (I.E., OBSERVABLE EVIDENCE)

1

2 2 2

3

Inputs to Domain Analysis

• National and state standards

• Latest research on student learning and assessment

• Curriculum Surveys

• Other relevant documents

• Expert judgment of panelists

Three Critical (re)Design Phases

Writing Claims and Evidence

• Claims start with the phrase “The student can” to reinforce the focus on what the student knows and can do

• Claims are comprised of content and/or skills that were identified in the domain analysis

• Evidence starts with the phrase “the work is characterized by…”

• Evidence includes nouns and adjectives (no verbs)

If it’s unobservable, it isn’t ‘evidence’

Examples of non-evidence:

• “a deep awareness of the importance of theorems”

• “understanding of differences and similarities between derivatives and integrals”

• What would a student have to do, say, or write in order to prove to you that he/she has a deep awareness of the importance of theorems? The answer to this question is the evidence.

• As evidence is written, necessary edits to the claims may be uncovered; iteration is key!

Achievement Level Descriptions – ALDs• Describe what students in each performance

category should reasonably know and be able to do

• Relate directly to the knowledge, skills, and abilities a student should know and be able to demonstrate to be classified into a particular score category

• Distinguish clearly from one performance category (e.g., AP Grade 2) to the next (e.g., AP Grade 3)

Three Critical (re)Design Phases

Achievement Level 3

Task Models: Take-home messages1) Claims and evidence are “fodder” for

item• Stimuli• Prompt• Response options• Rubrics

2) Many “like” items can be produced from same claim / evidence pair

Construct relevance as the target of measurement is

the claim/evidence

Item generation; Comparability

Form Assembly Specifications - Considerations• Multiple inputs

• Domain Model

• Experts’ ratings of importance of content and skills

• Psychometric criteria

• Structure of the domain

• Claims: skills-based versus integration of skills and content

• Content relationships

• Skill relationships

• Content and skill relationships

Challenges of ECD

• Iterative nature of work • Sometimes viewed as “doing it wrong” the first time, an

annoyance, but it IS the process• Resource and time intensive (6 years and counting),

especially development of task models based on iterative pilots

• Steep learning curve – difficult to think in terms of observable evidence

• Defining appropriate grain size for each piece

Lessons Learned and Ongoing Work

• We need to develop an evaluation process that specifically addresses the iterations between C&E, ALDs, etc

• Documents, captures, thinks about the iteration - this is the real work of figuring out the construct about how students learn as well as ECD

• We need to figure out the grain size issues and how everything fits together