Evaluating New Approaches to Assessing Learning FNL · 2005-03-24 · Evaluating New Approaches To Assessing Learning Richard J. Shavelson, Min Li, Maria Araceli Ruiz-Primo & Carlos

Evaluating New Approaches To Assessing Learning

Richard J. Shavelson, Min Li, Maria Araceli Ruiz-Primo & Carlos Cuauhtémoc Ayala

Keynote Address*

Joint Northumbria/EARLI Assessment Conference University of Northumbria at Newcastle, Longhirst Campus

28 August 2002

**Available at http//www.Available at http//www.stanfordstanford..eduedu/department/SUSE/SEAL/department/SUSE/SEAL

Overview

• Evaluating the Quality of Learning Assessments: Conceptual Framework

• Applying the Framework

• Concluding Comments

Conceptualizing Assessment: The Assessment Triangle

• Cognition: A model of that explains how students represent knowledge and develop competence

• Observation: Tasks or situations that prompt student to say, do, or create something to demonstrate knowledge

• Interpretation: A process for making sense of evidence

Observation Interpretation

Cognition

[Source: Pellegrino, Chudowsky & Glaser, 1999]

Evaluating Assessments: The Assessment Square Corners

• Construct: A working definition of what is to be measured

• Assessment: Systematic procedure for eliciting, capturing and scoring behavior

• Observation: Collecting and summarizing behavior in response to a task

• Interpretation: Inference from behavior on an assessment to the construct

Construct Interpretation

Assessment Observation

[Sources: Ayala, Yin, Shavelson & Vanides, 2002; Ruiz-Primo, Shavelson, Li & Schultz, 2001]

The Corners

Evaluating Assessments: The Assessment Square Analyses

• Conceptual Analysis: Identify domain of tasks and responses from construct definition

• Logical Analysis: Logical evidence that task will evoke in a student a problem space and response consistent with construct

• Cognitive Analysis: Empirical evidence on cognitive activities evoked by task/response

• Statistical and/or qualitative analysis: Bring quantitative and/or qualitative data to bear on proposed assessment interpretation.



Logical Analysis

Cognitive Analysis

Statistical and/ or Qualitative

Analysis

Warranted Inference?

Conceptual Analysis

The Analyses

Applying Framework To TIMSS’ Achievement Test:Construct and Assessment

• The Construct: Science Achievement• The Assessment: TIMSS Population 2

Science Test Items– Multiple-choice– Short-answer

• A combination of logical, cognitive and statistical analyses

[Source: Li (2001), Li & Shavelson, 2001]

Applying The Framework: Conceptual Analysis

• Declarative—knowing that

• Procedural—knowing how

• Schematic—knowing why

• Strategic—knowing when, where, and how to apply knowledge

Schematic Knowledge

Draws Upon

Declarative Knowledge

Strategic Knowledge Involved In

Procedural Knowledge

Applying The Framework: Logical Analysis of TIMSS Items

• Task Demands• Cognitive Demands—Assume

competent 14 year old• Item Openness• Complexity

Logical Analysis: Item Coding System

• Task Demands: What does the item ask student to do?– Terms, symbols, vocabulary, definition– Procedures, steps, actions, algorithms– Models, relationships, explanation, principles

• Cognitive Demands: What prior knowledge and cognitive processes examinee may use and reason with?– Visualize– Calculate– Perform experiment– Recall information– Reason and interpret with models and principles– Plan and monitor behavior– Guess or eliminate wrong options

• Item Openness: How free in shaping item response?– Hands-on v. paper-and-pencil– Selected v. constructed response– Constrained v. open response– One v. multiple solution paths– Follow steps in instruction

• Complexity: How familiar, relevant, reading difficult is item?– Textbook vs. ill-structured task– Inclusion of irrelevant background information– Long, reading demanding descriptions and complicated vocabulary– Answers contradict everyday experience

Logical Analysis: Item Coding System (Cont’d.)

Logical Analysis: Declarative Knowledge Item

• Assume competent 14 year old

• Task: Response expected to be a term, vocabulary (e.g., saliva), factual statement

• Cognitive Activity: Likely to be recall (question similar to form in which student learned content) with minimal reasoning to organize answer

• Openness: An open-ended question

P6. What digestive substance is found in the mouth? What does it do?

Logical Analysis:Schematic Knowledge Item

• Task Demands: Asks for explanation “why”; a model can be used to answer

• Cognitive Demands: Requires reasoning with a model (unless memorized/recalled)

• Openness: The information forms a complete question that allows examinees to finish the item without reading alternatives

• Complexity: Reasonable reading load

Q11. Which statement explains why daylight and darkness occur on Earth?

A. The Earth rotates on its axis

B. The Sun rotates on its axis

C. The Earth’s axis is tilted.

D. The Earth revolves around the sun

Logical Analysis:Procedural Knowledge Item

P1. The graph shows the progress made by an ant moving along a straight line.

A. 5cmB. 6cmC. 20cmD. 30cm

• Task Demands: Interpret diagram or apply algorithm

• Cognitive Demands: Apply the formula of Speed ÷Distance or extend line

• Openness: Constrained—can work backwards from alternative

• Complexity: Moderate reading

If the ant keeps moving at the same speed, how far will it have traveled at the end of 30 seconds?

Applying The Framework: Cognitive Analysis

• Assumptions: – Information processing model – Verbalization of working memory– Cognitive activity interpretation warranted

• Steps– Collect concurrent verbalization– Segment protocols– Code protocols– Analyze data

Cognitive Analysis: Sequence Of Think-Aloud Study

Session 1 Session 2-Session 4 Session 5

Introduction of the study, thinking aloud on the exercise problems-shoe tying

Solving a group of multiple-choice and free-response items with thinking-aloud & observations

Inter-view about solving the items

Solving the two performance assessment tasks with thinking-aloud & observations

Interview about solving the PA tasks and overall reflections

5-10 10-15 2-5 15-30 5-10

[Source: Li, 2001]

Cognitive Analysis: Protocol Analysis

• Collect concurrent verbalizations—participants were instructed to verbalize anything while responding to test items

• Segment each participant’s verbal protocol—Li used the entire response to each item or task no matter how many statements or types of knowledge

• Code participants’ segments—Li developed a system that captured evidence the four types of knowledge.

• Examine coding consistency (reliability)• Bring coded protocols data to bear on how participants

employed different types of knowledge to represent and solve problems

Cognitive Analysis: Link Between Logical And Cognitive Analysis

Based on the knowledge-type construct of science achievement, we expected participants’use of knowledge inferred from the protocols (cognitive analysis) to be congruent with the knowledge-types demanded by test items (logical analysis)

Pre-classified knowledge-typeType of knowledge used (n=9)* (n=10) (n=9) (n=2)

Declarative 8 11 048

Procedural 0 7 9

Schematic 9 16 0

Strategic 2 12 2

*Number of responses: 48 = 9 x 6 participants Chi-square = 208.12, p<.001

Declarative Procedural Schematic Strategic

54

41

10

Covariance Analysis: Links With Logical & Cognitive Analyses

DeclarativeKnowledge

bsmsa7e1

bsmsa9e2

bsmsa11e3

SchematicKnowledge

bsmsq11e20

bsssp5e19

bsssp2e18

bsmsb3e17

bsmsa12e15

bsmsa8e14

ProceduralKnowledge

bsmsr1 e28

bsmsp7 e27

bsmsp1 e26

bsmsb5 e25

bsmsq13e21

bsssq18e23

bsmsr2e24

bsmsb4e5

bsssp3e6

bsssp6e8

bsmsq14e9

bsssq17e10

bsssr4e12

bsssr5e13

.51

.84

.25

.33

.08.54

bsmsp4e7

-.11

.12

.40

.39.51.47

.41

bsmsq15e22

bsesr3e11

.27

.04.08

.36.21

.50.19

.70

.86

.26

bsmsb1e4

.23

-.06

.06.14

bsmsb2e16

-.17

-.10

.63.42

.44.37

.69.34

.37.42

.42

.11

.41

.00

-.24

.14

.02.29

Based on the knowledge-type construct of science achievement, we expected the emerged factors from the item scores (statistical analysis) to be congruent with the knowledge-types in logical analysis

Applying The Framework To Concept Maps:

Construct and Assessment• The Construct:

– Knowledge structure (declarative)

• The Assessment– Concept-Map – A graph with nodes

(concept terms) connected by labeled lines explaining relationship

[Source: Ruiz-Primo, Shavelson, Li, & Schultz, 2001]

Compound

Molecular Compound

Anions

Ternary Ionic Compound

Atoms

Elements

are composed of the same

Periodic Table

are organized on

with equal number of and protons have a

have a negativ

are charged

particles f

have a neutrahave either a

positive or a negative are formed

atoms lose or

are chemically

to form

made up of opposite charge

lose or gain electrons to

form

are of more thanone type of

is a ternary ioniccompound contains sulfite

which is a

are locatedon the left side of

with 3 or more elements

are elementslocated in the center of

containing 2 or elements are

is an example of

contain one or more

with two elements

are composed cations and

have to havehydroge

that begin with H

are composed H+ and

are composed metals

tend to

tend to

are located on upper right

of the

have a positive

have a negativ

form ions that can

have differen

will

is an example of

Metals

are atoms withmore protons electrons and

called

are atoms withmore than protonsand are

containing molecules

and sharingelectrons are

that are formed the attractionof cations and

anions are called

B

are composed of

Transition Metals

Non-

Charg

Ions

Polyatomic Ions

Cations

Electrons

Ionic Compound

N2O4

Sodium Sulfite

Binary Ionic Compound

Acids

Atoms & Molecules: Expert Map

The Study• Purpose:

– Compared “construct-a-map” and “fill-in-a-map” concept-mapping techniques

• Fill-in-the-nodes with structure given• Fill-in-the-line labels with structure given

– Motivation—fill in technique quicker, easier, and less expensive to score – Do both techniques provide equally valid measures of knowledge

structure?• Domain: Chemistry—atoms and molecules• Participants:

– Main study: 152 high-school students– Cognitive analysis expert-novice study:

• 3 low- proficient students• 3 high-proficient students• 2 chemistry teachers


Applying The Framework: Conceptual Analysis

Concept-map Assessment of Knowledge Structure:• Assumes long-term memory operates as a network

with linked nodes• Builds on empirical findings that experts’

knowledge, compared to novices’, is more:– Highly structured– Scientifically justifiable

Framework For Examining “Cognitive Validity” Portion Of Assessment Square

1. Cognitive activities evoked by assessment2. Relation between cognitive activities and performance

scores3. Impact of variation in technique on cognitive activities4. Correlation between scores on similar and different

assessments

Construct-A-Map Cognitive Processes Performance

Fill-In-A-Map Cognitive Processes Performance

(1) (2)

(3) (4)

(1) (2)


Assessment Cognitive Analysis Observation

Logical Analysis: Concept-Map Directedness

HighLow

Student Provides:

• Concept terms

• Linking Lines

• Linking Explanations

• Map Structure

Assessor Provides:

• Concept terms

• Linking Lines

• Linking Explanations

• Map Structure

Construct-A-Map

Fill-In-A-Map

[Source: Ruiz-Primo & Shavelson, 1996]

Cognitive Analysis: Think-Aloud

• Micro-level Analysis (Average reliability = 0.86)

– Segment protocol into small units—phrases– Code phrases

• Explanations (N2O2 is a molecular compounds because they are both nonmetals)

• Monitoring (“I can’t remember exactly what this is “)

• Conceptual errors (“Molecules are atoms”)

• Inapplicable events (Reads instructions)

• Macro-level Analysis (Average: agreement 95%/Kappa .92)– Segment protocol by entire map– Code planning from beginning of protocol– Code strategies for entire protocol attending to sequence

Comparison Of Micro-Level Cognitive Activities

Conceptual No-CodeTechnique n Explanation Monitoring Errors Applicable

Construct-a-map 8 39.08 28.22 9.78 22.91Fill-in-the- nodes 8 4.64 40.93 0.16 54.27Fill- in-the-lines 8 2.84 35.22 0.14 61.80




(1)(1) (2)

(3)(3)(4)

(1)(1) (2)


Comparison of Macro-Level Cognition: Strategy Sequence

3.1

4.2

2.1 2.1

1.1

2.4

1

2 2.1

3 to 20

2.324 2.2

25 26 27 28 29

30 324.3 33

4.131

3.3

4.2

2.1

1.2

2.1

1

2 2.13 4

2.4

5 to 7

2.48

22 to 23

10 2.2

27 to 28

11 123.3

13 15 16 17

4.2

18 2.114

2.4 2.1

2.23.3 2.2 2.2

2.42.2

19 2.420 21

2.12.1 24 2625

9 2.1

3.4 3.4

3.3 3.1 3.32 9 3230 31

21 to 23

3.1 4.2 2.4

2.1

High-Proficient Student

Low-Proficient Student

Codes:Starting Continuing Monitoring Ending

Shaded figures represent inaccuracy of content

• Select most general concept (1.1)

• Select concept related to general concept (2.1)

• Monitor performance by reading propositions (3.1)

• And so on…


Link Between Cognitive Activities and Performance Scores



(1) (2)(2)

(3)(4)

(1) (2)(2)


Correlations Between Verbal Proportion Scores and Performance Scores by Category (n=7) *

Mapping CategoryTechnique Explanation Monitoring Conceptual

Errors

.33 -.29 -.27

Fill-in-nodes .12 -.31 -a

.01 -.83b-

a

* Outlier deleted on each correlationNo variability in one of the variables.

b Correlation is significant at the .05 level.


a

Construct-a-map

Fill-in-lines

Comparison Of Scores On Mapping Techniques



(1) (2)

(3)(4)(4)

(1) (2)


Mean Scores and Standard Deviations Across the Mapping Techniques

Students TeachersMapping Max Mean S.D. Mean S.D.

Technique n = 6 n = 2Construct-a-map 1 0.63 0.31 0.74 0.17

Fill-in-the-nodes 12 10.67 1.97 12.00 0.00

Fill-in-the-lines 12 8.83 3.13 10.00 0.00


Comparison Of Scores On Mapping Techniques



(1) (2)

(3)(4)(4)

(1) (2)


Mean Scores and Standard Deviations Across the Mapping Techniques

Mapping CM LINTechnique (n = 5)

Construct-a-map -Fill-in-the-nodes .21

Fill-in-the-lines .91 .42


NOD

-

-

Concluding Comments



Logical Analysis

Cognitive Analysis

Statistical or Qualitative Analysis

Warranted Inference?

Conceptual Analysis

The assessment square:

• Provides a framework for examining new learning assessments

• Integrates conceptual, logical cognitive and statistical analyses of assessment-interpretation claims

Documents

Evaluating New Approaches to Assessing Learning FNL · 2005-03-24 · Evaluating New Approaches To Assessing Learning Richard J. Shavelson, Min Li, Maria Araceli Ruiz-Primo & Carlos