Windows Into The Mind Richard J. Shavelson & Maria Araceli Ruiz-Primo Stanford University Invited Talk University of Ancona Ancona, Italy June 26, 2000

Windows Into The Mind

Richard J. Shavelson & Maria Araceli Ruiz-Primo

Stanford University

Invited Talk

University of Ancona

Ancona, Italy

June 26, 2000

July 27, 2000 Shavelson & Ruiz-Primo: Windows Into The Mind

2

DENNIS THE MENACE

“GEE! I’M GLAD WE DID CONCEPT MAPS - NOW WE BOTH KNOW WHAT I LEARNED TODAY!!! ”

Assessing Knowledge Structure


3

Overview

• Cognitive theory underlying assessments of knowledge structure

• Direct and indirect methods of assessing “cognitive structure”

• Direct method: Concept maps

• Indirect method: Cognitive maps

• Link between concept and cognitive maps

• Concluding comments


4

Toward An Achievement Framework:Knowledge Types and Dimensions

Extent(How much?)

Structure(How is it organized?)

Others(Precision? Efficiency? Automaticity?)

Declarative Procedural StrategicKnowledge Knowledge Knowledge(Knowing the “that”) (Knowing the “how”) (Knowing the “which,” “when,” and “why”)

CognitiveCognitiveTools:Tools:PlanningPlanning

MonitoringMonitoring

Domain-specific content: • facts• concepts• principles

Domain-specificproduction

systems

Problem schemata/strategies/

operation systems

Proficiency

Low High


5

is an example of

Associative and Semantic Models of Memory

• Nodes represent concepts

• Lines represent relations among concepts

• Probe memory by asking about concepts (or lines) Associative Semantic

is a

contains


6

Concepts Maps

• Methods (Mapping Techniques)– Constructing A Map– Fill-In-The-Node Map– Fill-In-The-Line Map– Many Others

• Data structure and scoring methods depend on the technique


7

What Are Concept Maps?A Concept Map of a Concept Map!

graph

ConceptConcept MapMap

consist of consist of

is a

nodes labeledlines

concepts relation

denote arepresent

linked by

between a pair of


8

Concept Map Assessment Components

• TaskTask that invites a student to provide evidence bearing on his or her declarative knowledge structure in a domain

• FormatFormat for the student’s responseresponse

• Scoring systemScoring system by which the student’s concept map can be evaluated accurately and consistently


9

Are There Different Types of Concept Maps?

Yes, more than we would like!Yes, more than we would like!

Variations in the task, the response format, and the scoring system produce different concept

mapping techniques.


10

Concept Map Assessment: An Example

The following map shows the relationship among concepts related to genetic continuity. The concepts are arranged hierarchically and linked each other. Please examine the map and supply, in the space provided, a word or two for labeling each link such that association between concepts is made clear.

TaskTask

Response FormatResponse Format

Geneticcontinuity

Asexualreproduction

Sexualreproduction

Meiosis

Malegamete

Femalegamete

1. through2. producer3. requires4. called5. fuse into

Scoring SystemScoring System

• One point is given for each correct relationship

• One point is given for each level up until two levels beyond the last branching

• One point is given for the first branching where two or more concepts are connected to the concept above. Three points are given for any subsequent branching

• One point is given for each cross link

Source: Tamir, P. (1995). Science Assessment. In M. Birenbaum & F. J. R. C. Dochy (Ed.). Alternatives in assessment of achievements,learning processes, and prior knowledge. Boston: Kluwer Academic Publishers.12


11

Another ExampleTaskTask

Response FormatResponse Format Scoring SystemScoring System

Construct a concept map that reflects what you knowabout Ions, Molecules, and Compounds. The map shouldhave 10 concepts in it. Organize the terms in relation to one another in any way you want.

Focuses on the accuracy of the propositions:• Four points if the proposition is

outstanding and complete• Three points if the proposition is

correct and complete• Two points if the proposition is

incomplete but correct• One point if the proposition is correct

but does not show understanding between the two concepts.

• Zero if the proposition

Blank page for constructing the

map


12

On The Reliability & Validity of Concept Maps

• ReliabilityReliability: Can a student’s concept maps be consistently scored by different persons?

• Equivalence of Mapping TechniquesEquivalence of Mapping Techniques: Do all mapping techniques provide the same picture of a student’s declarative knowledge structure?

• Task DemandsTask Demands: Do different mapping techniques impose different cognitive demands on a student?

• Comparing Concept Maps With Traditional AssessmentsComparing Concept Maps With Traditional Assessments: Do concept maps measure something different than multiple-choice tests?


13

Study 1:Tasks

No-Concepts Provided Concepts Provided

• Construct a concept map about Ions, Molecules, and Compounds.

• Select another 7 concepts that you think are the most important in explaining the topic.

• Organize the concepts

• Construct a concept map about Ions, Molecules, and Compounds.

• Examine the 10 concepts and construct the map.

• Organize the concepts

• Redraw the map so someone else can read it.

Mapping Technique 1 Mapping Technique 2

LIST OF CONCEPTSacidsanions ...


14

Response Format

Blank page to construct the map

Construct-a-Map: No Concepts

Construct-a-Map: Concepts Provided


Concepts: Atoms, Compounds, Ions...


15

Scoring System

• Proposition Accuracy:Proposition Accuracy: Accuracy of propositions evaluated on a five-level scale--from 0 for inaccurate to 4 for accurate excellent

• ConvergenceConvergence: Proportion of accurate propositions in student’s map to the total possible accurate propositions in the criterion map

• Salience:Salience: Proportion of valid propositions to the total number of propositions in student’s map

Based on a criterion map, scoring focuses on:


16

Proposition “Quality”Quality ofProposition Definition

Outstanding proposition. Complete and correct. It shows a deepunderstanding of the relation between the two concepts.acids-compounds: < that gives off H+ when dissolved in water are

Excellent 4 points

GoodComplete and correct proposition. It shows a good understandingof the relation between the two concepts.acids-compounds: > are examples of

3 points

Poor Incomplete but correct proposition. It shows partial understandingof the relation between the two concepts.acids-compounds: < form

2 points

Don’t CareAlthough valid, the proposition does not show understandingbetween the two concepts.acids-compounds: > is a different concept

1 points

Invalid Incorrect proposition.acids-compounds: > made of

0 points


17

Generalizability(Reliability) of Scores

Percent of Total Variability and G Coefficients for Proposition Validity Total Scores

No-Concepts,Sample A, and B

Sample A & B

Person (p)Rater (r)Sample (s)p x rp x sr x cprc,e

2

71.64 78.67 .15 0 0 0 0 .7922.81 17.64 .01 .18 5.37 2.69

.89 .88 .89 .88

.78 .78

2 (nr=2,ns=3)

(nr=1,ns=1)

(nr=2,ns=2)


18

Discriminant Validity

Type ofScore No-Concepts Sample A Sample B

Proposition Validity

Congruence

Salience

Correlation Between Multiple-Choice Test and Concept Map Scores

.58

a

.45

.64

.66

.61

.63

.55

.50


19

Study 2:Tasks

Construct-a-Map Fill-in-the-Nodes Fill-in-the-Linking Lines

• Construct a map.

• Use the 20 concepts provided to construct the map.

• Organize the concepts any way you want.

• Check the map.

• Redraw the map so someone else can read it.

• Examine the skeleton map, the blank nodes and the terms provided.

• Select the term that corresponds to each blank node.

• Write the term inside the circle.

• Check that all blank nodes are filled-in.

• Examine the skeleton map, the blank linking lines and the linking words provided.

• Select the linking words that correspond to each blank linking line.

• Write the linking words on the blank linking line.

• Check that all blank lines are filled-in.


20

Response Format

Construct-a-Map Fill-in-the-Nodes

atoms

compounds

N2O4

Elements, Atoms, and Compounds

Fill-in-the-Linking Lines

Elements, Atoms, and Compounds

formed with cations

and anions areformed frommolecules sharing

example of

made upof opposite

composed of

that begins with H are

lose or gainelectrons to

form

with 3 or moreelements are

atoms

compounds

N2O4

made upof opposite

composed of

with 3 or moreelements are

ions

ioniccompounds

molecularcompounds

ternaryionic

acids

formed frommolecules sharing


Concepts: Elements, Atoms, Compounds...


21

Construct-A-MapConstruct a concept map that reflects what you know about what a concept map is. Organize the terms in relation to one another in any way you want.


22

Fill-In-The-NodeExamine the map, the blank nodes and the terms provided in the list below. Select the term that corresponds to each node and write it down inside the circle. Use each term only once.

Compounds

ELEMENTS, ATOMS, AND COMPOUNDS

Atoms

are composed of the same type of

Periodic Table

are organized on a

with equal number of electrons and protons have a neutral

have a negative

are negatively charged

particles of

have a neutralhave either a

positive or a negative

are formed when atoms lose or gain

are combined chemically

to form

made up of opposite charged

lose or gain electrons to

form

are composed of more than one type of

is a ternary ionic compound that contains sulfite

which is a

are located on the left side of the

with 3 or more elements are

are elements located in the center of the

containing 2 or more elements are called

is an example of a

contain one or more

with two elements are

are composed of cations and

have to have hydrogen

that begin with H are

are composed of H+ and

are composed of metals and

tend to form

tend to form

are located on the upper right side

of the

have a positive

have a negative

form ions that can

have different

will form

is an example of a

are atoms with more protons than electrons and are

called

are atoms with more electrons than protons

and are called

containing molecules

and sharing electrons are called

that are formed from the attraction of cations and

anions are called

A

are composed of two

Transition Metals

CationsPolyatomic Ions

N2O4

Electrons


23

Fill-In-The-Linking-LinesExamine the map, the blank lines and the linking words provided on the list below. Select the word(s) that corresponds to each line and write the number down in the line. Use each number only once.

Compounds

Molecular Compounds

Anions

Ternary Ionic Compounds

ELEMENTS, ATOMS, AND COMPOUNDS

Atoms

Elements

are composed of the same type of

Periodic Table

are organized on a

are negatively charged

particles of

have either a positive or a

negative are formed when atoms lose or gain

are combined chemically

to form

made up of opposite charged

are composed of more than one type of

is a ternary ionic compound that contains sulfite

which is a

are located on the left side of the

are elements located in the center of the

contain one or more

with two elements are

have to have hydrogenare composed of

H+ and

are composed of metals and

tend to form

are located on the upper right side

of the

have a positive

have a negative

will form

Metals

containing molecules

and sharing electrons are called

C

are composed of two

Transition Metals

Non-Metals

Charge

Ions

Polyatomic Ions

Cations

Electrons

Ionic Compounds

N2O4

Sodium Sulfite

Binary Ionic Compounds

Acids

is an example of a

with equal number of electrons and protons have a neutral

have a negative


24

Construct-a-Map Fill-in-the-Map

Scoring System

• Proposition Accuracy:: Accuracy of propositions evaluated on a five-level scale--from 0 for inaccurate to 4 for accurate excellent

• Convergence: Proportion of accurate propositions in student’s map to the total possible accurate propositions in the criterion map

• Salience:: Proportion of valid propositions to the total number of propositions in student’s map

• Correct/Incorrect: Student’s responses on the skeleton map were scored as correct or incorrect.


25

Are Fill-In-The-Map Scores Sensitive To The Nodes/Linking Lines Selected To Be Filled-In?

Type of Skeleton MapMean S.D.

Means and Standard Deviations by Type of Map and Sample

Fill-in-the-nodes

Sample 1 80 11.21 1.42

Sample 2 72 10.80 1.74

Fill-in-the-linking lines

Sample 1 78 9.77 2.74

Sample 2 73 8.99 3.09

(Max = 12)n


26

Are The Two Forms Of Fill-In-The-Map Equivalent?

Sequence

Fill-in-the-nodes Fill-in-the-linking lines

Mean S.D. Mean S.D.

Means and Standard Deviations by Type of Map and Sequence

1 Nodes 1-Lines 1 43 11.09 1.52 9.72 2.84

2 Nodes 1-Lines 2 36 11.03 1.33 9.31 3.06

3 Nodes 2-Lines 1 35 10.63 1.81 9.83 2.65

4 Nodes 2-Lines 2 37 10.97 1.67 8.68 3.13

Total 152 11.02 1.59 9.36 2.93

n


27

Can The Two Mapping Techniques Be Considered Equivalent?

Type of Assessmentn Max Mean S.D.

Means and Standard Deviations across the Three Types of Assessments

Construct-A-MapProposition Accuracy 152 135 53.91 22.17Convergence 152 1 .50 .19Salience 152 1 .73 .17

Fill-InFill-in-the-nodes 152 12 11.02 1.59Fill-in-the-lines 151 12 9.39 2.93

Multiple-Choice Test 150 30 24.05 3.74


28

Do The Mapping Techniques Provide Information Similar To That Provided By A

Multiple-Choice Test?

Type of Assessment

Construct-A-Map (C-M) (.99)

Fill-in-the-nodes-NOD (FI-N) .47 (.71)

Fill-in-the-lines-LIN (FI-L) .44 .40 (.85)

Multiple-Choice Test (MC) .44 .37 .53 (.74)

C-M FI-N FI-L MC

Structure Extent


29

Other Empirical Evidence

• Correlations tell you about rank ordering on different measures--that’s only part of the story

• Evidence is needed for the cognitive claims that different methods measure somewhat different aspects of achievement:– Talk aloud

– Focus group

– Group work

– Other?


30

Talk Aloud with Concept Maps: Method Variation

0

0.1

0.2

0.3

0.4

0.5

0.6

Explanation Monitoring ConceptualErrors

No-Code

Contruct-A-Map

Fill-In-Nodes

Fill-In-Lines

Verbal Codes

Pro

port

ion

Sco

res

Correspondence Between Directedness and Inferred Activity

Low Directed

High Directed


31

Tentative Concept Maps Conclusions • Students’ maps can be consistently scored by different persons even

when complex judgments, such as proposition quality, are required

• Different mapping methods provide different pictures of students’ declarative knowledge structure. The construct-a-map method provides opportunities to reveal students’ conceptual understanding

• Different mapping techniques impose different cognitive demands on students. Highly structured mapping techniques allow students to respond by elimination or guessing, whereas constructed response do not.

• Correlations between concept-map and multiple-choice scores are positive and moderate suggesting that these two types of assessment measure overlapping but somewhat different aspects of declarative knowledge


32

Cognitive Maps

• Methods– Word Association– Similarity Judgment– Card Sorting– Tree Building

• Data structure: Proximities or distances

• Scaling methods: Networks or hierarchical clusters


33

Example: Similarity Judgments

Rate the similarity of each pair of terms by circling one of the numbers provided or “DK -- Don’t Know

1. Central Tendency Mean(Closely Related) 1 2 3 4 5 6 7 (Unrelated) DK

2. Hypothesis Description(Closely Related) 1 2 3 4 5 6 7 (Unrelated) DK


34

Concept x Concept Proximity Matrix For A Student

Cent. Tend. Corr. Desc. Regr. ANOVA Hyp. Inf. Mean Median Mode Pop. Sample Stat. t-test Variab. Variance

Central Tendency 0Correlation 8 0Description 1 4 0Regression 9 3 8 0ANOVA 9 7 9 1 0Hypothesis 8 7 8 3 3 0Inference 8 8 5 2 3 1 0Mean 1 6 2 9 9 9 8 0Median 1 8 6 8 9 9 8 3 0Mode 1 8 4 9 8 9 8 3 2 0Population 8 8 8 3 3 2 1 8 7 7 0Sample 3 5 2 8 8 6 2 6 5 8 1 0Statistics 1 6 1 4 7 4 1 6 5 6 3 2 0t-test 8 8 7 2 1 4 2 7 8 9 4 9 7 0Variability 7 8 1 9 7 9 9 8 8 9 9 3 4 8 0Variance 8 7 3 8 5 8 7 7 9 8 8 3 6 8 1 0


35

Variance

Variability

Description

Mean

Central Tendency

Mode

Median

Statistics

Inference

Hypothesist-test

ANOVA

Regression

Correlation

Population

Sample

Network Representation of Proximities


36

Small Pilot Study (N=28) Of Similarity Judgments And:

• Concept maps

• Multiple choice

Terms: Variability, Variance, Standard Deviation

Variability

Variance

Std. Dev.

measuresmeasures

is the square root of

1. Which of the following refers to a procedure for making generalizations from a limited random sample of data?

a. Statistical Inferenceb. Intuitionc. Population Parameterizationd. Standardization

Source: Ed Wiley’s 2nd Year Project


37

Inference

Sample

Hypothesis

Population

ANOVA

LinearRegression

refers to relationships within the

uses sample to test ideas in the

is used

to te

st ide

as

abou

t popu

lation

in

is used to test

hypotheses in

is used to test population-related

addresses ideas about

variables in population

in the process of

is a procedure in statistical

is one special case of the general

linear model, along w

ith

is a

pro

cedu

re in

sta

tistic

al

Sample Portion Of Concept Map


38

Correlation Among Methods

Type of Assessment

PretestConcept Map .155 Similarity Judgment .554* .251

PosttestConcept Map .330Similarity Judgment .434* .706*

Multiple Choice Concept Map

* p < .01


39

What If We Could?

• Collect cognitive structure information indirectly by computer

• Immediately show a student a representation of her knowledge structure

• Permit her to modify the structure

• And label the lines?


40

Conclusions

• Concept and cognitive maps appear to tap a different aspect of declarative knowledge than do multiple-choice tests

• Concept and cognitive maps may provide similar representations of structure although this has not been tested directly

• By assessing different aspects of knowledge, we may broaden what gets taught!

Documents

Windows Into The Mind Richard J. Shavelson & Maria Araceli Ruiz-Primo Stanford University Invited Talk University of Ancona Ancona, Italy June 26, 2000