R.K. Thornton Effective methods for the use, creation, analysis, and interpretation of short- answer student conceptual evaluations. Ronald Thornton Professor

R.K. Thornton

Effective methods for the use, creation, analysis, and interpretation of short-answer student

conceptual evaluations.

Effective methods for the use, creation, analysis, and interpretation of short-answer student

conceptual evaluations.

Ronald Thornton Ronald Thornton

Professor of Physics and Education Professor of Physics and Education

Center for Science & Math TeachingCenter for Science & Math Teaching

Tufts UniversityTufts University

R.K. Thornton

What was I thinking?What was I thinking?

I’ll paint your house and I’ll paint your house and walk your dog as well.walk your dog as well.

R.K. Thornton

In Defense of Thoughtful Multiple Choice Conceptual

Assessment

In Defense of Thoughtful Multiple Choice Conceptual

Assessment





R.K. Thornton

Modest Suggestions from a Chemically

Illiterate Physicist.

Modest Suggestions from a Chemically

Illiterate Physicist.





R.K. Thornton

Center for Science and Math TeachingTufts University

Educational Research

Computer ToolDevelopment

Curriculum Development

Teacher & ProfessorEducation

R.K. Thornton

FundingFundingFundingFunding

NSF National Science Foundation

FIPSE Fund for the Improvement of Post Secondary Education

US Department of Education

R.K. Thornton

Wouldn’t it be nice if teachers could understand what students know from a simple conceptual evaluation?

Wouldn’t it be nice if teachers could understand what students know from a simple conceptual evaluation?

and they knew what to do and they knew what to do to help the student to help the student

learnlearn

R.K. Thornton

What use might this talk be?What use might this talk be? If you intend to If you intend to developdevelop a a chemistry concept inventory chemistry concept inventory these suggestions may help these suggestions may help you make it more useful.you make it more useful.

If you intend to If you intend to useuse a a chemistry concept inventory chemistry concept inventory these ideas should help you these ideas should help you pick a useful one.pick a useful one.

R.K. Thornton

We have spent yearsWe have spent years Creating effective learning environments for introductory science(physics) courses (curricula, tools, pedagogical methods, group structures)

And developing methods of conceptual evaluation to measure student learning and guide our progress.

R.K. Thornton

Why Multiple Choice?Why Multiple Choice? More easily administered to large numbers of More easily administered to large numbers of students.students.

Evaluation takes less time.Evaluation takes less time. Student responses can be reliably evaluated even Student responses can be reliably evaluated even by the inexperienced.by the inexperienced.

Can be designed to guide instruction.Can be designed to guide instruction. With proper construction, student views can be With proper construction, student views can be evaluated from the pattern of answers, changes evaluated from the pattern of answers, changes over time can be seen, frequency of student views over time can be seen, frequency of student views can be measured. can be measured.

Multiple choice combined with open response can Multiple choice combined with open response can help the teacher/researcher explicate the help the teacher/researcher explicate the students response. students response.

R.K. Thornton

Why not?Why not?

Every “good” educator knows multiple Every “good” educator knows multiple choice questions are no good.choice questions are no good.

Badly constructed multiple choice Badly constructed multiple choice can give misleading results.can give misleading results.

Unless very carefully constructed, Unless very carefully constructed, multiple choice will not identify multiple choice will not identify student thinking.student thinking.

The choices may be inappropriate The choices may be inappropriate when used with different audienceswhen used with different audiences

R.K. Thornton

First stepsFirst steps

Why do you want to make (use) Why do you want to make (use) a conceptual evaluation?a conceptual evaluation?

In what conceptual area do In what conceptual area do you want to know how students you want to know how students think?think?

R.K. Thornton

Why?Why?

There are pre-requisite areas There are pre-requisite areas of conceptual knowledge that of conceptual knowledge that students need to know to students need to know to actually understand actually understand chemistry.chemistry.

R.K. Thornton

What? Three modest suggestions.What? Three modest suggestions. Explore student beliefs in the atomic Explore student beliefs in the atomic nature of matter. (students may say nature of matter. (students may say atoms exist but few believe it in any atoms exist but few believe it in any functional matter)functional matter)

Explore student beliefs the dynamic Explore student beliefs the dynamic nature of equilibrium. (Most students nature of equilibrium. (Most students seem to have a static model)seem to have a static model)

Explore student beliefs about the Explore student beliefs about the difference between heat energy and difference between heat energy and temperature. (Most students do not temperature. (Most students do not clearly make this distinction.)clearly make this distinction.)

R.K. Thornton

Our research has shown.Our research has shown.

Student conceptual responses can be context Student conceptual responses can be context dependent.dependent.

Student domains of applicability can be Student domains of applicability can be different from those of a scientist.different from those of a scientist.

Students (and scientists) can hold apparently Students (and scientists) can hold apparently inconsistent views simultaneously. (and it inconsistent views simultaneously. (and it doesn’t mean they are stupid.)doesn’t mean they are stupid.)

Conceptual transitions are not instantaneous.Conceptual transitions are not instantaneous. There is statistical evidence of a hierarchy There is statistical evidence of a hierarchy of student conceptual views.of student conceptual views.

You can do more with large-scale conceptual You can do more with large-scale conceptual evaluation than just generating a single evaluation than just generating a single number.number.

R.K. Thornton

Good Practice for the Construction of Conceptual Multiple Choice

Good Practice for the Construction of Conceptual Multiple Choice All answers, "right or wrong," should All answers, "right or wrong," should help evaluate student views. help evaluate student views.

Derive the choices in the questions from Derive the choices in the questions from from student answers to free response from student answers to free response questions and from student interviews.questions and from student interviews.

Check to see students almost always find Check to see students almost always find an answer that they are satisfied with. an answer that they are satisfied with. Random answers should be few.Random answers should be few.

Ask similar questions in different Ask similar questions in different representations.representations.

Check results with different student Check results with different student populations.populations.

(more)(more)

R.K. Thornton

Good Practice (continued)Good Practice (continued) Look at correlations among questions and use Look at correlations among questions and use patterns to understand student thinking.patterns to understand student thinking.

Understand the implications of “correct” and Understand the implications of “correct” and “incorrect” answers to their performance on “incorrect” answers to their performance on other tasks.other tasks.

Check for gender differencesCheck for gender differences Identify circumstances for “false positive” Identify circumstances for “false positive” answersanswers

If at all possible, construct the evaluation If at all possible, construct the evaluation so it is useful to guide instruction. so it is useful to guide instruction.

R.K. Thornton

Multiple Choice Conceptual EvaluationMultiple Choice Conceptual Evaluation

Conceptual evaluation for Conceptual evaluation for kinematics (description of motion) andkinematics (description of motion) and dynamics (force and motion which is well dynamics (force and motion which is well characterized by Newton’s Laws). characterized by Newton’s Laws).

Force & Motion Conceptual Evaluation (FMCE)Force & Motion Conceptual Evaluation (FMCE) Conceptual evaluation for heat energy and Conceptual evaluation for heat energy and temperaturetemperature

The Heat and Temperature Conceptual Evaluation The Heat and Temperature Conceptual Evaluation (HTCE)(HTCE)

Both developed by the Center for Science and Math Both developed by the Center for Science and Math Teaching at TuftsTeaching at Tufts

R.K. Thornton

Using the FMCE as an exampleUsing the FMCE as an example Student answers correlate well (well above Student answers correlate well (well above 90%) with written short answers in which 90%) with written short answers in which students explain the reason for their choicesstudents explain the reason for their choices

Almost all students pick choices that we can Almost all students pick choices that we can associate with a relatively small number of associate with a relatively small number of student models. student models. (Conceptual Dynamics, R.K. (Conceptual Dynamics, R.K. Thornton in ICUPE proceedings edited by Redish)Thornton in ICUPE proceedings edited by Redish)

Testing with smaller student samples shows Testing with smaller student samples shows that those who can pick the “correct” graph that those who can pick the “correct” graph under these circumstances are almost equally under these circumstances are almost equally successful at drawing the graph correctly successful at drawing the graph correctly without being presented with choices.without being presented with choices.

R.K. Thornton

FMCE as exampleFMCE as example Because we are able to identify statistically Because we are able to identify statistically most student views from the pattern of answers most student views from the pattern of answers (and because there are very few random answers), (and because there are very few random answers), we are also able to identify students with less we are also able to identify students with less common beliefs about motion and follow up with common beliefs about motion and follow up with opportunities for interviews or open-ended opportunities for interviews or open-ended responses to help us understand student thinking.responses to help us understand student thinking.

The use of an easily administered and robust The use of an easily administered and robust multiple choice test has also allowed us and multiple choice test has also allowed us and others to track changes in student views of others to track changes in student views of dynamics and to separate the effects of various dynamics and to separate the effects of various curricular changes on student learning.curricular changes on student learning.

R.K. Thornton

FMCE as exampleFMCE as example Use multiple representations

The Force Graph questions The Force Graph questions require explicit knowledge of require explicit knowledge of coordinate systems and graphs coordinate systems and graphs but require little reading. but require little reading.

The Force Sled questions use The Force Sled questions use natural language and make no natural language and make no explicit reference to a explicit reference to a coordinate system or graphs.coordinate system or graphs.

R.K. Thornton

A sled on ice moves in the ways described in questions 1-7 below. Friction is so smallthat it can be ignored. A person wearing spiked shoes standing on the ice can apply aforce to the sled and push it along the ice. Choose the one force (A through G) whichwould keep the sled moving as described in each statement below.

You may use a choice more than once or not at all but choose only one answer for eachblank. If you think that none is correct, answer choice J.

E.

F.

G.

The force is toward the left and is

decreasing in strength (magnitude).

The force is toward the left and is of

constant strength (magnitude).

The force is toward the left and is

increasing in strength (magnitude).

Direction of Force

A.

B.

C.

The force is toward the right and is

increasing in strength (magnitude).

The force is toward the right and is of

constant strength (magnitude).

The force is toward the right and is

decreasing in strength (magnitude).

Direction of Force

D. No applied force is needed

1. Which force would keep the sled moving toward the right and speeding up ata steady rate (constant acceleration)?

2. Which force would keep the sled moving toward the right at a steady(constant) velocity?

3. The sled is moving toward the right. Which force would slow it down at asteady rate (constant acceleration)?

4. Which force would keep the sled moving toward the left and speeding up at asteady rate (constant acceleration)?

5. The sled was started from rest and pushed until it reached a steady (constant)velocity toward the right. Which force would keep the sled moving at thisvelocity?

6. The sled is slowing down at a steady rate and has an acceleration to the right.Which force would account for this motion?

7. The sled is moving toward the left. Which force would slow it down at asteady rate (constant acceleration)?

R.K. Thornton

Time

F

o

r

c

e

+

-

0

Time

F

o

r

c

e

+

-

0

Time

F

o

r

c

e

+

-

0

Time

F

o

r

c

e

+

-

0

Time

F

o

r

c

e

+

-

0

Time

A

B

C

D

E

F

GF

o

r

c

e

Time

F

o

r

c

e

+

-

0

Questions 14-21 refer to a toy car which

can move to the right or left along a

horizontal line (the positive part of the

distance axis).

+0

Assume that friction is so small that it

can be ignored.

You may use a choice more than once

or not at all. If you think that none is

correct, answer choice . J

A force is applied to the car. Choose the

one force graph ( through ) for each

statement below which could allow the

described motion of the car to continue.

A H

The car moves toward the right

(away from the origin) with a

steady (constant) velocity.

__14.

The car is at rest.__15.


and is speeding up at a steady rate

(constant acceleration).

__16.

The car moves toward the left

(toward the origin) with a steady

(constant) velocity.

__17.


and is slowing down at a steady rate


__18.

The car moves toward the left and

is speeding up at a steady rate


__19.

The car moves toward the right,

speeds up and then slows down.

__20.

The car was pushed toward the

right and then released. Which

graph describes the force after

the car is released.

__21.

None of these graphs is correct.J

F

o

r

c

e

+

-

0

Time

H

+

-

0

R.K. Thornton

Comparison with short answerComparison with short answer As with all the questions on the test As with all the questions on the test students who answered correctly were students who answered correctly were also able to describe in words why also able to describe in words why they picked the answers they did.they picked the answers they did.

Statistically one of the last Statistically one of the last questions to be answered in a questions to be answered in a Newtonian manner is the force on a Newtonian manner is the force on a cart rolling up a ramp as it reverses cart rolling up a ramp as it reverses direction at the top (question 9).direction at the top (question 9).

R.K. Thornton

Back to best practices. ConsiderBack to best practices. Consider All answers, "right or wrong," should All answers, "right or wrong," should help evaluate student views. help evaluate student views.

Derive the choices in the questions from Derive the choices in the questions from from student answers to free response from student answers to free response questions and from student interviews.questions and from student interviews.

Check to see students almost always find Check to see students almost always find an answer that they are satisfied with. an answer that they are satisfied with. Random answers should be few.Random answers should be few.

Look at correlations among questions and Look at correlations among questions and use patterns to understand student use patterns to understand student thinking.thinking.

R.K. Thornton

An example from the H&T Conceptual Evaluation

An example from the H&T Conceptual Evaluation Distinguishes different Distinguishes different student models for the student models for the relationship between heat and relationship between heat and temperature.temperature.

•

Room

Temperature

25°C

A B

100 g

45°C 50°C

80 g

Room

Temperature

25°C

A B

100 g 200 g

75°C50°C

Room

Temperature

25°C

A B

100 g

45°C 90°C

50 g

B

90°C

50 g

Oven

Temperature

70°C

Refrigerator

Temperature

10°C

A

100 g

20°C

____4. Cup A contains 100 grams of water and is initially at 10°C in a refrigerator. Cup A is heated until its temperature is 20°C.

Cup B contains 50 grams of water initially at 70°C in an oven. Cup B is heated until its temperature is 90°C. Which cup had more heat energy transferred to it?

____1. Cup A contains 100 grams of water and cup B contains twice as much water. The water in both cups was initially at room temperature. Cup A was heated to 75°C and cup B was heated to 50°C. Which cup had more heat energy transferred to it?

____2. Cup A contains 100 grams of water and cup B contains 50 grams of water. The water in both cups was initially at room temperature. Cup A was then heated to 45°C and cup B was heated to 90°C. Which cup had more heat energy transferred to it?

____3. Cup A contains 100 grams of water and cup B contains 80 grams of water. The water in both cups was initially at room temperature. Cup A was then heated to 45°C and cup B was heated to 50°C. Which cup had more heat energy transferred to it?

Questions 1 through 4 refer to two cups, A and B, which contain different amounts of water. The water in each cup is heated as described. In questions 1 through 3 the cups are in a room where the temperature is 25 °C. In question 4 the cups are in different environments. For each question choose one of the four answers A through D.

A) Cup A ha d more heat energy transferred

B) Cup B had more heat energy transferred

C) Both cups had the same amount of heat energy transferred

D) not enough information is given to determine the answer

R.K. Thornton

Results by categoryResults by category

R.K. Thornton

What about 1 number resultsWhat about 1 number results Not my favorite, but useful Not my favorite, but useful in some situationsin some situations

Let’s compare the performance Let’s compare the performance of 350 RPI students in the of 350 RPI students in the beginning physics course on beginning physics course on the FMCE and the FCIthe FMCE and the FCI

R.K. Thornton

Comparison of FMCE Gains

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Tufts Agebra + ILDs 1994, 1996, 1997 (N=325)

Oregon Algebra RTP labs F1991-94, Pre from 1989 (N=613)

Oregon Algebra + ILDs F1991, Pre from 1989 (N=79)

Dickinson Workshop Physics F1999-2000 (N=104)

Joliet Junior College Calculus RTP labs1997-2003 (N=199)

CU Calc +Peer & UW Tutorial S2004 (N=391)

Muhlenberg Col. Calculus + ILDs F1997 (N=87)

RPI Studio Physics + ILDs S1999 (N=311)

Mt. Ararat H.S. ILDs S1998 (N=33)

Sydney Calculus + ILDs 1999 (N=60)

RPI Studio Physics S1998 (N=145)

Sydney Traditional Calculus 1995 (N=472)

SUNY Albany Traditional Calculus F1998 (N=73)

Oregon Traditional Algebra 1988-1989 (N=236)

<g> (% Normalized Gain) .

Comparison of FMCE Gains

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Tufts Agebra + ILDs 1994, 1996, 1997 (N=325)

Oregon Algebra RTP labs F1991-94, Pre from 1989 (N=613)

Oregon Algebra + ILDs F1991, Pre from 1989 (N=79)

Dickinson Workshop Physics F1999-2000 (N=104)

Joliet Junior College Calculus RTP labs1997-2003 (N=199)

CU Calc +Peer & UW Tutorial S2004 (N=391)

Muhlenberg Col. Calculus + ILDs F1997 (N=87)

RPI Studio Physics + ILDs S1999 (N=311)

Mt. Ararat H.S. ILDs S1998 (N=33)

Sydney Calculus + ILDs 1999 (N=60)

RPI Studio Physics S1998 (N=145)

Sydney Traditional Calculus 1995 (N=472)

SUNY Albany Traditional Calculus F1998 (N=73)

Oregon Traditional Algebra 1988-1989 (N=236)

<g> (% Normalized Gain) .

R.K. Thornton

Still one numberStill one number

Let’s compare the performance Let’s compare the performance of 350 RPI students in the of 350 RPI students in the beginning physics course on beginning physics course on the FMCE and the FCIthe FMCE and the FCI

R.K. Thornton

Correlation between FCI Scores and FMCE ScoresRPI Studio Physics Pre-Instruction (N=361)

0.00

0.20

0.40

0.60

0.80

1.00

0.00 0.20 0.40 0.60 0.80 1.00

FMCE fractional score

Correlation Coefficient 0.791

R.K. Thornton

Correlation between FCI Scores and FMCE ScoresRPI Studio Physics Post-Instruction (N=350)

0.00

0.20

0.40

0.60

0.80

1.00

0.00 0.20 0.40 0.60 0.80 1.00

FMCE fractional score

Correlation Coefficient 0.8309

R.K. Thornton

Are the evaluations the same?Are the evaluations the same? Yes? Very high correlations (about 0.8 pre and Yes? Very high correlations (about 0.8 pre and

post with different instructional methods)post with different instructional methods) Yes? A high score on one implies a high score on Yes? A high score on one implies a high score on the other.the other.

No? FCI fractional scores are almost always No? FCI fractional scores are almost always higher than FMCE scoreshigher than FMCE scores

No? Evaluations are measuring different thingsNo? Evaluations are measuring different things No? A low score on the FMCE (non-Newtonian No? A low score on the FMCE (non-Newtonian student) does not imply a low score on the FCIstudent) does not imply a low score on the FCI

Lets look at a group of non-Newtonian studentsLets look at a group of non-Newtonian students

R.K. Thornton

FMCE Conceptual Categories for Low FMCE/High FCI StudentsFMCE<0.4, FCI≥0.6 (N=54)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Velocity Acceleration Coin Acc 1st & 2nd (nl) 1st & 2nd (g) Coin Toss Cart Ramp 3rd Contact 3rd Collision

R.K. Thornton

The conceptual threshold effect

(looking at pre-post correlations)

The conceptual threshold effect

(looking at pre-post correlations)

R.K. Thornton

Tufts University Calculus-based Physics (N=181)FMCE Post vs. Pre

0.00

0.20

0.40

0.60

0.80

1.00

0.00 0.20 0.40 0.60 0.80 1.00

Before Instruction

Spring 1994 (N=48)

Spring 1995 (N=37)

Spring 1997 (N=43)

Spring 1998 (N=53)

Pre/Post Evaluation--The Threshold EffectPre/Post Evaluation--The Threshold Effect

R.K. Thornton

University Physics Courses Before Instruction

University Physics Courses Before Instruction

100806040200

Velocity

Acceleration

Force

Before Instruction

Average College and University Results

% of Students Understanding Concepts

R.K. Thornton

100806040200

Velocity

Acceleration

ForceBefore Instruction

After Traditional Instruction



University Physics Courses After Normal Instruction

University Physics Courses After Normal Instruction

R.K. Thornton

We have evidence of substantial, persistent We have evidence of substantial, persistent learning of such physical concepts by a learning of such physical concepts by a large number of students in varied large number of students in varied contexts in courses and laboratories that contexts in courses and laboratories that use methods I am about to describe.use methods I am about to describe.

Such methods also work for students who have Such methods also work for students who have traditionally had less success in physics traditionally had less success in physics and science courses: women and girls, and science courses: women and girls, minority students, and those who are badly minority students, and those who are badly preparedprepared. .

Physics & Science Courses Physics & Science Courses Using New MethodsUsing New Methods

Physics & Science Courses Physics & Science Courses Using New MethodsUsing New Methods

R.K. Thornton

University Physics Courses University Physics Courses

After New MethodsAfter New MethodsUniversity Physics Courses University Physics Courses

After New MethodsAfter New Methods

100806040200

Velocity

Acceleration

ForceAfter New Methods



Before Instruction

After Traditional Instruc.

R.K. Thornton

“I still don’t have all of the answers, but I’m beginning to ask

the right questions.”

Our Instructional and Assessment Philosophy

Documents

R.K. Thornton Effective methods for the use, creation, analysis, and interpretation of short- answer student conceptual evaluations. Ronald Thornton Professor