Embed Size (px)
Citation preview
David EubanksDirector of Planning, Assessment, and Information Services
Coker CollegeHartsville, South Carolina
Assessing The
Elephant
map
G oal Test
C om plexityDialogue
A ssessing FA CS
The Goal
The College's goal is to graduate students with the ability to think analytically and
creatively, and to write and speak effectively.
The Unmoved Mover
[I]t is imperative […] to avoid narrow definitions of student learning or excessively standardized measures of student achievement.
The Unmoved Mover
Collegiate learning is complex, and the evidence used to investigate it must be similarly authentic and contextual.
The Unmoved Mover
But to pass the test of public credibility […] the evidence of student learning outcomes used in the accreditation process must be rigorous, reliable, and understandable.
All CHEA quotes are from: http://www.chea.org/pdf/StmntStudentLearningOutcomes9-03.pdf
public credibility
map
G oal Test
C om plexityDialogue
A ssessing FA CS
The question
The question
“Can Stanislav write effectively?”
should be answered in language that is rigorous, reliable, and understandable.
“…the meaning of a word is its usage in the language…”
-Wittgenstein
language
Context and perspective
A
B
C
D
map
G oal Test
C om plexityDialogue
A ssessing FA CS
Power of dialogue
Example: If we want to distinguish between 128 possibilities (where n = 7),
• Monologue requires 127 questions;
• Dialogue requires 7.
map
G oal Test
C om plexityDialogue
A ssessing FA CS
Complexity vs. reliability
Example: arithmetic
13 + 5 = 18 ?
This can be assessed with high reliability and high validity because it’s algorithmic.
Example: Turing Test
The Turing Test for Machine Intelligence:
If a computer, on the basis of its written replies to questions, could not be distinguished from a human respondent, then ‘fair play’ would oblige one to say that it must be ‘thinking’.
-Quoted from Alan Turing: The Enigma, by Andrew Hodges
map
G oal Test
C om plexityDialogue
A ssessing FA CS
Assessment as a Tire gauge
We can all agree that the tire pressure is about 30 psi.
Assessment as telescope
You don’t know what you’ll discover when you build the telescope, but we can create language to describe it.
map
G oal Test
C om plexityDialogue
A ssessing FA CS
Just The Facs, ma’am
The Faculty Assessment of Core Skills (FACS)
• Mission-based (effective writing, etc.)
• Rubrics in course syllabi
• Evaluation of authentic evidence each term
• Aggregation of results into group perception
features
• Uses authentic data in context
• Uses full perceptive ability of experts
• Fosters dialogue among students, faculty
• Creates a “Wittgensteinlich” language
• Has high prima fascia validity
• Has measurable reliability
• Is “low stakes”, reducing rater bias
Web survey
2003-2006 data
• 11,770 responses
• 1743 of 2054 traditional students (85%)
• Correlation with grades = .64
More Validity
Writing by Class and Library Use
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
Fall 05 Fall 04 Fall 03 Fall 02
Start Term
Avg
. W
riti
ng
Sco
re
> Library
< Library
Inter-rater Reliability
Analytical 50% on 820 studentsCreative 53% on 691 studentsWriting 50% on 799 studentsSpeaking 54% on 646 students
Probability of matching ratings between two raters for same student and skill:
Longitudinal DATA
Core Skills Over Four Years
1.0
1.2
1.4
1.6
1.8
2.0
2.2
1 2 3 4 5 6 7 8
Semester
Assessed
Level
Writing
Speaking
THE matthew effect
Distributions
2005-6 FACS Average Distribution
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Performance level
Fra
ctio
n o
f cl
ass
Fresh
Soph
Jr
Sr
Class effectiveness
Effective Speaking
-0.20
-0.10
0.00
0.10
0.20
0.30
0.40
0.50
0.60
-4 -3 -2 -1 0 1 2 3 4 5
COM 101
Other
Student/Advisor Report
AvgOfA AvgOfC AvgOfW AvgOfS1.29 1.25 1.14 1.50
Student has 104 Credits (about seven semesters)
Comparison group averages:
1.9, 1.8, 1.8, 1.9, respectively.
Program report
FEEDBACK AS Assessment
Contact Terri Flateby at [email protected]
Is this effective writing?
