View
260
Download
2
Category
Tags:
Preview:
DESCRIPTION
The second keynote I delivered at #eas14 continued the theme of Brookfield's lenses of reflection, but this time focussed on Assessment Analytics. The two examples identified were from the LIFTUPP project at Liverpool's Dental School, as well as a pilot from Liverpool's Medical School in using data from Turnitin and Grademark to analyse staff and student performance in written assignments.
Citation preview
VIEWING SUMMATIVE ASSESSMENT
THROUGH DIFFERENT LENSESPeter Reed
eAssessment Scotland Keynote Presentation
SELF REFLECTION
STUDENTS PEERS
CONTINUING THE THEME…
PROGRAMME SYSTEM(S)
LITERATURE
ANALYTICS
DATA CAPTURE
LIFTUPP TURNITIN
ANALYTICS @ LIVERPOOL
Curriculum Review – mapping to GDC domains
What should a Dentist be able to do upon qualification?
LIFTUPP
1 2 3 4 5 6
WORKING BELOW EXPECTED LEVEL
WORKING AT EXPECTED LEVEL
WORKING ABOVE EXPECTED LEVEL
DEVELOPMENTAL INDICATORS
STUDENT VIEW
Particular domain
STUDENT VIEW
Skills within domain & development indicator
STUDENT VIEW
Areas of the mouth & procedures
STUDENT VIEW
Materials used
STUDENT VIEW
STAFF VIEW
GENERATED DATA
TURNITIN / GRADEMARK
Pilot in School of Medicine SSMs
• Tutor name included in submission title;• SSMs marked against 7 objectives (intro, methodology, etc);• Objectives marked as Unsatisfactory (1), Fair (2), Good (3), Excellent (4);• Total possible marks was 7 * 4 = 28;• 6 Markers & 36 Student assignments;
TURNITIN / GRADEMARK
Unsatisfactory Fair Good Excellent0
20
40
60
80
100
120
Count of marks awarded
Coun
t
TURNITIN / GRADEMARK
Ob 1 Ob 2 Ob 3 Ob 4 Ob 5 Ob 6 Ob 70
5
10
15
20
25
Frequency of Grades Awarded for Each Objective
UnsatisfactoryFairGoodExcellent
TURNITIN / GRADEMARK
Objective 1 Objective 2 Objective 3 Objective 4 Objective 5 Objective 6 Objective 70
20
40
60
80
100
120
140
Number of QM Comments Made by Objective
TURNITIN / GRADEMARK
Marker 1 Marker 2 Marker 3 Marker 4 Marker 5 Marker 60
5
10
15
20
25
30
Mean Points awarded by marker
TURNITIN / GRADEMARK
1 2 3 4 5 6 70
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Mean Points Awarded per Objective, by Marker
Marker 1Marker 2Marker 3Marker 4Marker 5Marker 6
QUESTIONS
Is anybody doing anything with this information?
Should we?
Share with Staff? With Students?
Ethical considerations?
Hawthorne effect?
Predictive analytics?
Email Interviews - Do you think the data available from the system is useful and why?
EMAIL INTERVIEWS
“Yes, no Hawthorne for me, I didn't know marks were being assessed in that way. Helps reassure students and externals that we are monitoring for consistency.”
“I think it very useful as it will make it easy to identify conveners who mark outside normal parameters and identify who is giving feedback to the students and who isn't.”
EMAIL INTERVIEWS
“I’m not quite sure what I think about this. Obviously we’re all
interested in what our own performance is (although I’m not sure we
should be told!). Major inconsistencies between markers or between
objectives would be worrying, but I don’t think we have a solid basis
for expecting any particular result – to put it crudely, would ‘all
excellent’ be a good thing (great students and course-management)
or bad (slack marking)?”
ASSESSMENT ANALYTICS
For the Greater Good or Big Brother?
Recommended