Upload
simon-bates
View
1.093
Download
0
Embed Size (px)
DESCRIPTION
Citation preview
James Bond, internetmemes and partyingpenguins
(or, what happens when students write their own assessment content)
Simon BatesPearson Strategies for Success WorkshopToronto, May 2013
2
3
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
The University of EdinburghEdinburgh, Scotland
5th July, 2010
Paul Denny
PeerWisebridging the gap between online learning and social media
Department of Computer ScienceThe University of AucklandNew Zealand
• Web-based Multiple Choice Question repository built by students
• Students:– develop new questions with
associated explanations– answer existing questions and rate
them for quality and difficulty– take part in discussions– can follow other authors
peerwise.cs.auckland.ac.nz
>100,000>100,000
student contributors
>500,000>500,000unique questions
>10,000,000>10,000,000answers
14
As a question author…..
15
16
19
As a question answerer …..
20
21
22
23
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
Timeline
2010-11: UoE pilot study
2011-12: Multi-institution, multi-course
2012-13: UBC PHYS 101
Coursera MOOC
25
Pilot year (2010-11) – replace single handin
PeerWise was introduced
in workshop sessions
in Week 5
Students worked through
structured example task
and devised own Qs in groups.
All these resources are available online (see final slide)
26
An assessment was set for the end of
Week 6:
Minimum requirements:
• Write one question• Answer 5• Comment on & rate 3
Contributed ~3% to course assessment
27
28
We were deliberately
hands off.
• No moderation
• No corrections
• No interventions at all
But we did observe…..
29
• JISC project – SGC4L
N (students) ~800N (staff) ~10
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
Generally, students did
• Participate beyond minimum requirements
• Engage in community learning, correcting errors
• Create problems, not exercises
• Provide positive feedback
34
35
Generally, students did not
• Contribute trivial or irrelevant questions
• Obviously plagiarise
• Participate much beyond assessment periods
• Didn’t all leave it to the last minute
36
• Phys 101 uptake graph showing midterm
39
40
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
Correlation with end of course outcomes
Quartiles
Q4 – top 25%
Q3 – upper middle
Q2 – lower middle
Q1 – bottom 25%
22 students did not take the FCI
1st year Physics N=172University of Edinburgh
1st year Physics N=172University of Edinburgh
Overview
I. Motivation
II. Technology enabler: PeerWise
III. Use cases
IV. Engagement, learning, question quality?
Comprehensive categorisation of >50% of repository for two successive academic years
Principal measures to define a ‘high quality question’
- cognitive level of question- explanation quality- other binary criteria
Category Description 6 Create (synthesise ideas) 5 Assess 4 Analyse (multi-step) 3 Apply (1-step calcs.) 2 Understand 1 Remember
Cognitive level of question
Explanation
0 – Missing
1 – Inadequate(e.g. wrong reasoning / answer, trivial, flippant, unhelpful)
2 – Minimal (e.g. correct answer, but with insufficient explanation or justification, aspects may be unclear)
3 – Good/Detailed(e.g. clear and sufficiently detailed exposition of correct method and answer)
4 – Excellent (e.g. Describes physics thoroughly, remarks on plausibility of answer, use of appropriate diagrams, perhaps explains reasoning for distractors)
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
1 2 3 4 5 6
Taxonomic Category
Per
cen
tag
e o
f S
ub
mit
ted
Qu
esti
on
s
First semesterFirst semester N = 350 N = 350
Second semester N = 252Second semester N = 252
Results: Question level Physics 1A / 1B 2011
Results: Explanation Physics 1A 2010 and 2011
‘High quality’ question
1. At least 2/6 on cognitive level (“understand” and above)
2. At least 2/4 on explanation (“minimal” and above)
3. Clearly worded question (binary)
4. Feasible distractors (binary)
5. ‘Most likely’ correct (binary)
6. ‘Not obviously’ plagiarised (binary)
Results: Physics 1A 2010 and 2011
2 successive years of the same course (N=150, 350)
•‘High quality’ questions: 78%, 79%
•Over 90% (most likely) correct, and 3/5 of those wrong were identified by students.
•69% (2010) and 55% (2011) rated 3 or 4 for explanations
•Only 2% (2010) and 4% (2011) rated 1/ 6 for taxonomic level.
Literature
Bottomley & Denny Biochem and Mol Biol Educ. 39(5) 352-361 (2011)
•107 Year 2 biochem students •56 / 35 / 9 % of questions in lowest 3 levels.
Momsen et al CBE-Life Sci Educ 9, 436-440 (2010)
“9,713 assessment items submitted by 50 instructors in the United States reported that 93% of the questions asked on examinations in introductory biology courses were at the lowest two levels of the revised Bloom’s taxonomy”
• High general standard of engagement and student-generated questions
• Relatively few basic knowledge questions
• Transferable across disciplines / institutions
• We hypothesise scaffolding activities are critical for high level cognitive engagement
Summary
Acknowledgements
Physics 101 course team
Georg RiegerFiras MoosviEmily Altiere UBC CWSEI
@simonpbates
Ross GallowayJudy HardyKaron McBrideAlison KayKeith BruntonJonathan RiiseDanny Homer
Chemistry – Peter Kirsop
Biology – Heather McQueen
Physics – Morag Casey
Comp Sci – Paul Denny
Community: http://www.PeerWise-Community.org
JISC-funded multi institution study:https://www.wiki.ed.ac.uk/display/SGC4L/Home
UoE Physics Pilot Study: AIP Conf. Proc. 1413, 359 http://dx.doi.org/10.1063/1.3680069
RSC overview article
http://www.rsc.org/Education/EiC/issues/2013January/student-generated-assessment.asp
UoE Physics scaffolding resources http://www2.ph.ed.ac.uk/elearning/projects/peerwise/
Resources
Question quality analysis (1st year Physics University of Edinburgh)
Assessing the quality of a student-generated question repository, submitted to Phys Rev, ST Phys Educ Res.
Multi-institution, multi-course study
Student-generated content: Enhancing learning through sharing multiple-choice questions, submitted to
International Journal of Science Education
Scaffolding Student Learning via Online Peer Learning, submitted to International Journal of Science Education
Publications in preparation / review / press
Cop
yrig
ht 2
01
3 G
raha
m F
ow
ell /
Th
e H
itma
n,
re-p
rodu
ced
with
per
mis
sio
n,
Edu
catio
n In
Che
mis
try,
V
ol 5
0 N
o 1
(20
13
)
Photo credits
Photo credits
Community: http://www.flickr.com/photos/kubina/471164507/
Screen grab from Mwensch ‘A vision of students today’ http://www.youtube.com/watch?v=dGCJ46vyR9o
65