Upload
frederica-stewart
View
222
Download
1
Tags:
Embed Size (px)
Citation preview
Effective Methods for Educational Research
The value of prototyping, observing, engaging users
Diana Laurillard
Methods for educational software designAn iterative design-test-analyse-redesign approachThe focus is achievement of learning outcomesAlign outcomes – activities – assessment• Discover requirements – design – design tests• Prototype• Observe• Test• Engage• Evaluate
Prototyping
• Sketch the design idea• Design user tasks– Use materials to represent choices and actions– Allow users to construct the choices and actions
• Data collection– 6 target users (min)– User constructions (materials)– Users’ commentary (audio, observation notes)– Video (usually unnecessary)
A paper prototype
Proposed screen design
Proposed actions
A paper prototype
User defined actions
A paper prototype
Observing
• Individual– Observation notes– ‘Stimulated recall’ (Bloom1953) – video or
playback to act as a memory aide – Exercise: ‘what were you thinking at this point?’– Audio + notes
• Pairs – listen to them talking, audio + notes• Recording of screen activities • Video
Data capture and analysis tools
To capture screen activities
To capture eye movements, logged interactions, and for eye-tracking data analysis
To coordinate and analyse the multiple streams of data
User engagement
• Workshops – with frequent discussions• Focus-group - question then facilitated
discussion• Survey – Closed questions + open comment• Construct elicitation interviews to discover
users’ own categories of description
Learning analytics
Measures that correlate with final grade• Total discussion messages posted (+ve)• Total number of online sessions (+ve)• Total time online (+ve)• Time spent on assignments (+ve)• Time spent on assessments (-ve)
(MacFadyen and Dawson, 2010)
LMS tracking those at risk
Ave grade = 72%Ave grade = 63%Ave grade = 57%Ave grade = 44%
(MacFadyen and Dawson, 2010)
Predicting those at risk
Week of course
Aver
age
hour
s on
line
per w
eek
Learning analytics on outcomesPedagogically-driven analytics• Define the ecology of experiences and
resources in which the software is embedded• Design appropriate assessment of outcomes • Use the software tasks to generate tests• Formulate the pre-post tests that will capture
the differences the software should make• Interpret outcomes for re-design
Working groups
• 1. What kinds of problems do you see with methods of prototyping, observing and engaging?
• 2. What might be the advantages of using such methods?
• 3. How might an educational software developer can draw on evidence-based research?