Embed Size (px)
Citation preview
WLE Centre for ExcellenceInstitute of EducationUniversity of London20 Bedford WayLondon WC1H 0AL
Tel +44 (0)20 7911 5531Fax +44 (0)7092 288 882Email [email protected] www.wlecentre.ac.uk
Formative e-assessment: case stories, design patterns, and future scenarios
Norbert Pachler, Caroline Daly, Harvey Mellar, Yishay Mor
Institute of Education, University of London
OverviewScoping study commissioned by JISC
Short term, small budget, intended to inform future funding frameworks
Established a committed user group of higher-education teachers & researchers
Adopted and adapted the Planet Project's Participatory Methodology for Practical Design Patterns, and used the Planet platform
MethodologyDesk research
−Literature review
−Frameworks in which to situate formative e-assessment
5 Practical Enquiry Days
−Combination of collaborative reflection, report back from team, and guest plenaries
−Launch day, 3 Planet workshops, developers' day
to develop a domain map for formative e-assessment
to carry out a review of relevant literature
to delineate a set of key processes involved in effective formative e-assessment practice through case studies and patterns
to scope a vision for formative e-assessment
to make recommendations for practice and policy making
Project aims
Issues from the literature – consensus? Formative (e)-assessment is concerned with learners making
progress towards measurable attributes/knowledge/skills/understanding
It is about working with mechanisms and practices which allow the gap to close between what they are currently able to achieve and what they might be able to achieve
Increasing learners’ active responsibility for their part in the learning process is a main feature across a range of contexts and technologies
…but not much further consensus…differences tend to focus around whether ‘assessment’ is treated as an ‘event’ or a ‘process’.
Teacher and learner rolesCox et al 2008 (practice-based element of dentistry):
'a feedback process that provides information that can be used to fine-tune or modify what has already been done‘….but by whom?
What if…
the teacher/tutor is ‘monitoring’ rather than ‘changing’?
assessment processes are purely for self-assessment?
the responses are automated?
FeedbackBull & McKenna 2004 (computer-assisted assessment CAA)
‘…assessments which assist learning by giving feedback which indicates how the student is progressing in terms of knowledge, skills and understanding of a subject. In CAA this often takes the form of objective questions with feedback given to the student either during or immediately after the assessment. Formative assessment may be monitored by the tutor, used purely for self-assessment, or used to
contribute marks to a module grade’. (p. xiv) What if…
feedback is complex (Shute 2008), or a two-way or multi-way process?
- we contrast Bull with Nicol & Macfarlane-Dick’s 2006 criticisms of transmission-focused feedback:
−‘feedback messages are invariably complex and difficult to decipher’ in transmission contexts - students find it hard to take appropriate actions
−feedback tends to emphasis only cognition, not motivation and beliefs which are vital to internalization of knowledge and understanding
−transmission-based feedback is used in highly inventive ways, e.g. to direct learners within a carefully structured test environment towards appropriate pathways which enhance motivation and self-regulation (Winkley)
What types of knowledge, skills & understanding are to be included?
How do feedback patterns fit into broader conceptual frameworks for learning and teaching?
What is formative e-assessment?The contribution of more/faster/more frequent/
automated feedback to formative assessment (Conole & Warburton, 2005)??
The use of digital means to support formative assessment??
Formative features of assessment, which are afforded by specific features of digital media??
“An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the absence of that evidence”
(Dylan Wiliam)
Formative = feedback + moments of contingency
"... These create "moments of contingency," in which the direction of the instruction will depend on student responses. Teachers provide feedback that engages students, make time in class for students to work on improvement, and activate students as instructional resources for one another."
(Leahy, Lyon, Thompson, and Wiliam 2005)
Wiliam's 5 strategies
Conversational Framework (Laurillard)
Our Methodology
Focus on practitioner participation
5 practical enquiry days
+ on-line async. pre & post work
Case stories → design patterns → scenarios
15
Problem:
Bad Design
16
17
the limit on growth is not the capacity to produce, but the knowledge to do it right.
Problem: The Design Divide the gap between those who have the expertise to develop high-quality tools and resources and those who don’t (Mor & Winters, 2008*)
18
Solution...
Formative e-Assessment PED 5, Dec 2008
19
Design patterns
[describe] a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice(Alexander et al., 1977)
C o n t e x t
Problem Solution
Formative e-Assessment PED 5, Dec 2008
20
ProblemKeep the rain out
ContextCold, wet, poor.
Method of solutionThatched roof
RelatedTimber frame, Slanted roof,Chimney
Formative e-Assessment PED 5, Dec 2008
21
Design problemCommunity facilities scattered individually through the city do nothing for the life of the city.
Design solutionCreate nodes of activity throughout the community, spread about 300 yards apart.
http://www.uni-weimar.de/architektur/InfAR/lehre/Entwurf/Patterns/030/ca_030.html
example: activity nodes
22
Problem: acceleration
The world is changing. Fast. Faster.
Teachers are learners.
Students are researchers.
We are all designers of our own and our peer's learning experiences.
Son, this was my dad's mobile. I want you to have it.
23
24
Participatory Methodology for Practical Design Patterns
Problem
−Acceleration → need for effective protocols for sharing of design knowledge
Context
−interdisciplinary communities of practitioners engaged in collaborative reflection on a common theme of their practice.
−blended setting: co-located meetings + on-line collaborative authoring system.
25
Solution: a series of three* collaborative reflection workshops
Case Stories Workshop
−Engender collaborative reflection among practitioners by a structured process of sharing stories.
Pattern Mining Workshop
−Eliciting patterns by reflecting on and comparing case stories.
Future Scenarios Workshop
−Validating and enhancing patterns by applying them to novel problems.
27
Workshop I:
Sharing case stories
28
Problem: telling a good story is not so easy
Inexperienced story-tellers might -
−Take the context for granted
−Preach, apologise, market, or generalise
−Avoid inconvenient details
Interactive feedback should help, but peers might -
−Be reluctant to criticize
−Attribute misunderstanding to their own faults
−Loose attention
29
Three hats
Solution:
Three Hats
Some pointers for the case studies‘E-assessment…there is still much work to be done. One of the most important messages…is simply the variety and scope for imagining new forms of assessment – and of using technology to support it in imaginative ways’ (Whitelock and Watt 2008, p. 153)
‘Modernising assessment’ involves blurring the boundaries between formative and summative processes (Elliott 2008)
Rather than thinking in terms of ‘formative assessment’, it might be more appropriate to think in terms of how assessment can be used ‘formatively’ (Wiliam July 08)
HE under-graduate + post-16
selfinstant feedback to individual st
bespoke string comparator
accuracy of language items
String comparison
HE distance
tutor self-assess-ment
graphical feedback to tutors
web-based tool
tutor socio-emotive feedback
Open mentor
HE vet training WBL
self peer-peer t-st, t-group
reflection & multi-player feedback
social networking/ mob device
recording/ reflecting clinical exp.
Como: mobiles + flikr
HE under-graduate
t-stdeliver tutor feedback
Audacity & dicta-phones
concepts in sociology
Audiofiles
M-level ITE peer-peer t-st t-group
represent & share thinking
wikiacademic writing in teacher ed
Academic writing
Ca
ses
Asse
ssme
nt
focus
Te
chn
ology
use
d
Te
chn
ology
role
So
cio-pe
d
setting
Institutio
nal
setting
A few cases
Creature of the week
CoMo
Post 16 String Comparison
Open Mentor
...
Creature of the week (Judy Robertson)Situation
large class (138), first and second year computer science students. assignment: create a virtual pet in Second Life.
Task
Engage and motivate the students
show examples of good work which others could learn from
show students their work is valued.
build a sense of community.
http://purl.org/planet/Cases/creatureoftheweek
CoMo (Niall Winters, Yishay Mor)
Situation
Royal Vet College.
Hospital rotations as part of their training.
Task
Allow students to capture critical incidents in text and image.
Support sharing of clinical experiences and co-reflection.
http://purl.org/planet/Cases/CoMo
Post 16 string comparison (Aliy Fowler)
Situation Grammar school been piloting the ‘string comparison’ approach to
language teaching at post-16 for AS and A2 level students.
Sixth Form level, grammatical consolidation and whole-sentence translation.
Task
Allow students to practise written language independently and receive feedback on errors in order to improve their language skills.
http://purl.org/planet/Cases/Post16stringcomparison
Solution
A bespoke string (sequence) comparator was designed; uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error. If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received.
Open mentor (Denise Whitelock)
http://purl.org/planet/Cases/OpenMentor
A few patterns..
Try Once, Refine Once
Feedback on Feedback
Classroom display
Try Once, Refine Once
(Aliy Fowler)
http://pul.org/planet/Patterns/TryOnceRefineOnce
ProblemLack of immediate feedback for students leads to fossilisation of errors and misconceptions
providing immediate feedback in an iterative fashion can also hinder effective learning since students are able to "grope their way" step-by-step to a correct solution without necessarily having to think about each answer as a whole.
ContextClass size
Large (30-300)
Content
Skills \ facts
Mode of instruction
Blended / on-line. Computer tested.
Solution
Feedback on Feedback
(Linda McGuigan)
http://purl.org/planet/Patterns/FeedbackonFeedback
Good feedback should -
Alert learners to their weaknesses.Diagnose the causes and dynamics of these.Include operational suggestions to improve the learning experience.Address socio-emotive factors.
Tutors know this, but are pressed for time. Or not aware of their feedback strategies
Large teaching organisations are not equipped to provide tutors with personal feedback on their teaching
Problem
Context
Large scale, technology supported, graded courses
many tutors instructing many students.
Feedback is mediated by technology that allows it to be captured and processed in real time
Topic of study is subject to both grading and formative feedback.
SolutionEmbed a mechanism in the learning and teaching system
that regularly captures tutor feedback, analyses it, and presents them with graphical representation of the types of feedback they have given. Ideally, this should also include constructive advice as to how to shift from less to more effective forms.
In computer supported environments (e.g. VLEs), this mechanism could be integrated into the system, providing tutors with immediate analysis of their feedback, as well as long-term aggregates.
Classroom Display
http://purl.org/planet/Patterns/Classroomdisplay
Problem
Rewards participation.Relates to learner's personal experiences.Window on student conceptions.
Using learner generated Using learner generated content..content..
Needs to collate works in a single easy to access location.Learners uncomfortable about presenting their work in publicLegal or other restrictions on sharing work.
ContextClass size:
Small / medium (6-60)
Mode of instruction:
Blended (preferable)
Time frame
Continuous, over a period
Pedagogy
Involves construction / media production
Solution
Augmented domain map
Example scenarioWhen using Try Once Refine OnceTry Once Refine Once, there is a risk that
high-achievers do not receive feedback.
So -
Use Showcase Learning Showcase Learning to celebrate students’ work and provoke feedback from peers and tutors.
Use Feedback on Feedback Feedback on Feedback to alert tutors to the problem.
What does ‘e’ add to formative assessment?I. Speed Speed of response is often important in enabling feedback to have an effect Supports rapid iteration – in many cases the ability to give feedback quickly
means that the student’s next problem solving iteration can begin more quickly.
II. Storage capacity
Ability to access very large amounts of data (appropriate feedback/additional work/illustrations can be identified).
III. Processing Automation – in some situations the e-assessment system can analyse
responses automatically and provide appropriate feedback. Scalability – can often be the result of some level of automation. Adaptivity – systems can adapt to students.
IV. Communication
Often the advantage of the ‘e’ is that it enables rapid communication of ideas across a range of audiences, and the technology allows this range to be controlled it can be just one person, a group, a class or more
This communication aspect means that aspects of communication can be captured and given a degree of semi-permanence
Semi- permanence supports the sharing of intellectual objects.
V. Construction and representation Representation – the ability to represent ideas in a variety of ways and to move
and translate between these representations Technology can support learners in the construction of representations of their
own ideas.
VI. Mutability Shared objects are not fixed. They can be changed easily and quickly.
ConclusionsTip of the iceberg
Practitioners (educational / software) acknowledge the value of patterns, when served with side dishes of cases + scenarios
Collaborative elicitation of patterns from cases could be a potent form of professional development.
58
Thank youThe Formative e-Assessment project:
http://projects.lkl.ac.uk/feasst
Final report
http://telearn.noe-kaleidoscope.org/open-archive/browse?resource=1875
This event http://projects.lkl.ac.uk/feasst/april-28th/
This presentationhttp://www.slideshare.net/yish/formative-eassessment-a-scoping-study
