Learning-oriented Learning-oriented assessment and the assessment and the

development of student development of student learning capacitieslearning capacities

David CarlessDavid Carless

University of Hong Kong University of Hong Kong

OverviewOverview

• Introductory remarks Introductory remarks

• Student self-evaluation Student self-evaluation

• ““Learning-oriented assessment”Learning-oriented assessment”

• Exemplifying its 3 featuresExemplifying its 3 features

• Tensions in these processesTensions in these processes

• Implications for practice and future Implications for practice and future researchresearch

Hong Kong Hong Kong

The University of Hong The University of Hong Kong Kong

HKUHKU

Times Higher Times Higher Education Rankings Education Rankings 2010-112010-11

HKU ranked no. 1 HKU ranked no. 1 in Asia and 21st in Asia and 21st in the world.in the world.

English mediumEnglish medium

HKUStudent

population (2008-2009):

• 21,652– 11,962

(undergraduates)

– 7,326 (taught postgraduates)

– 2,364 (research postgraduates)

• 2,068 non-local students

The university comprises 10 faculties Faculties• Faculty of Architecture • Faculty of Arts • Faculty of Business and Economics • Faculty of Dentistry • Faculty of Education • Faculty of Engineering • Faculty of Law • Li Ka Shing Faculty of Medicine • Faculty of Science • Faculty of Social Sciences

English examination in Dongguan University of Technology Guangdong Province, July 9, 2007. The university authority organized 1200 students to take the examination in a large hall to prevent exam fraud.

From testing to productive student From testing to productive student learning: Implementing formative learning: Implementing formative assessment in Confucian-heritage assessment in Confucian-heritage settings settings

Three Wise MenThree Wise Men

Self-evaluation (Sadler, Self-evaluation (Sadler, 1989)1989)“ The possession of evaluative expertise is a necessary (but not sufficient) condition for improvement”“Instructional systems which do not make explicit provision for the acquisition of evaluative expertise are deficient”

(Sadler, 1989)

Sustainable assessment (Boud, Sustainable assessment (Boud, 2000)2000)

• Practices which meet Practices which meet students’ immediate students’ immediate assessment needs whilst assessment needs whilst not compromising the not compromising the knowledge, skills and knowledge, skills and dispositions they require dispositions they require to develop lifelong to develop lifelong learning capacities. learning capacities.

Dialogic feedback Dialogic feedback

• ““ Diverse expressions Diverse expressions of dissatisfaction with of dissatisfaction with written feedback can written feedback can be interpreted as be interpreted as symbols of symbols of impoverished and impoverished and fractured dialogue” fractured dialogue” (Nicol, 2010, p. 503). (Nicol, 2010, p. 503).

•Learning-oriented Learning-oriented assessment assessment

(Carless et al, 2006; Carless, (Carless et al, 2006; Carless, 2007)2007)

Learning-oriented Learning-oriented assessment assessment • Loa = assessment which places a primary Loa = assessment which places a primary

focus on improving student learning; focus on improving student learning;

• including summative assessment which including summative assessment which stimulates productive student learningstimulates productive student learning

(Carless et al. 2006; Carless, 2007)(Carless et al. 2006; Carless, 2007)

• Tensions in assessment given its multiple Tensions in assessment given its multiple functions (double duty, Boud, 2000)functions (double duty, Boud, 2000)

Three ProjectsThree Projects

Project 1: LOAP Project 1: LOAP (Learning-oriented (Learning-oriented assessment project)assessment project)

• 2002-2006 based in 2002-2006 based in Hong Kong Institute Hong Kong Institute of Education of Education

““How Assessment Supports How Assessment Supports Learning”Learning”

Collection of practicesCollection of practices

• 39 practices from undergraduate 39 practices from undergraduate education including commentaries education including commentaries and suggestions (Chapter 3 of the and suggestions (Chapter 3 of the book)book)

• Informed by learning-oriented Informed by learning-oriented assessment framework (Chapter 2) assessment framework (Chapter 2)

Project 2: SAFE projectProject 2: SAFE project

• Student Assessment and Feedback Student Assessment and Feedback Enhancement (SAFE)Enhancement (SAFE)

• Interviews with teachers about Interviews with teachers about feedback practicesfeedback practices

• Focus group interviews with students Focus group interviews with students

• Developmental work with colleagues Developmental work with colleagues on diversifying feedback practiceson diversifying feedback practices

Research MethodResearch Method

10 semi-structured interviews with 10 semi-structured interviews with award winners from each of the award winners from each of the ten Faculties of the universityten Faculties of the university

Project 3: IPSETProject 3: IPSET

• Interactive Pedagogy, Self-Evaluation Interactive Pedagogy, Self-Evaluation and Trust (IPSET)and Trust (IPSET)

• A case study of a “challenging” A case study of a “challenging” teacher identified from project 2 teacher identified from project 2

• Observations of two undergraduate Observations of two undergraduate courses in the Faculty of Businesscourses in the Faculty of Business

• Interviews with the teacher and Interviews with the teacher and studentsstudents

Three Components Three Components of Learning-of Learning-oriented oriented assessment assessment

Learning oriented Learning oriented assessment framework assessment framework

Learning oriented assessment

Assessment task design for

productive learning

Student involvement in peer feedback and self-

evaluation

Dialogic and/or Sustainable feedback

1. Task design principles 1. Task design principles

•Assessment tasks shouldAssessment tasks should- - capture sufficient study time and capture sufficient study time and

effort (Gibbs, 2006)effort (Gibbs, 2006)

- distribute student effort evenly across - distribute student effort evenly across topics and weeks (Gibbs, 2006)topics and weeks (Gibbs, 2006)

- stimulate complex learning (Knight, - stimulate complex learning (Knight, 2006).2006).

Example: course Example: course assessment assessment

• Case, class and blog discussion Case, class and blog discussion (40%)(40%)

• Individual written case assignments Individual written case assignments (30%)(30%)

• Group project: (30%) Group project: (30%) - Oral presentation Oral presentation - Written report 1 week laterWritten report 1 week later

Tensions in assessment Tensions in assessment designdesign• Reliability vs productive Reliability vs productive

learning learning

• Individual vs group Individual vs group

• Judgement vs measurementJudgement vs measurement

• Pre-set transparent criteria Pre-set transparent criteria vs emergent opaque criteriavs emergent opaque criteria

Teacher views on Teacher views on assessment assessment • ““ I don’t believe in exams, I I don’t believe in exams, I

think that is the worst thing a think that is the worst thing a university can do, they university can do, they encourage students to see encourage students to see education as just a product. education as just a product. Exams remove the whole Exams remove the whole process of learning and process of learning and discovery”.discovery”.

• ““The job of the teacher is to The job of the teacher is to come up with an assessment come up with an assessment method apart from exams” method apart from exams”

Student views on assessment Student views on assessment taskstasks

• ““ I like the assessment approach because I like the assessment approach because it promotes my critical and analytical it promotes my critical and analytical thinking skills”thinking skills”

• ““Your grade depends on his feeling, what Your grade depends on his feeling, what he thinks of you”he thinks of you”

• ““I am more advantaged by exams and I I am more advantaged by exams and I tend to get good grades in exams. I know tend to get good grades in exams. I know how to prepare for them, but for class how to prepare for them, but for class participation you can’t really prepare”participation you can’t really prepare”

Student views on blogStudent views on blog

• ““ Having some marks allocated to it, gives Having some marks allocated to it, gives some life to the blog”some life to the blog”

• ““I am worried about my grade, so I I am worried about my grade, so I actively participate in the blog”actively participate in the blog”

• ““A student set up a facebook group, A student set up a facebook group, because the blog will expire and he because the blog will expire and he wanted to go on discussing with people wanted to go on discussing with people interested in the topics”.interested in the topics”.

• Grading of blog: participation or insight?Grading of blog: participation or insight?

Learning oriented summative Learning oriented summative assessmentassessment

• Summative assessment needs to be Summative assessment needs to be characterized by experiment and characterized by experiment and innovation …to give space to innovation …to give space to individualityindividuality

• For students to embrace the For students to embrace the challenge of risk, uncertainty and challenge of risk, uncertainty and emotional responseemotional response

(Barnett, 2007)(Barnett, 2007)

2. Student involvement 2. Student involvement

• Participation in peer feedback and Participation in peer feedback and self-evaluationself-evaluation

• A process of engaging with quality in A process of engaging with quality in the disciplinethe discipline

Classroom atmosphereClassroom atmosphere• ““ He is very successful in building an He is very successful in building an

interactive relationship between him and interactive relationship between him and the students”the students”

• ““The whole intimacy of the course gives you The whole intimacy of the course gives you an environment where you can give an environment where you can give feedback more easily. One thing which feedback more easily. One thing which creates intimacy is the interactive teaching creates intimacy is the interactive teaching in which every single person really gives in which every single person really gives something during the class”something during the class”

Dialogic Dialogic FeedbackFeedback

Peer feedback and student self-Peer feedback and student self-evaluation of oral presentationsevaluation of oral presentations

I video-tape each student for five minutes … We I video-tape each student for five minutes … We show the video right after the presentation … show the video right after the presentation … Usually I get them to reflect first, “How do you Usually I get them to reflect first, “How do you think you did?” And then we give them feedback. think you did?” And then we give them feedback. I think they find it phenomenally useful because I think they find it phenomenally useful because they are able to give insightful analysis on their they are able to give insightful analysis on their own performance. … pictures don’t lie ... it’s own performance. … pictures don’t lie ... it’s effective, because they see the truth.effective, because they see the truth.

Challenge students intellectually and emotionally to develop their oral presentation skills

Student data: video Student data: video activityactivity

• ““ You have the feedback directly after You have the feedback directly after the presentation and you can see how the presentation and you can see how you are doing through other people’s you are doing through other people’s eyes”eyes”

• ““When I look at others presenting, I When I look at others presenting, I actually start noticing the things to actually start noticing the things to avoid”avoid”

• ““For myself, I really appreciate it, but For myself, I really appreciate it, but sometimes I think it is a waste of time”.sometimes I think it is a waste of time”.

TensionsTensions

• Time-consuming to view multiple Time-consuming to view multiple presentationspresentations

• Challenge of process versus Challenge of process versus delivering course content delivering course content

3. Feedback 3. Feedback

• “ “Good feedback causes Good feedback causes thinking.”thinking.”

(Black & Wiliam, 2003, p.631). (Black & Wiliam, 2003, p.631).

Feedback definitionsFeedback definitions

• Feedback: Feedback:

““All dialogue to support learning in both formal and All dialogue to support learning in both formal and informal situationsinformal situations”” (Askew & Lodge, 2000, p. 1) (Askew & Lodge, 2000, p. 1)

• Sustainable Feedback: Sustainable Feedback:

““Dialogic processes and activities which can Dialogic processes and activities which can support and inform the student on the current task, support and inform the student on the current task, whilst also developing the ability to self-regulate whilst also developing the ability to self-regulate performance on future tasksperformance on future tasks”” (Carless et al., 2011) (Carless et al., 2011)

Sustainability in feedbackSustainability in feedback

– Sustainability lies in the ability of Sustainability lies in the ability of students to improve the quality of their students to improve the quality of their work independently of the tutor work independently of the tutor

Features of sustainable Features of sustainable feedback feedback

• Involving students in dialogues about learningInvolving students in dialogues about learning

• Facilitating feedback processes through which Facilitating feedback processes through which students are stimulated to develop capacities students are stimulated to develop capacities in monitoring and evaluating their own learning in monitoring and evaluating their own learning

• Assessment task design: facilitating Assessment task design: facilitating engagement over time in which feedback is engagement over time in which feedback is used to enhance performance on multiple used to enhance performance on multiple stages of assignmentsstages of assignments

Interactive class dialogueInteractive class dialogue

• ““Sometimes his feedback is like a Sometimes his feedback is like a catalyst to promote the process, to catalyst to promote the process, to direct the discussion from one student to direct the discussion from one student to another. Often he was challenging another. Often he was challenging people to think more. He will ask the people to think more. He will ask the same student many why questions. It is same student many why questions. It is feedback showing that the teacher is feedback showing that the teacher is interested in your answer and wants to interested in your answer and wants to explore your answer more”.explore your answer more”.

Tensions in interactive Tensions in interactive teaching teaching • ““ Sometimes I don’t know what Sometimes I don’t know what

we are learning because he is we are learning because he is talking about very psychological talking about very psychological or philosophical things, or philosophical things, sometimes I don’t really know sometimes I don’t really know what that has to do with the what that has to do with the topic of the course”topic of the course”

Efficiency of feedback? Efficiency of feedback?

• ““The frequency of feedback is a good The frequency of feedback is a good thing because usually you may get thing because usually you may get some specific or general feedback at some specific or general feedback at some point, sosome point, so the hit ratio is a lot the hit ratio is a lot betterbetter.” .”

Effective FeedbackEffective Feedback• Is dialogic rather than uni-directionalIs dialogic rather than uni-directional

• Raises awareness of quality and Raises awareness of quality and standardsstandards

• Involves a judicious combination of Involves a judicious combination of peers, self and lecturerspeers, self and lecturers

• Encourages students to take Encourages students to take responsibility for their own learning responsibility for their own learning

ImplicationsImplications

The goalThe goal

• Students further developing their Students further developing their ability to self-monitor their own work; ability to self-monitor their own work;

• … … so that they are sensitive to and so that they are sensitive to and primed to make use of (what little) primed to make use of (what little) feedback we are able to provide; feedback we are able to provide;

• … … and more aware of what is good and more aware of what is good work in the disciplinework in the discipline

Implications for practice (1)Implications for practice (1)• Design learning activities which help Design learning activities which help

students to understand the nature students to understand the nature and characteristics of quality work and characteristics of quality work

• ActivitiesActivities: e.g. engagement with : e.g. engagement with exemplars, and associated exemplars, and associated assessment criteriaassessment criteria

Implications for practice (2)Implications for practice (2)

• Discuss with students the purposes of Discuss with students the purposes of feedback and their role in using feedback and their role in using productively (the limited) feedback productively (the limited) feedback they receivethey receive

• ActivitiesActivities: Students write on their : Students write on their assignment cover sheet, the areas on assignment cover sheet, the areas on which they would most like to receive which they would most like to receive feedback (e.g. Bloxham & Campbell, feedback (e.g. Bloxham & Campbell, 2010)2010)

Tensions in developing student Tensions in developing student self-evaluative capacitiesself-evaluative capacities

– Tensions between staff views Tensions between staff views of the desirability of of the desirability of stimulating students to stimulating students to develop self-evaluative skills develop self-evaluative skills vs some student unwillingnessvs some student unwillingness

– How far are we willing to How far are we willing to challenge students?challenge students?

– ““You ask too many questions, You ask too many questions, but you don’t tell us the but you don’t tell us the answers”. (student comment)answers”. (student comment)

Principled & pragmaticPrincipled & pragmatic

– Articulate to students Articulate to students that developing that developing enhanced self-enhanced self-evaluative capacities evaluative capacities leads to better quality leads to better quality learning learning

– … … And leads to And leads to higher higher gradesgrades. Do this and . Do this and your GPA will riseyour GPA will rise

•Ongoing workOngoing work- Student self-evaluative Student self-evaluative

capacitiescapacities- Sustainable feedback Sustainable feedback - TrustTrust

Student self-evaluationStudent self-evaluation

• How do students develop their self-How do students develop their self-evaluative learning capacities?evaluative learning capacities?

• What can we learn from how high What can we learn from how high achieving students monitor and achieving students monitor and improve their work and study improve their work and study behaviors?behaviors?

Draft revised definition of Draft revised definition of sustainable feedbacksustainable feedback

• Active involvement in dialogic Active involvement in dialogic activities in which students generate activities in which students generate and use feedback from peers, self or and use feedback from peers, self or others, as part of an ongoing process others, as part of an ongoing process of developing capacities as an of developing capacities as an autonomous self-regulating learnerautonomous self-regulating learner

The role of The role of trusttrust

• ““ I am a provocateur, bringing I am a provocateur, bringing them into the arena and them into the arena and involving them cognitively and involving them cognitively and emotionallyemotionally … It takes a while … It takes a while for them to get to know why for them to get to know why you’re doing things and then you’re doing things and then they’re going to become more they’re going to become more and more and more trustingtrusting””

• Trust as a key facilitating or Trust as a key facilitating or inhibiting factor in learning-inhibiting factor in learning-oriented assessment (Carless, oriented assessment (Carless, 2009)2009)