Assessment Careers and feedback profiling

Dr Gwyneth Hughes, Institute of Education, London

Outline

Assessment Careers JISC funded project at the IOE

Why analyse feedback-some current issues

Using a feedback analysis tool

Using assignment cover sheets for dialogue on feedback

Developing feedback principles

2

Some current issues with feedback in HE • What is feedback?-received not just given out• Feedback rarely scrutinised, when it is practice is

inconsistent • Lack of learner engagement with and understanding of

feedback (Lizzio & Wilson, 2008)• Inefficiency in terms of effort and impact• Transmitted feedback creates dependency on teacher

(Carless et al. 2011)

• Praise is not helpful to students unless accompanied by detail on why the praise is deserved. Encourages dependency on praise (Hattie & Timperley, 2007).

• Praise-critique does not encourage dialogue and self-reliance (Nicol & Macfarlane-Dick, 2005)

• Lack of information about progress but this could be motivating (Hughes, 2011)

• Formative feedback often encourages a focus on short-term grade improvement rather than longer-term learning.

4

Assessment Careers Project: taking action on feedback

• Year 1: Baseline and feedback analysis tool• Year 2 (2012/13): Five pilot programmes• Year 3 Institutional implementation

Assessment Careers: Institute of Education website:www.ioe.ac.uk/assessmentcareers and video clip:http://youtu.be/VSaGbPoXPh0

Tool to identify feedback categoriesP1 Giving praise

P2 Recognising progress (ipsative feedback)

Critique• C1 Correction of errors• C2 Factual criticisms• C3 Criticism of approach

Giving advice• A1 Specific to content current assignment• A2 General skills in current assignment• A 3 For future assignments

Q Clarifications and questionsO Other unclassified statements

Adapted from Orsmond & Merry, 2011 including Hughes, 2011.

Analysing feedback categories

• The score is the number of times a classification appears in the feedback

• The default unit for analysis was the sentence• Where a sentence contains clauses that make

distinct points, it was split into separate clauses, each of which was classified separately.

• Neutral comments that for example describe the piece of work, but do not make any judgement are unclassified.

8

You have clearly developed your thinking on how you might use technologies in your teaching but as I mentioned in the previous feedback, you have not reflected on your experience of using technology to identify the potential benefits and challenges.

You make your rationale for innovation clear, but less clear is the learning theory that underpins your pedagogic design. You have now explained X’s framework, but how does your design draw on this?

When you do mention Y’s model you do so uncritically, but you have responded to the feedback on your draft that you should discuss the limitations of applying model Z to your design.

Try to critique all models and frameworks in future.   

9

You have clearly developed your thinking on how you might use technologies in your teaching but as I mentioned in the previous feedback, you have not reflected on your experience of using technology to identify the potential benefits and challenges.

You make your rationale for innovation clear, but less clear is the learning theory that underpins your pedagogic design. You have now explained X’s framework, but how does your design draw on this?

When you do mention Y’s model you do so uncritically, but you have responded to the feedback on your draft that you should discuss the limitations of applying model Z to your design.

Try to critique all models and frameworks in future.   

The structure of the essay was very strong, with a clear introduction which nicely contextualised the general problem of extraction industries and the interest in social environments vs the needs for quite ‘quick and dirty’ (my phrase) data collection. The general aims of REA was well described, although I did have questions while reading this about how representative cultural samples could be drawn quickly, and what ‘representative’ meant in this context. In terms of limitations I felt that the essay had maybe missed a trick – if a closer comparison with the aims and debates in conventional ethnography had been conducted, then this may have opened a space for thinking about the extent to which REA can achieve those aims or not. The issues raised in the conclusion about ethics and trustworthiness for instance could have been discussed in some detail ... Overall, the essay was very clear and well written, but I think that the absence of relating this very particular method to the broader issues in ethnography perhaps made it limited in the level of analysis that it provided of REAs. An extremely interesting and competent essay all the same though. 10

Data• Analysed formative and summative assessment

feedback for modules on 5 postgraduate programmes at the IOE (total 228 pieces)

• Recorded the total number of comments in each category and the average per script

• Ranked the categories to obtain a feedback profile at programme level as well as an aggregate profile of the 5 programmes.

Baseline feedback profileTotal comments for 5 programmesSummative assessment (N= 165)

Category of feedback Average per script Rank

P1 Praise 4.4 1

P2 Ipsative (progress) 0 (negligible) 5

C1-C3 Critique 2.7 2

A1-A3 Advice for current or future assignments

1.9 (mostly for current assignment)

3

Q Questions and clarification requests

0.1 4

12

Baseline feedback profileTotal comments for 5 programmes (N= 165)

Category of feedback Average per script Rank

P1 Praise 4.4 1

P2 Ipsative (progress) 0 (negligible) 5

C1-C3 Critique 2.7 2

A1-A3 Advice for current or future assignments

1.9 (mostly for current assignment)

3

Q Questions and clarification requests

0.1 4

13

Overall you have written a really comprehensive and detailed review of the document. Your strength lies in your fantastic ability to contextualise the document with your political and pedigogical references. […] What do you understand by the term disruptive in this context? […] You seem to suggest ( or I am interpreting wrongly) that disruptive in this context is bad? […] Finally, you quote and cite almost in every sentence- but what is your personal view of 'the rise of K-12 blended learning‘ in primary and secondary education? Where is your personal voice? It scares me - or at least having a purely distance online learning component for such students worries me. What about you?

14

What do students do with feedback?Formative n=50 Summative n=35

didn't respond to the question 4 3

no feedback received 0 1

nothing 4 5

read it 3 10

revised assignment 32 3

systematically took forward 7 13

15

Overview of interventions

Pilot 1 MA Education, Health Promotion and International DevelopmentPilot 2 MRes in Educational and Social ResearchPilot 3 MA Clinical Education Pilot 4 MA/MSc Psychology of EducationPilot 5 PGCE Primary

Approx. 400 students and 30 staff in total.

  

Assignment cover sheets Thinking about the feedback on your draft of this essay, please

indicate what the key points were and what action you took to respond to this feedback to help you prepare for and write this essay

17

I would like tutor feedback on:1.2. 3.

Is ipsative feedback appreciated by students?“I made an effort to act on it (formative feedback). This

was acknowledged in the feedback on the final assignment.”

From another student:“I like the fact that the feedback demonstrates that the

tutor is aware not only of my current level of achievement, but my progress.”

18

Principles for Action on Feedback1. Feedback is directed towards students' concerns  2. Students are encouraged to see the value of feedback for both short-

term and longer-term learning3. Assessors initiate dialogue with students about feedback their work 4. Curriculum designers and assessors make it clear to students how

learning accumulates over a programme5. Programme teams are aware that feedback has different purposes and

agree which categories of feedback are appropriate for their students6. Students have experience early and frequent formative assessment

opportunities which are designed into their programme   

 

19

ConclusionFindings so far:• Feedback profiling is useful for getting programme teams

talking about feedback for both short and longer term learning.

• Assignment cover sheets are a useful way of engaging student in discussion of feedback.

• Feedback principles are useful for opening up discussion of feedback practice at the institution level.

• www.ioe.ac.uk/assessmentcareers

ReferencesAssessment Careers project Baseline Report see:

www.ioe.ac.uk/assessmentcareersCarless, D. Slater, D.Yang, M. and Lam, J. 2011. Developing sustainable

feedback practices. Studies in Higher Education 36, no.4: 395-407.Gibbs, G. & Simpson, C. (2004) Conditions Under Which Assessment

Supports Students’ Learning. Learning and Teaching in Higher Education, 1: 3-31

Hattie, J. & Timperley, H. 2007. The Power of Feedback. Review of Educational Research 77 no. 1: 81-112.

Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing Ipsative Assessment in Higher Education Studies in Higher Education 36 (3): 353 – 36721

Lizzio, A. & Wilson, K. 2008. Feedback on assessment: student’s perceptions of quality and effectiveness. Assessment & Evaluation in Higher Education 33 no.3:263-275.

Nicol, D. & Macfarlane,-Dick, D. 2006. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education 31 no. 2: 199-218.

Orsmond, P. & Merry, S. 2011. Feedback alignment: effective and ineffective links between tutors’ and students’ understanding of coursework feedback. Assessment & Evaluation in Higher Education. 36(2): 125-126.

22