Changing Feedback – Panel review session

5 July 2017: 07:00AM UTC/GMT Changing Feedback – Panel review session Justtoletyouknow:

Bypar'cipa'nginthewebinaryouacknowledgeandagreethat:Thesessionmayberecorded,includingvoiceandtextchatcommunica'ons(arecordingindicatorisshowninsidethewebinarroomwhenthisisthecase).Wemayreleaserecordingsfreelytothepublicwhichbecomepartofthepublicrecord.Wemayusesessionrecordingsforqualityimprovement,oraspartoffurtherresearchandpublica'ons.

Webinar Series

e-AssessmentSIG

Webinar Hosts ProfessorGeoffCrisp,PVCEduca'on,UniversityofNewSouthWalesg.crisp[at]unsw.edu.auDrMathewHillier,OfficeforLearning&Teaching,MonashUniversitymathew.hillier[at]monash.edu

Sessionchair:ProfessorSallyJordanOpenUniversityUK

Manchester, UK, 28-29 June 2017 About200delegatesfrom22countries.• Choiceof5masterclasses•  2keynotespeakers•  77researchpapers/prac'ceexchangesin10parallelsessions

• Posterandpitchsessions•  ForTwiZerac'vitysee#AssessmentHEConf“Thefriendlyconference”

Conference themes •  Exploringcontemporaryapproachestoassessmentandfeedbackinhighereduca'on.

•  Enhancingassessmentandfeedbackatprogrammeandins'tu'onallevel.

• Cul'va'ngassessmentliteracy.•  Integra'ngdigitaltoolsandtechnologiesforassessment.

• Developingacademicintegrityandacademicliteraciesthroughassessment.

• Assessmentforsocialjus'ce.

Keynote presentations

DrJanMcArthur,LancasterUniversityThedarkartsofassessment:FromSMARTtosocial

jus:ce

ProfessorDeniseWhitelock,TheOpenUniversityTechnologyenhancedassessment:Dowehaveawolf

insheep’sclothing?

Three research papers and practice exchanges to share with you…. CaroleSuIon,JaneCollingsandJoanneSellick,PlymouthUniversityModelsofExamina:onFeedbackJudyCohenandCatherineRobinson,UniversityofKentExploringtheeffectsofradicalchangetoassessmentand

feedbackprocesses:ApplyingTeam-basedlearningina

socialsciencemodule.LizAustenandCathyMalone,SheffieldHallamUniversityExploringstudentpercep:onsofeffec:vefeedback

ModelsofexaminaOonfeedback‘Notmanystudentswouldadmittoenjoyingtakingexamsbutifyou

wanttogetadegreethey’reanordealyouhavetosurvive.’

Guardian(2013)

JoSellick,CaroleSuIon,andJaneCollingsSchoolofLaw,CriminologyandGovernment

&TeachingandLearningSupport

Magne(2013)adaptedfromBrown(2007)

Feed-inWelldesignedassessment,briefing&prepara'on,DLEassessmentresourcesprac'cepapers&sessions,assessment&markingcriteria

Feed-forwardForma've‘feedback’,cues,discussion,mid-wayreviews,tutor,peer&self-reviewandfeedback

Feedbackendofexams,wriZen,verbal,mp3,YouTube,1:1,genericfeedback.construc'velegiblefeedback

Planningassessmentusingfeed-in,feed-forward&feedback

LearningSpace

ExaminaOonFeedbackProject

•  Pre-surveystudentexperiencestogainstudentvoice.

•  Literaturereviewinformedthedevelopmentofatoolkit.

•  Stafftoolkitproduc'on•  Post-surveystudentexperiences.

Outcomesoftheproject

1.ChangetotheUniversityofPlymouthAssessmentPolicy‘Receiveconstruc:vefeedbackaRerallassessments

includingexamina:ons.’hZps://www.plymouth.ac.uk/your-university/teaching-and-learning/guidance-and-resources/plymouth-university-assessment-policy-2014-20202.Examina'onstoolkitforstaff

– Genericfeedbackakerexams– 1:1feedbacksessionwithstudentswithexamina'onscripts

hZps://www.plymouth.ac.uk/uploads/produc'on/document/path/8/8424/University_of_Plymouth_Examina'on_Toolkit.docx

ExaminaOonFeedbackToolkit

KeyFactors•  Examtype&purposeoftheexam(linkedtolearningoutcomes)

•  CohortSize•  Loca'on&typeofstudent•  Timingand'mescales•  Accesstoexamscripts•  Withinamodule

– Forma'veandSumma'veassessmentapproach

ExamplesoffeedbacktoformaOveandsummaOveexaminaOonsandtests

SuggesOonshowto

GenericwriIen Summarycoveringminimumrequirementsandlinkedtolearningoutcomes–couldbebasedondis'nguishingbetweenclassifica'ons Email,

Moodle.‘Model’answersand/orexemplarsbankthatcanbeaccessedbystudentson-demandGenericverbal/audio Q&Asession Timetabledsession(note

poten'alofcontentcapture).Moodle.

Peerdiscussion/feedback(e.g.ofindividualforma'veand/orexemplars)Podcast/videoIndividualwriIen Template/summaryfeedbackformwithminimumrequirementsandlinkedtolearningoutcomeswithspecificreferenceonhowtoimprove/whattodonext

Hardcopy(collectedfrom/distributedbytutor/supportoffice).Moodle,PebblePad,Email

Individualverbal/audio 1:1appointmentwithexammarkerusingscript(forma'vemockand/orsumma've)

Generallypromotedwithon-demandappointmentsBlockedoutperiodsforappointments(‘examopenday’/‘feedbackweek’etc.).Moodle,PebblePad,Email

Mee'ngwithPersonalTutorwhohasreceivedreportsfrommarkersaspartofPDPprocessExamina'on‘openday’tolookthroughpaperswithtutorsIndividualaudiorecordingsuchaspodcast

Casestudy:SociologyProcessModule:Stage3:SocialandPoli'calTheory.AutumnSemesterAssessment:Seenexampaperessaystyle(2ques'onsin2hours)FormaOve

Feed-in:Dec/Jan:Examrelease,supportsessions,examansweroutlines

SummaOveFeedback:February:Groupfeedback,examscriptreview,invita'onfor1to1feedback

Impact

AcademicYear Avg.Exammark2008/09 53.84%2009/10 44.29%2010/11 49.50%2011/12 56.95%2012/13 54.16%2013/14 53.42%2014/15 63.33%2015/16 65.79%

Sociology:studentfeedback

–  Understandingoftheassessedlearningoutcomes

–  Expecta'onsandfocussedprepara'on

– Managinganxiety

–  Enhancedunderstandingofmarksachieved

–  Enhancedprepara'onleadingtoimprovedperformanceoverpreviousyears

–  Staffwereinterestedinmyperformance

–  Senseofachievement–  Reduc'oninneedtoseekaddi'onalstafffeedback.

Feed-in Feedback

StaffChallenges

•  Prac'calarrangements,including'metabling1:1slotsandschedulingforthere-sitexamina'on.

•  Timemanagementandlargecohorts.•  Tosensi'velybalanceexamscriptreturnwiththeneedforgradeprivacywhenconduc'nggroupfeedback.

•  Toensurefeedbackisasusefulasfeed-in.•  Establishingaclear,concretelinkbetweenexamfeedbackandimprovementsacrossacademicyears.

Emergingthemes•  Examsarejustanotherformofassessmentthatstudentsshouldexpectfeedbacktounderstandandimprovetheirperformance.

•  Logis'calconsidera'ons–  Reten'onoftheexamscriptdoesnotpreventfeedback–  Constrainedbyexam'metable–  Requiresadap'onoffeedbacktomeetthe'meconstraints–  Noequivalenceofe-submissionforexams–  Challengesforfeedback,especiallySemester2

•  Staffworkloadalloca'on–  adjustmentstoaligntootherformsofassessment

Exploring the effects of radical change to assessment and feedback processes / Applying Team-based learning in a social science module

Judy Cohen and Catherine Robinson (KBS)

Funded through the Faculty Learning and Teaching Enhancement Fund

Outline of the presentation

• Economics module

• What is Team Based Learning?

• How was it implemented?

• Lessons learned

• Next steps

Economics for Business 2

Stage 2 module (109) Optional for B&M

Diverse student base

Poor attendance Poor engagement Mediocre marks

(Image source: http://www.mbacrystalball.com/blog/economics/macroeconomics/)

The UK’s European University

What is Team-Based Learning?

•  More than group work

•  A form of flipped learning

•  Comprises of 3 key elements: •  The Readiness Assurance Process (RAP) •  Application exercises •  Peer evaluation

•  Applications need to meet the 4S criteria (Sibley and Ostafichuk (2014) •  Significant; Same; Specific;

Simultaneous


Assessment for Learning

•  Pre-reading

•  Readiness Assurance Process

•  Appeals

•  Corrective Instruction

•  4 S application…


Implementation

Challenges – working with what we’ve got...

•  Module choice •  Institutional

constraints of Ø  Physical space

Ø  Module assessment


Implementation: How did we do it? • 110 students split into teams of 6/7

•  Named after famous economists

• Lecture slots used to test the students’ readiness and the individual scores used as assessments

•  Continual assessment at 5 points during the 12 week course

•  Highest 3 scores were taken as their MCQ mark (worth 10% in total)

• This was administered using TurningPoint devices registered to individual students.

•  5 MCQs at the start, followed by team breakout time….


Was it worth it? 2015/2016 and 2016/2017

Attendance in seminars ↑  male 68.4% to 71%

↑  home students 69.3 to 79.8%

↓  female 80.4% to 75.3%

↓  overseas students 75.9% to 72.1

Performance in final exam ↑  Cohort average from 56.3% (standard deviation 12) to 63.0% in

2016/2017 (standard deviation 11.4)

↑  Anecdotal evidence of improved quality of writing


Student comments Viewpoint A Ø  This module was taught differently

which I enjoyed. I enjoyed the team working element which promoted and encouraged learning and participation.


Viewpoint B Ø  Never studied economics before, so layout of this module where you

read at home then come in and do test with no lectures was difficult for me. Difficult to understand terminology.

What I’d do differently

• More acknowledgement of the difference between novice and expert learners

• Changes to delivery •  2 hour workshop

• More on-line resources •  KentPlayer

•  Improvements to peer-evaluation system


Any questions?

Thank you


THE UK’S EUROPEAN UNIVERSITY

www.kent.ac.uk/kbs

Exploringstudentpercep'onsofeffec'vefeedback

[email protected]@lizaustenbooth;[email protected]

28

Aim

Topresentselectedfindingsofamixedmethodsresearchprojectwhichanalysedthecharacteris'csoffeedbackthatstudentsvalued.

29

ResearchMethodology

Thistriangulatedstudentopinionsaboutfeedback(collectedthroughfocusgroups)withtheirevalua'onsoffeedbacktexts.Studentevalua'onsinformedcorpusconstruc'onandcontras'veanalysis.FocusGroups:5ac'vitybasedstudentfocusgroups2016.•  n=28,4UGSocialSciencecourses,alllevels•  suppor'ngstudentannota'ons•  thema'ccodingofverbaldiscussions

StudentEvaluaOonsofFeedbackTexts•  95piecesofindividualwriZenfeedbackevaluated,

criteriaadaptedfromtheworkofNicol&MacFarlane-Dick(2007)

•  textsmarkedtwice,rankedtocreateminicorporaofhighandlowrankingfeedback

30

A

B

9570

251

Findings

31

Length Achievement

Detail ForwardorientaOonInterpersonal

Praise

Error

Praise:Theimportanceofmanagingaffec'veneedsofthereader"IfIhadgotthatbackIwouldhavegone'Thetutorhatesme.Idon’twanttogoback’."•  veryfewstudentscommentedonaposi'vetoneandthe

discussionfocusedonnega'vityandthedemo'va'ngimpactthishas

•  feedbackwhichhighlightedwhatwaswrong,withoutprovidingsupportfordevelopment,wasfrequentlydiscussedasalivedexperience

•  studentsdiscussedtheimportanceofreceivingpraise•  praisewasseenasnecessary,evenonworkthatwasofa

lowstandard•  clearcommentsaboutwhathasbeendonewellwas

neededtohighlightwhichaspectoftheworkcanberepeatedinforthcomingassessments

32

Ques'on:

Dostudentspreferlongerfeedback?Howlongistheop'mumfeedbackinyousubjectarea?

33

Length:Dostudentspreferlongerfeedback?

34

Thereisamoderatecorrela'on(0.51)betweenlengthoffeedbacktextandoverallstudentevalua'on•  althoughbeZerscoringfeedbacktendstobelonger,thelongest

piecesoffeedbackwereevaluatednega'vely"Toolong!"Textlength399

•  feedbacktextsbelow120wordsinlengthofplaintextaremorelikelytoelicitnega'vecomments"Tooshort,norealfeedbackgiven.Wouldn'tbeusefultome."Textlength66

•  short/nofreetextcommentsviewedas"rushed”

Text CorpusA(top25texts) CorpusB(boIom25texts)TotalWordcount 4840 2943Averagetextlength 194 118Rangeoftextlengths 116-321 0-402Averagesentencelength 20 14Rangesentencelength 10-29words/sentence 0-20words/sentence

ComparisonOveralllengthCorpusA&B

Achievement:Whatisiden'fiedas'good'?

•  'good'appearswithsimilarfrequencyinbothcorpora•  InCorpusAwhatisdescribedasgoodismuchmoredetailedandprecise.•  'good'isalsousedtolabeltopicdevelopment•  CorpusBgooddescribessomethingfairlygeneric,followedbyimmediatecri'cism

35

A B

Goodintroduc'on.(x1)goodsummariesoverallgoodargumentsheregoodlinkstothecausesofcrime/theriots(x2)agoodclarifica'onofkeytermsgoodbreadthoftheorygoodacademiccri'queyoustartedtodothisneartheend,whichwasgood...Yourrecommenda'onswereunusuallyclear,whichisgoodbutneedmorediscussionWhatistheevidencethatthesearegood?

goodpoint(s)x7good(cri'cal)thinkingx3goodwork(x2)goodconclusion(x1)goodanswer(x1)good,but

•  Whoappearsinyourfeedback?I/you/your.../we

36

Interpersonal:Whoisinthefeedback?You,I,we?

•  comparedtoBri'shNa'onalCorpuswriZen(BNC6million

words)'you'and'your'appearwithveryhighkeynessfiguresacrossthewholesample.

•  'we'doesn'toccurineithercorpusAnalysis:•  CorpusAismoreinterac'vethanB•  introduc'onoftheselfintothetextmakesthefeedback

morebalanced•  'I'isusedtosokencri'calstatements

37

CorpusA(top25texts) CorpusB(boIom25texts)Rank Freq Keyness word Keyness Freq Rank1 130 570.3 your 282.7 70 35 163 381.1 you 142.6 78 5

ConclusionsStudentevalua'onsoffeedbackrevealeddis'nctdifferencesinlanguageusedinthetwocorpora.Theseneedtobeinterpretedinthelightofstudentcommentaryontheprocess.•  Thisresearchhighlightstheimportanceoftheaffec've

responseofstudentreaderstofeedback,&howthismediatesstudentengagementwithfeedbackmessages.

•  ItalsodrawsaZen'ontoins'tu'onalprocessesthatencourageproduc'onofobjec've,structuredandcriteriabasedfeedback.

•  Differentaspectsoffeedbackallcontributetodiscursiveorienta'on,andagency,ofstudentreaderwithinthetext.

•  Intermsofimplica'onsforprac'cethesefindingsshouldbeconsideredholis'cally,ratherthanasseparatecomponentswhichfunc'onindependently.

38

SampleuseofFeedbackForm.Thesecommentsfollowamarkinggrid.

Yourbestworkwaswhen:Providedacri:queofthetheore:calapplica:ontopar:cularriotresponses.

Showedadetailedknowledgeofriotresponses.

Toimprove,youneedtoworkon:-More‘analysis’relatedtocri:calthinkingaboutpolicy(relatedtopunishment)inbroaderterms.

-Dis:nguishingbetweencriminaljus:ceandsocialpolicydevelopmentsinrespectoftrendsin

punishmentresponses.

-Showamoredetailedandnuancedunderstandingoftheoryitself.Useitforanevenmorepowerful

cri:que.

-Accuratelyreferenceallonlinesourcese.g.placeforbooksandpagenumbersforar:cles.

Resourcesrecommendedtohelpyouimproveyourwork(withhyperlinkswhereapplicable):TheDefini:veReferencingGuideavailableviatheLibraryGateway

CTCMScreencastLectureonCri:callyAnalysingPolicy(videos1&2)

CTCMScreencastLectureonCri:callyAnalysingCaseStudiesandReports

CTCMScreencastLectureonCri:callyAnalysingTheory(videos1,2,3,4,5,6&7)

RemembertodiscussthisfeedbackwithyourAcademicAdvisortoassistyouracademicdevelopment☺

39

Example:Astyleguide

40

Do Don'tLength Writeatleast100words-freetext

forsumma'veassessmentWritelessthan100ormorethan300wordsforsumma'veassessment.

PosiOve Iden'fywhatthestudenthasdonewell.Findonethingyoulikeaboutthispieceofwri'ng.

Evaluatefirstyearworkbythirdyearstandards.Don'tqualifyeveryposi'vestatementyoumake-itreadsascri'cism.

Detailed Commentwithprecision.Makeacleartheconnec'onbetweenassessmentcriteriaandthissampleofstudentwri'ng.Unpackit,explainwhyit'seffec've.

Usegenericcommentsthatapplyacrossgradeboundaries.

Tone Beposi've,encouraging,andinclusive. Besarcas'c,oroverlydirec'veorharshlycri'cal.

ForwardorientaOon

Sumupyourresponsetotheirworkin2/3takehomemessagesthatcanbeappliedtotheirnexttask.Makeexplicittheconnec'onbetweenperformanceintext(poorrangeofsources)andunderlyingstudyprac'ces(needtoreadmore)-thismayrequirealotofmodelling.

Don'tfocusexclusivelyoninadequaciesofthelasttext,andhowthatcouldhavebeendonebeZer.Don'tleavethemtointerpretyourfeedbackordeducefromdetailedtextualcommentarywhattheyshoulddodifferentlynext'me.

Error Spellcheck Useacronyms,orobscureterms

Discussion CaroleSuIon,JaneCollingsandJoanneSellick,PlymouthUniversityModelsofExamina:onFeedbackJudyCohenandCatherineRobinson,UniversityofKentExploringtheeffectsofradicalchangetoassessmentand

feedbackprocesses:ApplyingTeam-basedlearningina

socialsciencemodule.LizAustenandCathyMalone,SheffieldHallamUniversityExploringstudentpercep:onsofeffec:vefeedback

Forthefuture,watchoutfor:SpecialissueofPrac::onerResearchinHigherEduca:on(PRHE)Journal.Onedayseminar,28thJune2018,Manchester:“TransformingAssessmentinHigherEduca:on:The

LearningPowerofFeedback”.

Keynotespeaker:ProfessorKaySambellNextAHEConference,June2019.AHEwebhZps://aheconference.com/

Webinar Series

Webinar Session feedback With thanks from your hosts

Professor Geoff Crisp, PVC Education, University of New South Wales g.crisp[at]unsw.edu.au

Dr Mathew Hillier, Office of the Vice-Provost Learning & Teaching Monash University mathew.hillier[at]monash.edu Recording available http://transformingassessment.com

e-AssessmentSIG

Documents

Changing Feedback – Panel review session