Gamificaon and Analy’cs in Educaon - UAB

Gamifica'onandAnaly'csinEduca'on

JamesH.Willig,MD,MSPH

AssociateProfessorDepartmentofMedicine,DivisionofInfec'ousDiseases

AssistantDeanofClinicalEduca'onUniversityofAlabamaatBirminghamSchoolofMedicine

Mapfortoday…

•  Twothreadscoalesce–  ACGMEchanges–  Gamifica'on

•  Firstversion,firstgame

•  Toolsforbuildingbears…andthenow.

2003&2011ACGMEchanges:Effectoneduca'on

•  Somestudieshavesuggestedthatsomeeduca'onaloutcomeshavebeennega'velyimpactedbythedutyhourstandards.

•  Othershavesuggestedthatthedutyhourstandardsnega'velyimpactresidenteduca'oninthefollowingways:–  Fewerandmorenega'veclinicalexperiences–  Decreasedinstandardizedtestscores–  Decreasedpublica'onandpar'cipa'oninacademicmedicine–  DecreasedteachingbyaSendingphysicians.

Onaday,notunlikethisoneinTHservice….

•  Conversa'onaboutteachingundernewACGMEworkhours

•  Wonderedabout–  Hadthelimitedworkhoursalteredourteaching?–  Howhadweadapted?Werethereadverseeffects?–  Whotrainedus,totraininless'me?

•  Howdoyoutacklethe“stackedtomorrows”problem?

2011ACGMEchanges:Residentfocusgroups

•  “Imean…youhavethesameamountofworkwithless'metoputintoit.Something’sgottogo,andforthemostpartIthinkitistheteachingaspectofthingsjustsortofgetslostintheshuffle.”

•  Ifindthatthethirdyearresidentshavemuchless'metoteachthemedicalstudentsandtoteachtheinternsthantheyusedtohave.

•  “Attheendofourinternyear,Imean,minewasaliSledifferent,butprobablyeveryonehadbetween20and40lines,youknow,centrallines.Andnowwe'retalkingaboutinternswhohaven'tdonefive–”

SecondthreadàThemorethingschange…

Rounds 2001 Rounds Now

Gamifica'on

•  Theuseofgameelementsinnon-gameapplica'onstoimproveuserengagementandexperience.

•  Todate,gamifiedapplica'onshavebeenusedinavarietyofsocial,economic,andeduca'onalenvironments.

•  CouldweapplytheelementsofGamifica'ontoGraduateMedicalEduca'on?

Mapfortoday…

•  Twothreadscoalesce–  ACGMEchanges–  Gamifica'on

•  Firstversion,firstgame

•  Toolsforbuildingbears…andthenow

Kaizen

•  Kaizen:Japanesemanagementconceptforincremental(gradual,con'nuous)change(improvement)

•  Kai:change;Zen:tobecomegood

•  Lifephilosophy,underpinsmanymanagementconcepts

•  Lifelonglearning!

Somecharacteris'csofgames

•  Goal–  Clearlydefinedoutcomeplayerscanworktoachieve

•  Rules–  Rulesarelimita'onsonhowplayerscanachievethegoal

•  Feedbacksystem–  Playerscantrackprogress;iden'fyhowclosetheyaretogoal

•  Voluntarypar'cipa'on–  Playersknowandwillinglyacceptgamegoal,rulesandfeedbacksystem

Gamifica'onandrewardsystems

•  Commonlyused,mosteffec'vewhencombinesbothintrinsicandextrinsicrewardstoincreasemo'va'onandengagement.

•  Intrinsicrewards–  Self-efficacy:increasedknowledge,acquisi'onofanewskill–  Personalchallenge:problem-solving–  Socializa'on:collabora'onwithcolleagues,socialinterac'on

•  Extrinsicrewards–  Achievements:points,badges,levels–  Reputa'on:statuswithinthecommunityofusers

Content

•  Conciseques'ons–  Clearteachingpoint,focused–  Keypointsthatyoubelievemustbetaught–  Humor,mnemonics–anythingthathelpsusremember!

•  ImagesupportfromtheVisualDx–  www.visualdx.com

•  InternalMedicineResidencyProgram–  Integratedinto2011-2012academicyear;endofyearawards–  Threeseasons(8,6,6weeks)eachincludesIMplusthree

subspecial'es

Badgesandscoring

•  Correctques'ons–  Eachcorrectansweraddstoyourscore

•  Timeques'ons–  Doublepointswithin'melimit

•  Hotstreaks–  Numberofoverallques'onscorrectinarow

•  Marathons–  Numberofconsecu'vedaysdoingques'ons

•  Web-based

•  Alias

•  Rank

RankSystem:Increasedscore,increasedrank

Yourknowledgefesters,mul'pliesexponen'ally,disseminateitonthewards!

Hot?Cold?Acognoscen'caresnot!

•  Iden'fy&enlistgreateducators

•  Shortfocusedques'ons,followedbyconciseexplana'onkeypoint

•  Humorousifpossible

•  Twoperday–aimfor5minutes

•  Availabletodowhenever,accumulate

•  Teams

•  No'fica'ons

•  Scoringsystems

Scoring:HotstreakandMarathonSystems

HotstreakBadge:Likeavola'lemixtureofmardigras,alcohol,andaurethra,youareonfire!

MarathonBadge:Likeanovernightinternlookingatbreakfastbagels,nothingcanstopyou!

FirstgameinIMGMEprogram:Howdidwedo?

•  Round1:–  92residents;71%allques'onsanswered

•  Round2:–  90residents;69%allques'onsanswered

•  Round3:–  55residents;66%allques'onsanswered

•  Total16,427ques'onsanswered

53.1%answered5:01pmand7:59am

28.8%doneonSaturdaysandSundays

No'fica'onEffect:“Fanningtheflames”

Univariate Mul'variablePlayerClass

PGY1

PGY2

PGY3

1.0

1.99(0.80,4.97)

2.88(1.04,8.02)

1.0

1.54(0.49,4.79)

2.17(0.59,7.94)DebutSegment

UAB1.1

UAB1.21

Huntsville1.2

1.0

21.65(2.62,179.25)

0.60(0.16,2.30)

1.0

12.69(1.34,120.15)

1.33(0.29,6.15)Averageperformance2

1.39(1.04,1.86) 1.31(0.84,2.05)

Percentcompleted2

0.74(0.65,0.84) 0.81(0.69,0.95)

Predominantplaystyle3

Daily

Other

1.0

3.01(1.26,7.16)

1.0

2.82(0.86,9.28)

PlayerASri'onaverdebutsegment

Reten'onofKnowledge

Total TotalPlayers;AnswerPairs

Ini'alpercentcorrect1

Subsequentpercentcorrect1

MeanChange P-value2

Overall 50;1,046 40.5 52.3 +11.9 <0.0001Round1

8/20-10/13/1239;546 37.8 47.9 +10.2 0.0008

Round2

1/14–2/26/201232;500 46.2 63.6 +17.4 <0.0001

Mapfortoday…

•  Twothreadscoalesce–  ACGMEchanges–  Gamifica'on

•  Firstversion,firstgame

•  Toolsforbuildingbears…andthenow

Sothenofcourse,wesetouttobuildabear…

BuildingBears&TheNow

•  Techinneedofoverhaul(ThekindnessofstrangersITmodel)–  Acquiredinternalfunding

•  The“Build-a-Bear”–  Or,“Whatwouldhappenifwecouldputthesetoolsinthehandsof

othereducators”

Kaizen-ME

NevinCR,WesyallAO,RodriguezJM,DempseyDM,CherringtonA,RoyB,PatelM,WilligJH.Gamifica'onasatoolforenhancinggraduatemedicaleduca'on.PostgradMedJ.2014

•  Enhancedservers,redonefrontend

•  Teamlanding&communica'onspage

GameManager:The“Build-a-Bear”

•  Gamemanagersetstheirgameparameters:#players,#teams,badges,scoring,ques'ons,topics,etc.

•  Sharedques'ons

•  Timedornot

•  Selectownbadges(level,marathon,hotstreaks)fromsharedpool

Growthensues…

•  87educa'onalgamesacross15disciplines

•  Collec'vely5,816dis'nctKaizenusershavebeenregistered

•  Intotal,204,722ques'onresponses

Variedpartnerships

•  OtherGMEprograms–  OBGYN,ED,GeneralSurgery,ENT

•  Schoolofbusiness–  Connectwithalumniaverevents

•  UMEforintroduc'ontoclinicalmedicine–  Inlieuofformerweeklyquizzes

•  Den'stry–  Toiden'fyareasneedingaSen'onduringboardreview

PresidentandCEODr.RickValachovicAmericanDentalEduca'onAssocia'on

MonthlyNewsleSerJune2016

•  MichelleRobinson,D.M.D.,M.A.,isalsoenthusias'caboutthepoten'alofdigitalgamestoenhancedentaleduca'on.TheAssociateProfessorandAssociateDeanforHealthInforma'onandBusinessSystemsattheUniversityofAlabamaatBirmingham(UAB)SchoolofDen'stryrecentlydevelopedagameforresidentsinperiodon'cs.UsingtheKaizen(Japanesefor“con'nuousimprovement”)playormdevelopedattheUABmedicalschool,Michellecreatedanonlineknowledge-reviewgametopreparetheresidentsforamockstandardizedexam.Newcontentwasreleasedeachmorning,andstudentswhoplayedthegamedailywererewardedwithextrapoints.Thestudents'regularprac'ceappearstohavepaidoff.Michelletoldmetheprogramdirectorwas“floored”bytheimprovementshesawinexamscoresandplanstousethegameagainnextyear.

Variedpartnerships

•  Interna'onaleduca'on–  UABEDInterna'onaltrackfordistancetrainingofclinicians

•  SchoolofPublicHealth–  Pilottoprovecompetenciesacquiredviacurriculumtoaccredi'ng

body

•  Educa'onofunderservedpopula'onswithchronicillness–  DiabetesClinicPilot;approachtoques'ons

•  SchoolofNursing–  Usingin6graduateandundergraduateprograms

Reten'onofknowledgeinsecondroundamong151pairedresponsesamongNursingstudents

IniMalaNemptincorrectTotal(%)

SubsequentaNemptcorrect

Total(%)

GLIMMIXModel1

OR(95%CI)

49of151(32%) 25of49(51%) 1.8(1.0-3.4)

FinalExamPerformance–Educa'onalAnaly'cs?

ExamscoreMean±SD

p-value

PlayStyle

Engagedthroughout 52.6±2.3 0.045

Answeredinbursts 51.5±4.0

Justslightlyengaged 45.8±16.2

PERTerMle

Highest 52.4±3.2 0.017

Middle 51.1±3.9

Lowest 44.6±17.6

•  PlayStyle–  ArrivedatviaK-meanscluster

analysesofdaysplayedandques'onsanswered

•  PlayerEfficiencyRa'ng(PER)–  Globalreflec'onofplayer

effec'veness–  Combinesaccuracy(correct);

consistency(howdeepintoseasonplayed)and'meliness('mefromques'onposttoanswer)

InConclusion

•  WehavedonethisforTraininginRigorReproducibilityandTransparencytrainingattheUABCTSAwithgreatengagement

•  Gamifica'onprovidesamo'va'onalframeworkwithmanyapproachestoenhanceengagement

•  Tomaximizethebenefitsofgamifica'onforourstudents,wemustusetheresul'ngdatatopowereduca'onalanaly'cstouncoverthemosteffec'vestrategiestoenhanceengagement,reten'onofknowledgeandmaximizeperformance

ComingASrac'onsFall2017

NewNaviga'onforGameManagers

FeaturedandShare-ableGames

NewTeamBadges

BossBaSles!

•  SuperMarioBrothers

•  SpaceInvaders

•  MegaMan

•  Galaga

•  DuckHunt

•  Mrs.Pacman

•  LegendofZelda

•  DigDug Metagame

Kirby

Sonic

Piyall

VirtualFighter

Thankyouforyour'meandaSen'on!PleasefeelfreetocontactmeortheFollowingColleaguesformoreinfo:1.  JamesWillig:jwillig@uabmc.edu

2.  DonaldDempsey:ddempsey@uab.edu

3.  NancyWingo:nancy10@uab.edu

4.  CathyRoche:croche@uab.edu

5.  RyanOutman:routman@uabmc.edu

6.  DavidRedden:dredden@uab.edu