LearningAnalytics:HarnessingDataSciencetoTransformEducation
TimMcKay:UniversityofMichigan@TimMcKayUM,Blogat21stCenturyHigherEd.com
TheUniversityofMichigan
• 200yr oldpublicresearchintensiveuniversitywith19SchoolsandColleges&6,800faculty
• Highlyselectivegroupof29,000undergraduateand16,000graduatestudents
• Annualbudgetof$7billion,including$1.4billioninfederallyfundedresearch(#1public)
Particleastrophysics:Carefullymeasuringnothing…
ObservationalCosmology:MeasuringEverything…
Thisisobservationalscience:drawinginferencewithoutclassicalmethodsofexperiment…
Teachingthousandsofstudentsintroductoryphysicsofeverykind…
8yearsasDirectorofLSAHonorsProgram,caringfor2000students
acrossthedisciplines…
Myneedfordataaboutthesestudents&theirexperiencesledtocontinuallyexpandingeffortsin
LearningAnalytics
The20th Centurybeganwithanindustrialrevolution.Publichighereducationjoinedin:explodinginscaleandadoptedbureaucratic,industrialapproaches,including
standardizedtests,credithours,GPAs,majors,andminors.
Toooften,wepursuea20th century,industrialformofoptimization:seekingasinglesystemwhichmaximizeslearningacrossapopulationofstudents.
The21st Centurybeganwithaninformationrevolution.Weknowmoreaboutourstudentsthanweeverhaveandconnectthemwithus,information,oneanother,andtheworldinunprecedentedways.
Ourgoaltodayisa21st century,informationageformofoptimization:adaptingthesystemtoindividuallyoptimizelearningforeachstudent.
Whyhasn’tthishappenedalready?
Whysomuchvisceralresistancetotheideaofusingtechnologyto
personalizeeducation?
• >98%oftheseare:• Small&changeable• Taughtinidiosyncratic,engaged,andcreativeways
• <2%ofthemare:• Large&relativelystable• Taughtinindustrial,remote,andtraditionboundways
Onereason?Therearetwowaystoteachalargenumberofstudents…Wehave9200coursesatUM
Whypersonalize?Forthe735studentsinmyclass…
Nita
Frank
Weareoftenfooledbyafalsesenseofpersonalization.Thesetwostudentsare0.3%ofthe
wholepool…
FocusonEducation@Scale• EvenatbigUniversities,mostcourseshavewhattheyneedtobeexcellent.Largeintroductoryandmanyonlinecourses don’t– Theycouldbedramaticallybetterwithadifferentapproachtoinstructionandprofessionalsupport
• Improvingeducationatscaleisamajorsociotechnicalchallenge, requiringboth:– Newinformationtechnologyforpersonalization– Newsocialnormsforcoursedesign&delivery
It’shardtobeatBloom’stutorwhereshecanact.Learninganalyticsshouldfocusonimprovingthingswhereshecannot.
Fivethemesofourwork
Learninganalyticsforpersonalization
1. Ethics:whatarewedoingandwhy2. Measurement:datacollectionandmanagement3. Analysis:modeling,extractionofmeaning,
learningfromtheexperienceofall4. Action:decisionmaking,storytelling,creating
themotivationforchange5. Synthesis: Buildingalearninglaboratory
Informationethics:agrandchallenge
• Whatprinciplesshouldgoverncollectionanduseofdataaboutindividualsineducation?– Whatdataisrelevantforeducation?– Normsofconsent,privacy,autonomy?Howaretheydifferentwithinanacademiccommunity?
– Howareexperimentsineducationalpracticerelatedtoresearchnorms?
• WeneedtoensurethatcommercialEd-Techisapartof(&constrainedby)thisconversation
SixAsilomar principles
1. Respectfortherightsanddignityoflearners:transparency,consent,protectionofprivacy
2. Beneficence:maximizebenefits,minimizeharm3. Justice:benefitall,reduceinequalities4. Openness:learningandresearcharepublicgoods5. Thehumanityoflearning:insight,judgment,&discretionare
essential,weshouldkeeplearninghumane6. Continuousconsideration:ongoing,inclusivediscussionof
changingethicalcircumstances
http://asilomar-highered.info/
Wheretohavethisconversation?
Example3:MeasureableTypes
• Dataoftenusedtocategorize,collectingindividualsintogroups
• Theselabelsareoftenreductiveandinvisibletolearners
• Always incomplete,substitutingacategoryforindividuals
• Cheney-Lippold (2017)‘measureabletypes’≠complexsociallyconstructedclasses– gender≠‘gender’– text≠‘positive’
• Categorizationstoooftenonedimensional,excludingintersectionsofidentity
Howtolimittheimpactofreflexivelyreductivedatarepresentation?
IntersectionalityinLA
• Learnmethodologyofintersectionalityfromfeministscholarship*:usemultipleapproaches– Intercategorical:focusonvariationacrosssociallyconstructedprovisionalcategories
– Intracategorical:analyzevariationw/incategories– Anticategorical:realpersonalization,nocategories
• Betransparent,allowforagency:resistlabellingindividualsw/ounderstandingandconsent
*McCall,L.(2005).Thecomplexityofintersectionality.Signs:Journalofwomenincultureandsociety,30(3),1771-1800.
Whatdowemeasure?• Whatwemeasurenow:
– Admissionsinformation– Coursetaking&grades– Degrees&honors
• Whatwe’restartingtorecord(explosivegrowth)– Processoflearning:
clickstreams,discussions,video,coursestructures
– Productsoflearning:forumposts,essays,papers,presentations,theses
• Whatwewanttohave:Detailed,relevant,evolving
portraitsofeverystudent'sbackground,interests,goals,andaccomplishments
• Theseportraitsshouldbeusedtohelpstudents,faculty,administrators,staffbetterunderstandhighereducation
Justforstudentrecords,thereare157pagesofdatadescription…hundredsoforganicallyevolving,interactingtables…
1stchallenge:Datacleaning&aggregation
Examplepartialsolution:UMLearningAnalyticsDataArchitecture
A‘regularrelease’modelforcleanresearchdata.SimilartothoseinopenscienceprojectsliketheSDSS
orGAIAspacemission
Howtoreleaseinformation whileprotectingprivacy?
• Asmuchaspossible,weshouldleteveryonelearnfromtheexperienceofall– Restrictedreportingtools– accesstodigestedinformation,withintools(Ex:ART2.0,ECoach)
– Existingresearchprotocols– IRBoversight,anonymization=>theLARCapproach
• Newapproachesareemergingindatascience:syntheticdatacontainalltheinformationbutnoneofthedetails
• Personalprivacycan(&must)beprotectedwell.Weshouldrethinkinstitutionalprivacy…
Bettermeasuresoflearning• Grades:performance
measuresofunrecordedtasks,meanttoestimateunknownoutcomes,quantifiedonill-definedscales
• Weshouldbemeasuringlearning– increasesinwelldefinedknowledgeandskills– andfocusingonindividualgrowthovertime
• Direct:preandposttestingalignedwithlearninggoals.Goodforfoundationalcourses?
• Indirect:DataSciencetoolsforextractingmeaningfromproducts– Simple:IRT,topicmodeling
andbeyond– Complex:peerevaluation,
NLP,directrepresentationratherthandatareduction
IntellectualBreadth DisciplinaryDepth RangeofExperience
Engagement&Effort Social&ProfessionalNetworks
AcademicPerformance
Measuringwhatmatters:theTranscriptoftheFuture
Studentsconnectthroughcourses
Coursesconnectthroughstudents
Canwequantifyintellectualbreadth?
Exploreeachstudent’snetworkofconnection
• Courseco-enrollment:wellmeasured,largebipartitenetwork
• Betterrepresentationsofinteractioncoming
Comparemeasurednetworkstructurestoappropriaterandomgraphs– measurediversityofconnection
Exposesisolationofmajors,allowscomparisonofindividualswithinamajor
Threeexamplesusingdifferentmethodologies:
1. Areourclassroomsequitable?2. Dolearningcommunitieswork?3. Areplacementexamsusedwell?
BTEWTE
#1:Koester/Grom/McKayAreourclassroomsequitable?
Studentperformanceisinfluencedbybackgroundandpreparation.
Forexample:gradesinphysicsrelatedtogradesinothercourses.
ObservedcorrelationClassroomequity
Genderedperformancedifferences
<GPA– Grade>Male=0.32<GPA– Grade>Female=0.59
GPD=0.27
Theseperformancedifferencesremain
whenweaccountforallmeasuresofbackground
&preparation.Unexplainedperformancedifferenceslikethisaresignsofclassroominequity.Wemustlookforandaddressthesedisparateimpacts.
Koester,Grom,McKay:https://arxiv.org/abs/1608.07565
Classroomequity
AlllargeintroSTEMlecturecourses+Econ101/102
Measuredofgradepenalty&GPDacrossalllargecoursesatUM:Strikingpatternsofgenderedperformancedifference
Classroomequity
Datafrom2000– 2012foralllargeSTEMlectureandlabcourses
Labcourses
Lecturecourses
Details,includingtestsofmanyotherpossible
performancepredictors:arXiv 1608:07565
Intercategorical complexity
Classroomequity
Biology Chemistry
Math&Stats Physics
DatafromfiveBig10Schools:SimilarGPDpatternsacrosslecture&labSTEMcourses.
Matz etal. AERAOpeninpress
Thisanalysisusesbothhierarchicallinearmodelingandquasi-experimentalmatchingmethods
#2:Brooks/Morgan/Maltby - HSSPImpact
LivingLearning
Quasiexperimental design
HealthScienceScholarsProgram
Exampleresults
HSSPsignificantlyincreasedthelikelihoodofBSandadvanceddegreesforunderrepresentedand
first-generationstudents.
LivingLearning
Quasiexperimental design
Michigan1:2:1IntroductoryChemistryCurriculumModel:
Traditional2:2IntroductoryChemistryCurriculumModel:
2 Semesters General Chemistry 2 Semesters Organic Chemistry
Chemistry130
Chemistry210&215 Chemistry230
#3:Shultz/Gottfried/WinschelChemistryPlacementAnalysis
ChemPlacement
Regressiondiscontinuity
HowdoestakingGenChem firstmatter?
ChemPlacement
Shultz,GingerV.,AmyC.Gottfried,andGraceA.Winschel. JournalofChemicalEducation 92.9(2015):1449-1455.
Regressiondiscontinuity
Howtoputdatatowork…
Aspectrumofinformationagency…
Givestudents,advisors,facultythedata~directly.Letthemdecidewhattodo.
Give‘experts’thedata.Havetheminterpret,andmakedecisionsforstudents,advisors,
faculty.
Givethedatatoboth!Haveexpertshelpstudents,
advisors,facultyinterpretdata:shapedecisionsusingbehavioralscience,choice
architecture,nudges
Information&advisingsystemsalwaysfaceaspectrumofagency.What’snewisthe
richnessofinformationandanalysis.
Todothis,youneedtoolswhichprotectprivacy
whilesharinginformation,professionallydesigned
fortheirusers.
UMDigitalInnovationGreenhouse
Takegoodideasdevelopedoncampusfrominnovationtoinfrastructure,supportEd-TechR&D,personalize
educationatscale
UniversityTeaching
Community
UNIZIN
Startups
ExternalResearchFunding
ResearchFindings&
Pubs
Innovators&pioneeringadopters
DIGteamofDevelopers,
U/XDesigners,BehavioralScientists
Communitiesofpractice:faculty,students,staff
UniversityResearch
Community
UniversityIT:supportat
scale
DIG:ahomeforacademicR&D
DIGwasborninMay2015:Aplace,ateamofinnovators,
originallyinDEILabonWashington
DIGhasgrown,andnowlivesatopourlibrary
DIGteamconnectsfaculty/staff
FACULTYDIRECTORTimMcKay
OPERATIONSDIRECTORMikeDaniel
FACULTYCHAMPIONSGusEvrard(LSA)BarryFishman(SI)ElisabethGerber(Ford)AnneGere(Sweetland)TimMcKay(LSA)PerrySamson(ENG)GingerSchultz(LSA)
LEADBEHAVIORALSCIENTISTHollyDerry
LEADDEVELOPERSBenHaywardCait HolmanKrisSteinhoffChrisTeplovs
LEADINNOVATIONADVOCATEAmyHomkes-Hayes
BEHAVIORSCIENTISTCarlyThanhouser
DATASCIENTISTKyleSchulz
DEVELOPERSDaveHarlanKushank RaghavOliverSaundersKe Ye
UX&DESIGNMarieHooperKristinMillerMikeWojan
Plus15-20studentfellowsdrawnfromComputerScience,SocialPsychology,Art&Design,UIDesign,BehavioralScience,
Education,&more….
ART2.0– informationtoall
Coursecardswillbejoinedbyreportsoncoursesofstudy(majorsandminors)andpeople(students,faculty),alongwithtoolsforcurriculumexploration…
Expertinterpretationandadviceatscale
ECoach:computertailoredelectroniccoachingforequityandstudentsuccess
Experttailoredcommunication
• Builton20+yrs ofdigitalhealthcoaching• Aggregatesrichstudentinfofrommanysourcestotailorfeedback,encouragement,&advice
• Tailoringonbothwhat tosay,howtosayit,whospeaks:w/testimonialsfrompeers,etc.
• Allcontentwritten&testedbybehaviorchangeexperts,facultyfromdisciplines,students
• Atoolforhumanepersonalizationw/studentagency:allowingustospeak,sharedata,connect
ECoachisexpandingtomoreclasses,launchingatotherinstitutions,andsupporting
richarrayofresearchprojectsw/externalfunding.
Thisfall:8000studentsStats250EECS183,280Chem 130Physics140Bio171Econ101Engr 100,101ALA125FirstYear (6800more…)UCSantaBarbara
ECoachFuture ECoachsupportsresearchaddressingwidespreadgenderedperformancedifferences
inSTEMlecturecourses
ThisinterventionlaunchedthisfallasanRCTwithmorethan1000studentsineachofthetreatmentandcontrolarms.Firstofaseriesofupcoming
experimentsdeliveredinECoach
LearningtorespondtostudentwritingusingNLPetc.isa
majorgoalforthecomingyear.
FocusonFoundationalCourses• Large,relativelystable,
mostlyintroductorycourses• Servestudentswith
especiallyvariousbackgrounds
• Servestudentswithespeciallyvariousinterestsandgoals
• Foundationalcourseswhereweeducateatscaleareidealenvironmentsfortheapplicationofanalytics
• Bestlargecoursestaughtinmultigenerationalteamswithrolespecialization
• Rolesinclude:– Coursemanagement– Deliveryofinstructionon
largeandsmallscales– Instructionaldesign– Technology– Assessment&analytics– Studentsupport
• Coursesshouldbebroadly‘instrumented’forstudy
These‘foundational’coursesexistacrossmanydisciplines,mostofwhichareoutsidethenaturalsciences,sothisinitiativeiscampus-wide.
ALearningHigherEdSystem
DigitalInnovationGreenhouse
Largecourseteamdevelopsandsupportsatechnicalinfrastructureforresearch– gatheringdataandimplementinginterventionstudies
FoundationalCourseInitiative
CCDprocessprovidesasocialinfrastructureforsustainedresearchanddevelopment–practicegeneratingresearch
DataandTechnology People&SocialSystems
TranslationalResearchonEducationatScale:
TheFCIandDIGarecreatingthesociotechnicalframeworkneededtosupportrichtranslationaleducationresearch.
WearebringingresearchteamsintothisLearningLaboratory,intimatelyconnectingresearch&practiceintheauthentic,evolvingenvironmentofour
foundationalcourses.
Fiveyearsfromnow…
• Carefullydesignedandinstrumentedfoundationalcoursesestablishedasoneofthekeyelementsofalearninglaboratory
• Awellestablishedprogram,with~20-30coursesestablishedaspartofthislaboratory
• Weaimtoplayanimportantroleinestablishingarobustevidence-basisforlearninganalytics&personalizationatscale
Educating@scalein21st century• Teachingatscaleinthe
informationageaffordsunprecedentedopportunitiesforpersonalization
• Realizingtheseisamajorsociotechnicalchallenge
• WeareaddressingboththesocialandtechnicalchallengesassociatedwiththistaskatMichigan
• Ourcampusiscreatingalaboratoryforlearningatscale,connectingeducationresearch&practice
• Thiswillbecomea“learninghighereducation”communityinwhichwelearncontinuouslyfromexperienceincontext
• Itcanallbedoneinastudent-centeredwaywhileprotectingprivacy