The Problems and Promise of Big Data in Advising...

TheProblemsandPromiseofBigDatainAdvising

AdrienneSewellDirectorofAdvisingforRetentionandSophomoreInitiativesChiefEditor,JournalofAdvisingUniversityDivisionIndianaUniversityBloomingtonasewell@indiana.edu

Whenitcomestodata,wearen'talwayssurewhatwearelookingat….

Expectations….

ItwillsearchlikeGoogle!

ItwillmakerecommendationslikeNetflix!

Itwillfindwho’satrisksowecanretainthem!

BigDatacansolveanything!

• “BIGdataissuddenlyeverywhere.Everyoneseemstobecollectingit,analyzingit,makingmoneyfromitandcelebrating(orfearing)itspowers.….Bycombiningthepowerofmoderncomputingwiththeplentifuldataofthedigitalera,itpromisestosolvevirtuallyanyproblem— crime,publichealth,theevolutionofgrammar,theperilsofdating— justbycrunchingthenumbers.”

BUT• “Bigdataispronetogivingscientific-soundingsolutionstohopelesslyimprecisequestions.”

Eight(No,Nine!)ProblemsWithBigData,GARYMARCUSandERNESTDAVIS,NewYorkTimes

Predictiveanalyticsystemsinadvising

• Thesesystemspromiseto“increaseretentionandgraduationratesatatimewhenoutcomesareunderscrutinyandfundingforadditionalacademicsupportishardtocomeby.”(Korn,2012)

Predictiveanalytics,datamining,patternrecognition,machinelearning

PedroDomingos (TheMasterAlgorithm:HowtheQuestfortheUltimateLearningMachineWillRemakeOurWorld)FirstStage• Computerswereprogrammedbyus• Programmerslookedatdataandmadehypothesis/tests/redologic/etc

SecondStage• Computersprogramthemselves• Google:searchresults• Netflix:predictswhatyouwouldliketowatch• Smartphoneslearnaboutus:typos,voicerecognition,routesonGPS,howyouwalk

Machinelearning

• Toteachcomputerstohavetheabilitytolearnwithoutbeingprogramed• tonotonlyhavethelogictoanswerquestions,butalsowhatthequestionsaretobeginwith

• Relyon“masteralgorithm”• Datamining• Logictreesandinversededuction

• SomeconsideratypeofArtificialIntelligence• Ourjobistomakesuretheyaredoingwhatwewant

• Input(settinggoals)• Output(yougetwhatyouaskedfor)

• TheDanger:Machineswillgiveuswhatweaskforandnotwhatwewant

Whataboutethicsandprivacy?

• Privacyissuesinmachinelearning• Machineskeepingtrackofeverythingwedo

• “Peopleworrythatcomputerswillgettoosmartandtakeovertheworld,buttherealproblemisthattheyaretoostupidandtheyhavealreadytakenovertheworld.”

PedroDomingos,TheMasterAlgorithm

Predictionsareonlyasgoodasthealgorithm

Whatifpredictionsarewrong?

• “Consistently,andusingsomeofthemostsophisticatedpredictiveanalyticstoolsintheworld…theworldwasallbutcertainoftheoutcomeofthe2016American election.”• “Whatwelearnedfromthiselectionisthatrelyingexclusivelyon“BigData”canmissthegoldoftenhiddeninsomemuchsmallerandmuchdifferentdatapockets.”

KarenWebster,7DeadlyDataSinsofthe2016Election

PredictiveAnalyticsinAdvising

• "Old-schooladvisingisaboutwhoappearsinfrontofyou—it’sverylimited,"saysRichardD.Sluder,ViceProvostforStudentSuccess."New-schooladvisingisusingpredictiveanalyticstotargetaspecificgroup.“

SpotlightonRetention:Studentscan’tgraduateiftheydon’treturn

EricHoover,ChronicleofHigherEducation

• Butisthistrue?• Advisorsdataset:experiencewithotherstudents• UniversityDivision:outreach/rosterreview(reviewofstudentrecords)foryears

StudentSuccessCollaborative(SSC)

• EducationAdvisoryBoard(EAB)

SSCPilotFall2013atIUBloomington

• Onlyafewunits• Riskandsuccessmarkersnotveryuseful• Filtersuseful

• Fall2014fullrelease• Createdexpectationsforuse• Incorporatedfiltersforoutreach“campaigns”• Trackedusage• Reportedissues/errors

• Fall2017cancel/notrenew• Whatdidwelearn?• Wheredowegofromhere?

Whatwasaimedtobesimplecameacrossassimplistic

Issues

• Blackbox:whatarethecorrelations?• Predictionsforallstudentsnomatterhowweakthecorrelation• Needforadvisors’input

Wheredowegofromhere?

• Whendevelopingsystemsacknowledgethatadvisorshaveknowledge• Advocateforadvisorinput/testingindevelopment• Recognizethatnotallpredictionshaveequalaccuracy• Askquestions,checkoutputs,adjustquestions

Advisorsasexpertsofthetext(studentrecords)

CourseRecommendercreatedbyBloomingtonAssessmentandResearch• Creatingpredictionsforcommonlytakencoursestopredictsuccessinotherclasses• Calculusgrade:canitpredictoutcomeinBiology?Businessclasses?

• Testingagainstadvisorknowledge• Whatcategoriesofinformationdoadvisorsusetoaccessrisk?• Doesdotheanalyticsanything?

• AnApplicationofParticipatoryActionResearchinAdvising-FocusedLearningAnalytics,StefanoFiorini,AdrienneSewell,MathewBumbalough,Pallavi Chauhan,LindaShepard,GeorgeRehrey andDennisGroth

PredictionAccuracy

• Onlyshowspredictionswithastrongcorrelation• Validactionableinformation• Workingtoavoidrateoffalsepositives

• Mostsystemsprovideapredictionforallstudents• Lessaccuracy

CourseRecommender

Behavior

Difficult course

GPA

Inconsistent performance

Prerequisite

Quantitative

Repeat course

Similar course

Test scores

Withdrawal Incomplete

AAAD-A380

ABEH-A200

ANAT-A215

ANTH-A122

ARTH-A101

ARTH-A102

AST-A105

BIOL-L112

BIOL-L113

BIOL-L211

BIOL-L311

BIOL-L312BIOL-L318

BIOL-L321

BIOL-M200

BIOL-M250

BIOL-M315

BIOL-M485

BIOL-P451

BIOL-Z406

BIOL-Z460

BUS-A200

BUS-A201BUS-A202

BUS-F300

BUS-G300

BUS-K201

BUS-L201

Behavior

CHEM-C101

CHEM-C117

CHEM-C127

CHEM-C341

CHEM-C342

CHEM-C343

CHEM-S343

CJUS-K300

CLAS-C101

CLAS-C206

CLAS-L250

CMLT-C110

COGS-Q320

COLL-C103

COLL-C104

COLL-P155

CSCI-A110

CSCI-A290

Difficult course

EALC-E110

ECON-E201

ECON-E202

ECON-E370

EDUC-P248

EDUC-Q200

EDUC-X150

EDUC-X158

ENG-L204

ENG-L389

ENG-W103

ENG-W131

ENG-W231

FINA-N198

FRIT-F150

FRIT-F200

GEOG-G109

GEOG-G306

GEOL-G105

GNDR-G225GNDR-G335

GPA

HISP-S105

HISP-S200

HIST-A383

HIST-H102

HIST-H206

HIST-J300

INFO-I101

INFO-I201

INFO-I202

INFO-I210

INFO-I308

INTL-I204

Inconsistent performance

LATS-L396

LSTU-L100

MATH-D116

MATH-D117

MATH-M118

MATH-M119

MATH-M120

MATH-M18

MATH-M211

MATH-M311

MATH-V118

MATH-V119

MSCH-C101

MSCH-C207

MSCH-C226

MSCH-J300

MSCI-M216

MSCI-M375

MUS-Z111

PHIL-P150

PHSL-P215

PHYS-P150

PHYS-P201

PHYS-P202

PHYS-P221

POLS-Y100

POLS-Y103

POLS-Y109

PSY-K300

PSY-P101

PSY-P102

PSY-P304

PSY-P324

PSY-P335

PSY-P337

Prerequisite

QuantitativeREL-A235

REL-R160

Repeat course

SOC-S320

SOC-S321

SOC-S339

SOC-S371

SPEA-E162SPEA-E311

SPEA-H124

SPEA-H402

SPEA-K300

SPEA-V160

SPEA-V161

SPEA-V220

SPEA-V236

SPEA-V246

SPEA-V252

SPEA-V261

SPEA-V370

SPEA-V372

SPEA-V406

SPH-C213

SPH-E311

SPH-H150SPH-I187

SPH-K150

SPH-M333

SPH-M382

SPH-M418SPH-N231

SPH-N331

SPH-R142

SPH-R200

SPH-R210

SPH-W147

SPH-Y277

SPHS-A200

STAT-S100STAT-S301

STAT-S303 STAT-S320

SWK-S300

Similar courseTest scores

Withdrawal Incomplete

StudentSuccess

• Analyticsisonlyonepieceinastudentsuccesssystem,whichrequires“commitmenttopersistent,personalizedactionsandinterventionstoimprovestudentsuccessguidedbyanalytics-basedinsights.”(BaerandNorris,2013)

Ethicsinassessingrisk

• Whendoourdatapointsbecomeethicalissues?• Whatabouteconomicbackground,financialneed,race,etc?

• Canassessingriskbecomeaself-fulfillingprophecy?• Approachesmatter

Dataalonewon’tsaveus

• Datacanhelpanswerquestions• Datacanhelpusdescribe/discoveranpattern• Datacanhelpfiguringoutstudentstoreachoutto

Advisingmustcontinuallyadjustandusedatawisely• Innovatewaysofusingdata• Usedatatoinformapproaches• Approachesarecriticalinstudentsuccess!

Creativeusesofdata

Wordusagepatternsfromaprobationexercise(Voyant)

Aid+money+financial+scholarship+scholarships=39

Parents+family+mom+mama+dad=62

Kicked(out)+dismissed+expelled=57

Grades+gpa =64

Life+future=38

In-housereports

Tableaureports

Dataiscriticallyimportantsoweneedto…

• Haveabasicunderstandingofwhatthedataweareusingmeans• Learnhowtouseitwiselyandcreatively• Askgoodquestions• Questionoutputswhentheydon’tmakesense

Data…

• Baer,L.L.,&Norris,D.M.(2015).Whateveryleaderneedstoknowaboutstudentsuccessanalytics(WhitePaper).Civitas Learning.

• Bumbalough,M.,Chauhan,P.,Fiorini,S.,Groth,D.Sewell,A.,P.,Shepard,L.,Rehrey,,AnApplicationofParticipatoryActionResearchinAdvising-FocusedLearningAnalytics,(unpublished)(2017)

• Burns,B.(2016,January29).BigData’sComingOfAgeInHigherEducation.Forbes.Retrievedfromhttp://www.forbes.com/sites/schoolboard/2016/01/29/big-datas-coming- of-age-in-higher-education/#14fd578a2a32

• Buyarski,C.,Murray,J.,Torstrick,R.,LearningAnalyticsAcrossaStatewideSystem,NewDirectionsforHigherEducation(Fall2017)

• Domingos,P.TheMasterAlgorithm: HowtheQuestfortheUltimateLearningMachineWillRemakeOurWorld (2015)

• Korn,M.(2013,October8).CollegesMineDatatoHelpStudentsStayonCourse.WallStreetJournalhttp://www.wsj.com/articles/SB10001424127887324123004579057200819823872

• Neff,G.(2013).Whybigdatawon’tcureus.BigData,1(3),117–123.http://doi.org/10.1089/big.2013.0029• MARCUS,G.&DAVIS,B.Eight(No,Nine!)ProblemsWithBigData,NewYorkTimes• Webster,K7DeadlyDataSinsofthe2016Electionhttp://www.pymnts.com/big-data/2016/the-seven-deadly-data-sins-of-the-2016-american-election-for-president/