Developing a Code of Practice for Learning Analytics · Developing a code of practice for learning analytics. ... setting up appropriate governance structures, developing a taxonomy

(2016).Developingacodeofpracticeforlearninganalytics.JournalofLearningAnalytics,3(1),16–42.http://dx.doi.org/10.18608/jla.2016.31.3

ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 16

Developing a Code of Practice for Learning Analytics

NiallSclaterJisc/SclaterDigital,[email protected]

ABSTRACT:Ethicalandlegalobjectionsto learninganalyticsarebarrierstodevelopmentofthefield,thuspotentiallydenyingstudentsthebenefitsofpredictiveanalyticsandadaptivelearning.Jisc,acharitableorganizationthatchampionstheuseofdigitaltechnologiesinUKeducationandresearch,hasattemptedtoaddressthiswiththedevelopmentofaCodeofPracticeforLearningAnalytics. The Code covers the main issues institutions need to address in order to progressethicallyandlegally.Thispaperoutlinestheextensiveresearchandconsultationactivitiescarriedouttoproduceadocumentthatcoverstheconcernsof institutionsand,critically,thestudentsthey serve. The resultingmodel for developing a codeof practice includes a literature review,settingupappropriategovernancestructures,developingataxonomyoftheissues,draftingthecode,consultingwidelywithstakeholders,aswellaspublication,dissemination,andembeddingitininstitutions.Keywords:Learninganalytics,ethics,privacy,legalissues,codeofpractice

1 INTRODUCTION Ethicalandlegalissuesarealmostinvariablyraisedwhenevertheuseoflearninganalyticsisproposedininstitutions.Concernsareexpressedinparticularaboutpotentialinvasionsofstudentprivacyarisingfromthemisuseof theirdata,andabout theadverseconsequences thatmightarise frommonitoring theiractivity and predicting their future academic success. Such issues have become impediments to thedevelopmentandrolloutoflearninganalytics,insomeinstitutionshaltingtheimplementationoflearninganalyticscompletely.ThemostnotoriousexampleisinBloom,aninitiativedevelopedwith$100millionfunding from the Gates and Carnegie Foundations, which developed mechanisms for storing largeamounts of data relating to US schoolchildren and their learning activities. In the post-Snowden era,sensitivitiesaroundprivacywere runninghigh, communicationswerebadlyhandled,and familiesandprivacyadvocatesultimatelyforcedtheclosureoftheprogramme(K.N.C.,2014).Soonafterwards,Facebook’sfamous“moodexperiment”placedpositiveandnegativeitemsandimagesinthetimelinesof700,000userstofindoutifthesewouldaffectusers’moods.Thisresultedinahugebacklash from users and extensive negativemedia coverage, forcing changes to Facebook’s researchmethodsandpolicies(Shroepfer,2014).Whileconcernsaboutprivacy,dataprotection,andethicsraisedbystudentsandstaffateducationalinstitutionsaregenerallyvalidandmustbeaddressed,theensuinghiatusmeansthatlearnersarebeingdeniedthepotentialbenefitsoflearninganalyticsthatcanhelptoidentifyareasforimprovementandultimatelymakethedifferencebetweencompletingtheircourseanddroppingout.Usingpersonaldatatopresentanalyticsandinforminterventionsthatmaysignificantlyaffectstudents’



livesdoes,ofcourse,bringwithitseriousresponsibilities.KingandRichards(2014)arguethatweareinacriticalwindowandthatwhateverethicalpracticesareestablishednowinthefieldofbigdatawillaffectnotions of acceptability for years to come. The sheer amount of data that can now be collected onindividuals,andtheinsightthatcanbegainedfromitsanalysis,enablefarmoretobelearntaboutpeoplethanwaseveranticipated(PCAST,2014).Algorithmsnowexistthatcandiscoverthingsaboutyoubeforeyouknowthemyourself.Theimplicationsfortheprivacyoflearners,andthepotentialformisuseofthedatacollectedandtheanalyticsperformedonit,necessitatetheuseofcarefullyconsideredpoliciesthathelpinstitutionstoactethically,staywithinthelaw,andminimizeadverseimpactsonindividuals.Suchframeworks,however,shouldnotunnecessarilyholduptheprogressbeingmadeinthefieldoflearninganalyticsthatpromisesrealbenefitsforstudentsandinstitutions.Educationalresearchershavebeenarguingforsometimefortheneedforasetofprinciplesoracodeofethicsasthefieldoflearninganalyticsdevelops.Ferguson(2012)recommendsanethicalframeworktohelp institutionsmakedecisionsregardingtheownershipandstewardshipof learners’data.However,PardoandSiemens(2014)suggestthiswillbedifficultasinstitutionsalreadystruggletodefineprivacypolicies in other areas. Berg (2013) believes thatwithout a code of ethics, institutionsmay carry outanalyticsinarbitraryways,thusreducingconsistencyandfairtreatmentforstudents.Heproposesthatsuchadocumentwouldhelptoalleviatedifferencesinapproachestodealingwithanalyticsbyteachersandseniormanagers.Addintothemixtheviewpointofstudentswhomaynotwishtohavetheirpersonaldatacollected,letaloneacteduponthroughan“intervention,”andyouhaveatoxicmixofexpectations—unless commonunderstanding can be achieved throughmutually agreed policies. Indeed,withoutaddressingtheethicalissues,users(bothemployeesandstudents)mayactivelyresisttheintroductionoflearninganalytics(Greller&Drachsler,2012;Siemens,2012).Students are increasingly accustomed to having their data collected by commercial organizations orgovernmentagenciesforarguablyfarmoreintrusivepurposesthanlearninganalytics.Theymaythereforeshowlittleresistancetothecollectionanduseoftheirdatabytrustededucational institutionsforthepurposes of enhancing their education. However, developing a clearly articulated set of principles tohandle studentdataandany interventionsappropriately canbeapreventativemeasure,pre-emptingbacklashesfromuserssuchasthoseexperiencedbyInBloomandtheFacebookmoodexperiment.Ithasbeennotedtoothatemployeesusuallyprefertoworkfororganizationscommittedtoethicalstandards,andthatconsumersliketobuyfromcompanieswith“strongrecordsofadherencetostandardsofconductand socially sensitive behavior” (PABC, 2007). It is likely that employees and students of educationalinstitutions are no different in this regard; arguably, universities and colleges have an even strongerimperativetoactethicallyandlegallyforthebenefitoflearners,andtodemonstrateclearlyhowtheyaredoingso.Transparencyiskeyhere:itisintheinterestsofstudents,staff,andinstitutionsthattheusestowhichlearninganalyticswillbeputareexplainedasclearlyaspossible.Thereisarisk,asSlade&Prinsloo(2013)point out, that learning analyticswill fail to be adopted successfully unless perceptions aremanagedcarefully.Otherindustrieshavealreadydevelopedcodesofpracticefortheuseofdataandanalytics,as



havevariousprofessionsandcommunitiesofpracticewithineducation.ItisworthexaminingthebenefitstoorganizationslistedbytheUKInformationCommissioner’sOfficeoffollowingitscodeofpracticeforpersonalinformationonline(ICO,2010):

• Greatertrustandabetterrelationshipwiththepeopleyoucollectinformationabout• Reducedreputationalriskcausedbytheinappropriateorinsecureprocessingofpersonaldata• Better take-up of online services, meaning economic savings and greater convenience for

customers• MinimizedriskofbreachesandconsequentenforcementactionbytheInformationCommissioner

orotherregulators• Gaining a competitive advantage by reassuring the people you deal with that you take their

privacyseriously• Increasingpeople’sconfidencetoprovidemorevaluableinformation,becausetheyarereassured

thatitwillbeusedproperlyandkeptsecurely,and• Reducedriskofquestions,complaintsanddisputesaboutyouruseofpersonaldata.

Jisc, theorganization responsible formany aspects of IT infrastructure and learning technology inUKhigher and further education, has been leading an initiative to promote the effective use of learninganalytics(Sclater,2014a).Workingcloselywithstakeholdersaspartofa“co-design”process,1tacklingissuesrelatingtoprivacyandethicswasidentifiedearlyonasapriority.ItwasfeltthataCodeofPracticeforLearningAnalytics(Sclater&Bailey,2015)wasessentialinorderforprogresstobemade.Thiswouldidentify themain legal andethicalbarriers toprogressand suggestways for institutions toovercomethem.ThedevelopmentofJisc’sCodeofPracticehasinvolvedfivestages:

1. Anextensiveliteraturereviewofthelegalandethicalissuesaroundlearninganalytics2. TheformationofanadvisorygrouptooverseethedevelopmentoftheCodeofPractice3. Thedevelopmentofataxonomyofethical,legal,andlogisticalissues—anditsvalidationwith

nationalandinternationalexperts4. ThedraftingoftheCodeofPractice—anditsvalidationwithstakeholdersacrossthesector5. Thepopulationofanaccompanyingwebsitewithfurtherguidanceandcasestudies.

Thispaperdiscusseshowthesestageshavecontributedtoacodeofpracticeinformedbytheliteratureandbyexpertsinthefield,achievingconsensusthroughextensiveconsultation,andtheinvolvementofstudents. It also defines amodel for this process and briefly outlines how this could be used for thedevelopmentofothercodesofpractice.

1“Co-design”isJisc’scollaborativetechnologyinnovationmodelthatusescustomerprioritiestoidentifynewopportunitiesandaddresspressingissuesineducation.



2 CARRYING OUT THE LITERATURE REVIEW For the literature review (Sclater,2014b),publications frommanydifferentauthorsandorganizationswere gathered.Materialwasdrawn fromeighty-six documents,more thana thirdof thempublishedwithintheprecedingyear,fromsourcesincluding:

• Theliteraturearoundlearninganalyticsthatmakesexplicitreferencetolegalandethicalissues• Articlesandblogsaroundtheethicalandlegalissuesofbigdata• Afewpapersthatconcentratespecificallyonprivacy• Relevantlegislation,inparticulartheEuropeanDataProtectionDirective1995andtheUKData

ProtectionAct1998• Relatedcodesofpracticefromeducationandindustry

Expressingissuesasquestionscanbeausefulwayofmakingcomplexissuesmoreconcrete.Ninety-threequestionsfromtheliteraturethatauthorshadposeddirectlywereincorporated.ThecategorizationsofthesehighlightedinthewordcloudshowninFigure1giveaninstantflavourofthemainconcernsaroundtheimplementationoflearninganalyticsbeingraisedbyresearchersandpractitioners.

Figure1:MainconcernsintheliteraturearoundlearninganalyticsasidentifiedbySclater(2014b).

Attheendoftheliteraturereview,sixteencodesofpracticeandlistsofethicalprinciplesfromrelatedfieldswere reviewed. Itwas found that themain concepts their authors attempted to embodyweretransparency,clarity,respect,usercontrol,consent,access,andaccountability—allofwhicharehighlyrelevantandcorrespondwiththeconcernsbeingraisedbyresearchersandpractitionersinthefieldoflearninganalytics.3 DELIBERATIONS OF THE ADVISORY GROUP

AnimportantpartoftheprocessfordevelopingtheCodewastheformationofanadvisorygroupwithfifteenmembersfromthehigherandfurthereducationsectors(Sclater,2015a).Thegroupincludedaprovice-chancellor,academicswithexpertiseinpolicy,ethics,andlearninganalytics,seniorITstaff,alegalexpert,andarepresentative fromtheUK’sNationalUnionofStudents (NUS).2MemberswereabletoadviseontheapproachtoproducingtheCode,theareastobecovered,waystogainfurthervalidation2TheinstitutionsrepresentedwereCroydonCollege,EdinburghUniversity,HuddersfieldUniversity,LancasterUniversity,LondonSouthBankUniversity,LoughboroughUniversity,Jisc,NationalUnionofStudents,andTheOpenUniversity.



fromthecommunity,disseminationandadoption.ItwasagreedthatthemainpurposeoftheCodewouldbetohelpremovebarrierstotheadoptionoflearninganalytics,andthatitshouldprovideafocusforinstitutions todealwith themany legal andethicalhurdles that theywereencountering.Alongsideaconcisesummarydocument,theguidancecouldbepresentedasanevolving,dynamicsiteratherthanalengthyone-offdocumentthatwouldbelesslikelytoberead,letaloneadheredto.MembersalsoagreedtocritiquetheCodeasitwasbeingdevelopedandtoconsiderpilotingitattheirowninstitutions. 3.1 Methodology and Approaches Somedocumentsofthisnaturetakeaparticularmethodologicalorphilosophicalstance.For instance,SladeandPrinsloo’s(2013)socio-criticalapproach—wherelearninganalyticsisviewedasa“transparentmoralpractice”andstudentsareseenasco-contributors—hasinfluencedtheOpenUniversity’sPolicyonEthicalUseofStudentData(OpenUniversity,2014).TheadvisorygroupsuggestedthattheCode’semphasiswouldbe“positive,realisticandfacilitative”andthatitshouldemphasizethatlearninganalyticsisprimarilyforthebenefitofstudents.ThegroupconsideredthatoneofthemainchallengesofdevelopingtheCodewouldbetostrikeabalancebetweenapaternalisticapproachandrespectingstudents’privacyandautonomy.AnapproachthatputtheneedsoflearnersattheheartoftheCodewasthoughtlikelytoresultinabetter,morewidelyadopteddocumentandhelptoallaythefearsofstudentsandinstitutions,hencefacilitatingtheuptakeoflearninganalytics.TheinclusionoftheNUSinthisgroupwasthereforeparticularlywelcome.Wouldaseparatecodeofpracticeor“billofrightsforlearninganalytics”ownedbyandforstudentshelptogainacceptance?Orcouldthisdivergesomuchfromtheonerepresentinginstitutionalconcernsthatitwouldexacerbatethedifferencesandcreateconflict?Combiningallinterestsinonedocumentwouldrequireabalancedapproachandaseriesofcompromisesbuthopefullyencouragemutualunderstanding,resultinausableCode,andmovethefieldoflearninganalyticsforwardcollaboratively.Theadvisorygroupconcludedthatasingledocumentclearlysettingouttherightsandresponsibilitiesofstudents,institutions,andstaffwouldbepreferable.ExplainingwhattheCodemeansinpracticehoweverwould require separate advice for different stakeholders. At institutions, the Code should ideally linkcloselywithdocumentssuchasthestudentcharter,andensurebuy-infromthestudentunion.AnotherissueraisedwaswhethertheCodecouldbeatasufficientlyhighleveltomeettheneedsofallinstitutions while remaining specific enough to provide genuinely helpful guidance. Researchers andseniormanagerswithresponsibilityforimplementinglearninganalyticsatarangeofinstitutionshadbeenapproachedearliertoreviewthedevelopmentoflearninganalyticsattheirinstitutions(Sclater,2014d).From a series of semi-structured interviews, it had become clear that the potential uses of learninganalyticsandtheconcernsraisedvariedwidelyacrossinstitutions.TheadvisorygroupthoughtthattheCodeshouldbefairlyhighlevelinordertoproveusefultoall,butshouldbebackedupbycasestudiesandexamplesofhowinstitutionshavedealtwithparticularissues.Thecasestudiescouldbepresented



alongsidetheCode—foreachprinciple,therecouldbeexamplesofgoodpractice.Anotherquestionraisedwaswhether institutionsshouldbeencouragedtoadopttheCodewholesale,and therefore be able to claim conformancewith it, or to use itmore as a checklist of issues to beconsideredandcustomizedtofittheirowninstitutionalpolicies.Thelatterapproachseemedmorelikely,andseveraluniversitieshavealreadysuggestedthattheywilluseitasthebasisfortheirownlearninganalyticspolicies.Particular concern was expressed that the Code must reflect the human context and the need forintermediationoflearninganalyticsbystaff.Thisisacommonethicalthemeintheliterature.However,arepresentativefromTheOpenUniversitysaidthatthesheerscaleofthatinstitutionmakesitunfeasibletousehumanintermediationformanyofthepotentialusesoflearninganalytics.Meanwhiletherewasastrong recommendation that the language used to present analytics to students should be carefullyconsideredandthatdatashouldonlybeexposedwheninstitutionshavemechanismsinplacetodealwiththeeffectonstudents.ThepotentialimpactofanalyticsontheeducatoralsoneededtobereflectedintheCode.3.2 An Appropriate Format for the Code of Practice Most codes of practice are textual documents, normally provided in PDF. The members felt that adocumentoutliningtheprinciplesneededtobeprovidedinordertopresentittoinstitutionalcommitteesbutthataninteractivewebsitecontainingcasestudies,perhapsintheformofvideoedinterviewswithstaffandstudents,wouldbewelcome.Many codes of practice or “codes of ethics” are extremely lengthy and somewhat uninspiring papersstretchingtothirtypagesormore.OneofthemorereadableexamplesistheRespectCodeofPracticeforSocio-Economic Research (RESPECT Project, 2004). It is concise—only four pages— and reasonablyvisuallyappealing,thereforearguablymorelikelytobereadandabsorbedbybusypeoplethansomeofthelongercodes.However,giventhelargenumberofissuesidentifiedintheliteraturereview,fourpageswerethoughtunlikelytobesufficient.Theagreedapproachwastobackupaconcisesummarydocumentwithmoredetailedonlineguidanceforeachof theareas. The literature reviewcoversmostof theethical and legal issues likely tobeofconcerntostudentsandtoinstitutionswhendeployinglearninganalytics;thewordcloudsinthereviewcouldhelpprioritizethemainareastobeincludedinthedocument.Supportingcontent—e.g.,videoedinterviews—couldbedevelopedsubsequently,assistinraisingawarenessoftheCode,provideexamplesofhowitisbeingimplemented,andhelptokeepituptodate.Discussionforumscouldbeincludedoneachtopic,enablinguserstoraisefurtherissues,andotherstoprovideadviceonhowtheyhavetackledthat challenge. Thiswouldneed someongoingpromotion, facilitation, andmoderationby Jisc and/ormembersofthecommunity.



3.3 Dissemination and Rollout Asenseofownershipbyinstitutionsandbystudentswasconsideredessentialtoensureadoption.Howcouldthisbestbeachieved?Arangeofstakeholderorganizationswasproposedforconsultationandanumber of possible events to piggyback on were proposed as dissemination opportunities. SeveralmemberssaidtheywouldbekeentotrypilotingtheCodeattheirinstitutionstoo.Itwasalsosuggestedthat vendors should be included in the consultation process. It might help them when makingdevelopmentdecisions,encouragingthemforinstancetobuildconsentsystemsintotheirproducts.TheCodecouldhelptoensurethatsafeguards,suchasensuringprivacy,areincorporatedwithoutholdingbackinnovation.Onememberoftheadvisorygroupsuggestedthatitwouldbeusefultobetterunderstandtheprocessesinsideinstitutionsforgettingacademicpoliciesadopted,asthiswouldbekeytouptake.Inaddition,someeventsspecificallyaroundtheCodecouldbeheld,andpapersdeliveredatrelevantconferences.Itwasfelt that theCodeshouldbe launchedwithsome fanfareata largerevent to increaseawarenessandpotentialtake-up.4 DEFINING AND VALIDATING A TAXONOMY OF ETHICAL, LEGAL, AND LOGISTICAL ISSUES Afeweventshavecentredaroundissuesofethicsandprivacyinlearninganalytics,notablyaworkshoporganized by LACE and SURF in Utrecht, Netherlands, in October 2014 (Sclater, 2014c) and a pre-conferenceeventwithpeer-reviewedsubmissionsattheLearningAnalyticsandKnowledgeConference(LAK ’15) in Poughkeepsie,USA.3 In suchdiscussions at conferences andwithin institutions, the sameissues are continually raised but have generally already been covered somewhere in the growingcollectionof publicationson learning analytics. Sometimes the issue is expresseddifferentlybutboilsdowntothesameunderlyingproblem.TheliteraturereviewproducedfortheCodeofPractice(Sclater,2014b)isalargeandunwieldydocument,sotheissuesandquestionsdetailedinitweredistilledfromthetextlinebylineintoamoresuccincttabularformat.Thewordcloudsintheliteraturereviewwereusedasabasisforgroupingtheissues.TheresultingtaxonomyinTable1includeseighty-sixdistinctissues(Sclater,2015c).Eachisgivenanameandexpressedasaquestionthatattemptstocapturetheissueconcisely.Manyofthequestionscannotofcoursebeansweredsimply;almostallcouldberespondedtowith“Itdepends...”Anattemptwasmadetoclassifythemaseitherprimarilyethicalorprimarilylegalinnature.Mosthavebothanethicalandalegaldimension;aslawsareoftenunderpinnedbyethics,thisisnotsurprising.Whilesomewerereferredto in the literature as ethical issues, theywere however relatedmore to the logistics of carrying outlearninganalyticsininstitutionsthandoingwhat’sethicallyrightorkeepingwithinthelaw.Thus,whatstartedoutasacollectionofethicalandlegal issuesbecamealist incorporatinganumberoflogistical

3Detailsoftheeventareathttp://www.laceproject.eu/ethics-privacy-learning-analytics/



issuesaswell.Jisc,ApereoFoundation,andtheLACEProjectheldaworkshopinParisinFebruary2015(Sclater,2015b)to discuss the ethical and legal issues of learning analytics, focusing on the draft taxonomy. TwelveparticipantsfromFrance,Germany,theNetherlands,andtheUK,primarilyfromacademicbackgrounds,workedtogethertovalidateandrefinethelistofissuesandcommentontheapproach.Asaresult,thetaxonomyinTable1wasre-orderedtoreflectalifecycleviewoflearninganalytics,movingfromissuesofownershipandcontroltoseekingconsentfromstudents,encouragingtransparency,maintainingprivacy,ensuring validity in the data and the analytics, enabling student access to the data, carrying outinterventionsappropriately,minimizingadverseimpacts,andstewardingthedata.TheParisworkshopgroupsuggestedscoringtheissuesbasedontheirimportanceandstartedtheprocessofratingthemonascaleof1to5,highlightingthemost importantones.Thescalewassubsequentlyreducedtothreepoints,roughlyequatingto1)Critical,2)Important,3)Lessimportantormaynotarise.Theratingsarethesubjectiveviewofthegroupandtheauthorbasedontheirexpertiseandexperience.Amorerigorouswayofratingtheissues,seekingwiderinput,mighthavebeenhelpfulthoughtherankingwillalwaysbedependentonthenatureandprioritiesoftheinstitutionanditsstaffandstudents.Thegroupalsoaddedastakeholdercolumn.Theproblemwiththiswasthesignificantdifferencebetweenthe stakeholders most impacted and those responsible for dealing with the issue. Which should beincludedinthecolumn?Themostimpactedstakeholderswereusuallystudentssothecolumnturnedoutnottobeparticularlyhelpful.Thus,aresponsibilitycolumnwasincludedinstead,showingwhoisprimarilyresponsiblefordealingwiththeissue.Whilethismayhelpinstitutionstoassignresponsibility,again,thereisalevelofsubjectivityhereandtheseroleswillbeconstituteddifferentlydependingontheinstitution.The six typesof stakeholderwithprimary responsibility fordealingwith the issuesare categorizedasfollows:

1. Seniormanagement—theexecutiveboardoftheinstitution.2. Analyticscommittee—thegroupresponsibleforstrategicdecisionsregardinglearninganalytics.

This might be a learning and teaching committee, though some of the issues may be theresponsibility of a senior champion of learning analytics rather than a more representativecommittee.

3. Datascientist—whiletheanalyticscommitteemaydecideonparticularissues,thereisaneedfordatascientistsoranalyststoadviseonissuesrelatingtothevalidityofthedatasetandhowtointerpretit.

4. Educationalresearcher—someissueswouldbebestdealtwithbystaffwithdetailedknowledgeoftheeducationalaspectswhoareabletomonitortheimpactofanalyticsonstudents.Thisrolemaybecarriedoutbyteachersortutorsorthosemorededicatedtoeducationalresearch.

5. IT— the institutional information technology department will take primary responsibility forsomeaspectsoftheanalyticsprocesses.

6. Student—whilestudentsarepotentiallyimpactedbyalmosteveryissuehere,theyareprimarily



responsiblethemselvesfordealingwithafewofthem.

Table1:Ataxonomyofethical,legal,andlogisticalissuesforlearninganalytics.

GROUP NAME QUESTION TYPE RANK RESPONSIBILITY

Ownership&Control

Overallresponsibility

Whointheinstitutionisresponsiblefortheappropriateandeffectiveuseoflearninganalytics?

Logistical 1Seniormanagement

Controlofdataforanalytics

Whointheinstitutiondecideswhatdataiscollectedandusedforanalytics?


BreakingsilosHowcansilosofdataownershipbebrokeninordertoobtaindataforanalytics?

Logistical 2Analyticscommittee

Controlofanalyticsprocesses

Whointheinstitutiondecideshowanalyticsaretobecreatedandused?


OwnershipofdataHowisownershipofdataassignedacrossstakeholders?

Legal 1Analyticscommittee

ConsentWhentoseekconsent

Inwhichsituationsshouldstudentsbeaskedforconsenttothecollectionanduseoftheirdataforanalytics?

Legal/Ethical

1Analyticscommittee

Consentforanonymoususe

Shouldstudentsbeaskedforconsentforthecollectionofdatathatwillonlybeusedinanonymizedformats?

Legal/Ethical

3Analyticscommittee

Consentforoutsourcing

Dostudentsneedtogivespecificconsentifthecollectionandanalysisofdataistobeoutsourcedtothirdparties?


Clearandmeaningfulconsentprocesses

Howcaninstitutionsavoidopaqueprivacypoliciesandensurethatstudentsgenuinelyunderstandtheconsenttheyareaskedtogive?

Legal/Ethical

1Analyticscommittee

RighttooptoutDostudentshavetherighttooptoutofdatacollectionandanalysisoftheirlearningactivities?

Legal/Ethical

1Analyticscommittee

RighttowithdrawDostudentshavetherighttowithdrawfromdatacollectionandanalysisafterpreviouslygivingtheirconsent?


RighttoanonymityShouldstudentsbeallowedtodisguisetheiridentityincertaincircumstances?

Ethical/Logistical

3Analyticscommittee




Adverseimpactofoptingoutonindividual

Ifastudentisallowedtooptoutofdatacollectionandanalysis,couldthishaveanegativeimpactontheiracademicprogress?

Ethical 1Analyticscommittee

Adverseimpactofoptingoutongroup

Ifindividualstudentsoptout,willthedatasetbeincomplete,thuspotentiallyreducingtheaccuracyandeffectivenessoflearninganalyticsforthegroup?

Ethical/Logistical

1 Datascientist

Lackofrealchoicetooptout

Dostudentshaveagenuinechoiceifpressureisputonthembytheinstitutionoriftheyfeeltheiracademicsuccessmaybeimpactedbyoptingout?


Studentinputtoanalyticsprocess

Shouldstudentshaveasayinwhatdataiscollectedandhowitisusedforanalytics?


ChangeofpurposeShouldinstitutionsrequestconsentagainifthedataistobeusedforpurposesforwhichconsentwasnotoriginallygiven?


LegitimateinterestTowhatextentcantheinstitution’s“legitimateinterests”overrideprivacycontrolsforindividuals?


Unknownfutureusesofdata

Howcanconsentberequestedwhenpotentialfutureusesofthe(big)dataarenotyetknown?


Consentinopencourses

Areopencourses(MOOCs,etc.)differentwhenitcomestoobtainingconsent?

Legal/Ethical

2Analyticscommittee

Useofpubliclyavailabledata

Caninstitutionsusepubliclyavailabledata(e.g.,tweets)withoutobtainingconsent?

Legal/Ethical

3Analyticscommittee

TransparencyStudentawarenessofdatacollection

Whatshouldstudentsbetoldaboutthedatabeingcollectedaboutthem?

Legal/Ethical

1Analyticscommittee

Studentawarenessofdatause

Whatshouldstudentsbetoldabouttheusestowhichtheirdataisbeingput?

Legal/Ethical

1Analyticscommittee

Studentawarenessofalgorithmsandmetrics

Towhatextentshouldstudentsbegivendetailsofthealgorithmsusedforlearninganalyticsandthemetricsandlabelscreated?





Proprietaryalgorithmsandmetrics

Whatshouldinstitutionsdoifvendorsdonotreleasedetailsoftheiralgorithmsandmetrics?


Studentawarenessofpotentialconsequencesofoptingout

Whatshouldstudentsbetoldaboutthepotentialconsequencesofoptingoutofdatacollectionandanalysisoftheirlearning?


Staffawarenessofdatacollectionanduse

Whatshouldteachingstaffbetoldaboutthedatabeingcollectedaboutthem,theirstudents,andwhatisbeingdonewithit?


Privacy OutofscopedataIsthereanydatathatshouldnotbeusedforlearninganalytics?


TrackinglocationUnderwhatcircumstancesisitappropriatetotrackthelocationofstudents?


StaffpermissionsTowhatextentshouldaccesstostudentdataberestrictedwithinaninstitution?

Ethical/Logistical

1Analyticscommittee

Unintentionalcreationofsensitivedata

Howdoinstitutionsavoidcreating“sensitive”data,e.g.,religion,ethnicity,fromotherdata?

Legal/Logistical

2 Datascientist

Requestsfromexternalagencies

Whatshouldinstitutionsdowhenrequestsforstudentdataaremadebyexternalagencies,e.g.,educationalauthoritiesorsecurityagencies?

Legal/Logistical

2Seniormanagement

Sharingdatawithotherinstitutions

Underwhatcircumstancesisitappropriatetosharestudentdatawithotherinstitutions?

Legal/Ethical

2Analyticscommittee

AccesstoemployersUnderwhatcircumstancesisitappropriatetogiveemployersaccesstoanalyticsonstudents?


Enhancingtrustbyretainingdatainternally

Ifstudentsaretoldthattheirdatawillbekeptwithintheinstitution,willtheydevelopgreatertrustinandacceptanceofanalytics?


Useofmetadatatoidentifyindividuals

Canstudentsbeidentifiedfrommetadataevenifpersonaldatahasbeendeleted?

Legal/Logistical

2 Datascientist




Riskofre-identification

Doesanonymizationofdatabecomemoredifficultasmultipledatasourcesareaggregated,potentiallyleadingtore-identificationofanindividual?

Legal/Logistical

1 Datascientist

ValidityMinimizationofinaccuratedata

Howshouldaninstitutionminimizeinaccuraciesinthedata?

Logistical 2 Datascientist

Minimizationofincompletedata

Howshouldaninstitutionminimizeincompletenessofthedataset?


Optimumrangeofdatasources

Howmanyandwhichdatasourcesarenecessarytoensureaccuracyintheanalytics?


Validationofalgorithmsandmetrics

Howshouldaninstitutionvalidateitsalgorithmsandmetrics?

Ethical/Logistical

1 Datascientist

Spuriouscorrelations

Howcaninstitutionsavoiddrawingmisleadingconclusionsfromspuriouscorrelations?

Ethical/Logistical

2 Datascientist

Evolvingnatureofstudents

Howaccuratecananalyticsbewhenstudents’identitiesandactionsevolveovertime?

Logistical 3Educationalresearcher

Authenticationofpublicdatasources

Howcaninstitutionsensurethatstudentdatatakenfrompublicsitesisauthenticatedtotheirstudents?

Logistical 3 IT

AccessStudentaccesstotheirdata

Towhatextentshouldstudentsbeabletoaccessthedataheldaboutthem?


Studentaccesstotheiranalytics

Towhatextentshouldstudentsbeabletoaccesstheanalyticsperformedontheirdata?

Legal/Ethical

1Analyticscommittee

DataformatsInwhatformatsshouldstudentsbeabletoaccesstheirdata?


MetricsandlabelsShouldstudentsseethemetricsandlabelsattachedtothem?


Righttocorrectinaccuratedata

Whatdatashouldstudentsbeallowedtocorrectaboutthemselves?





DataportabilityWhatdataaboutthemselvesshouldstudentsbeabletotakewiththem?


ActionInstitutionalobligationtoact

Whatobligationdoestheinstitutionhavetointervenewhenthereisevidencethatastudentcouldbenefitfromadditionalsupport?

Legal/Ethical

1Analyticscommittee

Studentobligationtoact

Whatobligationdostudentshavewhenanalyticssuggestsactionstoimprovetheiracademicprogress?

Ethical 2 Student

Conflictwithstudygoals

Whatshouldastudentdoifthesuggestionsareinconflictwiththeirstudygoals?

Ethical 3 Student

Obligationtopreventcontinuation

Whatobligationdoestheinstitutionhavetopreventstudentsfromcontinuingonapathwaythatanalyticssuggestsisnotadvisable?


TypeofinterventionHowaretheappropriateinterventionsdecidedon?


Distributionofinterventions

Howshouldinterventionsbedistributedacrosstheinstitution?


Conflictinginterventions

Howdoestheinstitutionensurethatitisnotcarryingoutmultipleinterventionswithconflictingpurposes?


Staffincentivesforintervention

Whatincentivesareinplaceforstafftochangepracticesandfacilitateintervention?


FailuretoactWhathappensifaninstitutionfailstointervenewhenanalyticssuggeststhatitshould?


Needforhumanintermediation

Aresomeanalyticsbetterpresentedtostudentsviaatutorthanasystem?

Ethical 2Educationalresearcher

TriageHowdoesaninstitutionallocateresourcesforlearninganalyticsappropriatelyforlearnerswithdifferentrequirements?

Ethical/Logistical

1Analyticscommittee




TriagetransparencyHowtransparentshouldaninstitutionbeinhowitallocatesresourcestodifferentgroups?


OpportunitycostHowisspendingonlearninganalyticsjustifiedinrelationtootherfundingrequirements?


Favouringonegroupoveranother

Couldtheinterventionstrategiesunfairlyfavouronegroupoveranother?

Ethical/Logistical

2Educationalresearcher

Consequencesoffalseinformation

Whatshouldinstitutionsdoifastudentgivesfalseinformation,e.g.,toobtainadditionalsupport?


AudittrailsShouldinstitutionsrecordaudittrailsofallpredictionsandinterventions?


UnexpectedfindingsHowshouldinstitutionsdealwithunexpectedfindingsarisinginthedata?


Adverseimpact

LabellingbiasDoeslabellingorprofilingstudentsbiasinstitutionalperceptionsandbehaviourstowardsthem?


OversimplificationHowcaninstitutionsavoidoverlysimplisticmetricsanddecisionmakingthatignorepersonalcircumstances?


Underminingofautonomy

Isstudentautonomyindecisionmakingunderminedbypredictiveanalytics?


Gamingthesystem

Ifstudentsknowthatdataisbeingcollectedaboutthem,willtheyaltertheirbehaviourtopresentthemselvesmorepositively,thusdistractingthemandskewingtheanalytics?


AbusingthesystemIfstudentsunderstandthealgorithms,willtheymanipulatethesystemtoobtainadditionalsupport?


Adversebehaviouralimpact

Ifstudentsarepresentedwithdataabouttheirperformance,couldthishaveanegativeimpacte.g.,increasedlikelihoodofdropout?





Reinforcementofdiscrimination

Couldanalyticsreinforcediscriminatoryattitudesandactionsbyprofilingstudentsbasedontheirraceorgender?


Reinforcementofsocialpowerdifferentials

Couldanalyticsreinforcesocialpowerdifferentialsandstudents’statusinrelationtoeachother?


Infantilization

Couldanalytics“infantilize”studentsbyspoon-feedingthemwithautomatedsuggestions,makingthelearningprocesslessdemanding?


EchochambersCouldanalyticscreate“echochambers”whereintelligentsoftwarereinforcesourownattitudesandbeliefs?


Non-participationWilltheknowledgethattheyarebeingmonitoredleadtonon-participationbystudents?


Stewardship DataminimizationIsallthedataheldonanindividualnecessaryinordertocarryouttheanalytics?

Legal 1 Datascientist

Dataprocessinglocation

Isthedatabeingprocessedinacountrypermittedbythelocaldataprotectionlaws?

Legal 1 IT

Righttobeforgotten

Canalldataregardinganindividual(exceptthatnecessaryforstatutorypurposes)bedeleted?

Legal 1 IT

Unnecessarydataretention

Howlongshoulddataberetained? Legal 1Analyticscommittee

Unhelpfuldatadeletion

Ifdataisdeleted,doesthisrestricttheinstitution’sanalyticscapabilities,e.g.,refiningitsmodelsandtrackingperformanceovermultiplecohorts?


Incompleteknowledgeofdatasources

Cananinstitutionbesurethatitknowswhereallpersonaldataisheld?

Legal/Logistical

1 IT




Inappropriatedatasharing

Howcandatasharingbepreventedwithpartieswhohavenolegitimateinterestinseeingitorwhomayuseitinappropriately?

Legal 1 IT

5 DRAFTING THE CODE OF PRACTICE After the taxonomywasmadeavailableon theprojectblog,andwith further in-depth feedback frommembersoftheadvisorygroup,itprovedrelativelyeasytodrafttheCodeofPracticeusingthetaxonomyasitsbasis.Allthoseissueswithaprioritylevelof1or2wereincorporatedintotheCode.Giventhebriefnatureofthesummarydocument,guidanceinhowtodealwiththeissuesisbynecessityatabasiclevel,but,asstatedearlier,morein-depthadviceandcasestudieswillbeincludedinanaccompanyingwebsite,whichisunderdevelopment.5.1 Introduction TheCode is grouped into eight areas, basedon the categories in the taxonomy, andprecededby anintroductionthatemphasisestwoofthekeythemesoftheCode,i.e.,thatlearninganalyticsshouldbeforthebenefitofstudents,andthat itshouldbecarriedouttransparently. Italsonotesthatmanyoftheprocessesoflearninganalytics,particularlyintheareaofdataprotection,shouldbecoveredbyexistinginstitutionalpolicies.

Learninganalyticsusesdataaboutstudentsandtheiractivitiestohelpinstitutionsunderstandandimproveeducationalprocesses,andprovidebettersupporttolearners.

Itshouldbeforthebenefitofstudents,whetherassistingthemindividuallyorusingaggregatedand anonymized data to help other students or to improve the educational experiencemoregenerally.Itisdistinctfromassessment,andshouldbeusedforformativeratherthansummativepurposes.

Theeffectiveuseof learninganalyticswill initially involvethedeploymentofnewsystems,andchangestoinstitutionalpoliciesandprocesses.Newdatamaybecollectedonindividualsandtheirlearningactivities.Analyticswillbeperformedonthisdata,andinterventionsmaytakeplaceasaresult.Thispresentsopportunitiesforpositiveengagementsandimpactsonlearning,aswellasmisunderstandings,misuseofdataandadverseimpactsonstudents.

Complete transparency and clear institutional policies are therefore essential regarding thepurposesof learninganalytics,thedatacollected,theprocessesinvolved,andhowtheywillbeusedtoenhancetheeducationalexperience.

Thiscodeofpracticeaimstosetouttheresponsibilitiesofeducationalinstitutionstoensurethatlearninganalyticsiscarriedoutresponsibly,appropriatelyandeffectively,addressingthekeylegal,ethicalandlogisticalissueswhicharelikelytoarise.

EducationalinstitutionsintheUKalreadyhaveinformationmanagementpracticesandprocedures



inplaceandhaveextensiveexperienceofhandlingsensitiveandpersonaldatainaccordancewiththeDataProtectionAct(DPA)1998.

Bytransferringandadaptingthisexpertisetoregulatetheprocessingofdataforlearninganalytics,institutions should establish the practices and procedures necessary to process the data ofindividualslawfullyandfairly.

5.2 Responsibility Thenextsectionaimstomakeitclearthatresponsibilityintheinstitutionforthevariousprocessesoflearninganalyticsneedstobeallocatedappropriatelyandthatstudentsandotherstakeholdersshouldbeconsulted.

Institutionsmustdecidewhohasoverallresponsibilityforthelegal,ethical,andeffectiveuseoflearninganalytics.Theyshouldallocatespecificresponsibilitywithintheinstitutionfor:

• Thecollectionofdatatobeusedforlearninganalytics• Theanonymisationofthedatawhereappropriate• Theanalyticsprocessestobeperformedonthedata,andtheirpurposes• Theinterventionstobecarriedout• Theretentionandstewardshipofdatausedforandgeneratedbylearninganalytics

Student representatives and key staff groups at institutions should be consulted around theobjectives,design,development,roll-outandmonitoringoflearninganalytics.

5.3 Transparency and ConsentThenextpartof theCode is aboutbeingopenabout all aspectsof theuseof learning analytics, andensuring that students providemeaningful, informed consent. The area of requesting consent is notstraightforward, as the UK Data Protection Act (1998), derived from the European Data ProtectionDirective(EuropeanCommission,1995),doesnotalwaysrequireobtainingconsent,forexample,whendatacollectionisnecessaryforthe“legitimateinterests”ofanorganization.Meanwhileabalancemustbestruckbetweenobtainingmeaningfulconsentforlearninganalyticsbutnotbombardingstudentswithcontinual requests for permission on every aspect of data collection and use. The consensus of theadvisorygroupandvariousexpertsconsultedwasthatobtainingconsentfor interventionstobetakenbasedonastudent’sdataiskey.Allowingstudentstoopt-outofdatacollectionmay,insomecases,makethecarryingoutofnormaleducationalprocessesimpossible;forexample,virtuallearningenvironmentscollectdataonstudentactivitybydefaultandcannotfunctionwithoutdoingso.

Institutionswilldefinetheobjectivesfortheuseoflearninganalytics,whatdataisnecessarytoachievetheseobjectives,andwhatisoutofscope.Thedatasources,thepurposesoftheanalytics,themetrics used, who has access to the analytics, the boundaries around usage, and how tointerpretthedatawillbeexplainedclearlytostaffandstudents.

Institutions should also clearly describe the processes involved in producing the analytics tostudentsandstafformakethealgorithmstransparenttothem.



Studentswillnormallybeaskedfortheirconsentforpersonalinterventionstobetakenbasedonthelearninganalytics.Thismaytakeplaceduringtheenrolmentprocessorsubsequently.Theremayhoweverbelegal,safeguardingorothercircumstanceswherestudentsarenotpermittedtooptoutofsuchinterventions.Ifsothesemustbeclearlystatedandjustified.

New learninganalyticsprojectsmaynotbecoveredby the institution’sexistingarrangements.Collection and use of data for these may require further measures, such as privacy impactassessmentsandobtainingadditionalconsent.

Options for granting consent must be clear and meaningful, and any potential adverseconsequencesofoptingoutmustbeexplained. Students shouldbeableeasily to amend theirdecisionssubsequently.

5.4 Privacy TheCodehereemphasizesthataccesstostudentdatashouldbecarefullycontrolledanddataprotectionlegislation complied with. There is doubt as to whether in the age of Big Data it is ever possible toanonymizeanindividual’sdatasuchthattheycannotbere-identifiedatsomestage(e.g.,Bollier,2010),but institutions shouldmake every effort to do so if an individual’s data is to be used anonymously.Meanwhile,anysharingofdataoutsidetheinstitutionshouldbemadecleartostudentsandstaff.Itwasalso felt to be important to state that institutions may have legal obligations to override privacyrestrictions,forexample,whenanalyticsidentifyastudentwhoappearstobeatriskofsuicide.

Accesstostudentdataandanalyticsshouldberestrictedtothoseidentifiedbytheinstitutionashavingalegitimateneedtoviewthem.

Wheredataistobeusedanonymouslyparticularcarewillbetakenbyinstitutionstoavoid:

• Identificationofindividualsfrommetadata• Re-identificationofindividualsbyaggregatingmultipledatasources

Theuseof“sensitivedata”(asdefinedbytheDPA),suchasreligiousaffiliationandethnicity,forthepurposesoflearninganalyticsrequiresadditionalsafeguards.Circumstanceswheredataandanalytics could be shared externally — e.g., requests from educational authorities, securityagenciesoremployers—willbemadeexplicittostaffandstudents,andmayrequireadditionalconsent.

Institutionsshouldensurethatstudentdataisprotectedwhencontractingthirdpartiestostoredataorcarryoutlearninganalyticsonit.

Institutionsmayhavealegalobligationtointervene,andhenceoverridesomeprivacyrestrictions,where data or analytics reveal that a student is at risk. Such circumstances should be clearlyspecified.

5.5 Validity Predictiveanalyticsareworthlessunlessthedataisaccurateandthealgorithmsarevalid.Forthisreason,



itwasthoughtimportantethically,andpotentiallylegallytoo,thatexpertiseexistedintheinstitutiontoensure that the analytics processes,which could affect students’ careers and lives,were understood.Meanwhile,itisstressedthateveniftheanalyticsisvaliditneedstobeseeninthewidercontextofanindividual’sexperience.

Itisvitalthatinstitutionsmonitorthequality,robustnessandvalidityoftheirdataandanalyticsprocessesinordertodevelopandmaintainconfidenceinlearninganalyticsandensureitisusedtothebenefitofstudents.Institutionsshouldensurethat:

• Inaccuraciesinthedataareunderstoodandminimised• Theimplicationsofincompletedatasetsareunderstood• Theoptimumrangeofdatasourcesisselected• Spuriouscorrelationsareavoided

Allalgorithmsandmetricsused forpredictiveanalyticsor interventionsshouldbeunderstood,validated,reviewedandimprovedbyappropriatelyqualifiedstaff.

Data and analyticsmay be valid but should also be useful and appropriate; learning analyticsshouldbeseeninitswidercontextandcombinedwithotherdataandapproachesasappropriate.

5.6 AccessFewinstitutionsareyet inapositiontoprovideastudentwithacopyofallthedataheldaboutthem(includingtheanalyticsperformedontheirdata,which,also,isitselfdata).However,individualsinEuropedohavealegalrighttorequestthisinformation,andinstitutionsmustworktofacilitatethis.Thatdoesnotmeanprovidingstudentsautomaticallywithall themetricsontheiracademicprocess,but itdoesmeandoingsoifthestudentrequestssuchinformation.

Studentsshouldbeable toaccessall learninganalyticsperformedontheirdata inmeaningful,accessibleformats,andtoobtaincopiesofthisdatainaportabledigitalformat.StudentshavealegalrightundertheDPAtobeabletocorrectinaccuratepersonaldataheldaboutthemselves.[…] They should normally also be able to view themetrics and labels attached to them. If aninstitution considers that the analyticsmay have a harmful impact on the student’s academicprogressorwellbeingitmaywithholdtheanalyticsfromthestudent,subjecttoclearlydefinedandexplainedpolicies.However,thestudentmustbeshownthedataaboutthemiftheyasktoseeit.

5.7 Enabling Positive InterventionsAswiththeothersections,thisonereflectsconcernsexpressedintheliterature.Thefrequentlyquoted“obligationtoact”forbothinstitutionsandstudents(e.g.,Campbell,DeBlois,&Oblinger,2007;Kay,Korn,&Oppenheim,2012;Willis&Pistilli,2014) iscovered,asarethemainpotentialpitfallsof interveningdirectlywithstudentsbasedonanalytics.

Institutionsshouldspecifyunderwhichcircumstancestheybelievetheyshould intervenewhenanalyticssuggeststhatastudentcouldbenefitfromadditionalsupport.Thismayincludeadvising



students that they should not continue on a particular pathway. Students may also haveobligationstoactontheanalyticspresentedtothem—ifsotheseshouldbeclearlysetoutandcommunicatedtothestudents.The typeandnatureof interventions,andwho is responsible forcarrying themout, shouldbeclearly specified. Somemay require human rather than digital intermediation. Predictions andinterventions will normally be recorded, and auditable, and their appropriateness andeffectivenessreviewed.

Theimpactofinterventionsonstaffroles,trainingrequirementsandworkloadwillbeconsideredandrequiressupportfromseniormanagement. Institutionswillalsobeclearabouttheprioritygiventolearninganalyticsinrelationtootherrequirements.

Institutionswilldecidehowtoallocateresourcesforlearninganalyticsappropriatelyforlearnerswithdifferentrequirementsandensurethatdiversegroupsandindividualsaretreatedequitably.

5.8 Minimizing Adverse ImpactsSomeofthereservationsfrequentlyexpressedindiscussionsandintheliteraturearecoveredhere.Forexample,analyticsshouldnottreatanindividualasanumber,andinstitutionsshouldbecarefulnottoprejudice students’ chances by categorizing them (Campbell, DeBlois, & Oblinger, 2007; Greller &Drachsler,2012).Therearealsopotentialdetrimentaleffectswhenstudentprogressisbeinganalyzed,sotheseneedtobeunderstoodandminimized.MacCarthy(2014)pointsoutthatthealgorithmsusedforlearning analytics may reinforce discriminatory attitudes, while Swenson (2014) suggests thatinterventionsmightfavouronegroupoveranother(forexample,campusbasedoverdistancestudents).

Institutionsrecognizethatanalyticscannevergiveacompletepictureofanindividual’slearningandmaysometimesignorepersonalcircumstances.

Institutionswilltakestepstoensurethattrends,norms,categorizationoranylabellingofstudentsdo not bias staff, student or institutional perceptions and behaviours towards them, reinforcediscriminatoryattitudesorincreasesocialpowerdifferentials.

Analyticssystemsandinterventionswillbecarefullydesignedandregularlyreviewedtoensurethat:

• Studentsmaintainappropriate levelsof autonomy indecisionmaking relating to theirlearning,usinglearninganalyticswhereappropriatetohelpinformtheirdecisions

• Opportunitiesfor“gamingthesystem”oranybenefittothestudentfromdoingsoareminimised

• Knowledgethattheiractivity isbeingmonitoreddoesnot leadtonon-participationbystudentsorothernegativeimpactsontheiracademicprogressorwellbeing

• Adverseimpactsasaresultofgivingstudentsandstaffinformationaboutthestudents’performanceorlikelihoodofsuccessareminimised

• Staffhaveaworkingunderstandingoflegal,ethicalandunethicalpractice.



5.9 Stewardship of DataFinally,asectionwasincludedtoremindinstitutionsoftheirresponsibilitiestolookafterstudentdatacarefully.EuropeanlegislationdoescauseapotentialrestrictionhereontheuseofBigDataforlearninganalytics:theusestowhichthedatacanbeputareoftennotknowninadvance,sominimizingthedatathatiskept,anddestroyingitafteraperiod,mightrestrictinstitutions’abilitiestoobtainvaluableinsightonstudentbehaviour.However,institutionsdoneedtocomplywiththelegislationandshouldarguablyobtainadditionalconsentfromstudentsiftheywishtoretaintheirdataforlongerperiods.

DataforlearninganalyticswillcomplywithexistinginstitutionaldatapoliciesandtheDPA,andwillinparticularbe:

• Kepttotheminimumnecessarytodeliverthepurposesoftheanalyticsreliably• ProcessedintheEuropeanEconomicAreaor,ifelsewhere,onlyinaccordancewiththe

DPA• Retainedonlyforappropriateandclearlydefinedperiods

Onrequestbystudentsanypersonaldatausedfororgeneratedbylearninganalyticsshouldbedestroyedoranonymised,withtheexceptionofcertain,clearlyspecifieddatafieldsrequiredforeducationalorstatutorypurposessuchasgrades.

6 A MODEL FOR THE DEVELOPMENT OF A CODE OF PRACTICE Othershavewrestledwithhowbesttodevelopacodeofpractice.TheInstituteforBusinessEthics(IBE,2015)andtheInternationalFederationofAccountants(PABC,2007)eachsuggestanumberofbroadlysimilarstepstobetakeninthedevelopmentofacodeofethicsoracodeofconduct.Whilerelevant,theserelatetothedevelopmentofacodeforasingle institutionratherthananeducationalsectororprofession. Table 2 summarizes these steps and shows how they have been implemented in thedevelopmentofJisc’ssector-wideCodeofPractice.

Table2:ActionstakentodeveloptheCodeofPractice.Proposedstep ActiontakentodeveloptheCodeofPracticeGainthecommitmentofseniormanagementFindachampionandsetupacommitteetooverseedevelopmentandimplementationIdentifystakeholders,what

Inthiscase,“thecommitmentofseniormanagement” isanalogoustotheprioritizationbythefurtherandhighereducationcommunityintheUKthat theethicaland legal issuesof learninganalyticsneededtobeaddressedasamatterofurgency.AconsultantwasemployedtoleadthedevelopmentoftheCode,andanadvisorygroupwithexpertrepresentationfromthesectorwassetup.The advisory group was appointed, consisting of experts from



isimportanttothem,andwhoshouldbeinvolvedincodedevelopmentandimplementationAgreeonthepurposeofthecodeandhowitwillbeusedReviewexistingexternalandmulti-sectorconductguidelinesDraftthecontent,perhapsbyaworkinggroupcomprisingrepresentativesfromvariousfunctionsinanorganizationTest,pilot,andapprovethecodePublishandpublicizeitReviewthecodeanditsimpact,takingintoaccountchangingrequirementsandstakeholderconcernsProvidetrainingandcommunicationasameanstoembedthecodeConnectthecodetotheorganization’s(a)performancemanagementpoliciesandsystems,and(b)internalcontrols

universitiesandcolleges,andastudentrepresentativefromtheNationalUnionofStudents.TheadvisorygroupdecidedontheapproachtoproducingtheCode,theareastobecovered,waystogainfurthervalidationfromthecommunity,dissemination,andadoption.The literature review identifiedkeyaspectsofguidelinesandcodes inrelatedareas,aswellassummarizingthemainethicalandlegalissuesoflearninganalyticsarisingintheliterature.ThetaxonomyofissueswasdevelopedwithassistancefromApereoandLace.ThisprovidedthebasisfortheCodedraftedbytheconsultantandrefinedbytheadvisorygroup.The advisory group approved the Code; testing and piloting ininstitutionsisongoing.The Code was launched at a prominent event, published on the Jiscwebsite,anddisseminatedheavily.This will happen once the Code has been adopted by a number ofinstitutions.Somearealreadyreportingtheinfluenceitishavingonthedevelopmentofinternallearninganalyticspolicies.Conference presentations andworkshops are encouraging the Code’sadoption by institutions. Support materials, including a series ofpodcasts,havebeendevelopedfortheaccompanyingwebsite.ThiscrucialstepisbeginningtohappenasinstitutionsdeveloptheirowncodesthatbuildonJisc’sdocumentsandlinktootherinternalpolicies.



TheactivitiesinvolvedintheproductionoftheCodeofPracticeforLearningAnalyticspotentiallyprovidethebasis for a generalizablemodel fordeveloping codesofpractices inotherprofessionsor areasofeducation.AnoutlineofthemodelisprovidedinthediagramshowninFigure2.

Figure2:Amodelforthedevelopmentofacodeofpractice.

Fiveproductsaredevelopedduringthisprocess.First,aliteraturereview(1)capturesthemainethical,legal, and logistical concerns being raised. At this point, an advisory group is recruited, bringing inexpertizefromthesectorandrepresentingkeystakeholdergroups.Theadvisorygroupcancontinueuntilthecodeispublishedorcouldberetainedonanongoingbasistoensurethatthecodeandsupportingwebsiteremainrelevant.Itcouldevenbesetupearliertoguidetheproductionoftheliteraturereview,ifnecessary.Theliteraturereviewinformsataxonomyofissues(2),whichisrefinedthroughexpertconsultationinthesector.Thedraftcodeofpractice(3)isthenproducedandsenttoexpertsandstakeholdergroupsaspartofapublicconsultationforafixedperiod.Feedbackisincorporatedintoafinalcodeofpractice(4)document, which is released publicly both online and through associated events if appropriate. Asupportingwebsite(5)ispopulatedwithfurtherguidanceandcasestudies.Asinstitutionspilotthecodeofpractice,theyprovidefeedbackandsuggestedenhancementsthatcanbefedintoanewdraftforanupdatedversionofthecodeatalaterstage.Themodelrequiresanumberoffactorstobeinplaceinorderforittobeeffective;someofthesemaybeproblematicinothersectorsorindustries:

1. Anestablishedliteraturebaseoutliningtheissuesofconcern



2. Organizationswishingtocontribute,andtobepartofarepresentativeadvisorygroup,regardedascrediblebythesector

3. Communicationmechanismsenablingadequateconsultationwiththesector4. Institutionswillingtopilotthecodeofpracticeandprovidefeedbackforitsenhancement5. Anongoingcommitmenttoreviewingandupdatingthedocumentinordertoensureitremains

relevant

In addition, organizing the development of a code of practice will require financial input and timecommitmentfromrepresentatives.Itishopedthattheiterativeaspectofthemodeloutlinedabovewillbe tested by the development of a version 2.0 of the Code of Practice for Learning Analytics onceinstitutionshaveprovidedfurtherfeedbackonitsusefulnessandlimitations.Whilethismodelmaynotbeappropriateforthedevelopmentofothercodesofpracticeinitsentirety,thevariousproductsandstageshavehelped toensure that theCodewas rigorously researched,widelyconsultedon,andwelldisseminated. These factors should assist with its adoption in institutions, or at least with raisingconsciousnessabouttheissuesthatneedtobetackledattheorganizationallevel.7 CONCLUSION TheCodeofPracticeforLearningAnalyticsproducedbyJisccoversthemainconcernsforstudentsandinstitutions that commonly arise in the literature and in discussions. The rigorous and consensualapproachtodevelopingthedocumentthroughitsvariousstagesaimstohelpensureitsadoption.ThedraftCodewassenttoawiderangeofeducationalorganizationsandreceivedmuchdetailedfeedback,most of which was incorporated. Much of this attempted to tidy up ambiguities in the wording,particularlywhereitrelatedtolegalissues.ThefinalversionwasreleasedataneventinSalfordinJune2015(Sclater,2015d)andreceivedmuchinterestonline.TheprimarypurposeoftheCodeistohelpinstitutionsdealwithethicalobjectionsandlegaluncertainties,andtofacilitatethefurtherdevelopmentofthefieldoflearninganalytics.Thedocument’sbrevityandtheclearlanguageandformattingemployedaimtoencouragepeopletoreaditandtograspquicklythemostsalientissues.AdoptionoftheCodebyinstitutionsandtheiremployeesandstudents,however,isbynomeansguaranteed,andagreeingwiththedocumentisofcoursemucheasierthanapplyingittoallnecessary areas of institutional business. Itsmain valuemay be in raising the issues and providing achecklistforinstitutionstoconsider;ultimately,itistheirresponsibilitytoapplyitsprinciplestotheirownactivities, to embed the concepts in their policies as appropriate and to ensure that these areimplementedeffectively.Promisingsignsof its influencearealreadyemerging.TheNationalUnionofStudentshasproducedaguideforstudents’unionsthatbuildsontheCodeofPracticeandlinkstoit(NUS,2015).TheUK’sHigherEducationAcademyoutlines theeight areasof theCode in its LearningAnalytics Toolkit (HEA, 2015).MeanwhiletheUniversityofEdinburghlinkstotheCodeinitsemerging“Guidanceonlearninganalyticsanddataprotection”andsuggeststhatitis“agoodstartingpointforstaffinidentifyingrelevantissues



andhowtoaddress them, toensure that learninganalytics is carriedout responsiblyandeffectively”(UniversityofEdinburgh,2015).TheCodewasdevelopedintheUKcontextandreferstotheDataProtectionAct1998.However,mostofit is relevant to institutions wishing to carry out learning analytics anywhere, particularly in otherEuropean countries with similar data protection legislation. The development and release of thedocument has been of considerable interest elsewhere, notably in the USA, Australia, and theNetherlands.Inthelatter,a“GuidetoLearningAnalyticsunderthePersonalDataProtectionAct,”buildingonJisc’swork,hasbeenpublished(SURF,2015).Whileitsreceptionhasbeenoverwhelminglypositive,oneUSA-basedvendor,commentingonapubliclyavailabledraftoftheCode,feltthatitputunnecessaryrestrictionsoninstitutionsandwouldthusholdbackthedevelopmentoflearninganalytics.Thisispreciselytheoppositeofwhatisintended.Thelackofa codeofpractice for learninganalyticshasbeenparalyzingmany institutions,preventing them frommoving forward and hampering the development of a critical set of technologies and processeswithpotentially significant impacts on the lives of individual learners. Clarifying the issues and proposingtransparent,ethicalsolutionsthatcomplywithstrictdataprotectionlegislationshouldhelptobreakthedeadlock and enable the further development and rollout of learning analytics. Interest has beenexpressed in adapting the document for use in the Netherlands and France. Meanwhile, several UKuniversitieshavealreadyindicatedthattheywillusetheCodeasthebasisfortheirinstitutionalpoliciesonlearninganalytics.Thesupportingwebsiteiscurrentlybeingdevelopedandwillincludecasestudiesandmoredetailedandpracticalguidanceforinstitutionsonhowtodealwiththeproblemstheyencounter.AsJiscrollsoutitsbasiclearninganalyticssolutiontoUKinstitutions(Sclater,2015c),experienceoflocalimplementationswillbecapturedandfedintotheguidance,togetherwithexamplesofhowinstitutionselsewhereintheworldhavedealtwithethicalandlegalbarriers.REFERENCES Berg,A.(2013,September13).Towardsauniformcodeofethicsandpracticesforlearninganalytics.[Web

log post]. Retrieved from http://ict-innovatie.uva.nl/2013/09/13/towards-a-uniform-code-of-ethics-and-practices-for-learning-analytics/

Bollier,D.(2010).Thepromiseandperilofbigdata.Washington,DC:TheAspenInstitute.Retrievedfromhttp://www.aspeninstitute.org/sites/default/files/content/docs/pubs/The_Promise_and_Peril_of_Big_Data.pdf

Campbell, J.P.,DeBlois,P.B.,&Oblinger,D.G. (2007).Academicanalytics:Anewtool foranewera.EDUCAUSE Review, 42(4), 40–57. Retrieved fromhttp://www.educause.edu/ero/article/academic-analytics-new-tool-new-era

CouncilDirective95/46/EContheprotectionofindividualswithregardtotheprocessingofpersonaldataandonthefreemovementofsuchdata[1995]OJL281/31

Ferguson,R. (2012).Learninganalytics:Drivers,developmentsandchallenges. International Journalof



TechnologyEnhancedLearning,4(5–6),304.http://oro.open.ac.uk/36374/Greller,W.,&Drachsler,H.(2012).Translatinglearningintonumbers:Agenericframeworkforlearning

analytics.EducationalTechnology&Society,15(3),42–57.HEA (Higher Education Academy). (2015). “Learning analytics.” HEA Starter Tools. Retrieved from

https://www.heacademy.ac.uk/enhancement/starter-tools/learning-analyticsIBE (Institute for Business Ethics). (2015). Nine steps for preparing a new code. Retrieved from

http://www.ibe.org.uk/nine-steps-for-preparing-a-new-code/103/52ICO (Information Commissioner’s Office). (2010, July). Personal information online code of practice.

Retrievedfromhttps://ico.org.uk//for-organisations/guide-to-data-protection/Kay,D.,Korn,N.,&Oppenheim,C.(2012,November).Legal,riskandethicalaspectsofanalyticsinhigher

education, JISC CETIS Analytics Series, 1(6). Retrieved fromhttp://publications.cetis.ac.uk/2012/500

K.N.C. (2014, April 30). Withered inBloom. The Economist. Retrieved fromhttp://www.economist.com/node/21601484/print

King, J. H., & Richards, N. M. (2014, March 28). What’s up with big data ethics? Forbes.http://www.forbes.com/sites/oreillymedia/2014/03/28/whats-up-with-big-data-ethics/

MacCarthy,M.(2014).Studentprivacy:Harmandcontext.InternationalReviewofInformationEthics,21,11–24.Retrievedfromhttp://www.i-r-i-e.net/inhalt/021/IRIE-021-MacCarthy.pdf

NUS (National Union of Students). (2015, August). Learning analytics: A guide for students’ unions.Retrieved from http://www.nusconnect.org.uk/resources/learning-analytics-a-guide-for-students-unions

Open University (2014). Policy on ethical use of student data for learning analytics. Retrieved fromhttp://www.open.ac.uk/students/charter/essential-documents/ethical-use-student-data-learning-analytics-policy

ProfessionalAccountants inBusinessCommittee. (2007).Defininganddevelopinganeffectivecodeofconduct for organizations. International Federation of Accountants. Retrieved fromhttp://www.ifa.org.uk/files/PAIBcode-of-conduct.pdf

Pardo,A.,&Siemens,G. (2014).Ethical andprivacyprinciples for learninganalytics.British JournalofEducationalTechnology,45,438–450.http://dx.doi.org/10.1111/bjet.12152

President’sCouncilofAdvisorsonScienceandTechnology.(2014).Bigdataandprivacy:Atechnologicalperspective. Report to the President. Retrieved fromhttp://www.whitehouse.gov/sites/default/files/microsites/ostp/PCAST/pcast_big_data_and_privacy_-_may_2014.pdf

RESPECTProject.(2004).Codeofpracticeforsocio-economicresearch.InstituteforEmploymentStudies.Retrievedfromhttp://www.respectproject.org/code/respect_code.pdf

Shroepfer, M. (2014, October 2). Research at Facebook. Facebook Newsroom. Retrieved fromhttp://newsroom.fb.com/news/2014/10/research-at-facebook/

Siemens, G. (2012). Learning analytics: Envisioning a research discipline and a domain of practice.Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, 4–8.http://dx.doi.org/10.1145/2330601.2330605

Sclater,N.(2014a,September16).Takinglearninganalyticstothenextstage.EffectiveLearningAnalytics.Jisc. [Web log post]. Retrieved from http://analytics.jiscinvolve.org/wp/2014/09/16/taking-learning-analytics-to-the-next-stage/

Sclater,N. (2014b).Codeofpractice for learninganalytics:A literature reviewof theethicaland legalissues. Jisc. Retrieved from http://repository.jisc.ac.uk/5661/1/Learning_Analytics_A-_Literature_Review.pdf



Sclater, N. (2014c, October 29). Notes from Utrecht workshop on ethics and privacy issues in theapplicationoflearninganalytics.EffectiveLearningAnalytics.Jisc.[Weblogpost].Retrievedfromhttp://analytics.jiscinvolve.org/wp/2014/10/29/notes-from-utrecht-workshop-on-ethics-and-privacy-issues-in-the-application-of-learning-analytics/

Sclater, N. (2014d,November). Learning analytics: The current state of play inUK higher and furthereducation.Jisc.Retrievedfromhttp://repository.jisc.ac.uk/5657/1/Learning_analytics_report.pdf

Sclater,N.(2015a,January30).Developingthecodeofpractice.EffectiveLearningAnalytics.Jisc.[Weblog post]. Retrieved from http://analytics.jiscinvolve.org/wp/2015/01/30/developing-the-code-of-practice/

Sclater,N.(2015b,March3).Ataxonomyoftheethicalandlegalissuesoflearninganalyticsv0.1.EffectiveLearning Analytics. Jisc. [Web log post]. Retrieved fromhttp://analytics.jiscinvolve.org/wp/2015/03/03/a-taxonomy-of-ethical-legal-and-logistical-issues-of-learning-analytics-v1-0/

Sclater, N. (2015c, April 4). Explaining Jisc’s open learning analytics architecture. Effective LearningAnalytics. Jisc. [Web log post]. Retrieved fromhttp://analytics.jiscinvolve.org/wp/2015/04/04/explaining-jiscs-open-learning-analytics-architecture/

Sclater,N.(2015d,June4).Codeofpracticeforlearninganalyticslaunched.EffectiveLearningAnalytics.Jisc. [Web log post]. Retrieved from http://analytics.jiscinvolve.org/wp/2015/06/04/code-of-practice-for-learning-analytics-launched/

Sclater, N., & Bailey, P. (2015). Code of practice for learning analytics. Jisc. Retrieved fromhttp://www.jisc.ac.uk/guides/code-of-practice-for-learning-analytics

Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas.American BehavioralScientist,57(10),1509–1528.http://dx.doi.org/10.1177/0002764213479366

SURF (2015). Guide to learning analytics under the personal data protection act. Retrieved fromhttps://www.surf.nl/en/knowledge-base/2015/guide-to-learning-analytics-under-the-personal-data-protection-act.html

Swenson, J. (2014). Establishing an ethical literacy for learning analytics. Proceedings of the 4thInternational Conference on Learning Analytics and Knowledge, 246–250.http://dx.doi.org/10.1145/2567574.2567613

UniversityofEdinburgh.(2015).Learninganalytics:Guidanceonlearninganalyticsanddataprotection.Retrieved from http://www.ed.ac.uk/records-management/data-protection/guidance-policies/learning-analytics

Willis, J.E., III,&Pistilli,M.D. (2014,April).Ethicaldiscourse:Guidingthefutureof learninganalytics.EDUCAUSE Review Online. Retrieved from http://www.educause.edu/ero/article/ethical-discourse-guiding-futurelearning-analytics

Documents

Developing a Code of Practice for Learning Analytics · Developing a code of practice for learning analytics. ... setting up appropriate governance structures, developing a taxonomy