Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
This may be the author’s version of a work that was submitted/acceptedfor publication in the following source:
Philip, Robyn & Greener, Elizabeth(2016)Robotics MOOCs Evaluation Report 2015.
Queensland University of Technology, Australia.
This file was downloaded from: https://eprints.qut.edu.au/103552/
c© 2016 Queensland University of Technology and authors
This work is covered by copyright. Unless the document is being made available under aCreative Commons Licence, you must assume that re-use is limited to personal use andthat permission from the copyright owner must be obtained for all other uses. If the docu-ment is available under a Creative Commons License (or other specified license) then referto the Licence for details of permitted re-use. It is a condition of access that users recog-nise and abide by the legal requirements associated with these rights. If you believe thatthis work infringes copyright please provide details by email to [email protected]
Notice: Please note that this document may not be the Version of Record(i.e. published version) of the work. Author manuscript versions (as Sub-mitted for peer review or as Accepted for publication after peer review) canbe identified by an absence of publisher branding and/or typeset appear-ance. If there is any doubt, please refer to the published source.
Robotics MOOCs
Evaluation Report 2015
Writtenby:
DrRobynPhilipandMsElizabethGreener
foreLearningServices,TILS,QUT
Robotics MOOC Report 2015
1
Contents1.ExecutiveSummary.......................................................................................................................................3
2.KeyFindings...................................................................................................................................................4
3.Introduction...................................................................................................................................................7
3.1Backgroundandcontext........................................................................................................................7
3.2Coursedesign.........................................................................................................................................8
3.3Methodology..........................................................................................................................................93.3.1Aim................................................................................................................................................93.3.2Objectives......................................................................................................................................93.3.3Evaluationframeworkandtheelearninglifecycle........................................................................93.3.4Datacollectionandanalysis........................................................................................................10
4.Demographicsandpatternsofengagement...............................................................................................11
4.1Demographics......................................................................................................................................11
4.2Studentmotivations,completionsandpatternsofengagement........................................................134.2.1Motivationforenrolmentandpathways....................................................................................134.2.2Retentionandcompletionrates..................................................................................................134.2.3Engagementpatterns..................................................................................................................14
5.Responsetothecoursedesignandlearningandteachingstrategies.........................................................15
5.1Studentcoursesatisfaction..................................................................................................................15
5.2Successofthecoursedesign................................................................................................................16
5.3Levelofdifficulty:designingforamixedcohortofstudents...............................................................18
5.4Assessment..........................................................................................................................................19
5.5Engagementstrategiesandcommunity..............................................................................................195.5.1Thecentralityofthediscussionforum........................................................................................195.5.2Liveevents:GoogleHangouts.....................................................................................................205.5.3Otherengagementactivities.......................................................................................................21
5.6Accessandequity.................................................................................................................................21
6.Technicalandpedagogicalinnovations.......................................................................................................22
6.1TheEdCastplatformandcollaborationwithindustrypartners...........................................................226.1.1IntegrationofMATLABauto-gradingtool...................................................................................236.1.2Automatedonlinepeerreview...................................................................................................256.1.3Textbookpublishersupport........................................................................................................25
6.2ReuseofcourseresourcesforENB339................................................................................................26
7.Institutionalimplications.............................................................................................................................27
7.1Productionissues.......................................................................................................................................27
7.2Evaluationandlearninganalytics........................................................................................................28
7.3Staffdevelopmentandinnovation......................................................................................................28
7.4Marketing.............................................................................................................................................29
Robotics MOOCs Evaluation Report 2015
2
8.Conclusion...................................................................................................................................................30
8.1Generalconclusions.............................................................................................................................30
8.1Futureresearchandopportunitiesforinnovation..............................................................................31
Acknowledgements....................................................................................................................................31
References.......................................................................................................................................................32
Appendices......................................................................................................................................................33AppendixA:Objectives,sampleindicatorsanddatasources..............................................................33AppendixB:Datasources,samplinganddatatypes............................................................................34AppendixC:Detailsoftheevaluationelearninglifecyclefortheproject............................................35AppendixD:Thebestaspectsofthecourse........................................................................................36AppendixE:Qualityoftheresources...................................................................................................37AppendixF:Assessment.......................................................................................................................38
Related eLearning Services documents • CourseDesignPlan:IntroductiontoRobotics&IntroductiontoRobotVisionMassiveOpenOnlineCourse
(MOOC)(September2013)• EvaluationPlanandResearchDesignforRoboticsMOOCs(March2015)• EvaluationReportforENB339:ReuseofMOOCResourcesandtheFlippedClassroom(August2015)• MATLABReportforRoboticsMOOCs(October2015)• MOOCProgram:LessonsLearnedfromDesignandProduction(August2015)• QUTEthicsApprovalCertificate1500000201(June2015)• RoboticsMOOCsPreliminarySnapshot(July2015)
Robotics MOOC Report 2015
3
1. Executive Summary InFebruary2015,QUTlaunchedthefirstoftwoMassiveOpenOnlineCourses(MOOCs)inscienceandengineering:Introduction toRobotics(IR)andRoboticVision(RV).Aslarge-scale,online,distanceeducationcourses,theseMOOCsallowedthousandsofstudentstosimultaneouslyattendandengageinthesamecourseattheonetime.FundingforthecourseswasinitiallyallocatedfromtheVice-Chancellor’sTransformationFund.
Studentsforbothcoursescamefrom161countriesaroundtheworld.12,894enrolledinIntroductiontoRobotics,whichranfromFebruarytoApril2015;and7,824enrolledinRoboticVisionMOOC(ApriltoJune2015).Enrolmentwasnotrestrictedbyageorprioreducation,andnocoursefeesapplied.InbothMOOCs,themainreasonforenrollinginthecourseswasforgeneralinterestorenjoymentpurposes.Thesecondmostpopularreasonwasprofessionalinterest.
Intotal,1,096certificateswereawardedtostudentswhomettheassessmentrequirements(621forIR;475forRV),andthetopperformerswhogained90%oraboveweresentpersonalemailsofcongratulationsfromtheleadacademic,ProfessorCorke.Inaddition,27studentssuccessfullycompletedtheoptionalrobotbuildingprojectforthefirstMOOC.Toassesseachstudentprojectasystemofpeerreviewwasdesigned.ThesystemwasjointlydevelopedfromanopensourcetoolbyQUTstaffandtheplatformprovider,EdCast.
Overallcoursesatisfactionwasveryhigh.Studentscommentedontheeffectivedesignofthecourse.InparticularstudentscomplementedProfessorCorkeforhisenthusiasmforthesubjectmatterandclearexplanationofconcepts.Studentsalsolikedtheapplicationoftheorytorealworldexamples,andaseventhoughthemathematicsandprogrammingwerechallengingforsomestudents,acoregroupappreciatedthischallenge.Studentsalsocommentedonthesupporttheyreceivedviathediscussionforumfromtheteachingassistants(TAs),andtheinvaluablesupportofpeersonline.Somestudentsalsosaidthatcourseswerethebestonlinecoursesofthistypethattheyhadevertaken.
Thepurposeofthisdocumentistoreportontheeffectivenessoftheelearningenvironmentregardingthedesign,productionandimplementationofthefirsttwoQUTMOOCs.TheevaluationwasframedwithinthePhillips,KennedyandMcNaught(2012)elearninglifecycle.FindingsdiscussedhererelatetodesignandsupportoftheMOOCenvironment,learningandteachingissuesandprocesses,productionandtechnicalmatters,andorganisationalissues.ParticulartechnicalandpedagogicalinnovationsrealisedthroughtheprojectincludethecollaborationbetweenQUTanditsindustrypartnerstodevelopanautomatedpeer-reviewsystem,automatedassessmentforMATLABsoftwareprogramming,integrationofliveeventsintotheEdCastplatform,andthereuseofMOOCresourcesinthethird-yearroboticsunit,ENB339.QUT’svaluedindustrypartnersforthiscollaborationwereEdCast,MathWorks,andSpringer.
ThefindingsfromthisreportwillinformfutureiterationsoftheroboticsMOOCs,subsequentQUTMOOCsfromotherdisciplines,andonlinelearninggenerallyattheUniversity.Inaddition,thelargeamountofdatacapturedfromtheMOOCscanpotentiallyformthebasisformuchresearchintolearninganalytics.Thisisexpectedtobeofinteresttotheuniversitygivenitsstrategicgoalsregardinginnovationandresearch,asoutlinedintheQUTBlueprint4(QUT,2014).
Robotics MOOCs Evaluation Report 2015
4
2. Key Findings Thekeyfindingsfromthereportaresummarisedinthissection.Detailsofeachfindingareexplainedanddiscussedwithinthebodyofthereport. 1. Learningandteachingdesignprinciples
1.1. Sufficienttimeneedstobeallocatedfortheteachingteamtodevelop:(a)theirconceptualunderstandingofthespecialrequirementsofonlinelearning,(b)thedemandsoftheMOOCenvironment,and(c)therolesandresponsibilitiesofworkingwithateamoncoursedesign,productionandimplementation.
1.2. ThelengthofthistypeofMOOCisbestkeptshort,ideallyafewweeks,asstudentengagementandparticipationdropssharplyafterweeks1and2.However,wherestudentsneedtoacquireadditionallearningresourcesorinstallsoftware,extratimeshouldbefactoredintothecourseorientationperiodtoallowforthisessentialstudentpreparation.
1.3. Designissues,scopeandtimelinesneedtobecollaborativelynegotiatedandmanagedwithallteammembers.Rolesandresponsibilitiesoftheteachingandsupportstaff,thedesign,productionandtechnicalteamneedtobeclarifiedearlyinthedesignprocess.
1.4. Systemsformanaginganddisseminatingthefindingsderivedfromthelearninganalyticsneedtobeestablishedasearlyaspossible.Just-in-timereportinghelpswithmonitoringandtroubleshootingproblemsasthecourseprogresses,andallowsfortimelyfeedbackoncetheredesignanddeliveryprocesscommences.
2. Supportforstudentsandcommunitybuilding
2.1. Theroleofcommunitymanageriscriticalforsupportingcommunicationsbetweenandamongstthousandsofstudents.Itisanessentialpositionformanagingandcoachingtheteachingassistants,andbuildingandengagingapositivestudentcommunity.Oncetherolesandresponsibilitiesofthispositionhavebeendeterminedanddocumented,theactivitiescanbeallocatedtoeitheroneperson,ordesignatedtoteammemberswhohavebeencoachedintheironlineresponsibilitiesandtheneedforaccountability.
2.2. StudentcohortsmayvaryaccordingtotheMOOCsubjectfocusanddiscipline.ThestudentsintheroboticsMOOCswereoftenparticularlypassionatehobbyists,studentsorteachersdedicatedtothefield.
2.3. Asystemforspecialacknowledgementofinformallearningwaswellreceived.Informalacknowledgementcanbeforcontributionsoverandabovecourseassessmentrequirements.Forexample,inthefirstroboticsMOOCstudentswhomadeasignificantcontributiontothediscussionforumreceivedapersonalemailfromProfessorCorkethankingthemfortheirspecialparticipation.
3. Theplatformandrelationshipswithindustrypartners
3.1. Theplatformiskeytothewholeproject,soselectionoftheplatformandotherdigitaltechnologiesusedneedstocommencewellinadvanceofthecoursedesignprocess,andcontinueinparallelasthedesignevolves.
3.2. Itiscriticaltomaintaineffectiverelationshipswithindustrypartners.TheMOOCprojectwasauniqueopportunityforQUT,asanelearningprovider,toshapethetechnologyincollaborationwithitstechnologypartnersinordertomeettheUniversity’spedagogicalneeds.ThereisaneedtocontinuetoworkwithEdCastonthecommunicationtoolsinparticular,refiningtheuserinterfaceofthediscussionforum,andenablingdevelopmentofgroupandindividualmessagingtools.
Robotics MOOCs Evaluation Report 2015
5
3.3. Assystemsdevelop,continuousimprovementprocessesneedtobeapplied.Inparticular,thisappliestocopyrightacquisitionandmanagement,accessibilitycompliance,styleguidesandworkflowmanuals.
4. Production
4.1. Theelearningdesigncycleandproblemsolving:MOOCdevelopmentrequiresadesignapproach.Thisnecessarilyincludesphasesforiterativelygeneratingideas,conceptdevelopment,rapidprototyping,testingandqualityassurancepriortoimplementation.Itisessentialtofactorinpreparationandoperationaltimeforallstaff(design,productionandtechnical)forproblemsolving,researchingthefieldandsourcingalternativetechnicalandmediasolutions.
4.2. Conceptualdevelopment:TheMOOCteam(teaching,designandproductionstaff)needtimetobuildmutualtrustandtheirownconceptualunderstandingofthespecialrequirementsofonlinelearningintheMOOCenvironment.Rolesandresponsibilitiessharedamongstthedesignandproductionteam,andtheteachingteamneedtobenegotiated,andtimeallocatedforthisaspectoftheprocesstomature.
4.3. Qualityassurance:Theprocessofqualityassuranceisextremelyimportantforaprojectwith
suchhighpublicexposureasaMOOC.Thismeansthattimeandresourcesshouldbeallocatedforallstagesoftheelearninglifecycle,includingstudentandtechnicalsupportduringimplementationandevaluation.
4.4. Copyright:Tracingandsecuringcopyrightpermissions,ornon-copyrightalternativesisa
particularlytime-consumingprocess.Developersneedtobeawareofthisconstraintfromtheverybeginningoftheprocess.
4.5. Systems:WithanewelearninginitiativesuchastheMOOCproject,understandablynotallsystemswillbeinplacetocoverallaspectsoftheproject.Thisincludestechnicalprocesses,datastorageandmanagement,workflowsandguidelines,andlearninganalytics.Newsystemsneedtobecreatedandundergocontinuousimprovementtoensuresustainability,andtomaintainrelevanceandcurrency.
5. Analyticsandresearch
5.1. Streamlinedsystemsforcollecting,managingandanalysinglargedatasetsarecriticalfortroubleshootingduringimplementationandforsupportingfuturedevelopments.Thereisaroleforananalystwithexpertiseinlearninganalyticsandaknowledgeofteachingand/orlearningdesigninordertoenabletimelyreportingandefficientdisseminationofresults.
6. Professionaldevelopment,organisationallearningandstaffing
6.1. TheMOOCspaceisaninnovativeenvironmentwherestaffcanbeencouragedtoexperimentandinnovate.
6.2. Inprojectswherecomplexproblemsmustcontinuallybeaddressed,uncertaintyabounds,andinnovativeanswersarerequired.Muchlearningandresearchtakesplaceinformallyonthejob.
6.3. ThefindingsofthisreportconfirmthoseofGrajek(2015),thatoneofthetoptenITissuesof2015is‘hiringandretainingqualifiedstaff,andupdatingtheknowledgeandskillsofexistingtechnologystaff’(p.12).ThiswasthecasefortheroboticsMOOCs,whereflexiblemanagementstrategieswererequiredtolocate,retainandtrainstaffwiththerightmixofskillsfortheproject.
6.4. Inaninnovativeworkenvironment,wheredeadlinesareshortandthequalityofoutputiscritical,staffrewardandrecognitionareessential.Thishelpsmaintaingoodwillandcommitment.
Robotics MOOCs Evaluation Report 2015
6
7. Resourcing,marketingandfuturedirections7.1. Thereisalifecycleforelearningprojectssuchasthis(seeFigure2).Itisimportanttorecognise
thataftertheinitialcycle,itisnecessarytoensurethereareresourcesforongoingmaintenanceandredeliveryoftheproject.
7.2. TherelationshipwithMarketingandCommunicationswaskeytothesuccessoftheproject.ThereisnowaneedtocontinuebuildingawarenessoftheMOOCprogramwithintheuniversity,aswellasextendingitsnationalandinternationalreachandprofilebeyondQUT.
7.3. Thereisvalueincontinuingtoexploreopportunitiesandexperiments,capitalisingonthewaveofinterestgeneratedbythesetwoMOOCsonrobotics.ConnectionswithMOOCalumnishouldbemaintainedandstrengthenedinthisregard.
7.4. Theindustrypartnershipswereveryproductivewithregardtothisproject.ItwouldbeadvantageoustocontinuetoworkwiththeseexternalpartnerstoinnovatearoundtheEdCastplatform.ItmaybeworthconsiderationtoapproachLegotosponsoravailabilityoftheMindstormKitsfortherobotmakingproject.
7.5. TheremaybeacasefordesigningroboticsMOOCstargetingparticulardemographics,forexamplehobbyistsorpractitioners.Asagendergapwasnotedintheseroboticscourses,consistentwithmuchscience,technology,engineeringandmathematics(STEM)education,furtherresearchcouldbedirectedtowardsfindingstrategieswhichencouragemorefemalestoenrolinthesecourses.Also,buildingonthehighnumbersofpostgraduatestudentsinterestedintheroboticsMOOCs,theUniversitymaywishtoconsidercreatingonlinemodulesforthecorporateandpostgraduatemarket;theUdacityfee-payingNanodegreemodelmaybeonetoexplore(https://www.udacity.com/nanodegree).
Robotics MOOCs Evaluation Report 2015
7
3. Introduction
3.1 Background and context
TheMassiveOpenOnlineCourse(MOOC)projectwaslaunchedin2015.Itwasanewandexperimentalinitiativefortheuniversity.AlthoughQUThasastrongbackgroundinblendedlearningandmultimediadevelopment,theUniversityhaslessexperienceinthedesign,productionanddeliveryoffullyonlinecourses.Itsfocustodatehas,inthemain,centredonthesupportofon-campusstudents.However,asMOOCsbegantoemergeworldwideasanewformofonlinelearning(Gallagher&Garrett,2013),manyuniversitiesincludingQUTsawanopportunitytoinnovateandallocateresourcesforMOOCdevelopment.QUTinitiallyassignedfundingfordevelopmentoftwoMOOCsinrobotics.TheseMOOCsweredesignedanddevelopedthroughout2013and2014,andimplementedinFebruaryandApril2015.QUT’seLearningServicesandtheScienceandEngineeringFacultyworkedcooperativelytolaunchthetwoMOOCs,Introduction toRoboticsandRoboticVision,onthedeliveryplatformsuppliedbyEdCast(http://www.edcast.com).
FundingwasprovidedfromtheVice-Chancellor’sTransformationFund.TheMOOCprojectmetuniversitystrategicgoalsandbuiltonQUT’sinternationalresearchreputation.Theprojectalsocontributedtouniversitypriorities(QUT,2014),intermsofhighqualitylearningandteaching,engagementwiththecommunity,andtheUniversity’svision-onethatvaluesaspiritofexperimentation,innovation,andentrepreneurialism.Inaddition,theprojectbuiltonQUT’sinternationalreputationinthefieldofrobotics(seetheARCCentreofExcellenceinRoboticVision:http://roboticvision.org),andthequalityandrigourofitslearningandteachingattheUniversity.TheroboticsMOOCsprovidedimportantmodelsforthedesignanddevelopmentofMOOCsinothersubjectareas,andonlinelearninggenerallythroughouttheuniversity.
Figure1providesatimelineshowingtheprojectmilestones,fromconceptinitiationin2013,throughtoanalysis,design,development,prototyping,pilotingandtestingin2014,implementationanddeliveryofthetwoMOOCSinthefirsthalfof2015,summativeevaluation,andthenredeliveryofthesameMOOCsinthesecondhalfof2015(seeFigure1).
Figure 1: Project milestones for Robotics MOOCs, 2013-2015.
Robotics MOOCs Evaluation Report 2015
8
3.2 Course design
Thetwoseven-weekMOOCswerebasedonasinglethird-year,on-campusunit,ENB339:IntroductiontoRobotics.ENB339isathirteen-weekQUTundergraduateelective(https://wiki.qut.edu.au/display/cyphy/ENB339+Introduction+to+Robotics).Thetwocoursesweredesignedtobesequential.ThismeantthatIntroduction toRoboticsprovided afoundation forRoboticVision.StudentscouldattemptoneorbothMOOCs.Studentswereencouragedtocompletethecourses insequence, butthiswasnotmandated,asenrolmentwasopentoanyone,atanytime,andparticipantscouldcompleteeitherorbothMOOCSinwhateverordertheywished,andre-enrolasoftenastheywanted.
TheMOOCsweredesignedforindividual,self-pacedlearning.Interactionsviathediscussionboard(whichwasopenatalltimes),andtheliveevents(whichwereheldthreetimespercourse)wereoptional,andsupportedbyfourteachingassistants(TAs).Itwasexpectedthatstudentswouldspendabout4-8hoursperweekstudyingeitherofthecourses.Studentsworkedthroughtheactivitiesatatimeandplaceoftheirownchoosing,withintheconstraintsofassignmentduedatesandreleaseofonlineresourcesonaweeklybasis.Collaborationwithotherstudentswasvoluntaryandself-organisinggroupswereencouraged.Thecourseresourcesprincipallyincludedshortthreetotenminutelecture-stylevideos,narratedanimations,recordedvideotutorialsupports,excerptsfromthee-textbook(providedfreeforthedurationofthecoursebythepublisherSpringer),computerprogrammingtutorialsandassessmentsusingMATLABsoftware,andtheonlinediscussionforums.ParticipantscouldformgroupsiftheywishedandsomeopenedrelatedFacebooksites.Forstudentstoreceiveacertificateofcompletiontheywererequiredtosubmitandpassfourquizzesandfourprogrammingassignments.Theseweremachine-markedonsubmission.
ThefirstMOOChadanoptionalpracticalprojectwherestudentswhowereabletoaccesstheLegoMindstormNXTrobotmakingkitorsimilar,couldbuildandprogramaworkingroboticarm.Studentswererequiredtovideotheirassembledandprogrammedrobotarmasitcompletedasettask.TheresultingvideowasthenuploadedtoYouTubeorasimilarmedia-sharingplatform,andtheURLsubmittedasevidenceforthepeerreviewprocess.Forthosewithaccesstothenecessaryresources,successfullybuildaworkingrobotcapableofcompletingarequiredtaskwasexpectedtobeahighlightofthefirstcourse.
Robotics MOOCs Evaluation Report 2015
9
3.3 Methodology
3.3.1 Aim
TheaimoftheevaluationwastoinvestigatetheefficacyoftworoboticsMOOCs,IntroductiontoRoboticsandRoboticVision,anddeterminetheimplications,impact,issuesandconcernsforstudents,teachersandtheuniversityasaresultoftheinitiative.Datagatheredandanalysedwasexpectedtobeusedtorefinethetwocoursesforfuturedelivery,andinformthedevelopmentofotherMOOCs,andonlinecoursedevelopmentgenerallyatQUT.
3.3.2 Objectives
Theobjectivesoftheevaluationwereasfollows:
1. TodeterminethereachoftheprogramanddemographicsofMOOCparticipants.2. TodeterminepatternsofstudentengagementineachoftheMOOCS.3. TodeterminetheeffectivenessoftheteachingandlearningstrategiesemployedintheMOOCs
tosupportstudentlearning.4. Todeterminethesuitability(affordancesandfunctionality)oftheEdCastplatformasa
foundationfordeliveryandsupportoftheMOOCs.5. TodeterminetheimpactofissuesrelatingtouseoftheMOOCresources.6. Todetermineissuesofaccessandequitythatimpactthedesign,developmentanddeliveryof
theMOOCs.7. Todeterminefactors(e.g.technical,organisational,social,professional)thatimpactthedesign
andproductionoftheMOOCs.
Thisreportisframedinresponsetotheobjectives.Relevantobjectivesappearbeforeeachsection.AlistofindicatorsmappedtoeachobjectiveisprovidedinAppendixA.
3.3.3 Evaluation framework and the elearning lifecycle
TheframeworkthatunderpinsthisevaluationdrawsonthePhillips,McNaughtandKennedy(2012)designevaluationelearninglifecycle.Phillipsetal.arguethatinvestigationoflearningtechnologiesinvariablyincludesamixtureofevaluationandresearch,andtheyidentifyfivestageswhereevaluationand/orresearchtypicallyoccurinanelearningproject(seeFigure2).FromthePhillipsetal.perspective,researchisdefinedasinquirywhereinformationisgathered‘toinformourunderstandingofhowpeoplelearnusingane-learningartefactorenvironment’(p.1109);whereasevaluationismoreabout‘gatheringinformationtohelpmakejudgmentsaboutthevalueandworthofanelearningartefactorenvironment’(p.1109).Thisreportfallsintothelattercategory,andrepresentssummativeevaluationofanoperationalsystem(stage5).Itisexpectedthatfurthereffectivenessresearchintothelearningandteachingprocessesandoutcomeswillfollowinthenextstageoftheproject(stage6)whenthelargedatasetscollectedfromtheprojectcanbeinterrogatedmoredeeplyforwhattheycanrevealregardinga‘holisticunderstandingofhowlearnersengagewiththeelearningenvironment’(Phillipsetal.,2012,p.1112).
Figure2demonstratesthepositionofthissummativeevaluationintheelearninglifecycle.Thefigureshowsthatevaluationoccursthroughouttheproject,fromtheverybeginningwheretechnologiesandpedagogiesarechosen(stage0),throughthemanystagesofformativeevaluationateachpointinthedesigncycle(stages1-4),untilthelaststageswheresummativeevaluationtakesplace(stages5and6).Thecyclethenbeginsagain.AppendixCprovidesmoredetailonthelifecyclemodel.
Robotics MOOCs Evaluation Report 2015
10
Figure 2: The design evaluation elearning lifecycle for the robotics MOOC project, 2013-2015. Based on Phillips, McNaught and Kennedy (2012).
3.3.4 Data collection and analysis
Giventhecomplexityoftheissuesrelatingtothisevaluation,amixedmethodsapproachwasadoptedfordatacollectionandanalysis.Siemens,GasevicandDawson(2015)confirmthisapproachasasoundoneforMOOCresearchandevaluation.MultiplesourcesofdatawerecollectedandtriangulatedanddetailsofthemethodologycanbefoundintheeLSdocument:EvaluationPlanandResearchDesignforRoboticsMOOCs.Inthisprojecttherewerestagesofformativeandsummativeevaluation,reflectingtheevaluationlifecyclestagesidentifiedinthePhillipsetal.(2012)model(seeFigure2).Inaddition,alow-riskethicsapplicationwassubmittedtotheQUTEthicsCommittee(1500000201)toallowforpublicationofthefindings,andasthebasisforfurtherresearch.
Thetypeofdatacollectedfortheevaluation(quantitativeandqualitative),alongwiththenumberssampledisoutlinedindetailinAppendixB.Insummary,however,datasourcesincludedstatisticsfromtheEdCastplatformandGoogleanalytics,theYouTubechannel,ProfessorCorke’sMATLABToolboxwebsite(http://www.petercorke.com/Robotics_Toolbox.html),discussionforumpostsandemailstotheQUTMOOCsupportteam,pre-,mid-andpost-coursesurveys,andstatisticsfromtheQUTMarketingandCommunicationDepartment.
Descriptivestatisticalwerecalculatedforthequantitativedata,andthequalitativedatawasanalysedusingconstantcomparativemethods(Charmaz,2011).Todate,quantitativeanalysishasbeenlimitedtoreportingonstudentsasacohortontheplatform.Individualswerenottrackedthroughthesystemasthiswasbeyondthescopeofthefirstinvestigation.Giventherequisiteresources,however,thisisanareaforfutureresearch(stage6oftheelearninglifecycle).
Robotics MOOCs Evaluation Report 2015
11
4. Demographics and patterns of engagement Objective1:TodeterminethereachoftheprogramanddemographicsofMOOCparticipants.
4.1 Demographics
Thetwocoursesattractedhighnumbersofstudents,eventhoughQUTwasnotalliedwithconsortiasuchasEdXorFutureLearnthathaveestablishedglobalmarketingplatformsandstrategies,andMOOCcoursecatalogues.StudentrecruitmentreliedsolelyonapurposefulQUT-centricdigitalmarketingcampaign.Therewere12,894registrantsinIntroductiontoRobotics,and7,824registrantsforRoboticVision.Participantswerefrom161countries,withthetopthreecountriesbeingIndia,AustraliaandtheUnitedStates.ThetoptencountriesarelistedintheTable1.
Table1:Geographicallocationofstudents Introduction to
Robotics Robotic Vision
Enrolments from 161 countries (n=12,894)
161 countries (n=7,824)
Enrolment by country (top 10)
Country Students (%)
Country Students (%)
1 Australia 19 1 Australia 17 2 India 16 2 India 16 3 United States 14 3 United States 15 4 Egypt 3 4 Germany 3 5 Germany 3 5 Brazil 3 6 United Kingdom 2 6 Egypt 3 7 Canada 2 7 United Kingdom 3 8 France 2 8 Mexico 3 9 Mexico 2 9 Canada 2 10 Brazil 2 10 France 2
Note: Source: Google analytics Theglobalnatureofparticipationenabledaccessforstudentswhowouldnotnormallyhaveaccesstohighereducation.1
ThankyouProf.PeterCorkeincooperationwiththeQueenslandUniversityOfTechnologytoprovideopportunityforstudentindevelopingnationsespeciallyPapuaNewGuinea,whereROBOTICSisnottaughtasaCourseinComputerScienceandEngineering.(IR,forumpost)
Anothergreataspectisseeingpeoplefromallaroundtheworldparticipatingandhelpingeachother.AmazingtobeabletoparticipateinaglobalcourseinroboticsfromwayupinthewoodsinNorway.(IR,post-coursesurvey)
Themajorityofparticipantsforbothcoursesweremale(IR85%;RV88%),inthe20-30agerange(IR,52%;RV,53%),whoalreadyhadanundergraduatedegreeordiploma(IR&RV,38%)(seeFigure3,genderandage).ThisfitsthetypicalcourseregistrantprofileaccordingtoHoetal.(2014),butastheyexplainintheirHarvardXandMITxreport,registrantsarenotedfortheirdiversity.AnunusuallyhighnumberofregistrantsIntheQUTMOOCsdidnothaveuniversityqualifications(41%,seeFigure3,
1Note:Thespellingandgrammarofstudents’quotationshavebeenleftunedited.
Robotics MOOCs Evaluation Report 2015
12
educationalattainment),and20%wereunder20yearsofageforthefirstroboticscourse.Thecurrentpopularityofroboticsamongstyoungenthusiastsmaybethereasonforthis,andthedropinnumbersofparticipantsinthisagegroupforthesecondcourse(downto13%)mayreflectthechallengeoftheincreasedcomplexityofthesecondcourse(seeFigure3).
Figure 3: Demographics for Introduction to Robotics and Robotic Vision. Graphs show gender, age and highest educational attainment.
Robotics MOOCs Evaluation Report 2015
13
4.2 Student motivations, completions and patterns of engagement Objective2:TodeterminepatternsofstudentengagementineachoftheMOOCS.
4.2.1 Motivation for enrolment and pathways
SurveyresponsesregardingwhystudentsenrolledinthetwocoursesprovidedvaluableinformationaboutmotivationsforMOOCenrolmentsandconsiderationsforfuturecoursedesign.Forexample,surveyfiguresshowthatthemajorityenrolledinIntroductiontoRoboticsforreasonsofgeneralinterestorenjoyment(77%;n=5569).Thisfiguredroppedto25%(n=1689)forthemoreadvancedsecondroboticscourse,RoboticVision.Nonetheless,professionalinterestwasstillthesecondmainreasonforenrolmentinbothcourses(48%forIRand19%forRV;seeTable2).
Table 2: Motivation for enrolling (top 5 reasons)
Introduction to
Robotics %
Robotic Vision
% Survey responses (n) 5569 1689 General interest or enjoyment 77 25 Professional interest 48 19 I am studying university level robotics 34 12 Earning a statement of attainment 29 14 To help my employment prospects 24 9
Note: Source: Pre-course surveys. Participants could choose more than one option.
Asoneprogrammerexplained:
IamgladIdidthecourseasIhavebeenunemployedasaprogrammerfor2yearsandthiswillgivemealegup.BeforethisIhadtotryandgetagranttodoashortTAFEcoursebutwiththisbeingnearlyfree(datacostme$110)andself-serveredoingitIfeelIambetteringmyselfandgettingmyconfidencebackagain.StudentshelpeachotherwhereasjobapplicantsandcompetitorsarealwaysfightingsothiswasamuchbetterenvironmenttorebuildmyITandmathsskillsandconfidencetogetanITjobagain.Thanks.Greatideaandwelldone!(IR,post-coursesurvey)
CommentssuchastheoneaboveindicatethatopenonlinecoursessuchastheroboticsMOOCsmaywellhaveaplaceintermsofprofessionaldevelopmentandlearningpathwaysforstudents.ForsometheMOOCsfunctionedasrefreshercourses,oratasterofuniversity-levelstudyforthosethinkingaboutattendingregularundergraduatecourses.Eventhoughthiswasathird-yearcourseitclearlyprovidedopportunitiesandlearningpathwaysforanumberofstudents.ThissuggeststhatthereareopportunitiesforQUTtoconsiderotheropenonlinecoursesasuniversitybridgingcoursestargetingsecondarylevelandtechnicaltrainingmarkets.
4.2.2 Retention and completion rates
SinceMOOCsbeganattractingextraordinarilylargeenrolments,therehasbeenmuchdiscussionaboutretention(Freitas,Morgan,&Gibson,2015).Previousresearchhadindicatedthatcompletion ratescouldbeaslowas3–5%(Gallagher &Garrett,2013),7–9%(Knowledge@Wharton,November, 2012),andbelow13%(Onahetal.,2014).Thisofcoursedependsonhowretentionismeasured.
Robotics MOOCs Evaluation Report 2015
14
ConcernsaboutcompletionshaveoftenoverlookedthemultitudeofreasonswhystudentsenrolinMOOCs(refertopfivereasons,Table2).Forexample,thereareparticipantswhoregisterforaMOOCbutneverenterthecourse;othersmayenterthecoursebutsimplybrowseforgeneralinterestortoviewthecoursedesignanddeliverymodels;andsomeregistrantsmayhaveagenuineinterestinviewingorstudyingthecoursetopic,butdonotwishtobeassessed.MITusesthenomenclatureofregistrant,forthosewhoregisterforanaccountonly,andparticipantforsomeonewhoactuallyentersthecourse(Hoetal.,2015).GenerallyinMOOCsstudentscancomeandgofromthesecoursesasoftenastheyplease,attemptingcoursesasmanytimesastheywantand,whileacourseremainsopenandonoffer,thestudenthasinfiniteopportunitiestoregisterandbuildontheirlearningattheirownpace.SotypicallythosewhocompletetheassessmentsandreceiveacertificateofparticipationforaMOOCareintheminority.ThiswasthecasefortheroboticsMOOCs.
Inaddition,giventhelimitationsofthetechnicalsystemsandmeansofgatheringanalytics,wewereunabletomatchindividualmotivationstooutcomes.Wehaveonlybeenabletotrackcohortbehaviournotindividualbehaviouratthisstage.However,wecanindicatebehaviourandtrendswithinacohort.Wecansaythat29%(1615)ofthosewhocompletedtheIntroductiontoRoboticspre-coursesurveyindicatedthattheyaimedtoreceiveacertificateofcompletion.621ofthecohortattainedapassgradeandreceivedacertificateinthatcourse.Theymayormaynothavecompletedthesurvey.InRoboticVision14%(236)indicatedtheirmotivationtoreceiveacertificate.475certificatesofcompletionwereawardedforthesecondMOOC.
4.2.3 Engagement patterns
Theanalysisofpre-andpost-coursesurveyandGoogleanalyticsprovidedsomeindicationofengagementpatternsbutatthesametimehighlightedtheneedtogainmorefocusedanddetaileddatafromtheplatformand/orGoogleanalytics.Surveydataindicatedthetimestudentsspenteachweekonthecourse.Amajorityofsurveyparticipantsclaimedtospendonaveragefourtosixhoursperweekstudyingthecourse(IR43%;n=385;RV46%,n=254;post-coursesurvey),butasignificantnumberspentsevenormorehoursperweekonthecourse(IR36%;RV28%).
Googleanalyticsenabledaviewofhowthecohortallocatedtheirtimeandattentionoverthedurationofthecourse.InIntroductiontoRoboticsthenumberofparticipantsonanydaywashighatthebeginningofthecourseuntiltheendofweek3.Inweeks0–3therewereonaverage1,439,1,338and1,006participantseachday,afterwhichthenumberslevelledoffaround500-800,withanincreaseinthelastweek.InRoboticVisiontheaveragenumbersofdailyparticipantswassteadierforthefirst4weeks(591,536,460and405),beforedroppingoffslightlyinweeks5-7(278,220,207).Itisdifficulttodeterminehowindividualsinteractwiththecourse;howeverthedataindicatesthat53%(IR)and60%(RV)ofthedailyvisitorswereactiveforumparticipants.Thereappearedtobeastrongcoregroupwhohelpedeachother,interactedwiththeTAsandcompletedtheassessmenttasks.Othersmayhavepursuedthecourseinisolationorwithofflinecohorts,butwedonothavedatafromthesecoursestoconfirmthisexpectedbehaviour.
ThesemultiplepatternsofengagementreflectthepurposefulMOOCdesignthataimedtosupportdiverselearningstylesandmotivations.AsourownandEdCastmethodsofgatheringanalyticsmature,itisenvisagedthatwewillbeabletomapindividualstudentmotivation,engagement,achievementandsatisfactionwiththecourse,therebybetterinformingfutureeducationaldesignapproachesandimprovingtheuserexperience.
Robotics MOOCs Evaluation Report 2015
15
5. Response to the course design and learning and teaching strategies Objective3.TodeterminetheeffectivenessoftheteachingandlearningstrategiesemployedintheMOOCstosupportstudentlearning.Objective5.TodeterminetheimpactofissuesrelatingtouseofMOOCresources.
5.1 Student course satisfaction
Overallcoursesatisfactionwasveryhigh,indicatingthatthelearningandteachingstrategiesadoptedwereeffective.Inthepost-coursesurveys,studentsratedthecoursesaseitherexcellentorgood(96%IntroductiontoRobotics;92%forRoboticVision;seeTable3).Somestudentsalsosaidthatthecourseswereoneofthebestcoursestheyhadtaken(post-coursesurveycomments).Commentsonthediscussionforumalsoconfirmedthispositiveresponse.
Table 3: Overall course rating Introduction to Robotics (IR)
(n = 364) % Robotic Vision (RV)
(n = 246) % Excellent 62.9 53.7 Good 32.7 38.2 Satisfactory 3.0 6.5 Poor 0.8 1.2 Very poor 0.5 0.4 Total 100 100
Note: Post-course survey (IR: M = 4.57, SD = 4.43; RV: M = .650, SD = .707)
Onehighachieverrespondedasfollows:
Ithankyouprofessorforyourefforts.Thecoursewasattractiveandwellexplained.Mygradewasjustareflexionofthecourse'squalitycombinedwithastrongwilltolearnandachild'sloveforrobots.(email,IR)
Forsomestudentsitwasatransformativelifeexperience,asonestudentsaid:
It’sindeed[beena]lifechangingexperience,openingupnewvisionsandthoughtprocesses.(email,RV)
Theprojectisatreasurechestforenthusiasts.(RV,post-coursesurvey)
Theleadacademic’spassionforthesubjectwasalsooftencommentedupon,andconsideredbymanyasthebestaspectofthecourse:
Firstofall,thepassionofPeterCorkeforrobotics,it'sreallyreallymotivatingtobetaughtbyaprofessor,evenfaraway,inanotherlanguage,thatmakesyoufeelhe'sheretotakeyoutoanotherlevelofknowledge.(IR,post-coursesurvey)
Robotics MOOCs Evaluation Report 2015
16
Commentsaboutanindividual’spersonalgrowthand‘theexcitementoflearning’asonestudentputit(IR,post-coursesurvey)indicatedthedepthofengagementforsomestudents.Forsome,thecoursesopenedupopportunitiestheywouldnothaveotherwisehad,andafewstudentssaidtheywouldkeepworkingonmakingtheirrobotsafterthecoursefinishedastheyranoutoftimetocompletetheirprojectintimeforpeerreview.Forexample:
CongratulationProfCorke.Yourcourseisexcelent.Iwillcontinuetoworkonmyrobotuntiliwillcompletethegiventask,evenifitwillbeoutsidethetimelimit.Itismyfirstrobot!!!:-)))(IR,post-coursesurvey)
Itseemedtobeverypractical,comparedtootherMOOC'sI'vetakeninthepast.IfeellikeIlearnedsomethingthatIcanstartputtingtouseimmediately.(IR,post-coursesurvey)
I'mamechanicalengineerandIusedthisknowledgeinmyworklastweek:aclientsentusaCADfilewithadifferentcoordinateframethanthedrawing.ThedifferentposeswereincludedinthedrawingsoIwasabletorecreatethecorrectcoordinateframeintotheCADfile.Mybosswasquiteimpressed.Thepointisthattheknowledgeacquiredinthiscourseisusefulinmanydifferentaspectofengineering.(IR,post-coursesurvey)
5.2 Success of the course design Thecoursedesign,includingthelearning,teachingandsupportstrategies,wasacollaborationbetweenProfessorCorkeandeLSstaff.Itwasconductedoverthree-months,atthesametimeascourseresourceswerebeingcollaborativelydevelopedandtestedbyProfessorCorkeandtheeLSMOOCteam(seesection3.2forcoursedescription).Whiletheproductionofhighqualityresourceswasitselfaconsiderablechallenge,thepedagogicalchallengeswereequallysignificant.Themainissuesforthedesignteamcentredaroundstrategiesformaintainingengagementofadiversemixofstudentsrangingfromhobbyiststoexperiencedprofessionals,agedbetween10and70,andtodesignforinteractivity,automatedassignmentmarking,andmeaningfulfeedbackonlearningtothousandsofstudents.Addedtothesecomplexities,theratioofteacherstostudentswasapproximately5:12,000).Allmembersofthecourseteamwerechallengedintermsofproblemsolving,learningnewskills,timepressures,communicationandworkload.Atthetimethattheleadacademic,ProfessorPeterCorkewasdevelopingMOOCresourcesincollaborationwitheLSstaff(e.g.videoscriptingandrecording,writingquizzes,reviewingcontent),hewasalsopreparingforthenewblendedlearning/flippedclassroommodelintherelatedENB339roboticscourseinsemester2,establishinganewcentreofexcellenceinroboticsatQUTandtendingtoalltheusualadministrativeandcommunityoutreachtasksrequiredofanyacademic.Inaddition,whiletheeLSteamhadconsiderabledepthofexperienceandexpertiseinmediadesignandproductionofblendedlearningresources,itwasnotsetupintermsofinfrastructuretospecificallyleadandmanagefullyonlinelearning.eLSstaffmemberswerealsooftensimultaneouslyworkingontheMOOCprojectaswellasotherQUTprojects. Despitetheseconstraints,asaresultofthestrategicapproachestakenbytheMOOCteam,thecoursedesignwassuccessful,asevidencedfromthepost-coursesurveysconfirmed.ManystudentscommentedpositivelyaboutarangeofcoursedesignelementsasTable4indicates.Studentsratedthelecturevideos,thetextbook,theformativequestions,theforums,emailcommunicationsandtheGoogleHangoutsasexcellentorgood(seealsoappendicesDandEforfulldetailsandsamplecomments).
Robotics MOOCs Evaluation Report 2015
17
Table 4: Student rating of key course elements
MOOC course items rated as excellent or good IR (%)
RV (%)
Video 98 97
Textbook 88 88 ‘Check understanding’ questions 88 84
Forums 82 83
Email 82 83 Google Hangouts 80 82
Note: For full details see Appendix E, where percentages, means and standard deviations are provided.
Qualitativeresponsestoaquestioninthepost-coursesurveysaskingstudentswhattheythoughtwerethebestaspectsofthecoursewereanalysedintocategoriesandthenquantified(seeAppendixD).Themajorityofcommentsreferredpositivelytothequalityofthecoursematerials(IR38%;RV39%),inparticularthelecturevideos2,theanimationsandthedemonstrations.Studentswelcomedtherangeandchoiceoftopics(IR27%;RV22%).Theeffortputintothecoursedesignandproductiondidnotgounnoticed(IR36%;RV21%),andstudentscommentedonthetheclearinstructionsandexplanationofconcepts,themodularisedformat,theabilitytocontrolthespeedofthevideos,thetextbookasacomplementtothevideos,theinclusionofrealworldexamplesandthepacingoftheunit.TheyenjoyedinteractiveelementssuchastheliveandrecordedGooglehangoutsandtheopportunitytolearnwithandfromotherstudentsintheforumsandtheexperienceoftheTAs.
Thevalueoflearningtouseandapplymathematicalandprogrammingskillswasoftencommentedon(IR21%;RV19%),aswastheopportunitytoputtheoryintopracticeviatheformativeandassessablequizzes.TheyvaluedhavingaccesstothefreeMATLABlicenceforthedurationofthecourse,andProfessorCorke’sopensourceMATLABToolboxwhichextendedthefunctionalityofthebasicMATLABsoftware.Forthosewhohadtheresourcestodotherobotprojectitwaswellreceived.Studentsalsocommentedontheeffectivemixoftheoreticalandpracticalprogrammingtasks,andthefeedbackontheMATLABexercises.
ProfessorCorke’spassionforrobotics,hisskillexplainingcomplextopicsandabilitytorelatethetheorytoreallifescenarioswasoftencommentedon.Inaddition,thesupportprovidedbythewholeMOOCteamwasrecognised.Manystudentsalsocommentedonthegrowthintheirownlearningasaresultofparticipatinginthecourse.Afewstudentscommentedthattheywouldlikemoreroboticscoursesorthattheywouldre-enrolinthecoursestoimprovetheirlearning. Examplesofstudents’commentsonthecoursedesignfollow(alsoseeAppendixD):
Theeffortthathasgoneintothecoursereallyshows.(IR,postcoursesurvey)
Thevideocoursesbymr.peterweresoeffectiveandsoeasytounderstandbecauseheexplainsthemwithpicturesandtoysratherthanformulaeandequations.(IR,post-coursesurvey)
2 Foranexampleofthevideossee‘OutandAboutwithRobots’:https://www.youtube.com/watch?v=GF1DwjAFIWg
Robotics MOOCs Evaluation Report 2015
18
Thelogicalflowandthewaytheknowledgewaspresentedwasexemplary!Assignmentswereverynicelydesignedwhichreallytestedtheconceptslearnt.Itisagreatcourseforsomeonelookingtogetajumpstartinthefieldofroboticvision.(RV,post-coursesurvey)
Itwasjustlikeacompletestoryorcompletemovie.Thediverseinformationprovidedinaninterestingmannermakecourseveryattractive.Difficultconceptswereexplainedineasyandnicemanner.(RV,post-coursesurvey)
Thelectureswereextremelygood.Allcontentwaswellexplained.Alltheproblemsthatcouldbefacedbyuswastakenintoconsiderationbeforehandandsupportmaterialwasgivenaccordingly.(IR,post-coursesurvey)
5.3 Level of difficulty: designing for a mixed cohort of students
Whilesomestudentswerecomfortablewiththelevelofdifficultyandenjoyedthechallengeandthepaceofeitherorbothcourses,othersstruggled;anothergroupwantedevenmoredifficulty.Thisundoubtedlyreflectsthevarieddemographicofthecohort,andtheeducationalandlanguagebackgroundsofparticipants.ThecoursewasprovidedinEnglishonly,andassumedknowledgewasarticulatedontheregistrationpagesoftheMOOCwebsites.LinkswereprovidedtomathematicsandprogrammingtutorialsattheKhanacademy(https://www.khanacademy.org/),andMATLABtutorialswereprovidedbyMathWorks.Sothoughthesesupportswerehelpfulforsome,thelevelofdifficultywassetatthatsuitableforathird-yearundergraduateengineeringunit.Consequently,asolidfoundationinmathematicsandprogrammingwasessentialforthosestudentswantingtodomorethanwatchthevideos.CommentsinTable5showtherangeofabilitiesandthevariedstudentresponsestothestandardexpected(seeTable5).
Table5:Variationinresponsetothelevelofdifficulty
Level of difficulty Comment
Struggled with the difficulty
I'm spending many hours per week on the course but I'm still struggling to keep up. (IR, mid-course survey)
Appreciated the challenge
It is not that easy for someone like me lacking a technical background but it is still a lot of fun, hard but awesome (always loved math) and everything is very well done. (RV, mid-course survey) I'm also left with a lot of pride in the progress I made and all that I learnt. I left high school with the impression that I was no good at maths. In this course I really showed myself that I am capable of learning and applying maths - especially when there's a goal which is so compelling to me. (IR, post-course survey)
Course perceived as well within student’s capabilities
It is so far a great course. It is very basic. I have already completed a Robotics course in University, but this is an excellent refresher course for me. I am looking forward to the vision course after this. (IR, mid-course survey) The problems were great. Very interesting. The questions can be harder :) (RV, post-course survey) Recommended reading and links to background material make it suitable for diverse audience. (RV, post-course survey)
Robotics MOOCs Evaluation Report 2015
19
ResponsessuchasthoseaboveandtheexistenceofanenthusiasticroboticshobbyistcommunityindicatethatthereispotentialtoofferadditionalQUTMOOCspitchedatvariouseducationalentrylevels.ThiswouldalsosupporttheideaofpathwaysthroughfreeMOOCcoursesintofee-payingQUTundergraduateorpostgraduatedegrees.Further,theteachinganddevelopmentteamssuggestanotherinnovativeideaworthpursuingisthebuildingofaresourcebaseforrobotics,similartotheKhanAcademy(https://www.khanacademy.org/).
5.4 Assessment
Asexplainedabove(section4.2.2),courseregistrantscouldengagewiththecourseinanumberofways,butthoseparticipantswhosuccessfullycompletedallassessabletasksreceivedacertificateofparticipation.Theoverallassessmentwasworth240points,comprisingassessablequizzes(120points)andMATLABprogrammingtasks(120points).Toqualifyforacertificatestudentshadtoachieveanoverallscoreof50%(120points).Thequizzesandprogrammingtaskswereweightedequally,soitdidnotmatterhowthe120pointswereaccumulated.621studentsreceivedacertificateofparticipationforIR,and475forRV.Twenty-sevenstudentsparticipatedsuccessfullyinpassingtheoptionalpeer-reviewedrobotbuildingproject.(SeeAppendixFforadditionalfiguresregardingassessment.)
Theassessmentwaschallenging,andwassetatthird-yearengineeringstandard;however,theremaybeacaseforreviewingtherequirementsnecessarytoachieve‘completion’standard,versusverifiedaward‘credential’standard.Nonetheless,foronestudentwhometthestandardandthechallenge,therewardwassignificant:
IamdeeplygratefulandalsoveryproudtoearnthecertificateasIfinishedmystudy8yearsago(electricalengineeringinGermany)andfeelalittlebitoutofbusinessduetomyjobasflighttestengineer.MyjobhasalmostnothingtodowithwhatIstudiedandyourgreatcoursegivesmethepossibilitytowarmupagain:).Forme,itwasveryhelpfultohavethevideolecturesandtowatchthevideosseveraltimeshelpedalot.(IR,post-coursesurvey)
5.5 Engagement strategies and community
Considerableeffortwasputintodesigningandimplementinganumberofstudentengagementstrategiesinordertobuildasenseofcommunity,supportstudents’learningneeds,andtopromptlyresolveanytechnicalissuesthatarose.Thisincludedmonitoringoftheasynchronousdiscussionforumsandemailmessaging,threesynchronouseventsconductedusingGoogleHangouts,aroboticsT-shirtdesigncompetition,andface-to-face‘meetups’withProfessorCorkewhilsthewastravellingintheUSA.
5.5.1 The centrality of the discussion forum
Staffandstudentscommentedonthefunctionalityofthediscussionforumtool.Itwaslessthanintuitivetonavigateandonlyallowedforconversationsfromonetomany;noone-to-oneorprivatemessagingwaspossible.Nonetheless,theTAs,eLSsupportstaff,ProfessorCorke,andtheMATLABrepresentative,BrandonArmstrong,madeeveryefforttorespondtostudentqueriesandkeeptheforumsactive,livelyandproductive.
Theteachinganddevelopmentteamrecognisedthatmanagementofthelearningcommunityduringthedeliveryofthecourseswascritical.Thereforeacommunicationsplanwasdevisedforthemanagementoftheforums,andspecificcommunitymanagementroleswereassignedtomembersoftheteam.FourTAsweretrainedinhowtorespondmeaningfullyandstrategicallytothousandsofstudentpostsandmessages.TheTAsmetregularlywiththeMOOCteamtoshareexperiencesanddevelopstrategiestomanagethediversestudentcohort.TheTAsrespondedtostudentqueries,
Robotics MOOCs Evaluation Report 2015
20
moderatedthediscussions,andposteditemsofinteresttofocustopicdiscussions.Inadditiontheycollecteddataaspartofdocumentationandevaluationprocesses.
Asindicatedearlier,studentsratedthequalityandvalueoftheforumsasexcellentorgood(82%forIR;83%forRV;seeTable4andAppendixE).Theforumswerethemaincommunicationspaceforstudents,andanextremelyimportantself-helpmechanismforstudents.Thepost-coursesurveyscontainedmanycommentsabouttheimportanceoftheforumsasresourcesforstudentstohelplearnersachievetheirgoals.Someratedtheforumsandpeer-to-peerinteractionasoneofthebestaspectsofthecourse(seeAppendixD).Thefollowingstudentcommentsareillustrativeofthegeneralmoodofstudents.
Generallythecoursewasgood.Butthediscussionforumwasawesome.(IR,post-coursesurvey)
[Thebestthingaboutthecoursewas]beingabletogethelpfrompeers.(IR,post-coursesurvey)
ThankyousomuchProfessorCorkeandthesupportstaffforthiswonderfulcourse.AnequallybigthankyoutoalltheotherparticipantsinthecoursewithoutwhomiwouldnothavebeenabletoachievehalfthescoreIhave.Hopetoseeyouguysinthesequel.(IR,forum)
Theforumswerealife-saverasotherseitherhadsimilarideasandhelpedmeorIhelpedthem.IthinkforumsisvitalforMATLABandQuizzeswherethereislittleorpoordefinitionofquestions.YoucorrectedquestionsthathaderrorsinquizorMATLABafterfeedbackandgavepeopleasecondchance.(IR,post-coursesurvey)
StudentsidentifiedbytheTAsandthecommunitymanagementteamashavingmadeasignificantcontributiontoeithertheIRorRVcoursecommunitieswerealsoacknowledgedviaemail.
5.5.2 Live events: Google Hangouts
ThreesynchronousInterneteventsforeachMOOCwerehostedusingGoogleHangouts3.Liveone-houreventswerehostedeitherbyPeter,theTAsoramemberoftheMOOCteam.Thesewererecordedandwhileonlyasmallnumberofstudentsattendedtheeventslive(IR,143;RV,92),manyviewedtherecordingslateron(IR,2424;RV,944).Nodoubttimedifferencesacrosstheglobeaccountforlowliveattendancefigures.StudentscouldalsosendinquestionsaheadoftimeandProfessorCorkeandtheteamwouldanswertheseduringtheliveHangout.
Thehangoutswereratedhighly(excellentorgood;80%forIR;82%forRV;seeTable4andAppendixE).Studentsgavefeedbackthattheseopportunitieswerewellappreciatedasstudentscould,forexample,hearstoriesaboutProfessorCorke’sinterestingcareerwithrobots,orlearnabouttheTAsresearchprojects.
ParticipatingintheeventsandaskingquestionslivewerealotmorefunthanIanticipated.IespeciallylikedthatIwasabletounderstandandaskabouttheTAs'research.Atthebeginningofthecourse,Idon'tthinkIwouldhaveunderstoodthedetailsnorwouldIhavebeenabletoaskthequestionsthatIdidask.(RV,post-coursesurvey)
ThevideosofPeterCorkewereverygoodathangouts.Ienjoyedhimreminiscingabouthispastroboticdays.(IR,post-coursesurvey)
3GoogleHangoutsallowforlive,streamedvideochatandinstantmessaging.Uptotenparticipantscancontributesimultaneouslytoasinglevideosession;anynumbercanwatchthelivesession.
Robotics MOOCs Evaluation Report 2015
21
5.5.3 Other engagement activities
OtheractivitiesthathelpedbuildaspiritofcommunityandcreativeengagementincludedaroboticsT-shirtcompetitionwherestudentswereinvitedtosendintheirdesignsforaT-shirt.CourseparticipantsvotedonthebestdesignandthewinnerwasawardedafreeT-shirtwiththeirowndesignprintedonit.Anotheractivitythatsparkedsomefriendlyandhumorousrivalrywasalivelydiscussionaboutthecricketworldcup.AustraliaandIndiaaretwomajorcricketingnationsand,asAustraliaandIndiaarethecountriesfromwheremoststudentsoriginated,therewasalevelofinterestinthisinternationalsportingevent.
Also,duringthesecondMOOC,ProfessorCorketravelledtotheUSAforbusiness.Duringthattimehesentoutageneralinvitationtostudentsviatheforum,tomeetupwithhimatthevariouscitieshevisitedatdifferenttimes.Someenthusiasticroboticistswereabletotakehimuponthisoption.Thispreparednesstomeetstudentsshowsthededicationoftheleadacademic,theabilityoftheMOOCteamtocontinuetofacilitatethecourseintheabsenceoftheleadacademic,andthepowerofmobiledigitaltechnologiesthatallowforlearningandteachingtooccursuccessfullyfromalmostanywhereontheglobe.Thisclearlypaidoffandasonestudentsaid:
Iamveryimpressedwithallinvolvedintheteachingthiscourse.Iamateacherinmyfield,butthepassionanddedicationshownbyProfCorkeandhisteamisamazing.(IR,forum)
5.6 Access and equity
Objective6.Todetermineissuesofaccessandequitythatimpactthedesign,developmentanddeliveryoftheMOOCs.
Issuesofaccessandequityneedattentioninanyhighereducationcourse,butespeciallyintheonlineenvironmentwherethepurposeistoofferfreeandopenaccesstostudentswhoaregeographicallydispersed.Issuesthatwerenotedinthiscourse,forexample,relatedtoaccesstocoursematerialsfortheoptionalrobotmakingproject.StudentsexplainedthatforsomethecostoftheLegoMindstormNXTrobotmakingkit(c.$AU500)wasprohibitive,evenifpurchasedsecondhandfromeBay.Also,oncepurchased,timelydeliveryofthekittomeetcoursetimeconstraintsstillhadtobefactoredin.Asonestudentcommented:
ItistoohardtopurchaseaMindstormKit,insteadofthat,IorderedaArduinoMega...onebay,butitwilltaketimetoarrivetoSriLanka.(RV,mid-coursesurvey)
Andanother:
EventhoughIhavelotsofArduinoparts,Ihadtoskiptheoptionalproject.Thatbummedmeoutalittle.ButnothingsaysIcan'tgobackanddoitlater.:-)(IR,forum)
AnotherfactortobeconsideredinthedesignprocesswasthatcertainregionsororganisationsdidnotpermittheuseofYouTube,theonlinevideochannelthathousedmostofthelecturecontent.Whilethecoursewaspromotedas‘free’,andtherewerenoregistrationfees,therewerestillpersonalcoststhatastudentmustbeabletopayaroundInternetconnectivity,datacharges,andmaterialcosts.ThankstoMathWorks,theMATLABlicencewasprovidedfreetostudentsforthedurationofeachcourse,andexcerptsfromthetextbookwerealsoprovidedasPDFsatnochargebySpringeronthecoursewebsite.Toengageintheoptionalrobotbuildingprojectstudentsalsoneededacameraorcameraenabledmobilephonetorecordtheirrobotinaction.
Nonetheless,onestudentsaidthatthebestthingaboutthecoursewasthatitwas‘openforeveryone’(IR,post-coursesurvey):anyonecouldenrolinthecourse.OneofthemosthearteningstoriescamefromaSyrianstudentattendingthecourse.Havingescapedfromhisworntorncountryhewasverygratefulforthefreecourse,andevensentvideofootageofhisbombedhomebackinSyria.Thathe
Robotics MOOCs Evaluation Report 2015
22
couldcontinuelearninginsuchdramaticcircumstanceswasimmenselyinspiring,andshowsthepowerofeducationtobringaboutchangeandprovidehope.
TheprovisionofcoursessuchastheroboticsMOOCstostudentswithnoaccesstosimilarcoursesintheirowncountryisatremendousexampleofcommunityoutreachonthepartofQUT.Asonestudentcommented:
Trulyspeakingyoudidfantasticwork.Inmycountryroboticsisnotmuchdeveloped.Irarelyfindanycoursehere.Yourlectureislikeboontostudentlikeus.Videosofthislecturesareverysimpleandtothedepth.Itwillbegreatifyouprovidemorecoursesbecausewereallyneedit.THANKYOUagainsir.(RV,email)
6. Technical and pedagogical innovations Theplatformprovider,EdCast,andtheprogrammingsoftwareproviderMathWorksworkedhardtocollaborativewithQUTtodevelopandintegratetoolsintothenewlyengineeredplatform,usingemergenttechnologiesandavailablesocialmediaapplications.ThisensuredthatthelearningtechnologieswerecustomisedandsupportedtomeetQUT’spedagogicalneeds.TherewerethreeinnovationsintegratedintotheEdCastplatformthatdeserveparticularacknowledgment.Theywere:(1)theintegrationofGoogleHangoutsasliveeventsheldwithinthecourse;(2)thepeerreviewsystemfortherobotproject,whichtrainedstudentsineffectivepeerreviewandmanagedtheassessmentoftheoptionalstudentprojects;and(3)theMATLABauto-gradingassessmenttool.ThesethreeitemshadnotpreviouslybeenintegratedintotheEdCastplatform,andQUTdrovethisinnovation.Becauseofthiseffort,moststudentsfoundthesiteeasytouse,andwhilethereweresometeethingproblems,thesewereremediedasthecourseprogressed.Theexceptionwasthediscussionforumtoolthatwaslessthansatisfactory,butnonethelessstaffandstudentsmanagedtoworksuccessfullywithit.AnadditionalinnovationwasthereuseofMOOCresourcesintheoriginalundergraduateroboticscourseENB339.
6.1 The EdCast platform and collaboration with industry partners
Objective4.Todeterminethesuitability(affordancesandfunctionality)oftheEdCastplatformasafoundationfordeliveryandsupportoftheMOOCs.
Thechoiceofplatformwaskeytothewholeproject.AfterexplorationofvariousMOOCconsortiaoptions,QUTchosetheopensourceplatformofferedbyEdCast(www.edcast.com).EdCastisaStanfordStartXcompanybasedinCalifornia.EdCastworkedincollaborationwithQUTfrommid2014tobuildacustomisedplatform,whichincludednewtechnicalinnovationstailoredtoQUT’sneeds.TheserversusedtostorecoursedataweremaintainedonshoreinAustralia,inSydney.EdCastmaintainedthecourseplatformprogrammedwithOpenEdXsourcecode,andprovidedQUTwiththenecessaryMOOCservices,including24/7technicalsupport.
Whiletheplatformchoicemeanttherewereopportunitiesforinnovation,asEdCastwasnotpartofaMOOCconsortium(e.g.likeEdXorCoursera),theadvantagesofgroupmarketingwerenotachievable.(Thisisdiscussedfurtherinsection7.4.)However,bypartneringwithEdCast,QUThadclearownershipofthecoursecontent,andfullrightstorepurposeandre-useanyresourcescreatedontheplatform.
Becauseofthisproductivecollaborationwithindustrypartners,moststudentsfoundthesiteeasytouse,andwhilethereweresometeethingproblems,thesewereremediedasthecourseprogressed,orworkaroundsputinplace.
Aswithanytechnologytherearealwayslimitationsandaffordances.Twoparticularlimitationsoftheplatformwerethelookandfeelofthediscussionboard,andtheneedtodevelopalltheaccompanyingprocessesandprotocolsforanopensourceproduct.Whilegeneralprinciplesregardingthefunctionality
Robotics MOOCs Evaluation Report 2015
23
ofalearningmanagementsystemcanbetransferredfromoneplatformtoanother,eachplatformhasitsownspecificfeatures.Thereforetimewasrequiredforteaching,supportandtechnicalstafftotestoutandlearnthecapabilitiesofthenewsystem.Forfutureprojects,sufficienttimeforthisprocessmustbefactoredintothedevelopmentschedule.
TherelationshipwithEdCastwasakeypartnership,andwhilebothEdCastandQUTwerenewtoMOOCdesignanddevelopment,overallthepartnersworkedwellontheinitiative.EdCastprovided24/7technicalsupportforstaffandstudentswhichwasessentialforacourseofferedgloballyacrossmultipletimezones.QUTwouldhavepreferredmorenoticeanddocumentationaboutchangestotheplatformwhilethecourseswereinprogress,butthiscommunicationissueisnowbeingaddressed.Althoughdesignissuesemergedwiththelackofeaseofuseofthediscussionforumtool,functionalityofthequiztoolandreliabilityofthegradebooktool,errorswerecorrectedduringthecourse,andworkaroundsfoundforproblemsthatcouldnotbesolvedatthetime.
Overall,theEdCastplatformwassuitableforofferingaworld-classonlineroboticscourse,androbustenoughtodelivertwocoursestoover20,000students.Thiswasaconsiderableachievement.Thesameplatformisbeingusedtodeliverthesecoursesinthesecondhalfof2015,whichindicatesthattheplatformwasfitforpurpose.TheissuesaroundcommunicationtoolsremainthemainconcernandQUTisworkingwithEdCasttoimprovethefunctionalityanduserexperienceofthesetools.
6.1.1 Integration of MATLAB auto-grading tool
IncollaborationwiththeMATLABdevelopers(whomaintainedtheprogrammingsoftwareproducedbyMathWorks),QUTandEdCastpioneeredtheintegrationofanassessmenttooltoautomaticallygradeandprovidefeedbackontheweeklyprogrammingtasks.Thetoolhadtobereliableenoughtoreturnaccuratefeedbackonstudents’programming,androbustenoughtobescaleduptomanagetheassessmentofthousandsofstudents.ThecustomisedMATLABtoolalsohadtopassstudentassessmentresultsbacktotheEdCastplatformgradecentre.Whilesomeissueswereencounteredintheearlystagesofthefirstcourse,EdCast,MATLABandtheQUTMOOCteamworkedquicklytorectifyproblemssothatstudentswerenotdisadvantagedduetotechnicalglitches.Figure4showsthatgenerallythestudentsweresatisfiedwiththeirexperienceusingMATLABandthesupportingMATLABtutorials(seeFigure4).
Robotics MOOCs Evaluation Report 2015
24
Figure 4: Experience using MATLAB. Note: 1 = Good, easy to use, useful, no issues; 2 = mixed or qualified answer; 3 = poor, had problems or difficulties; 4 = other. The results are based on a post-course survey question: ‘Please describe your experience using the MATLAB component integrated in with the assignment’. Responses were open-ended. Comments were analysed and divided into 4 categories, and then assigned a percentage, relative to all the qualitative responses for that question.
Asdiscussedabove,MATLABgrantedthesoftwarelicencefreetostudentsforthedurationofeachcourse.MATLABalsopreparedcustomisedvideotutorialstohelpstudentsgainthemostoutofthesoftware,andthiscombinedwithProfessorCorke’sRoboticsToolboxforMATLAB(http://petercorke.com/Robotics_Toolbox.html)optimisedapproachestotheprogrammingtasks.AmoreindepthreportonstudentresponsetoMATLABcanbefoundintheeLSMATLABReport,butinsummarystudentsworkedwellwiththeMATLABtoolandthosewhousedtheMATLABvideosfoundthemhelpful.Whenaskedtodescribetheirexperienceusingthetoolstudentscommentedthatitwasan‘excellenttool’alongwiththeToolboxwhichwas‘amazing’(IR,post-coursesurvey).SomestudentshadfrustrationsandwantedmoredocumentationfortheToolbox,ormoreexamplesintheproprietaryvideotutorials.
Doingtheprogrammingassignmentsilearnedthemost.(IR,post-coursesurvey)
MATLABexercisesweregreatbutverytough.(IR,post-coursesurvey)
IthinkyourstyleofteachingwithMatlabandnotgoingintoverydeepisperfectforengineerslikeme.Ihavearegularjobintelecomengineeringandparticularlyin5Gdevelopment,butformeRoboticsisserioushobby.(IR,highachiever,email)
ProfessorCorkeisreallygood.I’mprettybadinmathbutImostlyunderstoodalloftheformulas.Ithinkisreallyimportanttoexplainwherethevariouspartofanequationarecomingfromsothestudentcanhaveagoodandvisualmappingbetweenmathandreality.Comparedtootherfreecoursesonlinethiswasbyfarthebest.InsteadoffeelingfrustratedbecauseIhaven’tunderstoodthecontentsofformulasthrownatme,afterthiscourseIactuallyfeelIlearnedalotandIhaveagoodfoundationtogodeeperonthissubject.(RV,post-coursesurvey)
0%
10%
20%
30%
40%
50%
60%
70%
1 2 3 4
IRPost-coursesurvey(n=223)
RVPost-coursesurvey(n=145)
Robotics MOOCs Evaluation Report 2015
25
6.1.2 Automated online peer review
OneofthemajorinnovationsoftheMOOCprogramwastheonlinepeerreviewsystem.IntroductiontoRoboticsincludedanoptionalpracticalprojectwherestudentswereofferedtheopportunitytobuildandprogramafunctioningrobotarmusingaLegoMindstormKit.Althoughthisdidnotcounttowardscertification,successfulstudentsreceivedacertificateofachievementformakingtheirrobotarm,alongwithapersonalemailofcongratulationsfromProfessorCorke.Twenty-sevenstudentscompletedthepeerreviewproject,andexamplesofcompletedprojectscanbeseenonYouTube:https://youtu.be/CpMm0j1zB8o.
Becauseofthesizeofthecourse,therobotprojectsneededtobeautomaticallymarked.EdCast,incollaborationwithQUTdesignedanditerativelydevelopedasuitablepeerreviewtoolandprocessforthis.
Thosestudentswhooptedtobuildtherobotarmandthecontrolsoftwarefortherobotwerementoredthroughtheprocessviaweekly,pre-recordedvideodemonstrations.Thedemonstrationswerecreatedusingvideoandscreencapturetechnologies,andpresentedonscreenbyaQUTPhDengineeringstudent(LiamO’Sullivan).Thetutorwalkedstudentsthroughtherobotbuildingandprogrammingtasks,replicatingthesupportstudentswouldreceiveinalaboratorysessionintheundergraduateunit,ENB339.
Itshouldbenotedthatifthecourseweretobeshortened,thetimeforstudentstosetupandprocurethenecessaryresourcesforthecoursewouldbeadesignconcern.MOOCstendtorangefrombetween2-12weeks,with6weeksbeingatypicallength(Bayne&Ross,2014),andashortMOOCispreferableformaximisingstudentretention.Withcoursessuchasroboticswherestudentsneedtoprocureequipment,lead-timeforthismustbefactoredintothecoursedesign.
OncestudentshadcompletedtheirrobotmakingforIntroductiontoRobotics,theywereaskedtovideotherobotperformingthesettask.ThevideoevidencewasuploadedtoYouTubeorasimilarvideohostingwebsite,andtheURLsubmittedsothatpeerreviewcouldtakeplace.Onlythosestudentswhohadalsocreatedarobotandsubmittedavideowerepermittedtoengageinthepeerreviewprocess.
Topreparestudentsfortheprocessofpeerreviewingothers’work,anautomatedpeerreviewtrainingprocesswasdevelopedbyEdCasttechnicalstaff,theQUTlearningdesignersandQUTTAs.Projectstobemarkedwererandomlyallocatedtoeachstudentbythesystem,andeachstudentgradedthreeprojectsbeforethegradecentrereleasedtheirownmark.
6.1.3 Textbook publisher support
Asindicatedearlier,anagreementwasrealisedwiththepublisherSpringer,allowingfreeaccesstoexcerptsfromProfessorCorke’stextbook(Corke,2011).Theseexcerptswereuploadedase-bookchaptersviathecoursewebsites.Purchaseofthetextbookwasoptional,howeverthepublishersofferedthetextbookforpurchaseasanebookorinsoftcovertextatadiscounttoregisteredstudents.
Robotics MOOCs Evaluation Report 2015
26
6.2 Reuse of course resources for ENB339
In2014,aselectionofMOOCresourceswaspilotedinanauthentic,livesetting.TheMOOCresourceswerereusedintheQUTroboticsengineeringunitENB339:IntroductiontoRobotics,theunitfromwhichtheyhadoriginallybeenderived.Thispilotingofdesignedresourcescorrespondstostage3oftheelearninglifecycleasoutlinedinFigure2andAppendixC.Principallythelecturevideosweresuccessfullyrepurposedinthethird-yearunitinmodularisedformat,aspartofablendedlearningapproach,incorporating‘flippedclassroom’4pedagogicalstrategies.
Becauseofthedecisiontoembedthemodularisedvideoresourcesinthesecondsemesterunit,therewasconsiderablepressuretopreparetheresourcesintimeforimplementationwiththethird-yearclasses.TheexperienceisdescribedindetailintheEvaluationReportforENB339:ReuseofMOOCResourcesandtheFlippedClassroom.Itissufficienttosayherethatthisformofcontentreuse,combinedwithawell-designedandsupportedflippedclassroommodelwassuccessful.Studentcoursesatisfactionratingoftheunitimproved(datafromQUTInsightsurvey),andlecturersandtutorssawadvantagesinthemodel.ThewillingnessoftheteachingteamandeLSstafftocollaborativelyengageindevelopingthisnewlearningandteachingmodelwaskey.Itwasclearthatstudentsneededongoingcoachinginthenewlearningandteachingapproach,andthisnewapproachsuitedsomebutnotallstudents.Somestudentsneededmoresupporttobecomeself-regulatedlearnersandbettermanagetheirtime.
OneofthemaintechnicalissuesrealisedthroughtheENB339trialofMOOCresourceswastheissueofstreamedvideocontentovermobileWi-Finetworks.Manystudentswouldhavepreferredtheoptiontodownloadthevideofilesastheycouldeasilyexceedtheirdataquotasonmobiledeviceswhenaccessingcoursecontentvia,forexample,theirsmartphone.TheissueofdatacostsandquotaswassimilarlycommentedonintheMOOCs,althoughthisdidnotemergeasamajorissue,basedonpost-coursesurveysandforumcomments.SoaswiththeMOOC,whilestudentsinENB339gainedgreaterflexibilityintermsofwhenandwheretheyaccessedtheresources,atthesametimeotheraccess,equityanddatacostissuesemergedasnewconsiderations.
Nonetheless,despitesometechnicaldifficultiesthemajorityofENB339studentsaffirmedthatthenewpedagogicalmodelwasagoodone,thatis,theflippedclassroomincorporatingrepurposedMOOCresources,andtheyratedtheunitas4.7(outof5)intheendofsemesterInsightSurvey,2014.
TheexperienceoftriallingtheuseoftheMOOCresourcesinamoremodestsettingwithundergraduatestudentswasanimportantphaseofthedesignanddevelopmentelearningcycle.TheENB339trialalsocomplementedthesubsequentpre-coursetrialofRoboticVisioninDecember2014.Importantly,hadinsurmountableproblemsforstudentsorteachingstaffemergedduringthetrialstudentswouldnothavebeendisadvantagedasthelecturerscouldalwayshaverevertedtothetraditionalmodelofunitdelivery(thelecture/tutorialmodel).Further,astheresourceswerereusedinthenextofferingofENB339,insemester2,2015,thisindicatestheefficacyoftheintegrationofMOOCresourcesandthenewblendedlearning/flippedclassroompedagogicalapproach.
4Theflippedclassroomisamodelofteachingwherelowerlevelcognitivetasks,suchasgaining‘firstexposuretoknowledge’(Brame,2013;Walvoord&Anderson,1998)occursoutsideofclass.Moretimeisdevotedinclasstohigherorderthinkingandapplyingandprocessingknowledgewithguidancefromtheteacher,andoftenincollaborationwithpeers(Brame,2013).
Robotics MOOCs Evaluation Report 2015
27
7. Institutional implications Objective7.Todeterminefactors(e.g.technical,organisational,socialandprofessional)thatimpactthedesignandproductionoftheMOOCs.
7.1 Production issues TheMOOCprojectsrequiredconsiderableresourcesintermsofstafftimeandeffort.Aninnovativeelearningenvironmentwascreatedandthecoursewassuccessfulfrommanyperspectives.Typicalofanydesign-basedinitiative,therewereaplethoraofunknownsanduncertaintiestobeaccommodatedandresolved.Managementhadtobeflexibleandresponsive,andsecuringandretainingtherightmixofstaffwiththenecessaryskillsfortheprojectwaschallenging.
AsFigures1and2show,theMOOCteamfollowedadesigncycleincludingphasesofanalysisoftheproblem,designoftheMOOC(educationalandinterfacedesign),prototyping,testinganddevelopment,piloting,andimplementation.Allstageswereunderpinnedbyevaluation,asindicatedintheelearninglifecycleinFigure2andAppendixC.Evaluationwaseitherformativeorsummative.Sooveratwenty-fourmonthperiod,theMOOCwentfromconceptionthroughtothefirstimplementation(approximately19months),anddeliveryofthefirsttwoMOOCs(5months).
Foranumberofreasonstherewasadelayinselectionoftheplatformandthisimpacteddesignandproductionschedules.NonethelesstheMOOCwasdeliveredontime.AttheconclusionofeachoftheMOOCs,adebriefingsessionwasheldwitheLSstafftocapturelessonslearnedandinformthenextiterationofeachcourse.
Thefindingsfromtheformativedebriefingsessionshavebeenwrittenupasaseparatereport(MOOCProgram:LessonsLearnedfromDesignandProduction),butsomeofthekeyissuesaresummarisedbelow.
1. TheelearningdesignlifecycleandproblemsolvingAdesignapproachnecessarilyincludesphasesforiterativelygeneratingideas,rapidprototypingandtesting,qualityassurance,implementationandsupportandevaluation.Sufficienttimemustbefactoredinforallstaff(design,productionandtechnical)tosolveproblemsandresearchthefieldasnecessarythroughouttheproject.Asdigitaltechnologieschangerapidly,soonthejoblearningisextremelyimportant.
2. ConceptualdevelopmentTheMOOCteam(teachinganddesignandproductionstaff)needtimetobuildtrustandconceptualunderstandingofthespecialrequirementsofonlinelearningintheMOOCenvironment.Rolesandresponsibilitiessharedamongstthedesignandproductionteam,andtheteachingteamneedtobenegotiated,andtimeallocatedforthisaspectoftheprocesstomature.
3. QualityassuranceTheprocessofqualityassuranceisextremelyimportantforaprojectwithsuchhighpublicexposureasaMOOC.Thismeansthatsufficienttimeandresourcesmustbeallowedforconceptdevelopment,coursedesignandproduction,securingofcopyrightpermissions,iterativetestingofallaspectsofthecourse,transcription,editingandproofing,andstudentandtechnicalsupportduringimplementation.
Robotics MOOCs Evaluation Report 2015
28
4. Copyright
Tracingandsecuringcopyrightpermissions,ornon-copyrightalternativesisaparticularlytime-consumingprocess.MediausedwithinaMOOC,inparticularartisticworks(e.g.photographs,maps,graphs)thatQUTdoesnotownarenotcoveredbythesamecopyrightandfairuseregulationsthatgovernstandardAustralianuniversitycourses(e.g.itemscoveredbyPartVBoftheCopyrightAct1968).Itcanbethecasethatevenaftermuchsearchingorwaitingforaresponsefromthecopyrightownerpermissionmaynotbegranted.
5. SystemsWithanewelearninginitiativesuchastheMOOCproject,therearefewsystemsinplaceforalloperationalandsupportmatters.Thisincludestechnicalprocesses,datastorageandmanagement,workflowsandguidelines,andlearninganalytics.Theseallassistwithstreamlinedandsustainablepractices.Wherethesesystemsdidnotexist,newsystemshadtobeestablishedandpiloted.
Thesetwoinnovativecoursesweredesigned,developed,implementedandsupportedwithanintensityofstaffeffortthatisnotsustainableoverthelongterm.Theriskisstaffburnout,andalossofexperiencedstaffmeansalossofvaluableprojectandcorporateknowledge.However,assystemsareputinplacetodocumentandregulateworkflowpatternsandprocesses,somedesignandproductionissueshavebecomeeasiertomanageandtrack.
7.2 Evaluation and learning analytics
Johnsonetal.(2011)definelearninganalyticsas‘theinterpretationofawiderangeofdataproducedbyandgatheredonbehalfofstudentsinordertoassessacademicprogress,predictfutureperformance,andspotpotentialissues’(p.29).TheoverwhelmingamountofdatarealisedthroughtheMOOCshasalreadybeenusedtoinformthedesignofthenextphaseofMOOCdesignandproductionand,astheMOOCsunfolded,statisticsweregatheredfromtheEdCastplatform,GoogleAnalytics,YouTube,socialmediasourcesandthecoursesurveystoinformlearningandteachingdecisionsasthecoursesprogressed(seeAppendixB).
AslearninganalyticsisanewareaofbusinessforQUTanduniversitiesingeneralthereisstillaneedtodeviseandrefinethenecessarysystemssoanalysisandreportingcanbetimelyandinformtheimplementationof,designandredesignofQUTMOOCs.eLSisestablishingprocessesandworkingwiththeQUTStrategicIntelligenceUnitinordertofurtheranalysecurrentandfuturedata.ThedatacanbeusedtodeterminenotonlycohortpatternsofengagementinQUTMOOCs,butalsoindividualbehaviours,preferencesandoutcomes.Thereisroomhereformuchresearch.
Oneofthemainissuesinthisareaoftheprojectwastheissueofrecruitmentandretentionofstaffwithexpertiseinlearninganalytics.ThisreflectswhatGrajek(2015)hasidentifiedasoneofthetoptenITissuesof2015,namely‘hiringandretainingqualifiedstaff,andupdatingtheknowledgeandskillsofexistingtechnologystaff’(p.12).
7.3 Staff development and innovation
Staffneedtimetofamiliarisethemselveswithanewlearningmanagementplatform(LMS),todiscoverandmasteritsaffordancesandlimitations.Thisincludes,academics,teachingassistants,learningdesigners,webdevelopers,videoproductionstaff,communityandtechnicalsupportstaff.Duringthedepartmentaldebriefingandfeedbacksessions,theeLSMOOCteamremarkedthattheyvaluedtheopportunitytocreativelyaddressandresolvechallenging,non-routineproblemsaspartoftheMOOCdesignandproductionprocess.Thisspacetoinnovateandtakerisks,toworkcollaborativelywith
Robotics MOOCs Evaluation Report 2015
29
industrypartnerstodeveloptechnologiesandprocessestosupportonlinelearningwashighlyvalued.ThiskindofprofessionallearningonthejobisinlinewiththevisionandvaluesoftheUniversity,asoutlinedinQUT’sBluePrint4(2014)whereitstatesthattheUniversityvalues‘aspiritofexperimentation,innovation[and]entrepreneurialism’(p.4).Toencourageongoinginnovation,itisimportantthatexperimentationissupportedintheMOOCenvironmentatQUT.Thismeansprovisionforstafftimetopursueandresearchdevelopmentsinthefield,andtoreflectonlessonslearnedtofurtherinformdevelopmentsinonlinelearning.
7.4 Marketing
QUTmarketingspecialistsdevelopedahighlyeffectivecustomiseddigitalmarketingcampaign.Recognisingthehighlevelofinterestinrobotics,thecampaigntargeteddomesticandinternationalmarkets.ThecampaignresultedinfrontpagecoverageinTheAustralian,andarticlesinmanyglobalmagazinesincludinginComputerWorld,TechworldandPDadvisor.ThesinglemosteffectivestrategywastheuseofaScienceAlertonFacebook(https://www.facebook.com/ScienceAlert),whichaccountedfor124,214uniquepageviews(1/12/2014–3/9/2015).Figure5showsthespikesincourseregistrationsfollowingpostingtoScienceAlert.Thebenefitsoffocussedmarketing,beyondthehighnumbersofregistrations,includeddrawinginterestedpartiestotheQUTwebsite,withtheQUTlandingpage(https://www.qut.edu.au/study/open-online-learning)receiving162,973uniqueviewsbetween1December2014and3September2015.EdCastdescribedtheQUTcampaignasexceptionallysuccessfulincomparisonwiththeirotherclients(seeFigure5).
Figure 5: Marketing campaign related to MOOC enrolments. Note: The Science Alerts were posted on Facebook on 7 Feb and 8 April 2015.
ThereisnowaneedtoincreasegeneralawarenesswithinQUTabouttheMOOCprogramandtheroleitplaysmorebroadlyinthedevelopmentofonlinelearningopportunities.AdditionallyQUTrequiresanongoingmarketingstrategytoraiseawarenessofitssuiteofMOOCs,andpathwaystootherQUTcoursesandresearchopportunities.ThereisalsoagrowingcommunityofMOOCparticipantalumniwithaspecificinterestinrobotics.ThissignificantcommunityoflearnersmaybeinterestedinfurthercontinuingeducationopportunitiesofferedbyQUT.
Robotics MOOCs Evaluation Report 2015
30
8. Conclusion
8.1 General conclusions
Thepurposeofthisdocumentwastoreportontheeffectivenessoftheelearningenvironmentregardingthedesign,productionandimplementationoftwoQUTMOOCs.Asetofsevenobjectivesfocusedtheevaluation.
Thefindingsindicatedthatthecourseswereofaveryhighstandard.Thiswasthecaseintermsofthequalityandappropriatenessoftheresourcesprovidedandtheeducationaldesignandsupport.Studentswereprovidedwithavariedsetofcourseresourcesthatincludeddemonstrationsandlectures‘inthefield’.Theoryandpracticewerecombinedinauniqueway,oftenusingexamplesfromtherealworldofrobotics.Studentswerechallengedthroughassessablemathematicalandprogrammingtasks,whileatthesametimebeingsupportedinafunandengagingway.Thediscussionforumwasakeycommunicationtoolforpeer-to-peerlearningandwaswellsupportedbytheMOOCteam.
TheroboticsMOOCsreachedadiverseinternationalaudiencein161countries,with20,718registrants.1,096certificatesofparticipationwereissuedforsuccessfulcompletionoftheassessmenttasks.Thecourseswerechallengingsoftwareengineeringcoursesbasedononeelectivethird-yearunit,andyettheyattractedhobbyistswithoutformalqualificationandpostgraduateswithvaryinglevelsofcompetencyinmathematicsandcomputerprogramming.Coursesatisfactionwasextremelyhighforbothcourses.
Asexpected,registrationsintheearlyweeksofeachcoursewerehigh,withactiveparticipationdroppingoffmarkedlyafterthefirstfewweeks.Themainreasonforenrollingineithercoursewasforpersonalinterest,whichsuggeststhatattaininganawardorpassingassessmenttaskswerenotthegoalsofmanyregistrants.Alargeproportionofstudentsmayhaveenrolledineithercoursefortheopportunitiesitofferedforinformalratherthanformallearning.
Someissuesofaccessandequitywerenoted,mainlyinrelationtoacquiringadditionalresourcesfortheoptionalrobotmakingproject.Howeveropenandfreeenrolmentintothecourses,andtheprovisionoffreelicencesfortheMATLABprogrammingsoftwareplusexcerptsfromthetextbookforthedurationofthecourse,maximisedpossibilitiesforwidestudentparticipation.
TherewereteethingproblemswiththeEdCastplatform,butEdCastprovidedasufficientlyrobustplatformforsuccessfuldeliveryofcoursestothousandsofsimultaneousparticipants.BypartneringwithEdCastandMathWorks,QUTpursuedopportunitiesforinnovationincourseprovisionandassessment.SolutionsdevelopedforthiscontextcannowbeusedtoinformonlineandblendedlearningatQUTgenerally.
TheefficacyofreusingresourcescreatedfortheMOOCsinanothercontextwasproven.ModularisedvideoswereintegratedbackintoENB339,theQUTunitfromwhichtheMOOCsoriginated.Thisreusewasaccompaniedbypedagogicalinnovationusingaflippedclassroomapproach.Itwasanexampleofinnovationsinpedagogyandtechnologytogetherdrivingchange.
TheMOOCprojecthighlightedinstitutionalissuesaroundproduction,learninganalytics,professionaldevelopmentandmarketing.ItwasclearthatproductionoftheMOOCswasachievedthroughtheeffortsofacommittedandinnovativeteam,whichincludedstafffromeLearningServicesandtheFacultyofScienceandEngineering.Theprojectdemonstratedthattomaintainquality,oncetheinitialprojecthadbeenestablished,therewasanurgentneedtocreatesustainableproductionandtechnicalsupportsystems.Manycomplextechnical,designandmanagementproblemswereencounteredinthisproject,andthecapacityofstafftoproactivelyfindandimplementworkablesolutionswasnotable.
Robotics MOOCs Evaluation Report 2015
31
Twofinalissuesemergedasimportantfindingsinthisevaluation.Firstly,theprojectdemonstratedthevalueoflearninganalytics,andtheirplaceininformingcoursedesign,andunderstandinglearnerbehaviourandmotivationsincontext.Secondly,theimportanceoffindingthemosteffectivecommunicationschannelformarketingthecourseswascrucial:locatingacommunitythatwaslikelytobemostinterestedintheseparticularcoursescontributedtocoursesuccess.ScienceAlertwasthesolutionforrobotics.
8.1 Future research and opportunities for innovation
ThesuccessfulimplementationofQUT’sfirstMOOCsinroboticssuggeststhattheUniversitycouldwelldevelopacatalogueofmassiveopenonlinecoursesthatalignwithdevelopmentsinQUT’sGraduateProfessionalEducation(GPE)program.Thiswouldbeanopportunitytobuildnewcoursesaroundcurrentundergraduateandpostgraduateunits.ThesuccessoftheENB339experimentregardingthereuseandrepurposingofMOOCresourcessuggeststhatsimilarinitiativescouldbeundertakenasMOOCcoursesaredevelopedforotherdisciplinesatQUT.Theseareallareasforfutureresearchandinnovation.
ThenextphaseofinnovationisexpectedtofocusontheintroductionofMOOCsfromotherdisciplines,capacitybuildinginthosealreadyinprogress,improvementofthelearnerexperienceandrefinementofassessmentandcommunicationtools.Itisanopportunityforpedagogicalandtechnologicalinnovation.StrategicthinkingisrequiredtodesignpathwaystoconnectcurrentQUTawardcoursesandthenewopenonlinecourses.Thisdiversificationofofferingwillrequireexaminationofcertificationoptionsanddevelopmentofnewsustainablebusinessmodels.Thecreationofaroboticsresourcebase,similartotheKhanAcademy,isanexampleofapossibleareaforfutureinnovation.Alongsidetheseopportunitiesforinnovation,QUTwillneedtosupportprocessesforcontinuousimprovementtoensurethatthecurrentMOOCcoursesmaintaintheirpositioninthesector,andthattheUniversitycontinuestoinnovateanddeveloprelationshipswithindustrypartners.
Thenextstagetobepursuedintheelearninglifecycleisstage6:Evaluationresearchonthematuresystem(seeFigure2andAppendixC).Thisiswhereamoreholisticunderstandingofhowlearnersengagewiththeelearningenvironmentcanbeinvestigated.Thelearninganalyticsalreadygatheredfromtheroboticscoursesshowthatthereisawealthofdatathatcouldformthebasisforfurtherinquiry.Inparticular,theUniversitymaywishtodirectefforttowardsuncoveringmorearoundthepictureofstudentdemographicsastheyrelatetoachievementandoutcomes.Thereisalsomuchtobelearntaboutstudentlearninginthiscontext.Asyetweknowlittleabouthowstudentsactuallylearninthesemassiveopenonlinecourses,andwhatengagementwithsuchonlinecoursesmeansforstudentsintermsofcareerandprofessionaldevelopment,andthereforeQUT’sroleasanelearningprovider.
Acknowledgements
Projectssuchastheserequirethetimeandcommitmentofalargeteam.Wegratefullyacknowledgetheexceptionaleffortsoftheacademicteam:ProfessorPeterCorke,teachingassistantsObadiahLam,JamesSergeant,JackSlaterandEdPepperell,andAssociateProfessorMichaelMilfordforhisENB339contributions.WethankQUTMarketingandCommunication,inparticularSimoneLong,andourindustrypartnersEdCast(especiallyDanielMcKelveyandTonyJiang),MathWorks(inparticularBrandonArmstrong),andSpringerpublishing.FinallywethanktheeLearningServicesteamfortheircommitmenttothedesign,production,technicalservicesandlearninganalyticscomponents:TabethaBozin,LuisaRossitto,LuzetteDonohue,PaulFenn,ChrisNewman,AndrewThomson,JasonShanks,JasonMarkland,LiamMoushall,RogerCook,DavidDennis,KushanYapa,DrTristanPhilip,DanielCrane,StephenBooth,KimManningandDesleyGorle.Wethankeveryonefortheirexceptionalenthusiasmandsupportforthisproject.
Robotics MOOCs Evaluation Report 2015
32
References Bayne,S.,&Ross,J.(2014).ThepedagogyoftheMassiveOpenOnlineCourse:TheUKview.Retrieved
fromhttp://www.heacademy.ac.uk/assets/documents/elt/HEA_Edinburgh_MOOC_WEB_240314.pdf
Brame,C.(2013).Flippingtheclassroom.CenterforTeaching,VanderbiltUniversity.RetrievedOctober
30,2015fromhttp://cft.vanderbilt.edu/guides-sub-pages/flipping-the-classroom/.
Charmaz,K.(2011).Constructinggroundedtheory.LosAngeles,CA:Sage.Corke,P.(2011),Robotics,visionandcontrol:FundamentalalgorithmsinMATLAB.Berlin,Germany:
Springer.
Freitas,S.I.,Morgan,J.,&Gibson,D.(2015).WillMOOCstransformlearningandteachinginhighereducation?Engagementandcourseretentioninonlinelearningprovision.BritishJournalofEducationalTechnology,46(3),455–471.doi:10.1111/bjet.12268v
Gallagher,S.,&Garrett,G.(2013,31July).Disruptiveeducation:Technology-enableduniversities.Retrievedfromhttp://ussc.edu.au/ussc/assets/media/docs/publications/130801_DisruptiveEducation_GallagherGarrett.pdf
Grajek,S.(2015).Top10ITissuesof2015:Inflectionpoint.EDUCAUSEReview,50(1),10-48.Retrievedfromhttp://er.educause.edu/articles/2015/1/top-10-it-issues-2015-inflection-point.
Ho,A.D.,Ho,A.D.,Reich,J.,Nesterko,S.,Seaton,D.T.,Mullaney,T.,...Chuang,I.(2014).HarvardXandMITx:Thefirstyearofopenonlinecourses,Fall2012-Summer2013,HarvardXandMITxWorkingPaperNo.1.Retrievedfromhttp://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263
Johnson,L.,Smith,R.,Willis,H.,Levine,A.,&Haywood,K.(2011).The2011horizonreport.Retrievedfromhttp://www.nmc.org/pdf/2011-Horizon-Report.pdf
Onah,D.F.O.,Sinclair,J.,&Boyatt,R.(2014,7-9Jul).Dropoutratesofmassiveopenonlinecourses:Behaviouralpatterns.InEDULEARN14Proceedings.6thInternationalConferenceonEducationandNewLearningTechnologies(pp.5825-5834).Retrievedfromhttp://wrap.warwick.ac.uk/65543/1/WRAP_9770711-cs-070115-edulearn2014.pdf
Phillips,R.,Kennedy,G.&McNaught,C.(2012).Theroleoftheoryinlearningtechnologyevaluationresearch.AustralasianJournalofEducationalTechnology,28(7),1103-1118.http://www.ascilite.org.au/ajet/ajet28/phillips.html
QUT(2014).BluePrint4.Brisbane,QLD:QueenslandUniversityofTechnology.Retrievedfromhttps://cms.qut.edu.au/__data/assets/pdf_file/0013/71113/qut-blueprint.pdf
QUT(2015).QUTDigitalRoadMap:Our2020Vision,v3.1,July.
Siemens,G.,Gasevic,D.,&Dawson,S.(2015).Preparingforthedigitaluniversity:Areviewofthehistoryandcurrentstateofdistance,blended,andonlinelearning.Retrievedfromhttp://linkresearchlab.org/PreparingDigitalUniversity.pdf
Walvoord,B.E.,&AndersonV.J.(1998).Effectivegrading:Atoolforlearningandassessment.San
Francisco:Jossey-Bass.
Robotics MOOCs Evaluation Report 2015
33
Appendices
Appendix A: Objectives, sample indicators and data sources Objective Sample Indicators and Data Sources
1. To determine the reach of the program and demographics of MOOC participants.
Number of unique ID logins by country; age, educational level, access patterns via device; EdCast/Google analytics, pre-, mid- & post-course survey results; discussion forum posts & emails
2. To determine patterns of student engagement in each of the MOOCS.
Comparison of access by level of activity (registrants, active participants, participants who received certificate); formative & summative assessments attempted; discussion forum posts & emails; feedback from teaching & MOOC team; surveys.
3. To determine the effectiveness of the teaching and learning strategies employed in the MOOCs to support student learning.
Certificates of completion issued, robot project completions, student responses on discussion forum & in surveys, feedback from TAs and Professor Corke (interviews, focus groups, discussion forums, emails, logs, groups)
4. To determine the suitability (affordances and functionality) of the EdCast platform as a foundation for delivery and support of the MOOCs.
Student affirmations on discussion forum, survey feedback, responses from TAs, Professor Corke, eLS team, and MATLAB rep (interviews, focus groups, debriefing sessions, emails, technical issues logged & resolved, data from MOOC pilot, 2014).
5. To determine the impact of issues relating to use of MOOC resources.
Student affirmations on discussion forum, survey feedback, responses from TAs, Prof. Corke, eLS team, & MATLAB rep (interviews, focus groups, debriefing sessions, emails, technical issues logged & resolved, MOOC pilot).
6. To determine issues of access and equity that impact the design, development and delivery of the MOOCs.
Student feedback from discussion forums & emails, surveys, staff feedback via interview/focus group, & the discussion forum, EdCast & Google analytics, data from MOOC pilot, 2014
7. To determine factors (e.g. technical, organisational, social, professional) that impact the design and production of the MOOCs.
Feedback and reports from eLS web, media and learning design staff, status reports, analytics and feedback from EdCast re issues, solutions & workarounds; data from the MOOC pilot, report from Marketing & Communications.
Robotics MOOCs Evaluation Report 2015
34
Appendix B: Data sources, sampling and data types Source Sampling Quantitative (Quan)
or Qualitative (Qual)
EdCast – student activity tracking; formative & assessable quizzes; MATLAB programming
Statistics collected weekly from student records (IR, N = 12894; RV, N = 7824)
Quantitative
Google analytics Statistics collected weekly from student records
(IR, N = 12894; RV, N=7824)
Quantitative
YouTube lecture streaming Sampled at intervals over the semester Quantitative
Robotics Toolbox http://www.petercorke.com/Robotics_Toolbox.html Quantitative
Pre-course survey IR, Feb-June 2015, N= 5569; RV, Apr-June 2015,
N= 1689
Quan & Qual
Mid-course survey IR, March 2015, N = 234; RV, May 2015, N = 102 Quan & Qual
Post-course survey IR, April 2015, N = 383; RV, June 2015, N = 256 Quan & Qual
Discussion forums Sampled on TAs feedback report sheet Quan & Qual
Emails from participants Issues collected on JIRA tracking system Qualitative
eLS debriefing workshop 13 participants; April 2015 Qualitative
eLS feedback session 8 participants; August 2015 Qualitative
Robotics MOOCs Evaluation Report 2015
35
Appendix C: Details of the evaluation elearning lifecycle for the project
Cycle Lifecycle stage Development activity
Evaluation Research Role of theory and design principles
0 Analysis of problem
Document the problem: online learning at a massive scale, for robotics
Baseline analysis: environmental scan and assessment of user needs.
Define teaching and learning problem based on scholarship
1 Design elearning artefact
Course design planned collaboratively – eLS and SEF.
Ongoing formative evaluation of design
Design based on principles of elearning best practice
2 Prototype elearning artefact
Prototype MOOC assets independently & on EdCast platform
Formative evaluation of the first MOOC and project management processes
Reflect on MOOC characteristics and potential for re-use of resources (conference poster)
Refine principles of elearning best practice for context
3 Piloting: Design elearning environment and conduct pilot study
Trial of MOOC on EdCast development site (Nov- Dec 2014) & in ENB339 (semester 2, 2014)
Formative evaluation of the elearning environment
Refine principles of elearning best practice for context
4 Implementation: Refine elearning environment and conduct full trial
IR MOOC released: Feb 2015; RV April 2015
Formative evaluation: eLS debriefing/feedback session; student surveys and data captured (analytics)
Effectiveness research into learning processes
Initial learning design principles
5 Evaluation research on operational system
Confirm effectiveness of elearning environment
Summative evaluation: Robotics MOOC Report 2015
Effectiveness research into learning processes and outcomes
Refined learning design principles
6 Evaluation research on mature system
Holistic understanding of how learners engage with the e-learning environment
Summative evaluation: Deeper investigation of MOOC learning analytics
Effectiveness research into learning processes and outcomes
Refined learning design principles
Note: This evaluation elearning lifecycle is for the robotics MOOC project for the period 2013- 2015. It is based on the Phillips, McNaught and Kennedy (2012) elearning lifecycle model. The table shows the development activities at each stage and the related evaluation and/or research activities that occur at any particular stage. This evaluation report represents Stage 5 activities, where the system is live and operational (see also Figure 2).
Robotics MOOCs Evaluation Report 2015
36
Appendix D: The best aspects of the course In an open-ended question participants were asked about the best aspects of the course. Each response was coded to one or more categories. Categories were derived from the data, not imposed from a pre-determined list. Comments could be positive, negative or relate in other ways to the category. Frequencies were then expressed as a percentage of the total responses to the question (IR = 247; RV = 154). Category Example comment IR
(n=247) (%)
RV (n=154)
(%) Course materials (lectures videos, animations, textbook, demonstrations)
‘The videos were excellent. Most lectures in and outside of the classroom consist of instructor reading bullet points, but professor Corke really knows how to visually complement his lectures.’
38 39
Design of the MOOC (e.g. pacing of the unit, imagery interactivity, modules, learning and teaching strategies, structure, explanations
‘All content was well explained. All the problems that could be faced by us was taken into consideration before hand and support material was given accordingly.’
36 21
Topics covered ‘Started at a base level of understanding, and covered a large range of topics in brief, to really support understanding and interest in the wider vision field.’
27 22
Mathematical and MATLAB learning tasks and programming
‘MATLAB exercises were great but very tough.’ 21 19
Prof. Corke’s passion and teaching approach, and/or support from the MOOC team
‘The passion of Peter Corke for robotics, it's really really motivating to be taught by a professor, even far away, in another language, that makes you feel he's here to take you to another level of knowledge.’
11 14
Students comments about their own learning
‘I think the best aspect of the course is that I learned a lot, I hope to use all this all this learning in a future. I think also that the material of the course is really good and liked the way the quizzes and Peter Corke teaches’
11 8
Forums, community, discussions & interactions with others
‘The socialization of knowledge regarding vision and robotics great importance in mechatronics.’
8 6
Everything – the whole course ‘The entire course was great. The level is perfect. The pace is great.’
7 5
Quizzes (formative and summative)
‘Good reinforcement of teaching material through quizzes and programs.’
6 4
Prof. Corke's MATLAB Toolbox ‘Another useful thing is the explanation of the toolbox it took years of coding and it is quite useful to understand using it.’
4 6
Robot project ‘Hands-on with Matlab and with real hardware in the optional project.’
4 1
Level of difficulty ‘I think the level of math was about right for an introductory course. Clearly there is a lot more math behind that covered in the lectures, but encapsulating much of that deeper math in MatLab routines allow us to focus on the concepts rather than getting lost in the math.’
2 3
Google hangouts ‘I really enjoyed the google hangouts. They made the course feel much more social than previous MOOC's I've taken.’
2 1
Flexibility the course offered ‘Ease of learning from anywhere.’ 0 2
Robotics MOOCs Evaluation Report 2015
37
Appendix E: Quality of the resources In the post-course surveys (IR and RV), participants were asked to rate the video lectures, the textbook, the ‘check for understanding questions, the discussion forums, email communications, and the Google Hangouts. The results are expressed as percentages of the total number of responses to each question.
Q4. Please rate the quality and value of the video lectures. IR (n= 382)
% RV (n=251)
% Excellent 77.7 72.5 Good 20.2 24.7 Satisfactory 1.6 2.0 Poor .5 .8 Very poor 0.0 0.0 Total 100.0 100.0 (IR: M = 4.75; SD = .5000; RV: M = 4.69, SD = .550)
Q4. Please rate the quality and value of the textbook.
IR (n=301) %
RV (n=185) %
Excellent 48.5 45.4 Good 39.2 42.7 Satisfactory 10.3 9.7 Poor 2.0 2.2 Very poor 0 0 Total 100.0 100.0 (IR M = 4.34, SD = .743; RV M = 4.31, SD = .737) Q4. Please rate the quality and value of the ‘check understanding’ questions.
IR (n=377) %
RV (n=249) %
Excellent 40.6 30.1 Good 47.7 54.2 Satisfactory 10.9 14.5 Poor .8 1.2 Very poor 0.0 0.0 Total 100.0 100.0 IR M = 4.28, SD = .685; RV M = 4.13, SD = .692
Q4. Please rate the quality and value of the discussion forums.
IR (n = 344) %
RV (n=2321) %
Excellent 37.2 39.4 Good 44.8 43.7 Satisfactory 14.2 13.0 Poor 3.8 3.9 Very poor 0 0
Total 100.0 100.0
IR M= 4.15, SD = .802; RV M = 4.19, SD = .805
Q4. Please rate the quality and value of the email communications.
IR (n=295) %
RV (n=188) %
Excellent 40.3 37.8 Good 42.0 44.7 Satisfactory 14.2 13.8 Poor 3.4 3.2 Very poor 0.0 0.5 Total 100.0 100
(IR M = 4.19, SD = .804; RV M = 4.16, SD = .818)
Robotics MOOCs Evaluation Report 2015
38
Q4. Please rate the quality and value of the Google Hangouts.
IR (n=206) %
RV (n=108) %
Excellent 36.4 38.9 Good 43.7 43.5 Satisfactory 16.5 13.9 Poor 3.4 2.8 Very Poor 0.0 0.9 Total 100.0 100.0 (IR M =4.13, SD = .807; RV M= 4.17, SD = .837)
Appendix F: Assessment Assessment: Introduction to Robotics
Quizzes (EdCast) - one per week Wk 1 Wk 2 Wk 3 Wk 4 Wk 5 Wk 6
Attempted 2280 1156 851 785 670 574
Passed 1713 766 735 622 538 381
Percentage passed 75% 66% 86% 79% 80% 66%
Tasks (MATLAB) - one per week
Attempted 1060 723 634 434 284 194
Passed 1060 723 597 319 242 157
Percentage passed 100% 100% 94% 74% 85% 81% Note: 621 certified participants received a certificate of participation having passed the assessable quizzes and MATLAB tasks. 27 passed the peer reviewed optional robot building project. Assessment: Robotic Vision
Quizzes (EdCast) - one per week Wk 1 Wk 2 Wk 3 Wk 4 Wk 5 Wk 6
Attempted 1246 858 682 598 536 481
Passed 1208 709 567 475 504 230
Percentage passed 97% 83% 83% 79% 94% 48%
Tasks (MATLAB) - one per week
Attempted 869 567 387 271 177 158
Passed 869 529 323 271 177 158
Percentage passed 100% 93% 83% 100% 100% 100% Note: 475 certified participants received a certificate of participation having passed the assessable quizzes and MATLAB tasks.