Individual Differences Related to College Students’ …Spehar, Adam, & Refkin, 2007; Macfadyen & Dawson, 2010), as well as the quality of these posts (2017). Individual differences

(2017).Individualdifferencesrelatedtocollegestudents’courseperformanceincalculusII.JournalofLearningAnalytics,4(2),129–153.http://dx.doi.org/10.18608/jla.2017.42.11

ISSN1929-7750(online).TheJournalofLearningAnalyticsworksunderaCreativeCommonsLicense,Attribution-NonCommercial-NoDerivs3.0Unported(CCBY-NC-ND3.0) 129

Individual Differences Related to College Students’ Course Performance in Calculus II

SaraA.HartDepartmentofPsychology

FloridaCenterforReadingResearchFloridaStateUniversity

[email protected]

MiaDaucourtDepartmentofPsychology,FloridaStateUniversity

ColleenM.GanleyFloridaCenterforResearchinScience

Technology,EngineeringandMath,LearningSystemsInstituteDepartmentofPsychology,FloridaStateUniversity

ABSTRACT. Inthisstudy,weexplorestudentachievementinasemester-longflippedCalculusIIcourse,combiningvariouspredictormeasuresrelatedtostudentattitudes(mathanxiety,mathconfidence, math interest, math importance) and cognitive skills (spatial skills, approximatenumber system), as well as student engagement with the online system (discussion foruminteraction, time to submission of workshop assignments, quiz attempts), in predicting finalgrades. Data from 85 students enrolled in a flipped Calculus II coursewas used in dominanceanalysis to determine which predictors emerged as the most important for predicting finalgrades.Results indicated that feelingsofmath importance,approximatenumber system (ANS)ability, totalamountofdiscussionforumposting,andtimegradingpeerworkshopsubmissionswas the best combination of predictors of final grade, accounting for 17% of variance in astudent’s final grade. The point of thisworkwas to determinewhich predictors are themostimportant inpredictingstudentgrade,withtheendgoalofbuildingarecommendationsystemthatcouldbeimplementedtohelpstudentsinthistraditionallydifficultclass.Themethodsusedherecouldbeusedforanyclass.

Keywords: Math performance, calculus, flipped classroom, math attitudes, cognitiveperformance,studentengagement

1 INTRODUCTION

Overthepastfewyears,theattritionofSTEM-focusedundergraduatesintheUnitedStateshasbecomea critical national concern,with “a substantial number of undergraduate students initially enrolled inSTEMdegreeprograms[dropping]outinthefirsttwoyears”(PCAST,2012).Inordertomaintainhigh-quality instructional practices in the face of STEMundergraduate attrition, growing demand for large



numbersofgraduates,anddiminishing financialandhumanresources,manycollegesanduniversitiesare turning to technologically aided teaching practices. This technological shift aims to integrate thetraditional classroomenvironmentwithonline course resources toenhance, replace, and supplementface-to-faceinstructiontoreachmorestudentsinacost-effectiveway(Garrison&Kanuka,2004).

Toaccommodatethisshifttowardtechnologicallyaidedteachingmethods,schoolshave implementedcampus-wideLearningManagementSystems(LMS),suchasMoodleandBlackboard.Theseweb-basedmanagementsystemsprovidedatarelatedtothe“user,”witheveryactionofthestudenttrackedandrecorded online. The underlying advantage of these LMS data is that they unobtrusively recordindividual student activity and interaction with course materials in real-time, providing a lens intotraditionallyunobservablelearning-relatedbehavioursandsilentlytrackingindividualstudents’learningprogression (Macfadyen & Dawson, 2010; Gašević, Dawson, & Siemens, 2015). In traditional collegecourses,manyperformancemeasures, forexamplemidtermor finalexams (Lee,Speglia,Ha,Finch,&Nehm, 2015;Macfadyen & Dawson, 2010), are taken too late in the semester to identify strugglingstudents in time toprevent course failure. In contrast, the real-timenatureof LMSdatamayprovideearlywarningsignsofpotentiallyat-riskstudents,enablingthe implementationof interventioneffortsbefore failure becomes inevitable (Milne, Jeffrey, Suddaby,&Higgins, 2012; Gašević et al., 2015). Bymoving away from summative assessments such as exams, and providing visualizable, real-timeinformationabout individualaspectsofstudentengagementandlearning(Cocea&Weibelzahl,2009),theuseof LMSdata, also called “academicanalytics” (Goldstein&Katz,2005;Macfadyen&Dawson,2010), allows educators to monitor individual academic progress step-by-step (Bienkowski, Feng &Means,2012).

With this study, we sought to determine the best individual differences predictors of studentperformance in a flipped Calculus II course. This course required students to interact with an LMSoutsideofclassbywatchingvideosofcoursecontentlecturesanddoingworkshopsandquizzes,aswellas come to class for guided problem solving. The course performance predictors we used includedattitudinal and cognitive factors commonly found to be associated with math achievement in theeducation and psychology literatures (Reyes & Stanic, 1988; Randhawa, Beamer, & Lundberg., 1993;House,1993;House,1995;Wigfield&Eccles,2000;Maloney&Beilock,2012).Beyondthesefactors,wealso incorporated LMS online-usage data as course performance predictors, to capture studentengagement and disengagement with the online portion of the course (Cocea & Weibelzahl, 2009;Tempelaar,Rientes,&Giesbers,2015;Gaševićetal.,2015).Thisstudyrepresentsasteptowardusinglearning analytics in combination with cognitive and attitudinal variables to inform potentialrecommendation systems based on predictivemodels to facilitate individualized learning and bolsterstudentsuccess(Tempelaaretal.,2015).

1.1 Attitudinal and Cognitive Factors Related to Math Achievement

Thereisarichhistoryofexaminingindividualdifferencespredictorsofmathachievement.Forexample,recentlyastudyfoundthatlearning-relatedemotionswerestrongpredictorsofundergraduatestudent



examscoresinamathandstatisticscourse(Niculescuetal.,2015).Ingeneral,researchacrossthisareahas found that attitudinal variables are typically weakly to moderately correlated with mathachievement(e.g.,meta-analysesformathanxietyandmathachievementfoundr=–.34forgrades5–12,Hembree,1990; r = –.27 for grades4–12,Ma,1999).Critical attitudinal variables related tomaththathavebeenexamined inpast research includemathanxiety,mathconfidence,math interest, andmath importance (e.g., Wigfield & Eccles, 2000). Math anxiety is defined as a fear or an adverseemotional response to the idea of doing math. Math anxiety has been negatively linked to mathachievementforseveralpotentialreasons,includingthepossibilitythatitactsasaproxyforpoormathability, that it is a source of worry that compromises cognitive load in a math testing situation, orbecauseitleadstomathavoidance,whichresultsinpoormathachievement(Ashcraft,2002;Ashcraft&Krause,2007;Ganley&Vasilyeva,2014;Hembree,1990;Hoet al., 2000;Ma&Xu,2004;Maloney&Beilock,2012).Ontheotherhand,mathconfidence,whichweconceptualizeas includingwhatotherstermexpectationsof success, academic self-concept, andperceived competence, hasbeen related tobettermathoutcomes(House,1993;House,1995;Randhawaetal.,1993;Reyes&Stanic,1988).Mathinterest (Köller, Baumert,& Schnabel, 2001;Marsh, Trautwein, Lüdtke, Köller,&Baumert, 2005) alsopositivelyrelatestomathachievementbecauseitmayincreaseengagement,asstudentsaremotivatedtoengagewithmaterialinwhichtheyareinterested(Cocea&Weibelzahl,2009).Thisrelationisfoundacrosseducationallevelsandwithmultipledifferentmathcontentareas(Richardson,Abraham,&Bond,2012). Math importance, or math utility value, is another critical variable to consider, as researchsuggeststhatwhenstudentsdonotseethevalueinlearningmaththeyhavelowermathachievementandarelesslikelytosignupformathcourses(Meece,Wigfield,&Eccles,1990;Simpkins,Davis-Kean,&Eccles,2006).

Differencesinmathachievementhavealsobeenpositivelycorrelatedwithcognitiveskills,includingtheApproximateNumberSystem(ANS)andspatialabilities.TheANSisapartofthenon-symbolicnumbersystem, lessprecise than counting, that improveswithage, andallowsus to representnumbersnon-verballyinordertoquicklydistinguishbetweentwoquantities(Halberda&Feigenson,2008).Althoughnotwithout controversy, a reviewof the literature links theANS tomath achievement (r=0.20, 95%confidenceintervals=0.14,0.26;Chen&Li,2014).Anothercognitiveskillbelievedtobeassociatedwithmathachievementisspatialability,ortheabilitytomentallyrepresentandmanipulateobjectsinspace.Inparticular,mentalrotationskills,ortheabilitytorotateobjects inspace, isconsistentlyfoundtoberelated tomathachievement (r = .35–.38 for female students, r = –.03–.54 formale students; Casey,Nuttall,Pezaris,&Benbow,1995).Someexperimentalworkhasalsoshownthattraininginspatialskillscan produce enhancedmath test performance (Cheng &Mix, 2014) and improved grades in amathcourse(Sorby,Casey,Veurink,&Dulaney,2013).

1.2 Factors Related to Achievement in Online Courses

Looking specifically at the online learning environment, research has revealed that positive learningdispositionsbolster studentengagement (Tempelaar,Niculescu,Rientes,Giesbers,&Gijselaers, 2012;Tempelaaretal., 2015), and that studentengagementand frequencyofparticipation stronglypredict



student performance (Chen& Jang, 2010; Davies & Graff, 2005; de Barba, Kennedy, & Ainley, 2016;Tempelaar et al., 2015;Kizilcec, Piech,& Schneider, 2013;Morris, Finnegan,&Wu,2005) and coursecompletion (Milne et al., 2012; Macfadyen & Dawson, 2010; Morris et al., 2005) in online courses.Althoughtheseresultsarenotsurprising,theyareimportantbecausetheysuggestthatwemaybeableto use LMS data to identify students who are not engaged with course material. This affords theopportunity todo things,using theonlineplatform, tohelpengagestudents in thematerial.Anearlystudy exploring the value of LMS student engagement data was the Course Signals programimplemented at Purdue University. The study revealed that students who received regular feedbackabout their likelihood of successful course completion via an online colour-coded risk assessmentsystemhadhigherretentionratesthancontrols(Yukselturk&Bulut,2007).

StudentengagementwithanLMSclasscantakedifferentforms, including learner–content interactionorsocialinteraction(i.e.,learner–learnerorlearner–instructorinteraction;Fulford&Zhang,1993),andthe studies analyzing LMS data have found inconsistent results about which engagement method ismostrelatedtoachievement.Somestudiesfindthatlearner–contentinteractionismostimportantforstudent success. For example, Morris and colleagues (2005) linked learner–content interaction,measuredbybothfrequencyanddurationofcontentviewing,tostudentachievement,concludingthathigher performing students aremore likely to dedicate their time to viewing relevant content ratherthancreatingtheirownoriginalcontent.Similarly,RamosandYudko(2008)foundthatthetotalnumberofuserhitsonanLMSpositivelypredictedexamoutcomes,butdiscussionboardparticipation,includingnumberofpostscontributedandnumberofpostsread,didnot.Ontheotherhand,Gašević,Dawson,Rogers, andGašević (2016) found that time spent on assignmentswas actually negatively associatedwithachievement,suggestingthatperhapsstudentswhospendmoretimeonassignmentsspecificallymaybeusingtheirresourcesinefficientlyoraretryingtomakeupforalackofcomprehension(Gašević,etal.,2016).

Other work suggests that learner–learner interaction is the more important form of studentengagement for student achievement. For example,multiple studies in different STEM content areashavefoundapositiverelationbetweenthenumberofdiscussionpostscontributedbyastudent(Huon,Spehar, Adam, & Refkin, 2007; Macfadyen & Dawson, 2010), as well as the quality of these posts(Romero, Lopez, Luna, & Ventura, 2013) and their course performance, suggesting that successfulstudentsaremore likely touseonline resources to facilitate learning throughsocialengagementwithpeers. Conversely, students who infrequently participate in discussion forums and show patterns ofdisengagement are more likely to fail (Milne et al., 2012; Romero et al., 2013). The strength of theassociation ranges from weak (Lauría, Baron, Devireddy, Sundararaju, & Jayaprakash, 2012; Chanlin,2012)tostrong(Macfadyen&Dawson,2010),andmereparticipationindiscussiondoesnotsignificantlydistinguish students with the highest grades from average performers (Davies & Graff, 2005).Furthermore, discussion participationmattersmost for students in danger of failing (Davies & Graff,2005), most likely because the feeling of being supported makes struggling students more likely topersist(Romeroetal.,2013).



Therise incomputer-assistedclassesrepresentsapushtoward learner-controlledand learner-centredlearningenvironments,inwhichstudentsarerequiredtomakedecisionsabouthowandtowhatextenttheyengagewithclassmaterials(Lust,Elen,&Clarebout,2013),butamajorityofstudentsfailtoutilizeresources effectively (Lust, Juarez-Collazo, Elen, & Clarebout, 2012; Lust, Vandewaetere, Cuelemans,Elen, & Clarebout, 2011). Evidence shows that the degree to which students engage in the learningprocess through effective online tool use is associated with their learning outcomes, even whenaccountingfordifferences incognitiveability(Shute&Gluck,1996).Thus,pinpointingthefactorsthatunderlie student engagement with learning materials is essential for understanding why studentssucceedorfailinanonlinecourse(Tempelaaretal.,2012).

Time management is defined as the ability to prioritize time in order to fulfill learning demands,completetasks,andmodifyplanstoaccommodateanychangesintimeordemands(Jo,Park,Yoon,&Sung,2016).Timemanagementhasbeenshowntobeasignificantpredictorofonlineachievement(Joet al., 2016; Kwon, 2009; Choi & Choi, 2012), although the association may be weak (r=.14, p=.00;Broadbent&Poon,2015).Otherstudieshavefounddirecteffectsoftimemanagementonachievement(Jo & Kim, 2013; Loomis, 2000). Conversely, poor time management, or procrastination, negativelyinfluencesonlineparticipation(Michinov,Brunot,LeBohec,Juhel,&Delaval,2011;Balduf,2009),withthe students least likely to participate also demonstrating the poorest performance (Michinov et al.,2011).

Pasteducationalresearchhasconsistentlyshownthatthebestpredictorofperformanceisperformanceitself and the learning analytics field is no different (Tempelaar et al., 2015). In studies that includeundergraduatestakingLMS-supplementedcourses,thestrongestpredictoroffinalexamgrades,whenaccounting for a variety of student dispositional and engagement variables,was grades on formativequizzes (Huon et al., 2007; Tempelaar et al., 2015). In addition, early quiz grades have the samepredictive power as later quiz grades, so these formative assessments can be used to createrecommendationsaswellas topredict finalexamgrades (Wolff,Zdrahal,Nikolov,&Pantucek,2013).Practicequizzesareoftenspecificallydesignedasexamreviewtools, so theassociationbetweenquizgradesandfinalexamgrades isnotsurprising.However,thefrequencyofuseofpracticequizzes,andnotjustthegrades,alsohaspredictivevalue(Huonetal.,2007).

Thus,severalpotentialfactorsrelatedtoLMSengagementhavepredictedsubsequentachievement,butthere are also some inconsistencies. These divergent results may be attributable to differences incontexts(i.e.,schoolsize,classtype),outcomevariables(i.e.,continuousgradesorpass/fail),choiceofcovariates,andpredictiontechniques,whichchallengesthegeneralizabilityofstudyresults(Gaševićetal.,2016).ThisstudywillusedominanceanalysistotrytofigureoutwhichfacetsofonlineLMSactivity,inconjunctionwithindividualstudentcognitiveandaffectivefactors,areimportantforstudentsuccessin a flipped Calculus II course. By focusing on predictors of performance, we hope to createrecommendations tailored specifically to future Calculus II classes, which will help overcome thechallengesofgeneralizability.



1.3 The Present Study

Althoughsubstantialliteraturesupportstheroleofthereviewedattitudinalandcognitivecorrelatesofmathachievement,aswellasLMSactivitydataasaproxyforstudentengagement,inpredictingstudentcourseachievement,ourstudy isuniquein itsvaried,multi-componentapproach.Asthepresentdatacome from a flipped course, we have both the attitudinal and cognitive factors and the studentengagement factors, andwe use these sources of information to predict student performance in theflipped Calculus II course. We build upon the “dispositional learning analytics infrastructure” thatproposes that learningattitudes canbegathered through self-report and combinedwith LMSactivitydata (Buckingham Shum & Deakin Crick, 2012). To this, we add cognitive factors to predict studentcourseperformance.

Wewillbuildonprevious studiesbyexploring studentachievement ina flippedclassroom, semester-long Calculus II course, combining various predictor variables often found to be related to studentattitudes(mathanxiety,mathconfidence,mathinterest,mathimportance)andcognitiveskills(mentalrotation, approximate number system), as well as student engagement with the online system(discussionforuminteraction,timeofsubmissionofworkshopassignments,timeofsubmissionofpeerreviews,quizattempts),inpredictingfinalgradesintheclass.Webeganthisprojectwiththeintentiontousethesedatatobuildarecommendationsystemintothecourseplatform,toprovidefeedbacktostudents about actions they can take to improve their likelihood of success in the class. As such,weweremost interestedinwhichfactorswerethebestatpredictingfinalgradessothatwecouldtargetthosefactors.Therefore,wewilluseamethodologicalapproachcalleddominanceanalysistorankordertherelativeimportanceoftheattitudinal,cognitive,andLMSactivitypredictorsonstudents’finalgradesinCalculusII.Thiswillallowustodeterminewhichpredictorsemergeasthemostimportant.

2 METHOD

2.1 Participants

Participantswere85studentswhocompletedaflippedCalculusIIcourseinSpring2014.About43%ofthe students were female. Approximately 87% of students were white, 7% were Black or AfricanAmerican,2%wereAsian,4%wereother,and17%wereHispanic/Latino.About51%ofstudentswerefirstyearstudents,22%weresophomores,21%werejuniors,and7%wereseniors.Thestudentswereonaverage19.75yearsold(SD=1.90,range=17.75–30.00).MoststudentsweremajorsinSTEMfieldsthatrequiredCalculusIItocompletethemajor.

In the course, students used the online course platform (WEPS)1 to watch lecture videos of coursecontentoutsideof class timeand thensolvedproblemswith theprofessorandother studentsduringclasstime.Eachcoursetopichadthreelecturevideosfilmedbythecourseinstructor,correspondingtothreedifficulty levels (gentle,normal, rigorous),andeachvideowasapproximately5to25minutes in

1 https://geom.mathstat.helsinki.fi/moodle/



length.All teachingcontentwasavailable to studentsatall times,althoughgraded itemshadspecifictimeframesofavailabilitybasedonduedates.Studentswerenotrequiredtowatchthevideosbeforeclass,butwerehighlyencouragedtodoso.

2.2 Measures

Measures for this studywere obtained frommultiple sources including the following: in-person datacollection(spatialskillsmeasureofmentalrotation);aQualtricssurvey(mathanxiety,mathconfidence,mathinterest,mathimportance,approximatenumbersystem);logdatafromtheonlinecoursesystem(timetodeadlineforonlineworkshops,timetodeadlineforgradingotherstudents’workshops,onlinequizattempts,activeforuminteraction,passiveforuminteraction);andclassrecords(coursegrade).Allattitudinal and cognitivemeasures used are open source and freely available, other than thementalrotationmeasure.2Alltheattitudinalandcognitivemeasureswerespecificallyincludedinthecourseforresearchpurposes.Participantscompletedinformedconsentproceduresandapencil-and-papermentalrotationtaskin-personatthestartofoneoftheirclassperiodsatthebeginningoftheSpring2014term.In total, our final n = 85, which represents the number of students who completed the course andreceivedafinalgrade,aswellasconsentedtobepartofthestudyandforustousetheironlinecoursedataandclassrecords.Ofthisn=85,onestudentdidnotcompletetheonlineQualtricssurvey.

2.2.1 Course gradeAstudent’sfinalcoursegrade(0–100)inCalculusIIwasusedastheoutcomevariable.Thisgradewasaweighted average of the scores on the workshops (30% of grade), scores on the two course exams(30%),andscoreonthefinalexam(40%).Quizgradeswerealsousedasbonuspoints(upto3%onthefinalgrade).

2.2.2 Math anxietyStudents’mathanxietywasmeasuredwith theMathAnxietyRatingScale–Revised (MARS–R;Plake&Parker,1982).Thisscalehas24itemsratedonafive-pointscaleinwhichstudentsareaskedtoindicatetheamountofanxietytheyfeelindifferentsituations(e.g., lookingthroughthepagesonamathtext)from“notatall”to“verymuch.”Theinternalconsistency(α)forthescalewas.95.

2.2.3 Math confidenceStudents’ math confidence was measured with the 12-item confidence subscale of the Fennema-ShermanMathAttitudesScales(Fennema&Sherman,1976). Itemswereratedonaseven-pointscalefromStronglydisagreetoStronglyagree(e.g.,IamsureIcoulddoadvancedworkinmathematics).Theinternalconsistency(α)forthescalewas.92.

2.2.4 Math interestWe measured math interest with four items adapted from Wigfield and Eccles (2000), from theEducational Longitudinal Study (Ingels et al., 2007), and from the Early Childhood Longitudinal Study

2 TheMentalRotationTestisfreebutmustberequestedfromDr.MichaelPeters.



(Tourangeau,Nord,Lê,Pollack,&Atkins-Burnett,2006). Itemswereratedonaseven-pointscalefromStrongly disagree to Strongly agree. Sample items are “I like math” and “I find working on mathassignmentstobeveryinteresting.”Theinternalconsistency(α)forthescalewas.91.

2.2.5 Math importanceWemeasured math importance with six items, two of which are adapted fromWigfield and Eccles(2000) and four ofwhich are researcher-developed (e.g.,What I learn inmath is useful). Itemswereratedonaseven-pointscalefromStronglydisagree toStronglyagree.The internalconsistency(α) forthescalewas.91.

2.2.6 Mental rotationTheMentalRotationTest(Vandenberg&Kuse,1978)wasadministeredtostudents.Thetestconsistsof24itemsdividedintotwoblocks.Eachitemincludesapictureofathree-dimensionalobjectpresentedon the left (i.e., the target object). On the right, there are four other pictures of three-dimensionalobjects.Twoofthemdepictobjectsidenticaltothetarget,onlypresentedfromadifferentperspective.Theother twodepict either amirror imageof the targetor anobjectwith slightlydifferent features.Students are asked to identify which two of the four objects are the same as the target. Using thestandard scoring for thismeasure, students received 1 point only in those cases when both of theirchoiceswerecorrect.Theyreceived0points foranyothertypeofresponse(e.g., if theyselectedonecorrectandoneincorrectchoice)tohelpaccountforguessing.Internalconsistency(Cronbach’salpha)forthemeasurewas.89.

2.2.7 Approximate number system (ANS) The Approximate Number System (ANS), or intuitive recognition of number, wasmeasured using anonlinetestonthePanamathwebsite3(Halberda,Mazzocco,&Feigenson,2008).Thistestmeasurestheability of an individual to non-verbally represent numbers, or understand and manipulate numericalquantities non-symbolically (Halberda et al., 2008; Halberda & Feigenson, 2008; Libertus & Brannon,2010). In the task, participants were shown brief displays (600 milliseconds) of intermixed blue andyellowdotswithfiveto20dotspercolour,andaskedtodetermineifthereweremoreblue(bypressingthe“b”key)oryellowdots(bypressingthe“y”key). Intotal,theparticipantsweregiven120trialsofvariousratiosofdotquantities(~5–7minutesoftestingtime),andaccuracyandresponsetimeforeachtrialwasrecorded.PanamaththencalculatestheparticipantWeberfraction(w-score),whichrepresentsthe smallest ratio that can accurately be discriminated by a given individual, and reports it in ahyperlinked .pdf. The participant was asked to provide the hyperlink, from which the w-score wasobtained. Due to the additional step of having to navigate to a differentwebsite and then copy thehyperlinkinQualtrics,13participantsdidnotreportw-scoredata,thusthereweren=71participantswithANSdata.Additionally,uponlookingattheinitialdata,4datapointsweredeemedtobeoutliers(w-scoresgreaterthan.85,wherethenexthighestscorewas.44),sothescoresweresettomissing,asscoresthisoutofrangearelikelyreflectiveoftheparticipantnotaccuratelycompletingthetask.

3 http://panamath.org/expt5_fsu/



2.2.8 Online forum interaction The online learning platform (WEPS) included forums inwhich students could ask, answer, and readquestions about the course material. We created two different scores from forum information. For“discussionforumposting,”wescoredthenumberoftimesstudentsactivelyinteractedwiththeforumbycountingeachtimetheywroteapost, includingwhentheywrotetheoriginalpostorrespondedtoanother student’s post. Upon looking at the initial data, 1 data pointwas deemed to be an extremeoutlier(correspondingto101posts,versusthenexthighestof36posts),sowesetthisscoretomissing.For“discussionforumviewing,”wealsoscoredthenumberoftimesstudentspassivelyinteractedwiththeforumbycountingthenumberoftimestheyviewed(butdidnotcontribute)contentontheforum.

2.2.9 Time to deadline for online workshopsAs part of the course, students had to complete and submit 13 workshops, which were essentialhomework problem sets, over the course of the semester.We created a score that represented theaveragenumberofhoursremainingbeforethedeadlineatthetimetheysubmittedtheirassignment.

2.2.10 Time to deadline for grading other students’ workshops Students also had to grade the workshop assignments of five other students in their class for eachworkshop. They were given a specific amount of time to do this and again, we coded the averagenumber of hours remaining before the deadline at the time they submitted the graded workshopassignments.Theiraccuracyingradingmadeup20%oftheirgradeforeachworkshop.

2.2.11 Online quiz attemptsStudentstooksevenonlinequizzesoverthecourseofthesemester,andtheywereallowedtotakethequizzesanunlimitednumberoftimesuntiltheywerehappywiththeirgrade.Becauseofthis,usingtheirgrade on the quiz was less meaningful (and was also included in the calculation of final grade), soinsteadweexaminedthemeantotalnumberofattemptsacrossthesevenquizzes.Uponlookingattheinitial data, 1 data point was deemed to be an extreme outlier (corresponding to a mean of 8.34submissions per quiz, versus the next highest of 3.15 submissions per quiz), so we set this score tomissing.

3 RESULTS

DescriptivestatisticsarepresentedinTable1.Onevariablewaspositivelyskewed(skew>2.00,withnoobviousoutliers;seeTabachnick&Fidell,2013),andthereforethisvariablewaslog-transformedforusein all remaining analysis (skew was 0 after log-transformation). Following calculation of descriptivestatistics,ProcMI (i.e.,multiple imputation;Rubin,1987)wasused to fill in themissingdata foreachpredictor.Weelected todo this stepbecausedominanceanalysis, themainanalysis,dropscases list-wisewhenthereismissingdata.UsingProcMI,1000imputeddatasetswerecalculatedwithplausiblevaluesforeachmissingdatapoint,afterwhichwecalculatedameanscoreofall1000datapointsthatbecamethepredictorvalue.Thismeant thatevery individualhadacompletedataset (i.e.,nomissingdata)inallsubsequentanalyses.



Table1:DescriptiveStatisticsVariable na M SD

Min.

Max.

Possiblerange skew

Mathanxiety 84 1.97 0.63 1.00 3.78 1–5 0.53Mathconfidence 84 5.56 0.94 3.00 7.00 1–7 –0.72Mathinterest 84 5.14 1.30 1.00 7.00 1–7 –0.97Mathimportance 84 5.53 1.09 1.83 7.00 1–7 –1.03Mentalrotation 85 12.66 5.12 1.00 23.00 0–24 0.04ANS 67 0.20 0.06 0.13 0.44 0–¥ 1.48Discussionforumposting 77 8.75 8.86 0.00 36.00 0–¥ 1.47Discussionforumviewing 78 193.60 196.53 19.00 978.00 0–¥ 2.21/0.00bWorkshopsubmissions(hrs) 78 164.00 10.84 138.12 197.83 0–varying 0.64Gradingpeerworkshopsubmissions(hrs)

78 69.44 21.69 23.27 128.04 0–varying 0.84

Quizattempts 77 1.63 0.45 1.00 3.15 0–α 1.42Finalgrade 85 84.38 11.84 33.28 100.00 0–100 –1.33anisreportedastotalcasesavailableforeachvariable,outofatotalpossiblen=85.bskewisreportedasbeforelog-transformation/afterlog-transformation.

Table2:PearsonCorrelationsamongAlltheMeasures 1 2 3 4 5 6 7 8 9 10 111.Mathanxiety 1

2.Mathconfidence –.43* 1 3.Mathinterest –.24* .66* 1 4.Mathimportance .17 .58* .65* 1 5.Mentalrotation –.05 –.05 .10 .07 1 6.ANS –.13 –.03 .21 .07 .04 1 7.Discussionforumposting –.02 .15 .28* .08 –.06 –.21* 1

8.Discussionforumviewing .16 .03 .15 .04 –.10 –.28* .54* 1

9.Workshopsubmissions –.18 .03 .06 –.09 –.10 .09 .10 .11 1 10.Gradingpeerworkshops .02 .14 –.13 –.24* –.36* .01 .14 .12 .26

* 1

11.Quizattempts .14 .10 .01 –.11 –.25* –.19 –.05 .06 .17 .33* 112.Finalgrade .02 .24* .15 .19 –.03 –.21 .24* .19 .01 .19 .09*p<0.05.PearsoncorrelationsamongthemeasuresareinTable2.Forthemostpart,correlationswithfinalgradewerelowtomoderateinmagnitude,withonlythecorrelationsofmathconfidenceandactivediscussionforum activity with final grade being statistically significant. Sensitivity analysis conducted usingG*Power suggested that we were only powered to detect a statistically significant correlation co-efficientofr=.26andgreaterwithoursamplesizeof85,analpha-errorprobabilityof.05,andpower=.80.Therefore,weadvisethatthemagnitude,andnotstatisticalsignificance,beconsidered.Thereare



four correlation coefficients, for example, that are .19 or higher, but are not statistically significant(mathinterest,ANS,discussionforumviewing,gradingpeerworkshopsubmissions).

As a first step, we used all-subsets regression to reduce the total predictor variables that would beenteredintothedominanceanalysis(asdominanceanalysisiscomputationallyintensiveandlimitedtoa maximum of 10 predictor variables). All-subsets regression computes all possible R² values for allpossiblesetsizesofpredictors(i.e.,1predictor,2predictors,etc.)regressedontofinalgrade,andthenrank-orderstheobtainedR²inorderofhighesttolowestwithinsetsize(Miller,2002).OncethehighestR²valuesforeachsetsizewasobtained,weuseda“diminishingreturns”techniquetoavoidoverfittingthedominanceanalysis,mirroring amethod laidout in Speeceet al. (2010).Using this approach,wedeterminedwhen going from a set ofnpredictors to a set ofn+ 1 predictorswould not give us animportant increase inR²value for the additional set increase. Themodelswith 1 to all 11 predictorsincludedyieldedmaximumR²valuesof.06,.10,.15,.17.17,.17,.18,.18,.18,.18,and.18(respectively).Wedeterminedthatafter thesetsize four, therewerediminishingreturnsofR² increases, in that foreachsetsizeincreaseafterfour,therewasatmostonly1%morevarianceexplainedthanthepreviousset.Table3displaysthetop10models,basedontotalR²values,ofpossible4predictormodels.LookingatTable3,onlyonemodelaccountedforthehighestproportionofvarianceaccountedforinfinalgrade(17%ofthevariance),representingthepredictorsofmathimportance,ANS,discussionforumposting,andgradingpeerworkshopsubmission.Theseweremovedforwardintothedominanceanalysis.

Table3:Top10ModelsofPredictorsofCalculusIIFinalGradeforSetSizeFour

Dominance analysiswas used to rank order the predictors by contributive importance to final grade.DominanceanalysisusesbootstrappingtocomputetotalanduniqueR²forallthepossiblecombinationsoftheenteredpredictorsoftheoutcomevariable,herefinalgradeinCalculusII.Indominanceanalysis,a series of different regression models, called “subset models,” are run and used as a whole todetermine dominance order of the predictor variables. The total number of regressionmodels run isbasedonacombinatorialruleofprobability(Hays,1994).Aswehadfourpredictors,weran15differentmodels:foursinglepredictormodels,sixcombinationsoftwopredictormodels,fourmodelswiththree

Predictors R² AdjR²

Mathimportance,ANS,discussionforumposting,gradingpeerworkshopsubmission .17 .12Mathimportance,ANS,discussionforumviewing,gradingpeerworkshopsubmission .16 .11Mathinterest,mathimportance,ANS,gradingpeerworkshopsubmission .15 .11Mathconfidence,mathimportance,ANS,gradingpeerworkshopsubmission .15 .11Mathimportance,mentalrotation,ANS,gradingpeerworkshopsubmission .15 .11Mathanxiety,mathimportance,ANS,gradingpeerworkshopsubmission .15 .11Mathimportance,ANS,workshopsubmission,gradingpeerworkshopsubmission .15 .11Mathimportance,ANS,gradingpeerworkshopsubmission,quizattempts .15 .10Mathconfidence,ANS,discussionforumposting,gradingpeerworkshopsubmission .15 .10Mathinterest,ANS,discussionforumposting,gradingpeerworkshopsubmission .14 .10



predictors, and one model with four predictors. This was done using a macro in SAS 9.4 (Azen &Budescu,2003).

There are three types, or levels, of dominance: complete, conditional, and general (Azen& Budescu,2003; Budescu, 1993). In the most strict, complete dominance, a predictor variable is consideredcompletelydominantoveradifferentpredictorvariable if it contributesadditionaluniquevariance tofinal grade in both the pairwise comparison, as well as when all other possible combinations ofpredictorsareaddedintothemodel.Thatis,apredictoriscompletelydominantoveranotherpredictorinitsassociationwithfinalgradewhenitpredictsuniquevarianceinfinalgradewhencompetedagainstallotherpredictorsinallpossiblesubsetmodels.Conditionaldominanceisaweakerformofdominancefromcomplete,inthatapredictorvariableisconsideredconditionallydominantoveranotherpredictorvariablewhenitcontributesuniquevariancetofinalgradewithineachmodelsize(i.e.,averagingacrossall the subsetmodels with two predictor variables in themultiple regression). General dominance isweaker still, in that general dominance is achieved when a predictor variable’s unique varianceaccountedfor isgreaterthananotherpredictorvariable’s,averagedacrossallpossiblesubsetmodels.Achievingcompletedominanceisideal,butoftenundeterminable(i.e.,totaluniquepredictionaboveallothervariablesisdifficulttoachieve),sotheweakerformsarethenusedtorankorderpredictors(Azen&Budescu,2003).Ifavariableshowscompletedominance,itwouldthenlogicallyalsoshowconditionalandgeneraldominance,andsoon.

Table 4 presents the total and unique R² values for each variable, or variable combination, in eachsubsetmodel.Thesubsetmodelswithonepredictoraccountedfor4–6%ofthevarianceinfinalgrade,the subsetmodelswith twopredictors accounted for 8–9%of the variance in final grade, the subsetmodelwiththreepredictorsaccountedfor11–15%ofthevarianceinfinalgrade,andthesubsetmodelwithallfourpredictorsaccountedfor17%ofthevarianceinfinalgrade(seeR²columninTable4).ThecolumnsinthefarrightofTable4representtheadditionaluniquevarianceeachvariablewouldaccountfor in the presence of the other predictors in the model, if they were added into the model. Forexample,thesubsetmodelwithonlymath importanceaccountedfor4%ofthevariance inCalculus IIfinal grade. After controlling for the variance attributable to math importance, ANS would haveaccounted for an additional 5% of variance in final grade if it were added as a second predictor,discussionforumpostingwouldhaveaccountedforanadditional5%ofvarianceinfinalgradeifitwereaddedasasecondpredictor,andtimeofsubmissionofgradingpeerassignmentswouldhaveaccountedforanadditional6%ofthevarianceinfinalgradeifitwereaddedasasecondpredictor.

Ineachofthebootstrappedsamples,thedominancevalueisDijobtainedforagivenpairofpredictors,XiandXj,whichcorrespondstooneofthreevalues:1, ifXidominatesXj;0, ifXjdominatesXi;and0.5, ifdominancecannotbeestablishedbetweenthetwopredictors.Asweused1000bootstrappedsamples,Dij_meanrepresentstheexpecteddominancelevelinthepopulationofXioverXj.Finally,asdominanceanalysis uses this bootstrapping, it does not employ the more traditional approach used in multipleregressionofproducingp-values todetermine“significance.” Instead, thecloserDij_mean is to1or0,thestrongerthecaseforcleardirectionaldominance,andthecloserDij_meanisto0.5,thestrongerthecasefor indeterminatedominance.Additionally, theseanalysesalsoproducea“reproducibility”value,



which represents the proportion of bootstrapped samples that the given dominance pattern isproduced.Thecloserthereproducibilityvalue isto1.00,thegreatertherobustnessofthedominanceresults.

Note.Thefarfourrightcolumnsrepresentthevariancethateachgivenvariablewouldcontributetothemodellisted,ifitwereaddedtothemodelasanadditionalvariable.Table5presentstheDijresults,aswellasDij_mean(andcorrespondingstandarderror),Pij(proportionofsampleswereDij=1.0),Pji(proportionofsampleswereDij=0.0),Pijno(proportionofsampleswereDij=0.5), and the reproducibility value. As described inAzen andBudescu (2003), complete dominance isestablished if the additional contribution of a given predictor is higher than another predictor in allsubsetmodels where the two predictor variables are competing head-to-head. Complete dominance

Table4:R²ContributionsacrossSubsetModelsofPredictorsofCalculusIIFinalGrade UniqueContributionof

PredictorSubsetmodel R² 1. 2. 3. 4.Nullmodelwithzeropredictors — .04 .04 .06 .04Modelswith1Predictor

1.Mathimportance .04 — .05 .05 .062.ANS .04 .04 — .04 .043.Discussionforumposting .06 .03 .03 — .024.Gradingpeerworkshopsubmissions .04 .06 .05 .05 —1predictoraverage .04 .04 .05 .04

Modelswith2Predictors Mathimportance–ANS .09 — — .03 .06Mathimportance–Discussionforumposting .09 — .03 — .04Mathimportance–Gradingpeerworkshopsubmissions .09 — .05 .04 —ANS–Discussionforumposting .08 .03 — — .03ANS–Gradingpeerworkshopsubmissions .08 .07 — .03 —Discussionforumposting–Gradingpeerworkshopsubmissions .08 .05 .03 — —2predictoraverage .05 .04 .03 .04

Modelswith3Predictors Mathimportance–ANS–Discussionforumposting .12 — — — .05Mathimportance–ANS–Gradingpeerworkshopsubmissions .15 — — .02 —Mathimportance–Discussionforumposting–Gradingpeerworkshopsubmissions

.13 — .04 — —

ANS–Discussionforumposting–Gradingpeerworkshopsubmissions

.11 .06 — — —

3predictoraverage .06 .04 .02 .05Modelswithall4predictors

Mathimportance–ANS–Discussionforumposting–Gradingpeerworkshopsubmissions

.17 — — — —

Overallaverage .05 .04 .04 .04



wasestablishedformathimportanceovergradingpeerworkshopsubmissions(asseenbytheDij=1.0,Dij_mean=.55),buttherewasnotenoughevidenceforanyotherpair-wisecompletedominancetobeestablished(asseenbytheDij=0.5).Nopairspassedthetestforconditionaldominance,sothereforetheremainingpairswererankedbythegeneraldominancecriteria(which,bydefinition,shouldalwaysfinddominancepatternsasitsimplyrankordersthetotalamountofvarianceaccountedfor,averagedacrossallsubsetmodels,fromhighesttolowest).Theresultsindicatethefollowinggeneraldominancepattern: math importance > grading peer workshop submissions > ANS > discussion forum posting.Reproducibilityvaluestoppedoutat.66,suggestingthatsomecautionshouldbegivenwhenacceptingthegeneraldominancepattern,particularlyconsideringAzenandBudescu’s(2003)recommendationtogivemoreweighttothemuchstrictercompletedominance.

4 DISCUSSION

Asubstantialbodyofworkhasidentifiedparticularstudentattitudinalandcognitivefactorsrelatedtomathperformance.Moreover,researchthatexaminesstudentengagementvariables,usingLMSactivitydata fromonline courseshasgrown.We sought to combine these researchareas,determiningwhichpredictors,outofstudentattitudinalandcognitivefactors,andindicatorsofstudentengagementintheonlinecourse,emergeas themost important inpredicting final grades ina flippedCalculus II course.The goal of this workwas to build a list of factors that predict performance in Calculus II, so that a(future) recommendationsystemcanbebuilt toprovide feedback to studentsaboutactions theycantaketoimprovetheirlikelihoodofsuccessintheclass.

Somemightfind itsomewhatsurprisingthatonlytwoofourpredictorswerestatisticallysignificant intheircorrelationwithfinalgrade.Thisreactioniscertainlywarranted,butoursmallsamplesizeof85islimiting our ability to obtain statistically significant results. Seven out of the 11 predictors hadcorrelationcoefficientsthatwereover.15,andmanyofthesehadp-values=.05–.08.This,coupledwiththe possibility ofmulti-collinearity, is why we elected to use all-subsets regression, which takes intoaccountvarianceexplainedratherthanstatisticalsignificance.Ourgoalwastoexplainasmuchvarianceaspossibleinfinalgrade,sowefocusedmoreoneffectsizesthanstatisticalsignificance.

The all-subsets regression indicated that believingmath is important, having a stronger approximatenumber system (ANS), contributing more discussion forum posts, and submitting peer grading ofworkshops earlier together represent the best combination of predictors of final grade while alsobalancingmodelcomplexitywhenaddingadditionalpredictors.Thesevariableswerethenchosentobeusedinthedominanceanalysis.Theresultsofthedominanceanalysiswerenotoverlycompellingforaspecific ordering of importance of the four final predictor variables in predicting final grade. Mathimportancewasestablishedascompletelydominantovertimeofsubmissionofgradingpeerworkshopsubmissions.Otherwise,dominancewasonlyestablishedusingtheleaststrictform,generaldominance.Theseresultssuggestedthatmathimportancewasmoreimportantthantimeofsubmissionofgradingpeerworkshopsubmissions,whichwasmore importancethanANSability,whichwasmore importantthantheamountanindividualdiddiscussionforumposting.Overall,wetaketheseresultstomeanthat,ingeneral,thesevariablesareeachsimilarlypredictive.



Table5.ResultsfromtheDominanceAnalysisofAllPairwisePredictorsacrosstheThreeLevelsofDominance,PredictingCalculusIIFinalGrade

Xi Xj Dij Dij_mean SE(Dij) Pij Pji Pijno ReproducibilityComplete

Mathimportance ANS 0.5 .63 .42 .53 .26 .22 .22Mathimportance Discussionforumposting 0.5 .56 .41 .41 .29 .30 .30Mathimportance Gradingpeerworkshopsubmissions 1.0* .55 .47 .50 .40 .10 .50ANS Discussionforumposting 0.5 .43 .44 .32 .46 .23 .23ANS Gradingpeerworkshopsubmissions 0.5 .42 .43 .30 .47 .23 .23Discussionforumposting Gradingpeerworkshopsubmissions 0.5 .49 .39 .30 .32 .39 .39

Conditional Mathimportance ANS 0.5 .64 .43 .55 .27 .18 .18Mathimportance Discussionforumposting 0.5 .56 .42 .41 .29 .30 .30Mathimportance Gradingpeerworkshopsubmissions 1.0 .55 .48 .51 .41 .08 .05ANS Discussionforumposting 0.5 .43 .44 .33 .46 .21 .21ANS Gradingpeerworkshopsubmissions 0.5 .41 .44 .31 .50 .19 .19Discussionforumposting Gradingpeerworkshopsubmissions 0.5 .49 .40 .30 .33 .38 .38

General Mathimportance ANS 1.0* .66 .48 .66 .34 .00 .66Mathimportance Discussionforumposting 1.0* .55 .50 .55 .45 .00 .55Mathimportance Gradingpeerworkshopsubmissions 1.0 .55 .50 .55 .46 .00 .55ANS Discussionforumposting 1.0* .42 .49 .42 .58 .00 .42ANS Gradingpeerworkshopsubmissions 0.0* .40 .49 .40 .60 .00 .60Discussionforumposting Gradingpeerworkshopsubmissions 0.0* .49 .50 .49 .51 .00 .51

Note:iandj=variablesthatarecompeting;Dij_mean=averagenumberoftimesvariableidominatedvariablejoverallbootstrapsamples;Pij=proportionofbootstrapsamplesinwhichidominatedj;Pji=proportionofbootstrapsamplesinwhichjdominatedi;Pijno=proportionofbootstrapsamplesinwhichnodominancewasestablished.Reproducibilityistheproportionofbootstrapsamplesthatreplicatedthereportedeffect.∗indicatesthehighestlevelofdominanceachievedandimpliesallsubsequentlevelsofdominancearealsoachieved.



Thepredictorvariableofmathimportanceisfromascalemeasuringthoughtsabouttheusefulnessofstudyingmath. It is interesting to consider that students in Calculus II have already chosen a collegecoursepaththatisSTEMrelated,asthiscourseisonlyrequiredofcertainmajors.Thus,thesestudentsmustalreadybelieveat some level thatmath is important.Yet, individualdifferences in theextent towhich students believe math is important is associated with course performance. Perhaps mathmajors/minorsbelievethatmathismoreusefulthanbiologymajors,andpossiblymathmajors/minorsare themselves better at calculus than biologymajors based solely on the fact they have chosen topursueadegreein it.Or it ispossiblethatstudentsabletomoreclearlyseetheconnectionsbetweenwhattheyarelearninginCalculusIIandtheirchosenfielddobetter(i.e.,utilityvalue;Wigfield&Eccles,2000).Wewere not able to disentangle this nuance here, although this resultmight lend support toremindingstudentsinCalculusIIabouttheutilityoftheclassinachievingtheircareergoals.

ANSabilityisameasureofnon-symbolicnumerosity,oranintuitiverecognitionofnumber.Researcherssuggest that even as early as infancy, individuals develop the ability to make approximations anddiscriminate between largenon-symbolic values (e.g., ten versus eight dots, objects, syllables, andorshapes;e.g.,Halberdaetal.,2008).WefoundthatstudentswithanabilitytodifferentiatesmallersetsizesquicklydidbetterintheCalculusIIclass.TheANSisthoughttobethefoundationonwhichmathabilitybuilds(Verguts&Fias,2004),althoughsomeworkhasfoundnosignificantrelationbetweentheANSandfuturemathperformance(e.g.,deSmedt,Noël,Gilmore,&Ansari,2013).Ameta-analysishasfoundthattheredoesappeartobeasmallbutstablerelationbetweentheANSandmathperformance(Chen&Li,2014)butthisrelationappearstobenon-linear(Bonny&Lourenco,2013;Purpura&Logan,2015)with theANSmorerelated toearlier skills thanwith laterskills (Chu,vanMarle,&Geary,2015;Libertus,Feigenson,&Halberda,2013).Given thismixed literatureon the roleof theANS incomplexmathperformance,we found it surprising that theANSwasoneof themost important predictors ofperformanceinCalculusII.

We also found it surprising that mental rotation was not an important predictor (or even a strongcorrelateof)finalgradeinCalculusII.Spatialskillsareundeniablyimportantinmathperformance.ThebestwecangatheristhatbythetimestudentsareenrolledinCalculusII,spatialskillsthemselvesarenolonger an important predictor of performance, as these students likely, at the very least, have goodenough spatial skills tohavegotten that far inmath.Theaverage scoreof this sampleon thementalrotationtaskseemstobehigherthanageneralundergraduatesamplefromthesameschoolwehavecollectedthesamemeasurewith(approximately2itemscorrectmore),butatthispointthisideaisjustconjecture. We also remind the reader that spatial skills are not necessarily specifically needed foransweringtheproblemsinCalculusII(comparedtosayCalculusIIIorothermathcourses).Someoftheunits in this Calculus II class did requiremore overt spatial skills than others, andwe anticipate thatperformanceonthoseunitsmightbemorerelatedtospatialskillsthanoverallgrade.

Beyondtheattitudinalandcognitivepredictors,wefoundthattwooftheonlineengagementvariableswere important predictors of Calculus II performance. The total number of posts that a studentcontributedtotheonlinecoursediscussionboardwaspositivelyrelatedtotheirfinalcoursegrade.This



variableiscoarse,inthatwearenotabletosayifthestudentwasgeneratingtherootdiscussionpost(e.g.,askingthequestion),oriftheywereansweringotherstudent’squestions/posts.Therefore,wearenot sure if this variable represented engagement in the course, or if studentswho posted questionswereabletoreceivemorehelpthatledtoahighergrade,oranyothernumberofpossibleexplanations.Interestingly,thisfindingreplicatespreviousworkthatalsoindicatedthatthetotalnumberofpoststoadiscussionboardwasassociatedwithsuccessinanonlinebiologycourse(Macfadyen&Dawson,2010).Somehavesuggestedthatinteractingwiththecourseinthisengagedwaymaydeepencomprehensionof the course content (Evans & Sabry, 2003), a conclusion that fits with the finding here that morediscussionboardpostswasassociatedwithhighergrades.

We also found that students who submitted their grading of peer workshop assignments earlier didbetter in the course.Onemight suggest that this variable could be thought of as a “procrastination”variable.Procrastinationingeneralisthoughttobenegativelyassociatedwithcourseperformance,andthis might explain our finding (e.g., Tice & Baumeister, 1997). Other studies have found that timemanagement inonlinecoursespredictsachievement (Joetal.,2016;Kwon,2009;Choi&Choi,2012).Alternatively,thisvariablecouldsimplyreflectthatstudentswhowerestrugglinginthecoursefounditdifficult to grade these assignments (i.e., it’s hard to determine if something is right orwrong if youyourself are unsure of the right answer), and therefore took longer to submit it. We are unable todisentanglethesepossibleexplanationswiththecurrentlyavailabledata.

Forbothof theonlineengagementvariablesthatwerefoundtobe important,weareconjecturingatbestastowhatthesevariablesrepresent.Butwecansaythatsimplytrackingauserthroughanonlinecourse, here with two variables, predicted 8% of the total variance in final grades. To put this inperspectivewithbroadereducationalresearch,thevariancein individualreadingperformancedirectlyassociatedwith a child’s teacher has been suggested to be similar inmagnitude (Byrne et al., 2010).Althoughsmall,this8%ismeaningfulinaneducationalcontext.

4.1 Limitations

There are three points wewould like to caution readers about. First, certainly any of the predictorslistedinTable3arelikelyinterchangeablewiththefourweselectedfromthemostpredictivemodel,asthereisnostatisticaltestofthedifferencebetweenthecorrelationsofthepredictorvariablesincludedinto the dominance analysis and those that were not (Schatschneider, Fletcher, Francis, Carlson, &Foorman, 2004). In particular, math importance is likely interchangeable with some of the otherattitudinalpredictors,includingmathinterest,andmathconfidence—variablesthatwerefairlyhighlycorrelatedwithmathimportance.Wetakefromthisthatincludinganattitudinalvariableisimportantinpredicting end of course grades, and is likely useful for readers interested in using studentcharacteristicstoincorporatingpredictivesystemsintheironlinecourses(e.g.,Yukselturk&Bulut,2007)to include at least one rather thannone. These attitudinal surveys are easy to administer online andtake5–10minutesatthemost,andcontributeanon-trivialamountofvarianceinpredictingfinalgrade.Interestingly, previous work looking at predictors of success in math classes have found attitudinal



predictors to be important (e.g., math confidence; House, 1995), supporting their role especially inpredictingstudentperformanceinonlinemathclasses.

Second,thedominanceanalysiswasnotconvincingforaclearlyimportantdifferenceinstrengthofthepredictors,andsubsequentlywebelievethatallfourpredictorsareimportantforthemodelpredictingfinalgrade.Readersshouldfocusonhowacombinationoffourpredictors,acrossattitudinal,cognitive,andonlinestudentengagementvariables,accountedfor17%ofthevarianceinfinalgradeinCalculusII.Althoughfarfrom100%,wefeelthisamountisimpressivegiventhatnoneofthepredictorsareobviousindicatorsofperformance inCalculus II.Certainly,mathperformancewouldbeadominantpredictor,including previous grades. We sought to test measurable student characteristic variables that couldfeasibly be added into prediction software attached to online courses. With that in mind, testing astudent’sANSabilityisactuallynotasfeasibleasareadermightwant,giventhatthetaskwouldeitherneed tobeprogrammed into the courseplatform,oruserswouldbe required tonavigate to a third-partywebsiteandharvesttheirresponse(i.e.,ourmethod).Wenoteourdatashowsthatwepredicted13%ofthevariance instudentfinalgradewithout includingANS,anoptionthatmightbeconsidered.Equallyimportant,notallinstructorsareabletoharvestLMSactivitydatabecauseofprivacyconcerns;therefore,thefullrangeofstudentengagementvariablesmightnotbepossible.Ourdatashowsthatwepredict12%ofthevarianceinstudentfinalgradewithoutincludinganyoftheLMSactivitydata.Thoughthesevariablesalonedonotgivethefullpicture,somethingistobegainedevenwiththis incompleteinformation.

Finally,itisalsoimportanttopointoutthatthefourpredictorswedeterminedtobemostimportantinourdatamightnotbeequally important inothersimilardataorforothercourses.Theseanalysesarefundamentally sample specific. Therefore, we reiterate that acknowledging that student attitudes,cognitiveperformance,andonlineengagementvariablesareallimportanttoconsiderwhenpredictinggradeperformanceandshouldbeconsideredtogether,whenfeasible,orethicallypossible.

4.2 Conclusion

In conclusion, we sought to predict student final grades in a Calculus II from a battery consisting ofattitudinal,cognitive,andonlinestudentengagementvariables.Wefoundthatamixofvariablesacrossallthreecategoriesofvariablespredictedanon-trivialproportionofvarianceinfinalgrade.Theaimofthisworkwastodeterminewhicharethemostimportantpredictorsofstudentgrade,withtheendgoalofbuildingarecommendationsystemthatcouldbe implementedtohelpstudents inthistraditionallydifficult class. The methods used here could be used for any class, with the intention to determinestudentperformanceearly,andpotentiallyallowaninstructortoidentifystudentswhomayneedmoreintensivehelpearlierinthesemesterwheninterventioncanbemoreeffective.



ACKNOWLEDGEMENTS

ThisprojectwasmadepossiblebythetirelesseffortsofDr.MikaSeppälä,whodiedbeforehecouldseethesuccessfuloutcomeofhiswork.Mikawasapioneer inonline teaching,andwaspassionate inhiseffortstomaketheundergraduatemathcurriculummoreaccessibletoallstudents.WethankDr.OlgaCaprottiandYahyaAlmalkifortheireffortsincontinuingMika’swork,includingtheirimportantworkinharvestingthestudentengagementvariablesfromtheWEPSsystemforusafterMikadiedsothatthisprojectcouldcontinue.

This material is based upon work supported by the National Science Foundation under Grant No.1338509and1450501.

REFERENCES

Ashcraft, M. H. (2002). Math anxiety: Personal, educational, and cognitive consequences. CurrentDirections in Psychological Science, 11(5), 181–185. https://dx.doi.org/10.1111/1467-8721.00196

Ashcraft, M. H., & Krause, J. A. (2007). Working memory, math performance, and math anxiety.PsychonomicBulletin&Review,14,243–248.http://dx.doi.org/10.3758/BF03194059

Azen,R.,&Budescu,D.V.(2003).Thedominanceanalysisapproachforcomparingpredictorsinmultipleregression.PsychologicalMethods,2,129–148.http://dx.doi.org/10.1037/1082-989X.8.2.129

Balduf,M.(2009).Underachievementamongcollegestudents.JournalofAdvancedAcademics,20,274–294.https://dx.doi.org/10.1177/1932202X0902000204

Bienkowski,M., Feng,M., &Means, B. (2012). Enhancing teaching and learning through educationaldata mining and learning analytics: An issue brief.US Department of Education, Office ofEducationalTechnology(pp.1–57).

Bonny,J.W.,&Lourenco,S.F. (2013).Theapproximatenumbersystemand itsrelationtoearlymathachievement: Evidence from the preschool years. Journal of Experimental Child Psychology,114(3),375–388.http://dx.doi.org/10.1016/j.jecp.2012.09.015

Broadbent,J.,&Poon,W.L.(2015).Self-regulatedlearningstrategies&academicachievementinonlinehigher education learning environments: A systematic review. Internet and Higher Education,27,1–13.https://dx.doi.org/10.1016/j.iheduc.2015.04.007

BuckinghamShum,S.,&DeakinCrick,R. (2012). Learningdispositionsand transferablecompetencies:Pedagogy,modellingandlearninganalytics.Proceedingsofthe2nd InternationalConferenceonLearningAnalyticsandKnowledge (LAKʼ12),29April–2May2012,Vancouver,BC,Canada(pp.92–101).NewYork:ACM.http://dx.doi.org/10.1145/2330601.2330629

Budescu,D.V. (1993).Dominanceanalysis:Anewapproach to theproblemof relative importanceofpredictors in multiple regression. Psychological Bulletin, 114(3), 542–551.http://dx.doi.org/10.1037/0033-2909.114.3.542



Byrne,B.,Coventry,W.L.,Olson,R.K.,Wadsworth,S.J.,Samuelsson,S.,Petrill,S.A.,Willcutt,E.G.,&Corley, R. (2010). “Teacher effects” in early literacy development: Evidence from a study oftwins.JournalofEducationalPsychology,102,37–42.http://dx.doi.org/10.1037/a0017288

Casey,M.B.,Nuttall,R.,Pezaris,E.,&Benbow,C.P. (1995).The influenceofspatialabilityongenderdifferencesinmathematicscollegeentrancetestscoresacrossdiversesamples.DevelopmentalPsychology,31(4),697–705.http://dx.doi.org/10.1037/0012-1649.31.4.697

Chanlin, L.-J. (2012). Learning strategies in web-supported collaborative project. Innovations inEducation and Teaching International, 49(3), 319–331.http://dx.doi.org/10.1080/14703297.2012.703016

Chen, K. C.,& Jang, S. J. (2010).Motivation in online learning: Testing amodel of self-determinationtheory. Computers in Human Behavior, 26(4), 741–752.https://dx.doi.org/10.1016/j.chb.2010.01.011

Chen,Q.,&Li,J. (2014).Associationbetweenindividualdifferenceinnon-symbolicnumberacuityandmath performance: A meta-analysis. Acta Psychologica, 148, 163–172.https://doi.org/10.1016/j.actpsy.2014.01.016

Cheng, Y. L., &Mix, K. S. (2014). Spatial training improves children’s mathematics ability. Journal ofCognitiveDevelopment,15(1),2–11.http://dx.doi.org/10.1080/15248372.2012.725186

Choi,J.I.,&Choi,J.S.(2012).Theeffectsoflearningplansandtimemanagementstrategiesoncollegestudents’ self-regulated learning and academic achievement in e-learning. Journal ofEducationalStudies,43(4),221–244.

Chu, F. W., vanMarle, K., & Geary, D. C. (2015). Early numerical foundations of young children’smathematical development. Journal of Experimental Child Psychology, 132, 205–212.http://dx.doi.org/10.1016/j.jecp.2015.01.006

Cocea, M., & Weibelzahl, S. (2009). Log file analysis for disengagement detection in e-Learningenvironments.User Modeling and User-Adapted Interaction,19(4), 341–385.http://dx.doi.org/10.1007/s11257-009-9065-5

Davies,J.,&Graff,M.(2005).Performanceine-learning:Onlineparticipationandstudentgrades.BritishJournal of Educational Technology, 36(4), 657–663. http://dx.doi.org/10.1111/j.1467-8535.2005.00542.x

deBarba,P.G.,Kennedy,G.E.,&Ainley,M.D.(2016).Theroleofstudents’motivationandparticipationin predicting performance in a MOOC. Journal of Computer Assisted Learning, 32, 218–231.http://dx.doi.org/10.1111/jcal.12130

deSmedt,B.,Noël,M.-P.,Gilmore,C.,&Ansari,D.(2013).Howdosymbolicandnon-symbolicnumericalmagnitudeprocessingskills relate to individualdifferences inchildren’smathematical skills?Areviewofevidencefrombrainandbehavior.TrendsinNeuroscienceandEducation,2(2),48–55.

Evans,C.,&Sabry,K. (2003). Evaluationof the interactivityofweb-based learning systems:Principlesand process. Innovations in Education and Teaching International, 40, 89–99.http://dx.doi.org/10.1080/1355800032000038787

Fennema, E., & Sherman, J. A. (1976). Fennema-Sherman mathematics attitudes scales: Instrumentsdesigned to measure attitudes toward the learning of mathematics by females and males.



Journal for Research in Mathematics Education, 7(5), 324–326.http://dx.doi.org/10.2307/748467

Fulford, C. P., & Zhang, S. (1993). Perceptions of interaction: The critical predictor in distanceeducation.American Journal of Distance Education,7(3), 8–21.http://dx.doi.org/10.1080/08923649309526830

Garrison,D.R.,&Kanuka,H.(2004).Blendedlearning:Uncoveringitstransformativepotentialinhighereducation.TheInternetandHigherEducation,7(2),95–105.

Gašević,D.,Dawson, S.,& Siemens,G. (2015). Let’s not forget: Learning analytics are about learning.TechTrends,59(1),64–71.https://doi.org/10.1016/j.iheduc.2004.02.001

Gašević,D.,Dawson,S.,Rogers,T.,&Gašević,D.(2016).Learninganalyticsshouldnotpromoteonesizefits all: The effects of instructional conditions in predicting academic success. Internet andHigherEducation,28,68–84.https://doi.org/10.1016/j.iheduc.2015.10.002

Ganley,C.M.,&Vasilyeva,M.(2014).Theroleofanxietyandworkingmemoryingenderdifferencesinmathematics. Journal of Educational Psychology, 106(1), 105–120.http://dx.doi.org/10.1037/a0034099

Goldstein, P. J., & Katz, R. N. (2005). Academic analytics: The uses of management information andtechnology in higher education. Educause Center for Applied Research, 1–12.https://net.educause.edu/ir/library/pdf/ecar_so/ers/ers0508/EKF0508.pdf

Halberda, J.,& Feigenson, L. (2008).Developmental change in the acuityof the “number sense”: Theapproximatenumbersystemin3-,4-,5-,and6-year-oldsandadults.DevelopmentalPsychology,44(5),1457–1465.http://dx.doi.org/10.1037/a0012682

Halberda, J.,Mazzocco,M.M.M.,&Feigenson, L. (2008). Individual differences innonverbal numberacuitycorrelateswithmathsachievement.Nature,455,665–668.

Hays,W.L.(1994).Statistics.London:HarcourtBraceJovanovich.Hembree, R. (1990). The nature, effects, and relief of mathematics anxiety. Journal for Research in

MathematicsEducation,21(1),33–46.http://dx.doi.org/10.2307/749455Ho,H.-Z.,Senturk,D.,Lam,A.G.,Zimme,J.M.,Hong,S.,Okamoto,Y.,Chiu,S.-Y.,Nakazawa,Y.,&Wang,

C.-P. (2000). The affective and cognitive dimensions of math anxiety: A cross-national study.Journal for Research in Mathematics Education, 31(3), 362–379.http://dx.doi.org/10.2307/749811

House, J. D. (1993). Achievement-related expectancies, academic self-concept, and mathematicsperformance of academically underprepared adolescent students. The Journal of GeneticPsychology,154(1),61–71.http://dx.doi.org/10.1080/00221325.1993.9914722

House, J. D. (1995). The predictive relationship between academic self-concept, achievementexpectancies, and gradeperformance in college calculus. Journal of Social Psychology,135(1),111–112.http://dx.doi.org/10.1080/00224545.1995.9711411

Huon,G.,Spehar,B.,Adam,P.,&Rifkin,W.(2007).Resourceuseandacademicperformanceamongfirstyear psychology students.Higher Education, 53, 1–27. http://dx.doi.org/10.1007/s10734-005-1727-6

Ingels,S.J.,Pratt,D.J.,Wilson,D.,Burns,L.J.,Currivan,D.,Rogers,J.E.,&Hubbard-Bednasz,S.(2007).



EducationLongitudinalStudyof2002:Base-YeartoSecondFollow-upDataFileDocumentation(NCES2008-347).U.S.DepartmentofEducation.Washington,DC:NationalCenterforEducationStatistics.

Jo,I.,&Kim,Y.(2013).Impactoflearner’stimemanagementstrategiesonachievementinane-learningenvironment: A learning analytics approach. Journal of Korean Association for EducationalInformationandMedia,19(1),83–107.

Jo,I.H.,Park,Y.,Yoon,M.,&Sung,H.(2016).Evaluationofonlinelogvariablesthatestimatelearners’timemanagementinaKoreanonlinelearningcontext.TheInternationalReviewofResearchinOpenandDistributedLearning,17(1),195–213.http://dx.doi.org/10.19173/irrodl.v17i1.2176

Kizilcec, R. F., Piech, C., & Schneider, E. (2013). Deconstructing disengagement: Analyzing learnersubpopulationsinmassiveopenonlinecourses.Proceedingsofthe3rdInternationalConferenceonLearningAnalyticsandKnowledge(LAK’13),8–12April2013,Leuven,Belgium(pp.170–179).NewYork:ACM.http://dx.doi.org/10.1145/2460296.2460330

Köller,O.,Baumert,J.,&Schnabel,K.(2001).Doesinterestmatter?Therelationshipbetweenacademicinterest and achievement in mathematics.Journal for Research in Mathematics Education,32(5):448–470.

Kwon,S.(2009).Theanalysisofdifferencesoflearners’participation,procrastination,learningtimeandachievementbyadultlearners’adherenceoflearningtimescheduleine-Learningenvironments.JournalofLearner-CenteredCurriculumandInstruction,9(3),61–86.

Lauría, E. J. M., Baron, J. D., Devireddy, M., Sundararaju, V., & Jayaprakash, S. M. (2012). Miningacademicdatato improvecollegestudentretention:Anopensourceperspective.Proceedingsof the2nd InternationalConferenceonLearningAnalyticsandKnowledge (LAK ʼ12),29April–2May2012,Vancouver, BC, Canada (pp. 139–142).NewYork:ACM.http://dx.doi.org/10.1145/2330601.2330637

Lee,U.J.,Sbeglia,G.C.,Ha,M.,Finch,S.J.,&Nehm,R.H.(2015).Clickerscoretrajectoriesandconceptinventory scores as predictors for early warning systems for large STEM classes. Journal ofScienceandEducationalTechnology,24,848–860.http://dx.doi.org/10.1007/s10956-015-9568-2

Libertus,M.E.,&Brannon,E.M.(2010).Stableindividualdifferenceinnumberdiscriminationininfancy.DevelopmentalScience,13(6),900–906.

Libertus,M.E.,Feigenson,L.,&Halberda,J.(2013).Isapproximatenumberprecisionastablepredictorof math ability? Learning and Individual Differences, 25, 126–133.http://dx.doi.org/10.1111/j.1467-7687.2009.00948.x

Loomis, K. D. (2000). Learning styles and asynchronous learning: Comparing the LASSImodel to classperformance.JournalofAsynchronousLearningNetworks,4(1),23–31.

Lust,G.,Elen,J.,&Clarebout,G.(2013).Students’tool-usewithinawebenhancedcourse:Explanatorymechanisms of students’ tool use pattern. Computers in Human Behavior, 29, 2013–2021.https://dx.doi.org/10.1016/j.chb.2013.03.014

Lust,G.,Juarez-Collazo,N.A.,Elen,J.,&Clarebout,G.(2012).Contentmanagementsystems:Enrichedlearning opportunities for all? Computers in Human Behavior, 28(3), 795–808.



https://doi.org/10.1016/j.chb.2011.12.009Lust, G., Vandewaetere, M., Cuelemans, E., Elen, J., & Clarebout, G. (2011). Tool use in a blended

undergraduate course: In search of user profiles. Computers & Education, 57, 2135–2134.https://doi.org/10.1016/j.compedu.2011.05.010

Ma, X. (1999). A meta-analysis of the relationship between anxiety toward mathematics andachievementinmathematics.JournalforResearchinMathematicsEducation,520–540.

Ma,X.,&Xu, J. (2004).Thecausalorderingofmathematicsanxietyandmathematicsachievement:Alongitudinal panel analysis. Journal of Adolescence, 27, 165–179.https://dx.doi.org/10.1016/j.adolescence.2003.11.003

Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” foreducators: A proof of concept. Computers and Education, 54, 588–599.https://doi.org/10.1016/j.compedu.2009.09.008

Maloney, E. A.,&Beilock, S. L. (2012).Mathanxiety:Whohas it,why it develops, and how to guardagainst it.Trends in Cognitive Sciences, 16, 404–406.https://dx.doi.org/10.1016/j.tics.2012.06.008

Marsh,H.W.,Trautwein,U.,Lüdtke,O.,Köller,O.,&Baumert,J.(2005).Academicself-concept,interest,grades, and standardized test scores: Reciprocal effects models of causal ordering.ChildDevelopment,76(2),397–416.http://dx.doi.org/10.1111/j.1467-8624.2005.00853.x

Meece, J. L.,Wigfield,A.,&Eccles, J. S. (1990).Predictorsofmathanxietyand its influenceonyoungadolescents’ course enrollment intentions and performance in mathematics. Journal ofEducationalPsychology,82(1),60–70.

Michinov,N.,Brunot,S.,LeBohec,O.,Juhel,J.,&Delaval,M.(2011).Procrastination,participation,andperformance in online learning environments. Computers & Education, 56, 243–252.https://dx.doi.org/10.1016/j.compedu.2010.07.025

Miller,A.J.(2002).Subsetselectioninregression,2nded.NewYork:Chapman&Hall.Milne,J.L.,Jeffrey,M.,Suddaby,G.,&Higgins,A.(2012).Earlyidentificationofstudentsatriskoffailing.

InM.Brown,M.Hartnett,&T.Stewart(Eds.),Proceedingsofthe29thAnnualConferenceoftheAustralasian Society for Computers in Learning in Tertiary Education (ASCILITE 2012), 25–28November2012,Wellington,NewZealand,Volume1.

Morris,L.V.,Finnegan,C.,&Wu,S.(2005).Trackingstudentbehavior,persistence,andachievementinonline courses. Internet and Higher Education, 8, 221–231.https://dx.doi.org/10.1016/j.iheduc.2005.06.009

Niculescu, A. C., Tempelaar, D., Leppink, J., Dailey-Hebert, A., Segers, M., & Gijselaers, W. (2015).Feelingsandperformanceinthefirstyearatuniversity:Learning-relatedemotionsaspredictorsof achievement outcomes in mathematics and statistics. Electronic Journal of Research inEducationalPsychology,13(3),431–462.

Plake, B. S., & Parker, C. S. (1982). The development and validation of a revised version of theMathematics Anxiety Rating Scale.Educational and PsychologicalMeasurement, 42, 551–557.https://doi.org/10.1177/001316448204200218

PCAST(President’sCouncilofAdvisorsonScienceandTechnology).(2012).Engagetoexcel:Producing



onemillionadditionalcollegegraduateswithdegrees inscience, technology,engineering,andmathematics. http://www.whitehouse.gov/sites/default/files/microsites/ostp/pcast-executive-report-final_2-13-12.pdf

Purpura, D. J., & Logan, J. A. (2015). The nonlinear relations of approximate number system andmathematical languagetoearlymathematicsdevelopment.DevelopmentalPsychology,51(12),1717–1724.http://dx.doi.org/10.1037/dev0000055

Ramos,C.,&Yudko,E.(2008).“Hits”(not“discussionposts”)predictstudentsuccessinonlinecourses:A double cross-validation study.Computers & Education,50(4), 1174–1182.https://dx.doi.org/10.1016/j.compedu.2006.11.003

Randhawa,B.S.,Beamer,J.E.,&Lundberg,I.(1993).Roleofmathematicsself-efficacyinthestructuralmodel of mathematics achievement.Journal of Educational Psychology,85, 41–48.http://dx.doi.org/10.1037/0022-0663.85.1.41

Reyes,L.H.,&Stanic,G.M.A. (1988).Race,Sex,SocioeconomicStatus,andMathematics.Journal forResearchinMathematicsEducation,19(1),26–43.

Richardson,M.,Abraham,C., &Bond,R.(2012). Psychological correlates of university students’academicperformance:Asystematicreviewandmeta-analysis.PsychologicalBulletin,138,353–387.http://dx.doi.org/10.1037/a0026838

Romero,C.,Lopez,M.-I.,Luna,J.-M.,&Ventura,S.(2013).Predictingstudents’finalperformancefromparticipation in on-line discussion forums. Computers & Education, 68, 458–472.https://doi.org/10.1016/j.compedu.2013.06.009

Rubin,D.B.(1987).Multipleimputationfornonresponseinsurveys.NewYork:JohnWiley&Sons.Schatschneider,C.,Fletcher, J.M.,Francis,D. J.,Carlson,C.D.,&Foorman,B.R. (2004).Kindergarten

prediction of reading skills: A longitudinal comparative analysis. Journal of EducationalPsychology,96(2),265–282.http://dx.doi.org/10.1037/0022-0663.96.2.265

Shute, V. J., & Gluck, K. A. (1996). Individual differences in patterns of spontaneous differences inpatterns of spontaneous online tool use.The Journal of the Learning Sciences,5(4), 329–355.http://dx.doi.org/10.1207/s15327809jls0504_2

Simpkins, S. D., Davis-Kean, P. E., & Eccles, J. S. (2006).Math and sciencemotivation: A longitudinalexaminationofthelinksbetweenchoicesandbeliefs.DevelopmentalPsychology,42(1),70–83.http://dx.doi.org/10.1037/0012-1649.42.1.70

Sorby,S.,Casey,B.,Veurink,N.,&Dulaney,A. (2013).The roleof spatial training in improvingspatialandcalculusperformanceinengineeringstudents.LearningandIndividualDifferences,26,20–29.https://dx.doi.org/10.1016/j.lindif.2013.03.010

Speece,D. L.,Ritchey,K.D., Silverman,R., Schatschneider,C.,Walker,C. Y.,&Andrusik,K.N. (2010).Identifyingchildreninmiddlechildhoodwhoareatriskforreadingproblems.SchoolPsychologyReview,39(2),258–276.

Tabachnick, B. G., & Fidell, L. S. (2013). Using Multivariate Statistics, 6th ed. New Jersey: PearsonEducation.

Tempelaar,D.T.,Niculescu,A.,Rienties,B.,Giesbers,B.,&Gijselaers,W.H.(2012).Howachievementemotions impact students’ decisions for online learning, and what precedes those emotions.



InternetandHigherEducation,15(3),161–169.Tempelaar,D.T.,Rienties,B.,&Giesbers,B.(2015).Insearchforthemostinformativedataforfeedback

generation: LearningAnalytics inadata-rich context.Computers inHumanBehavior,47, 157–167.https://dx.doi.org/10.1016/j.chb.2014.05.038

Tice,D.M.,&Baumeister,R.F.(1997).Longitudinalstudyofprocrastination,performance,stress,andhealth: The costs and benefits of dawdling. Psychological Science, 8(6), 454–458.https://dx.doi.org/10.1111/j.1467-9280.1997.tb00460.x

Tourangeau,K.,Nord,C.,Lê,T.,Pollack,J.M.,&Atkins-Burnett,S.(2006).EarlyChildhoodLongitudinalStudy, Kindergarten Class of 1998–99 (ECLS-K), combined user’s manual for the ECLS-K fifth-grade data files and electronic codebooks (NCES 2006–032). U.S. Department of Education.Washington,DC:NationalCenterforEducationStatistics.

Vandenberg, S. G., & Kuse, A. R. (1978). Mental rotations, a group test of three-dimensional spatialvisualization. Perceptual and Motor Skills, 47, 599–604.https://doi.org/10.2466/pms.1978.47.2.599

Verguts, T., & Fias, W. (2004). Representation of number in animals and humans: A neural model.JournalofCognitiveNeuroscience,16(9),1493–1504.

Wigfield,A.,&Eccles, J. S. (2000).Expectancy-value theoryofachievementmotivation.ContemporaryEducationalPsychology,25,68–81.https://doi.org/10.1006/ceps.1999.1015

Wolff,A.,Zdrahal,Z.,Nikolov,A.,&Pantucek,M.(2013).Improvingretention:Predictingat-riskstudentsby analysing clicking behaviour in a virtual learning environment. Proceedings of the 3rdInternational Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013,Leuven,Belgium(pp.145–149).NewYork:ACM.http://dx.doi.org/10.1145/2460296.2460324

Yukselturk, E., & Bulut, S. (2007). Predictors for student success in an online course. EducationalTechnology&Society,10(2),71–83.

Documents

Individual Differences Related to College Students’ …Spehar, Adam, & Refkin, 2007; Macfadyen & Dawson, 2010), as well as the quality of these posts (2017). Individual differences