Integrating Assessment with Instruction: A Look Forward

Integrating Assessment with Instruction: A Look ForwardKen KoedingerPittsburgh Science of Learning CenterHuman-Computer Interaction and PsychologyCarnegie Mellon University

Neil T. HeffernanComputer ScienceLearning Sciences and Technology ProgramWorcester Polytechnic Institute

We care about improving education because we believe it is at the core of social progress.

We are enthusiastic about educational technology in the 21st century.Intelligent tutors & on-line courses that individualizeEnglish learning games on mobiles around the worldVirtual labs & on-line collaborative science lots of potential benefits of ed tech, but the most important benefit is the data we will get ...2Why data is important to improving student learningIf we knew everything about students learning challenges, we would not need data

But, there is a lot we do not know about student learning

In fact, theres a lot we dont know about our own learningYouve had lots of experience with the English language. You might say you know English.But, do you know what you know?You cant design for what you dont know!

But, how to design for what you dont know you know?Answer:

Data => DiscoveryYou can design instruction & assessment for what you do know you know

Technology Enhanced AssessmentASSISTments combines assessment with tutoring assistance to students. Teachers and school leaders get customizable reports. There are many TEAs available. Today for this talk we are going to use examples from ASSISTments.5

Students here are working on laptops but any device can be used.6

This is an ASSISTment.7

This is the assistance part.8

Fabulous report with lots of great details.9

Perimeter correct but not divided by 6.Area divided by 6 instead of perimeter divided by 6.Common wrong answers can help teachers understand the errors made by students.10TEAs help researchersHelp researchers do stuff here is how they are useful in research. Cog sci paper - 11Three question typesOne of the eight cover stories used in the studies with the three different variants. VersionSample AnswerSample questionP1 (Symbolizing)Accuracy 25% 800 40mAnn is in a rowboat in a lake that is 2400 yards wide. She is 800 yards from the dock. She then rows for "m" minutes back towards the dock. Ann rows at a speed of 40 yards per minute. Write an expression for Ann's distance from the dock.

P2 (Articulation)Accuracy 26%800 40*3Ann is in a rowboat in a lake that is 2400 yards wide. She is 800 yards from the dock. She then rows for "3" minutes back towards the dock. Ann rows at a speed of 40 yards per minute. Write an expression for Ann's distance from the dock.

P3 (Computation)Accuracy 70%680Ann is in a rowboat in a lake that is 2400 yards wide. She is 800 yards from the dock. She then rows for 3 minutes back towards the dock. Ann rows at a speed of 40 yards per minute. Compute Ann's distance from the dock.

Heffernan, N. T. & Koedinger, K. R. (1998). A developmental model for algebra symbolization: The results of a difficulty factors assessment. In M. Gernsbacher & S. Derry (Eds.) Proceedings of the Twentieth Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Erlbaum. pp. 484-489. Heffenan and Koedinger three questions12Story ProblemAs a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work?

Word ProblemStarting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with?

Equationx * 6 + 66 = 81.90Which kind of problem is most difficult for Algebra students?This is important to understand because many educators and assessment creators have blind spots.13

0102030405060708090100ElementaryTeachersMiddleSchoolTeachersHigh SchoolTeachers% Correctly ranking equations as hardestNathan & Koedinger (2000). An investigation of teachers beliefs of students algebra development. Cognition and Instruction.Expert Blind Spot!Koedinger & Nathan (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences.Data contradicts common beliefs of researchers and teachersWe need science to help us see what we are missing.141. Assessment Value

2. Learning Value

3. Research Value

TEAs can provide greaterTransition Slide15Assessment ValueTwo kinds of Assessment Value from TEAs

Predictive AssessmentsRicher types of questions can be asked2004-2005 Data, N = 391 studentsOnline dataSept 04 - May 05267 minutes (sd = 79); 9 days; 147 items (sd = 60)8th grade MCAS scores (May 2005)2005-2006 Data, N = 616 studentsOnline dataSept 05 May 06196 minutes (sd = 76); 6 days; 88 items (sd. = 42)8th grade MCAS scores (May 2006)

Feng, Heffernan, & Koedinger (2009). Addressing the assessment challenge in an online system that tutors as it assesses. User Modeling and User-Adapted Interaction: The Journal of Personalization Research.Predicting MCAS from ASSISTments useResults of Regression2004-2005 data2005-2006 data(Constant)32.4143.284IRT_Proficiency_Estimate26.832.944Scaffold_Percent_Correct 20.42721.327Avg_Question_Time -0.17-0.102Avg_Attempt-10.5Avg_Hint_Request-3.217Question_Count0.072Avg_Item_Time0.045Total_Attempt-0.044Correlation with MCASR = .84R = .85Cross-year validation:Prediction models are robust!

Assessment ValueRicher questions can be asked with TEAsSimulations and gamesSimulations = microworlds which can be supported by ASSISTments either embedded or students can be sent to the microworld. Screen shot of PHET inside ASSISTments20Richer Types of Questions

21Assessment of Microworlds

This is a assessment quesiton that has a micro world22Learning ValueComparing learning: TEA vs.Business as UsualHomeworkMastery LearningAutomatic Reassessment and Relearning SystemParent Involvement PortalComparing Learning in Middle School N = 1, 344 urban 7th grade students Only took students who also had a 6th grade MCAS score for as well Assigned at the student level either to experimental or control condition

Koedinger, K.R., McLaughlin, E. & Heffernan, N. (2010). A quasi-experimental evaluation of an on-line formative assessment and tutoring system. Journal of Educational Computing Research. 43(1), 489-510.

ADD MORE DETAILS24ResultsBoth groups benefitted from ASSISTments uses and special needs students benefited even more.

Adjusted post-test means for 7th grade student groups. This was completed in classrooms during regular math instruction.25HomeworkStudents in eight 8th classes were assigned to either TEA or Business as usual conditionsStudent level assignment Students were then switched to the other group

Quality of evidence high enough to qualify ASSISTments to get an Efficacy Study funded via the US Dept of Eds IES.

Singh, R., Saleem, M., Pradhan, P., Heffernan, C., Heffernan, N., Razzaq, L. & Dailey, M. (2011). Improving K-12 Homework with Computers.Proceedings of the Artificial Intelligence in Education Conference.Mendicino, M., Razzaq, L. & Heffernan, N. T. (2009). Comparison of Traditional Homework with Computer Supported Homework.Journal of Research on Technology in Education, 41(3), 331-359.

Homework done in a TEA also has learning benefits. 26ResultsWhen students received immediate feedback on their homework, they learned more compared to business as usual with feedback in the morning.The two studies had a .4 and .6 sd effect size.This translates to about a half of a letter grade.

Mastery LearningMastery Learning is possible in TEALiterature suggests is a good thingSee Kulick and Kulic (1990), Bloom (1984)

Kulik, C., Kulik, J., & Bangert-Drowns, R. (1990). Effectiveness of mastery learning programs: A meta-analysis.Review of Educational Research, 60(2), 265-306.Shows from Cris what ML looks like check the sales pitch28ReportPellegrino, Goldman, Heffernan, Heffernan, Deitz & Sofffer (in preparation) . The Positive Effects of Mastery Learning on Student Achievement in the Connected Mathematics Program

29Automatic Reassessment and Relearning SystemN = 128 8th grade studentsARRS implements spaced practice

Experimental: Given practice until three right in a row and then continual re-assessment and re-training Control: Only practiced until they get three right in a rowHeffernan, N., Heffernan, C., Dietz, K., Soffer, D., Pellegrino, J. W., Goldman, S. R. & Dailey, M. (2012). Improving Mathematical Learning Outcomes Through Automatic Reassessment and Relearning. AERA 2012

Results

Parent NotificationN = 86 parents of 8th grade studentsExperiment: Parents received an automatic notice each Friday about the results of all the assignments.Control: Did not receive a notice.

Broderick, Z., DeNolf, K., Dufault, J., Heffernan, N. & Heffernan, C. (2011). Increasing Parent Engagement in Student Learning Using an Intelligent Tutoring System with Automated Messages. Journal of Interactive Learning Research, 20(4), 467-490.

ResultsParents felt more connected (p