Tweaking the pilot

Tweaking the pilotTweaking the pilotA Case Study from DVMT 100 at Frostburg State UniversityA Case Study from DVMT 100 at Frostburg State University

Dr. Megan E. BradleyDr. Megan E. Bradley

DVMT 100 @ FSUDVMT 100 @ FSU

• Intermediate algebra, developmental math Intermediate algebra, developmental math coursecourse

• 3 credits, does not count toward graduation 3 credits, does not count toward graduation or GPA*or GPA*

• Must take if need MATH 102 (College Algebra) Must take if need MATH 102 (College Algebra) or MATH 106 (Algebra with Calculus – or MATH 106 (Algebra with Calculus – Business majors)Business majors)

• About 450 students per yearAbout 450 students per year• 1078% increase since inception in 19851078% increase since inception in 1985

Course IssuesCourse Issues

• Failure rate with gender gap in DVMT 100:Failure rate with gender gap in DVMT 100:• 41% failure rate overall41% failure rate overall• 44% rate for males; 35% rate for females44% rate for males; 35% rate for females

• Failure rate in next math course:Failure rate in next math course:COURSE DVMT DWF RATE

NON-DVMT DWF RATE

DIFFERENCE

MATH 102 56% 39% 16%

MATH 106 43% 33% 11%

Course IssuesCourse Issues

• Staffing issuesStaffing issues• Relied solely on undergraduate students to Relied solely on undergraduate students to

teachteach

• Course DriftCourse Drift• Delivery: ½ sections all face-to-face (f2f); other Delivery: ½ sections all face-to-face (f2f); other

½ all computer lab½ all computer lab• Different textbook, syllabus, point systemDifferent textbook, syllabus, point system• No system for checking reliability of gradingNo system for checking reliability of grading

What we didWhat we did

• Emporium ModelEmporium Model

• Hired new staff to serve as lead instructorHired new staff to serve as lead instructor• Undergraduates became ULAs, shifting role to Undergraduates became ULAs, shifting role to

lab assistantlab assistant

Pilot – spring ‘11Pilot – spring ‘11• Traditional lecture Traditional lecture

• all face-to-face (f2f) classesall face-to-face (f2f) classes• no online workno online work• taught by trained undergraduatestaught by trained undergraduates• point system for course gradepoint system for course grade• 1 final exam but could have earned other points with 1 final exam but could have earned other points with

previous assignments to make final exam not have much previous assignments to make final exam not have much weightweight

• RedesignRedesign• Lecture 1x/week by instructor & lab 4x/week with trained Lecture 1x/week by instructor & lab 4x/week with trained

Undergraduate Learning Assistants (ULAs) using Undergraduate Learning Assistants (ULAs) using ASAlgebra by PlatoASAlgebra by Plato

• 3 modules & corresponding exams3 modules & corresponding exams• Mastery learning – retake exams until passedMastery learning – retake exams until passed

• Pass course by passing all 3 modules with 80% or higherPass course by passing all 3 modules with 80% or higher• Extra credit for attending and doing online homework & Extra credit for attending and doing online homework &

evaluatesevaluates

assessmentassessment

• Pass/Fail ratesPass/Fail rates

• Scores on “core questions” Scores on “core questions” • Questions that show up on the redesign module Questions that show up on the redesign module

exams & the final exams for the traditional exams & the final exams for the traditional sectionssections

• Focus groupsFocus groups

Pilot resultsPilot results

• Pass/failPass/fail• Historical failure rate: : Historical failure rate: :

41% 41%

• Redesign failure rate: Redesign failure rate: 47.2% which was 47.2% which was significantly worse than…significantly worse than…

• Traditional failure rate: Traditional failure rate: 22.6%22.6%

• Males failed more than Males failed more than femalesfemales


• Core questionsCore questions• Difficult to use final grades due to different Difficult to use final grades due to different

grading systemsgrading systems• Considering all core questions, a one-way Considering all core questions, a one-way

ANOVA of Type of Classroom (2: Redesign versus ANOVA of Type of Classroom (2: Redesign versus traditional) by Core Qs (All) was significant, F = traditional) by Core Qs (All) was significant, F = 37.429, p = .000, eta2 = .327. 37.429, p = .000, eta2 = .327.

• Redesign students (X = 87.98%) Redesign students (X = 87.98%) performed significantly better than performed significantly better than traditional students (X = 63.14%). traditional students (X = 63.14%).

• Core questionsCore questions• Difficult to use final grades due to different Difficult to use final grades due to different

grading systemsgrading systems• Considering all core questions, a one-way Considering all core questions, a one-way

ANOVA of Type of Classroom (2: Redesign versus ANOVA of Type of Classroom (2: Redesign versus traditional) by Core Qs (All) was significant, F = traditional) by Core Qs (All) was significant, F = 37.429, p = .000, eta2 = .327. 37.429, p = .000, eta2 = .327.

• Redesign students (X = 87.98%) Redesign students (X = 87.98%) performed significantly better than performed significantly better than traditional students (X = 63.14%). traditional students (X = 63.14%).


• Core questionsCore questions• Below is a breakdown of core questions per Below is a breakdown of core questions per

module, module, • Students from the redesign section scored Students from the redesign section scored

significantly higher than traditional sections for significantly higher than traditional sections for all three modules:all three modules:

• M1: M1: Redesign (X = 86.20%) > traditional Redesign (X = 86.20%) > traditional (X=83.66%) (X=83.66%)




• Regression indicated which of course Regression indicated which of course activities significantly related to student activities significantly related to student grade on core questions.grade on core questions.• Attendance: correlated but weakAttendance: correlated but weak• Online homework: correlated but weakOnline homework: correlated but weak• Online Online evaluatesevaluates: strongly correlated: strongly correlated

• Homework & evaluates: needed 80% to pass Homework & evaluates: needed 80% to pass and move onand move on• Evaluates: Often only had 4 questions so needed Evaluates: Often only had 4 questions so needed

to get perfect score.to get perfect score.

Additional resultsAdditional results

• We examined students’ time on task and We examined students’ time on task and when they were using software.when they were using software.

• Reviewed focus group suggestions.Reviewed focus group suggestions.

• Compared student performance on certain Compared student performance on certain items in traditional sections.items in traditional sections.

• Created hypotheses and tested them out as Created hypotheses and tested them out as best as we could.best as we could.

• Reassessed the teamReassessed the team

Issues & tweaksIssues & tweaks

1.1. Students compared DVMT Students compared DVMT 100 sections. 100 sections.

2.2. Redesign students did not Redesign students did not effectively use their lab effectively use their lab time wisely. time wisely.

3.3. Redesign students did not Redesign students did not have enough deadlines – have enough deadlines – 1x/module, night before 1x/module, night before exam.exam.

4.4. Students fell behind next Students fell behind next module while retaking module while retaking previous module exam.previous module exam.

1.1. Fall 2011 – full Fall 2011 – full implementation.implementation.

2.2. Changed lab to 2x/wk and Changed lab to 2x/wk and used technology to block used technology to block other sites.other sites.

3.3. Created several deadlines Created several deadlines with last deadline before with last deadline before test review day. test review day.

4.4. Added retake week after Added retake week after Mod1.Mod1.

1.1. Fall 2011 – full Fall 2011 – full implementation.implementation.

2.2. Changed lab to 2x/wk and Changed lab to 2x/wk and used technology to block used technology to block other sites.other sites.

3.3. Created several deadlines Created several deadlines with last deadline before with last deadline before test review day. test review day.

4.4. Added retake week after Added retake week after Mod1.Mod1.


1.1. The grading system in the The grading system in the redesign confused redesign confused students. students.

2.2. Redesign students found Redesign students found and exploited a loophole and exploited a loophole about retaking modules about retaking modules next semester. next semester.

3.3. Lab assistants were Lab assistants were scattered across different scattered across different labs. labs.

4.4. No pedagogy to address No pedagogy to address gender gap.gender gap.

1.1. Revised to be based on Revised to be based on weights that required and weights that required and rewarded important course rewarded important course aspects.aspects.

2.2. Modified retaking of Modified retaking of modules.modules.

3.3. Assigned lab assistants.Assigned lab assistants.

4.4. Created Train Your Brain Created Train Your Brain ProgramProgram

1.1. Revised to be based on Revised to be based on weights that required and weights that required and rewarded important course rewarded important course aspects.aspects.

2.2. Modified retaking of Modified retaking of modules.modules.

3.3. Assigned lab assistants.Assigned lab assistants.

4.4. Created Train Your Brain Created Train Your Brain ProgramProgram


1.1. Failure rate on first Failure rate on first version of module exam version of module exam was very poor:was very poor:

• Mod1 = 27% passedMod1 = 27% passed



1.1. Implemented PreModule Implemented PreModule ExamExam

• Earn 85% or higher – no need Earn 85% or higher – no need to take Module examto take Module exam

• Reward studying & doing Reward studying & doing wellwell

1.1. Implemented PreModule Implemented PreModule ExamExam

• Earn 85% or higher – no need Earn 85% or higher – no need to take Module examto take Module exam

• Reward studying & doing Reward studying & doing wellwell

•Fall 2011 pass/fail rate•20.3% failure rate overall•19.7% rate for males; 21.3% rate for females•Gender analyses NOT statistically significant.

•Fall 2011 pass/fail rate•20.3% failure rate overall•19.7% rate for males; 21.3% rate for females•Gender analyses NOT statistically significant.

Full implementation Full implementation resultsresults

• Remember this?Remember this?• Failure rate with gender gap:Failure rate with gender gap:

• 41% failure rate overall41% failure rate overall• 44% rate for males; 35% rate for females44% rate for males; 35% rate for females• Pilot redesign failure rate: 47.2%Pilot redesign failure rate: 47.2%

Impact of changesImpact of changes

• Deadlines = large % students completed Deadlines = large % students completed deadlinesdeadlines

• Weights & lab changes = better attendance Weights & lab changes = better attendance and time on taskand time on task

• Train Your Brain = no gender gap, better Train Your Brain = no gender gap, better performance overallperformance overall

Impact of changesImpact of changes

1.1. Failure rate on first Failure rate on first version of module exam version of module exam was very poor:was very poor:




1.1. PreModule Exam resultsPreModule Exam results• Module 1Module 1

• Premod: 36% passedPremod: 36% passed

• Version 1: 72% passedVersion 1: 72% passed

• Module 2Module 2

• Premod: 16.2% passedPremod: 16.2% passed





1.1. PreModule Exam resultsPreModule Exam results• Module 1Module 1

• Premod: 36% passedPremod: 36% passed








Overall Overall recommendationsrecommendations

• Look, look, look.Look, look, look.

• Add structure.Add structure.

• Improve based on evidence (from pilot, from Improve based on evidence (from pilot, from other redesigns, from published research)other redesigns, from published research)

• Add psychologyAdd psychology• Provide incentivesProvide incentives• Spacing effectSpacing effect• Practice effectPractice effect• Mastery learningMastery learning

Your rewardYour reward

Documents

Tweaking the pilot