Upload
juan-munoz
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
On-line auto-graded assignments in economics: A test of their
effectivenessBrendan Kennelly1, John Considine2 and
Darragh Flannery1
1Department of Economics, NUI Galway2Department of Economics, UCC
Structure of Presentation
• Background
• Experiment
• Descriptive statistics
• Econometric analysis
• Discussion
Background: What is Aplia?
• Designed by Paul Romer in 2000
• Course management tool
• Facilitates learning and assessment– Using diagrams– Practice Assignment– Instant Feedback
Income doubles for this consumer. Place new budget constraint on diagram.
Background: Previous Work
• Technology– Teaching/Learning– Cost to Instructor
• Aplia– Collins et al (2008)– O’Dea and Ring (2008)
• Aplia in Nat. Univ. of Ireland, Galway– Kennelly and Duffy (DEE 2007)
• All Assignments on Aplia• Evaluated By Student Survey
A New ‘Experiment’
Brendan Kennelly and David Duffy. 2007. Poster Presentation to Developments in Economic Education Conference. Evaluating the Use of Aplia as a Substitute to Alternative First Year Tutorials Using Student Survey
Darragh Flannery (econometrics) and John Considine (student opinion) replace David Duffy
Aplia being used as a complement to tutorials for Second Year students
Evaluate by Student Survey and Focus Groups
Evaluate by examining grade outcomes
A New ‘Experiment’
• 390 students took managerial economics• Analysis mainly about 254 Commerce
Students• 10 assignments; all did 1 and 2 on Aplia• Thereafter ¼ did a paper assignment
instead of Aplia• Exam in 8 sections corresponding to the
last 8 assignments• Students had to do 6 sections in the exam
Assignment grade based on best 8 out of 10
Examination grade based on 6 question out of 8
Variable Mean Standard Deviation
Minimum Maximum Scale
Overall Exam Score 116.8 46.27 24.75 254.75 0-300
Assignment 1 30.58 9.45 2.27 47.72 0-50
Assignment 2 26.59 10.46 3.12 50 0-50
Assignment 3 33.31 8.39 4.54 50 0-50
Assignment 4 32.52 8.40 7.17 50 0-50
Assignment 5 29.85 12.09 1.19 50 0-50
Assignment 6 30.42 10.60 4.16 49.30 0-50
Assignment 7 30.12 15.19 2.17 50 0-50
Assignment 8 34.25 10.42 4.41 50 0-50
Overall Assignment Mark 61.92 18.48 6 97 0-100
CAO points 438.85 37.45 390 590 390-600
1st Year Economics Result 57.83 10.76 30 88 0-100
Overall Course Total 178.73 58.91 43.5 348.75 0-400
Summary Statistics
Descriptive Statistics
• Table 2: completion rate was lower for the written assignments than for the Aplia ones.
• Table 3: no consistent pattern in the marks across assignment format (Aplia or written)
• Table 4: no consistent pattern in the average mark per examination section by each group
• Table 5: average mark varied considerably across exam section
A1 A2 A3 A4 A5 A6 A7 A8
0.90 0.94 0.95 0.94 0.67 0.94 0.92 0.94
0.92 0.88 0.93 0.86 0.96 0.74 0.91 0.91
0.93 0.92 0.86 0.85 0.91 0.94 0.80 0.95
0.93 0.88 0.92 0.75 0.93 0.94 0.92 0.69
Table 2: Percentage of Group Completing Assignment
Econometric analysis: overall exam performanceVariable Coefficient P-value
Constant -117.43 0
Overall Assignment Mark 1.4 0
CAO Points .2 0
Results in 1st Year Economics 1.85 0
Gender .28 .94
Variable Coefficient P-value
Constant -158.85 0
No. of Assignments Complete 8.8 0
CAO Points .23 0
Results in 1st Year Economics 2.2 0
Gender .33 .94
Observations 254
Overall exam performance
• The results are in line with expectations. We replaced overall assignment mark with a variable that measured the number of assignments attempted. This variable is also significant and has a very large coefficient. However there is relatively little variation in this variable.
Econometric analysis: section specific regressions
1 2 3 4 Constant 4.7
(.58) -14.42
(.1) -24.58 (.01)
-27.35 (.01)
CAO Points
-.0045 (.83)
.0166 (.48)
.055 (.03)
.075 (007)
1st Year Result
.227 (.007)
.5 (0)
.214 (.024)
.29 (.006)
Marks in Ass i
.144 (.094)
.04 (.58)
.244 (.03)
.08 (.48)
Gender 2.65 (.062)
-.38 (.784)
-1.15 (.48)
2.12 (.24)
Aplia/paper Dummy
.475 (0.778)
3.9 (.026)
1.42 (0.45)
-2.7 (.19)
Observations 221 174 172 159
R2 .1 .31 .195 .22 p-values in parentheses
Econometric analysis: section specific regressions
5 6 7 8 Constant -23.74
(.03) -23.74 (.03)
-32.93 (0)
-33.17 (0)
CAO Points
.04 (.09)
-.001 (.96)
.053 (.02)
.035 (.035)
1st Year Result
.28 (.005)
.575 (0)
.395 (0)
.36 (0)
Marks in Ass i .035 (0.66)
-.01 (.88)
.1 (.49)
.14 (.017)
Gender -1.01 (.54)
-.9 (.5)
.18 (.9)
.075 (.95)
Aplia/paper Dummy
-2.89 (.129)
2.79 (.122)
1.91 (.27)
.39 (.788)
Observations 105 162 170 180
R2 .22 .32 .34 .3 p-values in parentheses
Section specific regressions
• Aplia/paper dummy is statistically significant only in Section 2. In that case students who did the assignment by Aplia performed somewhat better in that section of the exam.
• The number of marks that a student received in an assignment is statistically significant with a positive sign in the regressions on Section 3 and Section 8 and is weakly significant in the regression on Section 1.
Panel analysis
• The variation across students and across sections in our dependent variable meant that our data could be analysed using panel data techniques
• Pooled data
• Fixed effects
• Random effects
Econometric analysis: Pooled DataVariable Coefficient P-value
Constant -16.9 0
Assignment Mark .09 0
Aplia/Paper Dummy .66 .288
Dummy for Section 2 3.0 0
Dummy for Section 3 -1.57 .1
Dummy for Section 4 3.85 0
Dummy for Section 5 -7.6 0
Dummy for Section 6 -3.6 0
Dummy for Section 7 -4.3 0
Dummy for Section 8 -3.03 0
CAO Points .033 0
Results in 1st Year Economics .35 0
Gender .23 .65
Observations 1343
Econometric analysis: fixed effectsVariable Coefficient P-value
Constant 21.56 0
Assignment Mark .009 .766
Aplia/Paper Dummy .57 0.334
Dummy for Section 2 3.1 0
Dummy for Section 3 -1.88 .04
Dummy for Section 4 3.22 .001
Dummy for Section 5 -7.25 0
Dummy for Section 6 -4.11 0
Dummy for Section 7 -5.01 0
Dummy for Section 8 -3.1 0
Rho .43
Observations 1343
Econometric analysis: Random EffectsVariable Coefficient P-value
Constant -17.1 0
Assignment Mark .07 .011
Aplia/Paper Dummy .64 .274
Dummy for Section 2 3.13 0
Dummy for Section 3 -1.65 .07
Dummy for Section 4 3.61 0
Dummy for Section 5 -7.52 0
Dummy for Section 6 -3.85 0
Dummy for Section 7 -4.6 0
Dummy for Section 8 -3.1 0
CAO Points .034 0
Results in 1st Year Economics .36 0
Gender .18 .77
Rho .136
Observations 1343
Discussion: Results
• Very little evidence that Aplia v Written format of the assignment matters
• Little evidence of a direct link between performance in an assignment and mark on the corresponding section in the exam.
• If there is no difference in learning outcome then– Cost (student & instructor) becomes important– Students’ preferences become important
Discussion: Issues
• Where might learning from Aplia take place?– Between practice assignment and actual
assignments?– Incrementally so that ALL students do a better
examination – all have used Aplia 6 times?
• Does the examination test learning from Aplia?• Is the experiment testing the benefits of
technology and/or assignments rather than Aplia?
• Is there cheating and/or cooperative learning?
Discussion: Future Work
• More analysis of existing data– Including two stage estimation
• Change Experiment– Do not allow practice assignments for some of
2009-10 assignments• Compare with other 2009-10 assignments• Compare with exact same 2008-9 assignment
• Is Aplia fit-for-purpose?– What purpose?