Embed Size (px)
Citation preview
An Algorithm for Assessing Student's Reading Levels
Mark Brierley
Kosuke Adachi
Shinshu University
What’s an algorithm?
• a set of rules for solving a problem in a finite number of steps– variant of algorism, (from arabic al Kuwarismi)
Why do we need one?
• Recommending books for students to read
• Making an adaptive reading text
What do we know?
• Student ratings (R)
• Reading times (T)
• Comprehension (C)
• Effective reading speed/Comprehensible WPM (E) = WPM x Comprehension
The data
• Online ER system.– 12,575 reviews by – 937 students on– 634 graded readers
• ERFOPT pre-beta trials– 512 attempts by– 115 students on– 21 texts and questions
The nature of the data
• Stringy– different students read different texts
• Unreliable– opinions, emotions, characters…
(welcome to the real world!)
• Even a fuzzy string vest has a definite shape!
Assumptions
• Each text has a level of difficulty– Harry Potter is easier than Da Vinci Code– Noam Chomsky is more difficult than Bill
Bryson
• Each reader has a reading ability– I’m not very good at reading
What level are the texts?
• Nominal level (N)– How did the students do?– If that was the students’ level, what level were
the texts?– repeat...
P1 Levels
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
11.0
12.0
13.0
14.0
1 2 3 4 5 6 7 8 9 10 11
comp
time
rating
comp wpm
C5 Levels
3.0
4.0
5.0
6.0
7.0
8.0
9.0
10.0
11.0
12.0
13.0
14.0
1 2 3 4 5 6 7 8 9 10 11
comp
time
rating
comp wpm
WK Level
5.0
6.0
7.0
8.0
9.0
10.0
11.0
12.0
13.0
14.0
1 2 3 4 5 6 7 8 9 10
comp
time
rating
comp wpm
text code
number of attempts (n)
nominal level
comp. estimate (c )
impress. estimate (i)
time estimate (t)
effective speed estimate (w)
average variation (c t i)
average variation from nominal
words questions
O2 12 8 6.3 9.1 6.0 5.9 1.0 0.9 551 8
O4 13 8 6.5 4.5 6.4 7.1 0.7 1.1 387 9
E1 14 5 6.8 4.9 5.4 6.1 0.6 0.5 500 5
O6 17 10 7.3 12.0 5.6 6.2 2.0 1.7 647 9
O3 21 8 6.7 11.5 6.4 7.7 1.7 1.0 699 9
C5 24 10 6.1 13.5 6.2 5.0 2.6 2.0 664 7
O7 24 11 7.0 10.4 6.0 6.7 1.4 1.9 666 9
C3 26 5 6.2 4.9 6.0 6.5 0.4 0.5 496 9
P2 33 6 6.8 6.0 7.2 7.1 0.4 0.7 529 9
C4 33 7 6.1 5.3 6.4 6.7 0.3 0.5 514 11
C1 39 4 5.6 4.6 6.3 6.0 0.5 0.9 314 10
C2 39 5 6.3 6.1 5.7 6.2 0.2 0.5 518 10
E4 41 3 5.7 4.6 6.4 6.4 0.6 1.4 374 7
E2 43 4 5.6 4.8 4.9 4.7 0.3 0.5 500 5
O1 46 7 6.5 5.6 6.2 6.9 0.3 0.4 629 9
P1 55 6 7.3 4.6 6.1 7.7 0.8 0.6 348 8
average 479 5.5 5.6 5.9 5.4 5.6 0.6 0.8 463.0 7.5
Correlations between tests
Nom. Comp. Imp. Time Eff.Var. (c t i)
Var from nom.
Ww Qq
Nominal 100% 65% 81% 9% 16% 76% 64% 74% 23%
Comp. 65% 100% 48% 8% 50% 40% 26% 60% 17%
Imp. 81% 48% 100% -1% -7% 94% 73% 80% 3%
Time 9% 8% -1% 100% 65% -1% 7% -11% 54%
Eff. 16% 50% -7% 65% 100% -13% -15% 12% 57%
0.05 16.00 49%(pink cells: Significant correlations)
Significant correlations within texts
16 textsComp. Imp. Time Eff.
Comp.= 2 0 11
Imp.2 = 1 4
Time0 1 = 16
Eff. 11 4 16 =
Student level correlations
Time Comp. Imp. Eff. EPER E YL
Time 100% 7% 9% 52% 25% 8%
Comp. 7% 100% 53% 53% 39% 33%
Imp. 9% 53% 100% -13% 37% 46%
Effective 52% 53% -13% 100% 33% 7%
sig. 0.05 106 19%
sig. 0.05 64 25%
What about the algorithm?
• What text should we show the student next?
• What level is the student at?
• What level is a new text?
The algorithm
How easy was it?
• 1- Very difficult• 2-• 3-• 4-• 5- Very Easy
• What does this tell us?
• S < Book
• Depends on student...
• S >= Book
Can they read it?
• Logic– If T < Tm ... AND C > Cm – If T < Tm ... OR C > Cm
• Score– (C/Cmax) x (W/T)
Can they read it?
Speed
Comprehension
Fast Average Slow
High
Average
Low
Yes
Probably
?
Maybe not
No!
0
1
2
3
4
5
6
7
8
9
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
0
1
2
3
4
5
6
7
8
9
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
0
1
2
3
4
5
6
7
8
9
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
0
1
2
3
4
5
6
7
8
9
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
Questions
• What do we trust? – Speed– Comprehension– Effective speed– Student impressions
• Logical or arithmetic
Conclusions
• Publisher level ≠ Performance
• Performance ≠ impression of difficulty
• Publisher level ≈ impression of difficulty
Reading ability multidimensional
• Speed
• Comprehension
• Self-image?
• Reading strategies
• Imagination
• World knowledge– Independent variables?
Strategy
• Keep independent data for each text
• Calculate several levels for each student
• Take an average to estimate their level
