Embed Size (px)
Citation preview
Introduction to
Value-Added Data
Robert ClarkNeil Defty
Nicola Forster
Theory and Stats bits…
Subject X
20
40
60
80
100
120
4 5 6 7 8
Baseline
Out
com
e
.
-ve VA+ve VA
Trend Line/Regression Line
Raw Residual
Measuring Value-Added – Terminology
Exa
m g
rade
BASELINE SCORE
Measuring Value-Added – An Example
Low Ability Average Ability High Ability
Baseline Score
A*
U
B
C
D
E
F
G
Res
ult
Aldwulf Beowulf
Cuthbert+ve (+ 2 grades)
-ve (- 2 grades)
National Trend
‘Average’ Student
The position of the national trend line is of critical importance
Subject A
Subject B
40
60
80
100
120
140
5 6 7 8
Average GCSE
Gra
de
Photography
Sociology
English Lit
Psychology
Maths
Physics
Latin
Some Subjects are More Equal than Others….
A-Level
>1 grade
A*ABC
A
A*
B
C
D
E
Some Subjects are More Equal than Others….
BTec National Diploma
Art & DesignBusinessHealth & CareHospitalityIT PractitionersMedia ProductionMusicPerforming ArtsPublic ServicesScienceSportSportTravel & Tourism
BCE
DDM
DDD
DMM
MMM
MMP
MPP
DPPP
Average GCSE Score
Gra
de
Some Subjects are More Equal than Others….
International Baccalaureate
4
5
6
7
C B A A*
Average (I)GCSE Score
Gra
de
Biology
Business and Management
Chemistry
Design Technology
Economics
English_A1
Film
French_B
Geography
History
Mathematics
Music
Philosophy
Physics
Psychology
Spanish_B
Theatre Arts
Visual Arts
F
E
D
C
B
A
A*
Test Score
GC
SE
Gra
des
Art & DesignBiologyChemistryEconomicsEnglishFrenchGeographyGermanHistoryIctMathematicsMedia StudiesMusicPhysical EducationPhysicsReligious StudiesScience (Double)Spanish
Some Subjects are More Equal than Others….
GCSE
Definitions:• Residual – difference between the points the student attains and
points attained on average by students from the CEM cohort with a similar ability
• Standardised Residual – the residual adjusted to remove differences between qualification points scales and for statistical purposes
• Average Standardised Residual – this is the ‘Value Added Score’ for any group of results
• Subject VA – average of standardised residuals for all students’ results in the particular subject
• School VA – average of standardised residuals for all students’ results in all subjects for a school / college
• Confidence Limit – area of statistical uncertainity within which any variation from 0 is deemed ‘acceptable’ and outside of which could be deemed ‘important’
Burning Question :
What is my Value-Added Score ?
Better Question :
Is it Important ?
SPC Chart
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year
Performance inline with expectation
VA Score
Performance below expectationProblem with Teaching & Learning ?
Performance above expectationGood Practice to Share ?
-0.2 -0.2 -0.1
0.1
-0.6
0.1
-0.2
-0.7 -0.7-0.4 -0.4
-0.2
-2.2
-0.5
2.0
-0.7-0.5
0.10.4
-4.0
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
Art
Bio
logy
Che
mis
try
Des
ign
and
Tech
nolo
gy
Dra
ma
Eng
lish
Lang
uage
Eng
lish
Lite
ratu
re
Fre
nch
Geo
grap
hy
Ger
man
His
tory
Hom
e E
cono
mic
s
Info
rmat
ion
Tech
nolo
gy
Mat
hs
Med
ia S
tudi
es
Mus
ic
Phy
sics
Rel
igio
us S
tudi
es
Spa
nish
Av
era
ge
Sta
nd
ard
ise
d R
es
idu
al
Subject Summary
Standardised Residual Graph
The Scatter Plot
Baseline Score
Gra
de
Po
ints
Eq
uiv
alen
t
Look for Patterns…
General Underachievement / over achievement ?
Do any groups of students stand out ?
– high ability vs low ability ?
– male vs female ?
Other things to look for…
Why did these students do so badly ?
Why did this student do so well ?
How did they do in their other subjects ?
Worked Example
Which Subjects Cause Most Concern ?
Danger of Relying on Raw Residuals Without Confidence Limits
Additional A
pplied Science
Additional S
cience
Art &
Design
Biology
Business S
tudies
Chem
istry
Design &
Technology
Dram
a
English
English Literature
French
Geography
Germ
an
History
Mathem
atics
Music
Physical E
ducation
Physics
Religious S
tudies
Science
Spanish
Short C
ourse Religious S
tudies
-4
-3
-2
-1
0
1
2
3
4
0.00.8 0.5
-0.3
1.1
-0.4
1.00.2 0.4 0.1 0.1
0.0
0.0 0.1
0.0
0.0
-0.3
0.2 0.5
-0.3
0.70.2
-2.9
0.0
Average Standardised Residuals by Subject
Ave
rag
e S
tan
dar
dis
ed R
esid
ual
Which subjects now cause most concern ?
Business Studies
Religious Studies
Summary of Process
• Examine Subject Summary• Determine ‘interesting’ (i.e. statistically significant) subjects• Look at 3 year average as well as single year if available• Look at trends in ‘Interesting Subjects’• Examine student data –Scatter graphs• Identify students over / under achieving (student list or Paris)• Any known issues ?• Don’t forget to look at over achieving subjects as well as under
achieving• Phone / Email CEM when you need help understanding /
interpreting the data / statistics !
Baseline Choice
Same School - Spot the Difference ?
GCSE as
Baseline
Test as
Baseline
GCSE as
Baseline
Test as
Baseline
A2 Biology
GCSE as
Baseline
Test as
Baseline
A2 Biology
GCSE as
Baseline
Test as
Baseline
Student A
Student A
Student B
Student B
A2 Biology
Student A
Baseline Baseline Score
Grade Points ‘Predicted’ Points
VA (Residual)
GCSE 7.8 A 120 122.4 -2.4
Test 1.7 A 120 90.6 29.4
Student B
Baseline Baseline Score
Grade Points ‘Predicted’ Points
VA (Residual)
GCSE 8.0 A* 140 128.1 11.9
Test 2.3 A* 140 100.1 39.9
How well did these students perform ?
National or School Type Specific ?
Comparison to all schools
Comparison to Independent Schools Only
Comparison to all schools
Comparison to FE Colleges Only
Questions:
→ How does the unit of comparison used affect the Value-Added data and what implications does this have on your understanding of performance ?
→ Does this have implications for Self Evaluation ?
Thank You
Robert Clark – [email protected] Defty – [email protected]
Nicola Forster – [email protected]
