Embed Size (px)
Citation preview
Introduction to CEM
Secondary Information Systems
Dr Robert Clark
ALIS Project Manager
The Projects
The Projects
Year 7Year 7
Year 8Year 8(+ additional)(+ additional)
Year 9Year 9
MidYISMidYIS Computer Adaptive Baseline Test(paper test also available)
Year 10Year 10
Year 11Year 11 YellisYellis Computer Adaptive Baseline Test(paper test also available)
Year 12Year 12
Year 13Year 13 AlisAlis GCSE Computer Adaptive Baseline Test(paper test also available)
GCSE
A / AS / Btec / IB etc
INSIGHTINSIGHT Combines curriculum tests with developed ability
The baseline score is measured as early as possible and the information sent to the CEM Centre
The institution then receives feedback from the CEM Centre: baseline test results and predictions
The public examination results are sent to the CEM Centre when received by the institution
The value added data is available from September via the Institution’s website at the CEM Centre
Typical Timeline
Early Testing & Predictions
Testing is available from June at end of previous academic year i.e. Y6 / Y9 / Y11
Predictions
.
Subject X
020406080
100120140
4 5 6 7 8
Baseline
Out
com
e
0 2 4 6 8
Subject X
How CEM ‘Predictions’ are made…
A / B
C
Regression Line (…Trend Line, Line of Best Fit)
Outcome = gradient x baseline + intercept
Correlation Coefficient (~ 0.7)
40
60
80
100
120
140
5 6 7 8
Average GCSE
Gra
de
Photography
Sociology
English Lit
Psychology
Maths
Physics
Latin
Some Subjects are More Equal than Others….
>1 grade
A*ABC
A
A*
B
C
D
E
F
E
D
C
B
A
A*
Test Score
GC
SE
Gra
des
Art & DesignBiologyChemistryEconomicsEnglishFrenchGeographyGermanHistoryIctMathematicsMedia StudiesMusicPhysical EducationPhysicsReligious StudiesScience (Double)Spanish
Some Subjects are More Equal than Others….
GCSE (MidYIS or Yellis)
1 grade
Baseline Measurement
Problems with Key Stage Baselines
1. Not all students have KS baselines
No KS3
Foreign Students
Vocational Students
Adult Learners
2. KS exams do not always represent ‘Start of Course’ ability
Post-16 : Year(s) out or intermediate years
3. Prior Value-Added
Can you add value at every Key Stage ?
Under achievement leading to under expectation
One teacher’s output = another teacher’s input
4. Teaching to the test
Does performance represent ability, effort or exam technique ?
5. Aptitude & fluency vs Achievement & knowledge
6. Transfer of Data
Although Key Stage baselines can be a very good indicator of potential attainment, by themselves they are not sufficient.
Key Stage baselines are confounded by the effects of prior treatment.
The Computer Adaptive Test
• Test performed online – results automatically transmitted to CEM.
• Minimal installation / setup required - if any.• Adaptive – difficulty of questions changes in relation to ability of
student.• Efficient – no time wasted answering questions that are far too
easy or difficult.• Wider range of ability• Less stressful on students – more enjoyable experience than
paper test.• Less demanding invigilation.• Cheaper !
In 2010/ 2011 over 200,000 students across yrs 7-13 sat this test
Try it yourself at www.intuproject.org/demos
Baseline Feedback
Reports, Graphs
& Predictions
IPRs (Individual Pupil Record Sheets)
Look for sections that are
inconsistent
Also available based on MidYIS, Alis, SOSCA & INSIGHT scores
Intake Profiles
Also available based on MidYIS, Yellis, SOSCA and INSIGHT scores
Intake Profiles (Historical)
Predictions – MidYIS example
Similar spreadsheets available from Yellis, SOSCA, INSIGHT
Predictions - Alis example
English Language - band D
2
10
23
3124
9
1 0 00
10
20
30
40
50
U G F E D C B A A*
Grade
Perc
ent
English Language - band C
1 26
20
3529
71 0
0
10
20
30
40
50
U G F E D C B A A*
Grade
Perc
ent
English Language - band B
0 0 27
24
40
21
50
0
10
20
30
40
50
U G F E D C B A A*
Grade
Perc
ent
English Language - band A
0 0 0 1
8
26
35
23
7
0
10
20
30
40
50
U G F E D C B A A*
Grade
Perc
ent
Chances Graphs
Predictions
vs
Targets
What is a ‘Prediction’ ?
NOT a forecast of the grade the student will get
An indication of the grade (points score) achieved on average by students of similar ability in the previous year
Targets ?• Minimum Targets – Round CEM prediction down ?
• Realistic Targets – Use nearest CEM grade ?
• Challenging Targets – 75th percentile ? Prior Value Added ? Arbitrary grade fraction ?
75th Percentile Predictions• If all students attain 75th percentile predictions, School VA will be at top 25% • Provided by all Projects in addition to 50th percentile• Alis : Excel spreadsheet - ‘Predictions – Spreadsheet (75th Percentile)’• Pre-16 : Adjust button on predictions spreadsheet
Prior Value Added• Only where prior VA is positive ?• 1 year or 3 ?• Reasonable to use raw residual figures as this is an approximate measure and
raw residuals give grade fractions• Alis : Can be calculated using PARIS software• Pre-16 : Adjust button on predictions spreadsheet
Data used to inform, not replace, professional judgement
75th Percentile
Prior Value Added
Step 1
75th Percentile
PriorValue-Added
Value-Added Feedback
Reports & Graphs
Subject X
20
40
60
80
100
120
4 5 6 7 8
Baseline
Out
com
e
.
-ve VA+ve VA
Trend Line/Regression Line
Raw Residual
Measuring Value-Added – Terminology
Exa
m g
rade
BASELINE SCORE
Measuring Value-Added – An Example
Low Ability Average Ability High Ability
Baseline Score
A*
U
B
C
D
E
F
G
Res
ult
Alf Bob
Chris+ve (+ 2 grades)
-ve (- 2 grades)
National Trend
‘Average’ Student
The position of the national trend line is of critical importance
Subject A
Subject B
Standardisation of Residuals
• (Raw) Residuals can be used to examine an individual’s performance
• Standardised Residuals are used to compare performance of groups
• Standardised Residuals are independent of year or qualification type
• For a class, subject, department or whole institution the Average Standardised Residual is the ‘Value-Added Score’
• Standardised Residual = Residual / Standard Deviation (National Sample)
• When using Standardised Residuals then for an individual subject
• 95% Confidence Limit = 2.0 x Standard Error• 99% Confidence Limit = 2.6 x Standard Error• 99.7% Confidence Limit = 3.0 x Standard Error
N
1ErrorStandard where N = number of results in the
group
(for combinations of subjects consult the relevant project)
Burning Question :
What is my Value-Added Score ?
Better Question :
Is it Important ?
Value Added Feedback…
Statistical Process Control (SPC) Chart
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Year
-0.2 -0.2 -0.1
0.1
-0.6
0.1
-0.2
-0.7 -0.7-0.4 -0.4
-0.2
-2.2
-0.5
2.0
-0.7-0.5
0.10.4
-4.0
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
Art
Bio
logy
Che
mis
try
Des
ign
and
Tec
hnol
ogy
Dra
ma
Eng
lish
Lang
uage
Eng
lish
Lite
ratu
re
Fre
nch
Geo
grap
hy
Ger
man
His
tory
Hom
e E
cono
mic
s
Info
rmat
ion
Tec
hnol
ogy
Mat
hs
Med
ia S
tudi
es
Mus
ic
Phy
sics
Rel
igio
us S
tudi
es
Spa
nish
Av
era
ge
Sta
nd
ard
ise
d R
es
idu
al
Subject Summary
Standardised Residual Graph
The Scatter Plot
Baseline Score
Gra
de
Po
ints
Eq
uiv
alen
t
Look for Patterns…
General Underachievement / over achievement ?
Do any groups of students stand out ?
– high ability vs low ability ?
– male vs female ?
Other things to look for…
Why did these students do so badly ?
Why did this student do so well ?
How did they do in their other subjects ?
PARIS Software
PARIS is …..•Software to install and use in school
•An interactive reporting tool
•Included free with Alis / Yellis / MidYIS
PARIS provides …..•Student level reports
•Subject Level reports
•Institution Level Reports
PARIS analyses …..•Potential Performance
•Intermediate Performance
•Actual Performance
Attitudes
There is more to school / college than exams….
• Student attitudes• Student Welfare & Safety• Non-academic activities• Support• Social and personal development• Parental Survey• Induction Survey
Attitudinal MidYIS INSIGHT Attitudinal Yellis Full ALIS
Self Evaluation (Every Child Matters)
Try it yourself at www.intuproject.org/demos
Other Issues
Points to mull over…
• Independence – no agendas
• Transparency of Analysis
• Self Evaluation
• Straightforward and standardised administration
• Prompt Feedback
• Full working hours phone / email support
• Student focus
• Replacement for KS3
• Innovative online adaptive testing available – student experience
• Longitudinal analysis with appropriate error backgrounds
• Non-curriculum embedded baselines available
• Attitudinal surveys available – Every Child Matters…
