-
7/28/2019 Cambridge University Letter on AS Levels
1/3
Dr Mike Sewell
Director of Admissions for the Cambridge Colleges
31 May 2013
Kevin Brennan MP
House of Commons
London
SW1A 0AA
Dear Mr Brennan,
As promised, we should like to offer some further reflection on
the analysis put forward to you byDavid Laws MP in his letter of 14
May.
An initial analysis of the DfE study suggests that this could be
a case of policy-based evidence.No attempt appears to have been
made to recognise the effect of the variables at any level ofdetail
(as far as can be determined given the lack of information about
the base data employed),and that in turn suggests that the original
proposition being explored was either poorly consideredor loaded
towards a particular result or, more charitably, that the raw data
the researchers hadto work with was simply not up to the job.
In our judgment the study has three fundamental design
flaws:-
- The authors choose two oversimplified measures of GCSE and AS
attainment, andcompare them to a crude measure of degree success.
Selective HEIs do not admit on a chanceof getting a II.1. They seek
to admit the students who are likeliest to perform really well
whichmeans a good II.1 (65%+) or First.
- Extrapolating from logistic regression in the manner in which
this study does is highlyproblematic, since the binary outcome
treats the lowest II.1 as equivalent to the highest First,
andsimply asks whether AS and KS4 performance can correctly predict
which side of the II.2/II.1divide a student will end up on. A
linear regression model incorporating the full range of
degreeresults (including failures) allows for finer distinctions
between student performances, and sowould have been much more
appropriate to the question of whether AS Levels provide
universityadmissions officers with useful additional information
about student potential, over and aboveGCSE.
- The model fails to adjust for the fact that, at any given
institution, students will tend to have
roughly similar levels of prior attainment. This is because
students in large part select HEIs basedon their own level of
academic attainment. Since, as the study admits, a II.1 is a
relative measureof performance within an institutions cohort rather
than a nationally calibrated standard, lumpingall universities
together, as this analysis does, is destined significantly to
weaken correlations. Thisis because, while students obtaining II.1s
at a highly competitive-for-entry university might averagegrades
AAAA at AS Level, students obtaining II.1s at a less
competitive-for-entry university mightonly average grades BBBB. A
serious and rigorous analysis would have run multiple
separateregression models to analyse the relationships within each
institution.
Cambridge Admissions Office
Fitzwilliam House
32 Trumpington Street
Cambridge CB2 1QY
Tel: +44 (0) 1223 333308
Fax: +44 (0) 1223 746868
Email: [email protected]
Cambridge Admissions Office
-
7/28/2019 Cambridge University Letter on AS Levels
2/3
Page 2 of 3
The reporting of the studys findings is incomplete in a number
of respects. In particular the reportprovides very little
information about the factors considered when securing base data
andconducting the analysis.
A complete report should, for example, provide information on
how the analysts counted GCSEresults and AS results: which grades
did they include, A*-C, or A*-F? Did they exclude subjectslike
General Studies and Critical Thinking which are not accepted by
most selective universities?Did they include short courses, and did
they cap the total number they counted? If they used apoints system
for each grade, what points did they allocate to each grade?
Similarly for AS level, the key information should include
whether capping was applied and thepoints allocated per grade.
Finally, does this analysis include degree failure or drop-out?
A good sense of what the data being analysed actually is and how
it is being used is essential to asound statistical report. We
therefore requested the missing information, and the data-set,
fromthe DfE in order to carry out our own modelling.
The DfEs reply confirmed that the analysis is based on uncapped
average scores in both cases,that KS4 qualifications are being used
rather than GCSE in isolation, and that no subjects havebeen
excluded.
The University of Cambridge uses an average score calculated
across the best three, or threemost relevant, AS UMS scores
(depending on whether the application is for arts or sciences),
andexcludes General Studies and Critical Thinking.
While we await a decision on our request for the full data-set,
we have run a similar logisticregression model to that to which Mr
Laws refers using our own data, and using the yardstick of agood
II.1/First, which around 50% of our students achieve.
In this case we see a clear effect when adding UMS to GCSE: the
predictive validity of the modelis improved and the GCSE
coefficient is rendered insignificant. That is, if we have UMS,
GCSEdoes not add value, but if we already have GCSE, UMS adds value
over GCSE while alsoeradicating any value GCSE had.
So far as Mr Laws argument that GCSE and AS tell us the same
thing goes, we can show thatUMS tells us everything that GCSE does,
and much more besides.
Our position remains that AS UMS is key to accurate and fair
university admissions. If otheruniversities also adopted UMS as a
key selection-tool, we believe that they too would find that itwas
much more valuable than GCSE.
We hope that this is helpful.
As ever, if you would like to discuss the issues in more detail,
please dont hesitate to get in touch.
Yours sincerely,
Jon Beard Dr Mike Sewell
Director of Undergraduate Director of AdmissionsRecruitment for
the Cambridge Colleges
cc:
David Willetts MP, Minister of State for Universities and
Science, Department for Business,
-
7/28/2019 Cambridge University Letter on AS Levels
3/3
Page 3 of 3
Innovation & Skills, 1 Victoria Street, London SW1H 0ET
Michael Gove MP, Secretary of State for Education, Department
for Education, Sanctuary
Buildings, Great Smith Street, London SW1P 3BT
David Laws MP, Minister of State for Schools, Department for
Education, Sanctuary Buildings,Great Smith Street, London SW1P
3BT
Dr Julian Huppert MP, 16 Signet Court, Swann Road, Cambridge CB5
8LA
Dr Wendy Piatt, Director General, The Russell Group, 1
Northumberland Avenue, TrafalgarSquare, London WC2N 5BW
Nicola Dandridge, Chief Executive, Universities UK, Woburn
House, 20 Tavistock Square, LondonWC1H 9HQ
Mary Curnock Cook, Chief Executive, UCAS, Rosehill, New Barn
Lane, Cheltenham GL52 3LZ
Janet Graham, Director, SPA, Rosehill, New Barn Lane, Cheltenham
GL52 3LZ
