Assessment Matters …

Monday December 5th 2011Student Union Academic Council

Andy Lloyd, Assessment Project Manager

Assessment Matters …

• Do you know how your degree result and the classification will be worked out?

• Do you care how your degree and its classification will be worked out?

• Do you want to be able to work out what class of degree you will be getting from your module marks?

‘Assessment defines what students regard as important, how they spend their time and how they come to see themselves as students.’

George Brown,LTSN Assessment Series 3A guide for lecturers

Variations across the University

• To date there have been about 300 different sets of award rules across the University;

• Considerable variety in both processes and procedures; e.g.– Rounding of marks at different

levels in different subjects;– Weightings within and

between modules;– Contributions from different

years;– Examining Boards;– Extenuating circumstances;– The operation of ‘discretion’.

Why we are where we are?

• Senate Regulations provide a framework in which schools define their own rules;

• Schools have developed a myriad of variations – often to ensure students are not disadvantaged;

• The rationale for differing approaches to regulations has become obscure. (Universities UK 2007, p. 28)

• Where institutional audit reports have noted that institutions are operating multiple sets of assessment and classification arrangements, they have generally tended to recommend measures to reduce the number of such arrangements; (QAA 2006, p. 2)

“Students value highly processes that enable them to understand better the required standards and receive timely input on their progress towards those standards”

(Carless, 2004)

[http://p.ondero.us/funny-exam-answers/]

Consistent rules to govern modular taught programmes (i.e. not MEDIC and DENTL undergraduates)

• The ways in which students progress between the different levels in their programme;

• The ways that marks are rounded (modules, final mark);

• The rules that govern ‘condonement’;• The introduction of anonymity at

Examining Boards;• The processes used to arrive at final

awards;• The award of ‘Merit’ and ‘Distinction’

to taught postgraduate students.

Approved by Senate 30/11/11

“Anonymous marking for exams and coursework is crucial to ensure equality and fairness for students.”

(NUS)

Timetable for implementation of the new award rules

• It is planned that …• Modular Taught Postgraduate

programmes will adopt the new rules in session 2012/13;

• Modular Undergraduate programmes will adopt the new rules in session 2013/14;

Why the Urgency?• Benefit to students• QAA Institutional Review• ‘Not in the Guard’s van but not in the

Vanguard’

What impact might this have on current students?

• Taught postgraduates– Very few current students will be

effected;– Part-time students;– Programmes that operate for longer

than 12 months.

(About 1,000 students in total)• Undergraduate students

– 1st year students in Psychology;– 2nd year students on 4-year taught

Master’s Programmes (e.g. MEng, MPharm etc.);

– 2nd year students who take a year-abroad or a year in industry.

(About 1,300 students in total)

What difference might this actually make?

VERY LITTLE• Research findings;

– Published literature;– Outcomes from modelling;

• A hypothetical question– Assessments and student

behaviour will change;– The ways in which academic

staff approach marking will change.

[http://p.ondero.us/funny-exam-answers/]

Effects of the award rules on degree classifications(Outcomes from the literature)

• “The adoption of a reduced number of degree algorithms is unlikely, in itself, to have a significant influence on the spread of classifications, given the wide range of other factors that have a statistically significant impact in this area.” (Lloyd 2010, p. 1)

• “Changing the weighting seems to have less effect than the dropping of modules in the classification algorithm.” (Yorke … et al. 2004, p. 411)

• Altering the weighting of Year 2/3 from 50/50 to 25/75 would change around 10% of classifications by one band, the majority upwards; (Yorke … et al. 2004, p. 411)

[http://p.ondero.us/funny-exam-answers/]

0 10 20 30 40 50 60 70 8020.0

30.0

40.0

50.0

60.0

70.0

80.0Modelling the Effect of Non-Final Year Contribution on Final Mark for Engineering

Students Graduating 2009/10

Original Mark

70:30

60:40

Student

Fina

l Mar

k (%

)

There is no statistically significant difference between these data-sets.

Paired, two-tailed student T-tests gave P-values of greater than 0.05 when comparing the original marks with the 70:30 or 60:40 non-final year contribution

Effects of the award rules on degree classifications(Outcomes from the modelling undertaken)

• Altering the non-final year contribution had no significant effect on the Final Weighted Mark in CARBS, ENCAP or ENGIN;

• The proposed new algorithm results in minimal perturbation to the original degree classifications awarded irrespective of whether the non-final year contribution used is 70:30 or 60:40;

• Only ~10% of students in CARBS, ENCAP and ENGIN could potentially have been awarded a different degree classification using the proposed new algorithm. A proportion of these students had Extenuating Circumstances which may have influenced their original degree classification;

• An increased number of students would fail to progress using the new algorithm as they are currently carrying more than 20 failed credits. However, this outcome is not predictive as in reality these students would have resit in order to allow them to progress;

• In both CARBS and ENCAP the distribution of ethnicity, disability and gender are extremely similar for students whose degree classification would change using the new algorithm as in the general population in those Schools (in ENGIN the student numbers are too low for this type of analysis to be meaningful);

Overall, modelling the effect of the proposed changes to the calculation of degree classifications awarded has so far indicated minimal perturbation.

• “These changes are in students’ interests” – Prof. Jonathan Osmond (Senate 30/11)

• You will know what marks you will need to get to attain a specific outcome;

• You will NOT know how your marks would have been combined under the previous rules and what discretion the Examining Board might have used to arrive at the final outcome;

• Assessment methods in individual modules may change;

Effects of the award rules on degree classifications(Assessment and student behaviour may change)

• Schools are being encouraged to make better use of the ‘full mark range’;

• Schools are being encouraged to introduce ‘categoric’ marking (e.g. 62, 65, or 68 etc.);

• Improved moderation of marks (Working with schools on blind double marking etc.);

• Clearer marking criteria.

Effects of the award rules on degree classifications(Marking and staff behaviour may change)

• Taught postgraduate students in 2012/13;

• Undergraduate students in 2013/14;

• Writing to all students who will be effected by the new rules to notify them of the proposed changes, how they will benefit them;

• Writing to all students who will NOT be effected by the new rules to notify them of the potential impact they will have should they e.g. change programme, repeat a year, take an ‘Interruption of Study’ etc.

New Award Rules (Supporting their implementation)

What can student reps do?

• Put this on the agenda at the next staff:student panel meeting;

• Provide information / feedback to your fellow students;– Meetings– Emails

• Help manage the “grapevine”;

• Send any questions that you have to either Sam or myself;(LloydA@Cardiff.ac.uk)