Formative thresholded assessment: Evaluation of a faculty-wide change in assessment practice

Sally Jordan and Janet HaresnapeFaculty of ScienceThe Open University, UK

Assessment in Higher Education Conference, June 2013

This is a practice exchange…

• So please interrupt

Our plan

Previous practice

Drivers for change

What do we mean by formative thresholded assessment?

Evaluation and stumbling blocks

Early findings

The UK Open University• Founded in 1969;• Supported distance learning;• 150,000 students, mostly studying part-time;• Undergraduate modules are completely open entry, so

students have a wide range of previous qualifications;• Normal age range from 18 to ??• 10,000 of our students have declared a disability of

some sort;• 25,000 of our students live outside the UK.

Historic OU Science Faculty practice

• Tutor-marked assignments (TMAs) and sometimes interactive computer-marked assignments (iCMAs) combine together into overall continuous assessment score (OCAS);

• Examination and/or end-of-module assessment (EMA) gives overall examination score (OES).

OCAS is integrated and interactive

Continuous assessment• Has a useful pacing function;

• Students are concerned about the minutiae of the grading;

• Even though for most students their final grade is determined by their performance in the examination or end-of module assessment;

• A considerable amount of time and effort goes into producing new TMAs for each presentation of each module;

• And we don’t always get it right.

Drivers for change• “when assessments serve both formative and

summative purposes… formative work will always be threatened due to the dominance of summative requirements” (Brearley & Cullen 2012 with echoes of Black & Wiliam 1998, Gibbs 2006, Snyder 1971)

• Lack of alignment of tutor and student understanding of our assessment strategies.

• Lack of alignment of tutor and student understanding of the purpose of continuous assessment.

• Saving time and money.

Formative thresholded assessment – two models

• OCAS is formative, but students have to demonstrate engagement by scoring more than 30% in x out of y TMAs and iCMAs; final score is determined by OES alone.

• Students have to reach threshold (usually 40%) for OCAS (overall), but their module result is then determined on the strength of their OES alone.

Risks

• Students will feel insufficiently prepared for the examination or end-of-module assessment;

• Students will not engage sufficiently in the formative thresholded OCAS and so will lose valuable formative opportunity;

• Assignments will be reused, so there may be more plagiarism. For formative assessment, does this matter?

DISCUSSION POINT – what do you think about our move to formative thresholded assessment?

Evaluation

• A series of practitioner-led mini projects;• Strengths: we are practitioners and so close to the

student experience; we will help each other;• Dangers: we are practitioners so we have considerable

pressures of other work; we won’t deliver: we will have insufficient expertise.

DISCUSSION POINT: Does anyone have experience of research or evaluation carried out in this way?

Evaluation• Quantitative and qualitative;• Janet – looking at behaviour of students who are taking

two modules, one with summative and one with formative thresholded assessment;

• Lynda – looking at two new modules with subtly different assessment strategies;

• iCMA usage can give a ‘signature’ of student engagement;

• TMA submission rates (before and after the change of assessment strategy);

• What do students think?• What do tutors think?

Evaluation - complications• We need to compare current with historic data; we need

to compare current with historic student perceptions;

• Confounding variables: other changes to assessment strategy; changing student population; over-committed students.

TMA submission rates

How many TMAs did students omit

Correlations between exam score and omitted TMAs

• 2011J – for students who submitted all TMAs, mean exam score = 57.5%; for those who did not submit all TMAs, mean exam score = 41.3%;

• 2012J – for students who submitted all TMAs, mean exam score = 53.4%; for those who did not submit all TMAs, mean exam score = 44.4%;

• Should this worry us? (DISCUSSION POINT)• Not a causal relationship.

Engagement with iCMAs

Engagement with iCMA feedback – same question, different mode of use

Similar assignment, different students

Outcomes from the evaluation• Overall findings (and thus our plans for the future) – still

too early to say; • A beneficial side effect is that we are thinking about our

assessment strategy for current and future modules in a more coherent manner.

Sally JordanSenior Lecturer and Staff Tutor

Deputy Associate Dean, AssessmentFaculty of Science

The Open University

sally.jordan@open.ac.uk

blog: http://www.open.ac.uk/blogs/SallyJordan/