Grading criteria

and marking schemes

Liz Norman

Massey University

Australian and New Zealand College of Veterinary Scientists

Examiner Workshop, 14-15 February, 2015

Validity again…

• The questions must elicit the behaviour we want

to evaluate

• Different markers need to award similar/scores

for the same candidate response

• Markers need to reward features we want to

evaluate and not reward features we do not

want to evaluate

Purpose of marking schemes

• To help you during Q writing

– What content is important

– Whether the Q asks what you intended it to ask

– Whether it is do-able in the time available

• To help you during Q marking

– helps you decide whether an answer is good enough

to be awarded a mark

– facilitates reliable and fair marking

Types of marking scheme

1. Model answer – “ideal” answer

2. Point-based schemes

3. Criteria- & level-based schemes

4. Schemes with incorporated principle(s) for

discriminating levels

Model answers

• Not recommended as a sole component of a

marking scheme

– Usually more than would be expected to be given by

any candidate

– No guidance on how to assess alternative answers to

the model provided

– No guidance on how to award marks

• Can be a useful adjunct to a marking scheme

Point-based schemes

• Points for each objectively identifiable content

point

• Does not indicate the relative importance of the

points it awards

• Sum can be more or less than the whole

• Rewards quantity not quality

Prestructural

Unistructural Able to identify, list, name, enumerate but does not describe,

explain, relate or elaborate multiple aspects of a response

Multistructural

Relational

Extended

abstract

Quantitative change

Qualitative change

Criteria- & level-based schemes

• Criteria – different dimensions of performance

• Level – different quality/standards of

performance on a given criteria

Criteria- & level-based schemes

• Generic vs specific for the Q

• Explicit vs implicit weighting

Standards

Very poor Poor Fair Excellent C

rite

ria

Analysis and

interpretation

of results

Total 8 marks

0-2 marks

Interpretation not

provided or incorrect.

2-4 marks

Lacks one or more key

elements.

4-6 marks

Adequate interpretation

that addresses key

elements. Misses nuances

of interpretation or

uncommon differentials

6-8 marks

Thorough accurate

interpretation of results.

Well justified and

appropriately prioritised

list of differentials.

Quality of

planning

Total 8 marks

0-2 marks

No plans provided or

plans not appropriate

or dangerous

2-4 marks

Plans miss some key

aspects or overly

general

4-6 marks

Adequate plans that

address all key

differentials. Some

displaced in priority or not

pragmatic

6-8 marks

Thorough detailed and

well-prioritised and

pragmatic plan that

addresses all defined

differentials.

Knowledge of

current

literature

Total 5 marks

0-2 marks

Little or no literature referred to or incorrectly

referred to.

3-4 marks

Answer refers to some of

the key literature

4-5 marks

Answer refers to current

literature including

controversies and

comparative work from

other species.

Logical

presentation

Total 4 marks

1 mark

Answer is disorganised

and includes a large

amount of irrelevant

material

2 marks

Answer is somewhat

disorganized and

includes some

irrelevant material

3 marks

The answer is relatively

well organized and

contains little irrelevant

material.

4 marks

The answer shows a high

degree of logical thought

and well-constructed

argument.

Analytical vs holistic schemes

• Both are valid

• Analytical (criteria scored separately)– Better agreement between examiners

– Insufficient criteria

– Overlapping criteria

– Really just lots of smaller holistic decisions

• Holistic (scored as a whole)– Don’t straightjacket examiners

– Challenging, especially for longer answers

– Less agreement between examiners

Prestructural Question may be rephrased as the answer; almost completely

misses the point of the question.

Unistructural Able to identify, list, name, enumerate but does not describe,

explain, relate or elaborate multiple aspects of a response

Multistructural Able to list as well as describe distinct aspects of a response (such

as being able to describe aetiology, clinical features, management

of thrombotic stroke) but unable to explicitly explain causes for

observations; unable to present cause-effect relationships.

Relational Able to describe multiple aspects of a process and additionally

explain or elaborate observations into cause-effect relationships;

able to compare similarities and differences between apparently

distinct phenomena. This level is taken as suggesting that the

learner has understood.

Extended

abstract

Highly developed; able to explain mechanisms of phenomena and

apply this information to a novel context — able to develop novel

hypotheses, theories, and deduce principles; creative thinking.

SOLO levels in marking schemes

Prestructural The task may be engaged, but the student is distracted or

misled by irrelevant aspects

Unistructural The student focuses on the relevant domain and works with a

single aspect

Multistructural The student provides correct material with discrete, separate

pieces of information that may be combined to provide a

composite picture

Relational The student offers an integrated understanding of the

information. The whole has a coherent structure and meaning

Extended

abstract

Abstract general principles or hypotheses are provided

Scholten I, Keeves JP, Lawson MJ. High Educ 44:233–255, 2002.

Q examples at different SOLO levels

Inviting a unistructural response:

• Identify descending motor pathways that primarily mediate upright posture and reflex postural adjustments.

• List four examples of long acting calcium channel blockers.

Inviting a multistructural or even a relational response:

• Describe the structure of the eukaryotic cell membrane.

• Describe the clinical features of ischemic stroke.

Inviting a relational response:

• Explain the mechanisms of each of the following in a patient with cirrhosis of the liver: ascites, splenomegaly, hematemesis, and neurologic deterioration.

Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149

Incorporating principles/rules

• Ideal is a specific criteria- & level-based

schemes with incorporated principle(s) for

discriminating levels

AQA GCSE Science A Physics 1 Foundation Tier Physics 1F Specimen Mark Scheme

http://www.aqa.org.uk/subjects/science/gcse/physics-4403/past-papers-and-mark-schemes

Incorporating principles

Calais has a warmer winter and a cooler summer than Wroclaw. Explain why. (3 marks)

Marking scheme: Looking for answers related to distance from the sea therefore latitude is not credited.

– Land heats up quicker than sea (1 mark)

– A clear distinction between land and sea heating (2 marks)

Ahmed & Pollitt (2011) Improving marking quality through a taxonomy of mark schemes. Assessment in Education:

Principles, Policy & Practice 2011;18:259-278.

Incorporating principles

Marking scheme excerpt:

Discussion should focus on strategies adopted to

ensure sustainability and an evaluation of these with

regard to whether or not or to what extent the Sahel can

be sustainably managed. The discussion will depend on

the content and whether the overall view is optimistic or

pessimistic.

AQA (2013) General Certificate of Education (A-level) Geography Unit 1: Physical and Human Geography

http://www.aqa.org.uk/subjects/geography/a-level/geography-2030/past-papers-and-mark-schemes

Quality vs quantity

• More complex and unstructured the Q the more

assessing quality not quantity

• In very constrained tasks only judging how

correct the answer is

• In very open tasks, “correctness” is less

important and its quality that's judged instead

Prestructural Question may be rephrased as the answer; almost completely

misses the point of the question.

Unistructural Able to identify, list, name, enumerate but does not describe,

explain, relate or elaborate multiple aspects of a response

Multistructural Able to list as well as describe distinct aspects of a response (such

as being able to describe aetiology, clinical features, management

of thrombotic stroke) but unable to explicitly explain causes for

observations; unable to present cause-effect relationships.

Relational Able to describe multiple aspects of a process and additionally

explain or elaborate observations into cause-effect relationships;

able to compare similarities and differences between apparently

distinct phenomena. This level is taken as suggesting that the

learner has understood.

Extended

abstract

Highly developed; able to explain mechanisms of phenomena and

apply this information to a novel context — able to develop novel

hypotheses, theories, and deduce principles; creative thinking.

Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149

Content vs connections

Lucander et al. (2010). European Journal of Dental Education, 14(3), 145-150.

Writing marking schemes

• Select and organise the criteria/dimensions

• Develop clear descriptions for each

level/standard of each criteria

• Need to think about poor answers as well as

good ones

Revising mark schemes in use

• Hopefully all types of answer are anticipated

• Sometimes though it is not – can indicate unanticipated problems with the Q

• Marking schemes might need revising after first few candidates marked

• HSE should moderate marking and should encourage team members to report marking issues early in the piece.

Key points

Marking schemes

• Give an indication of the key criteria for

discriminating passing answers from failing ones

• Then add in what would be a better performance

and what would be a lesser performance

• Think about both quantitative and qualitative

aspects of an answer

Marking schemes

• Quantitative:

– Which parts of the answer are essential

– How many errors in these parts are tolerated

• Qualitative

– Thinking processes you want to examine

– What connections are essential/important

– How the structure of an answer will help you

differentiate a structured, related, connected thought

process from a multistructural list of unrelated facts

Marking schemes

• Think about how a candidate who includes

all/most of the right stuff but also includes wrong

stuff will be graded

• What it indicates about the knowledge structure

if there are contradictions/irrelevancies

in the answer

Marking schemes

• Keep them brief

• Think about them as specifying principles to

apply rather than exhaustive detail

• Concentrate on specifying the differentiating

characteristics and how much of this there

needs to be

• Don’t use them to educate examiners about the

topic

Next session’s task

Next session’s task:

• For each of your questions:

– Think about how you will discriminate a good from a

poor answer

– Form this into a marking scheme

• Aim is to have a full set of drafted Qs and

outlined marking schemes by 5:30 pm