Deakin University CRICOS Provider Code: 00113Bdeakin.edu.au/learning/enhancing-your-practice/cradle

The ‘unknown unknowns’ of assessment rubrics in practice, policy and research

Associate Professor Phillip DawsonAssociate Director

Centre for Research in Assessment & Digital Learning (CRADLE)Deakin University, Melbourne, Australia

@phillipdawson

Deakin University CRICOS Provider Code: 00113B

Deakin University CRICOS Provider Code: 00113B

A couple of decades ago, rubric began to take on a new meaning among educators. Measurement specialists who scored students’ written compositions began to use the term to describe the rules that guided their scoring. They could have easily employed a more readily comprehensible descriptor, such as scoring guide, but scoring guide lacked adequate opacity. Rubric was a decisively more opaque, hence technically attractive, descriptor. (Popham 1997, p. 72)

Deakin University CRICOS Provider Code: 00113B

Hands up if rubrics make you feel:• Confident• Concerned• Relieved

• Confused• Excited• Bored• …another feeling

you’d like to share?

Deakin University CRICOS Provider Code: 00113B

Pre-1997

19982000

20022004

20062008

20102012

20140

1000

2000

3000

4000

5000

6000

7000

Mentions on Google ScholarMentions in books

Cumulative frequency, 28 April, 2015.

Deakin University CRICOS Provider Code: 00113B

What is a rubric?• A tool used in the process of evaluating

student work, which usually includes:– evaluative criteria– quality definitions for those criteria at

particular levels – a scoring strategy.

Per Popham (1997)

Deakin University CRICOS Provider Code: 00113B

This workshop• Overview of framework• Practice sharing• Scenarios

Deakin University CRICOS Provider Code: 00113B

Framework overview• 14 ‘design elements’ representing choices

made in rubric design• Brief overview of each – stop me if you

want more detail

Deakin University CRICOS Provider Code: 00113B

Specificity: the particular object of assessment

• Generic– Including faculty-wide

or course level• Task-type• Task-specific

Deakin University CRICOS Provider Code: 00113B

Secrecy: who the rubric is shared with, and when it is shared

• Top-secret rubrics• Rubrics returned with marks/feedback• Rubrics shared with task instructions

Deakin University CRICOS Provider Code: 00113B

Exemplars: work samples provided to illustrate quality

What does a good one of these look like?

Why is this one good?

Deakin University CRICOS Provider Code: 00113B

Scoring strategy: procedures used to arrive at marks and grades

Analytic• Adding up• Logic, conditions,

thresholds• Combining scores from

multiple markers

Holistic• Assessor uses all

information to make a single overall quality judgement

Deakin University CRICOS Provider Code: 00113B

Evaluative criteria: overall attributes required of the student

• (usually the leftmost column)• Mandatory (or not?)• Connector between indicators of quality

and task/subject outcomes

Deakin University CRICOS Provider Code: 00113B

Quality levels: the number and type of levels of quality

• Grade levels (P/C/D/HD)

• Learning outcomes taxonomies

• Levels of study

• Fail?• Every criterion at

every level?

Deakin University CRICOS Provider Code: 00113B

Quality definitions: explanations of attributes of different levels of quality

“Mixture of qualitative and

quantitative feedback sources”

Deakin University CRICOS Provider Code: 00113B

Judgement complexity: the evaluative expertise required of users of the rubric

Pass Credit Distinction

Clarity of expression

Bad Acceptable Good

Deakin University CRICOS Provider Code: 00113B

Judgement complexity: the evaluative expertise required of users of the rubric

Pass Credit Distinction

Clarity of expression

Five or fewer grammatical errors

Two or fewer errors

Zero grammatical errors

Deakin University CRICOS Provider Code: 00113B

Users and uses: who makes use of the rubric, and to what end

Teachers• Marking• Providing feedback• Communicating

standards• Understanding standards

Students• Planning work• Understanding standards• Self assessment• Peer assessment• Grade grubbing• Planning improvement to

future work

Deakin University CRICOS Provider Code: 00113B

Creators: the designers of the rubric

• Teachers• Textbook publishers• Internet

communities• Students

• Rubric specialists• Researchers• Professional bodies

Deakin University CRICOS Provider Code: 00113B

Quality processes: approaches to ensure the reliability and validity of the rubric

• Formal reliability and validity processes• Informal review, piloting and revision• Haven’t got time for that, just see how it

goes

Deakin University CRICOS Provider Code: 00113B

Accompanying feedback information: comments, annotation or other notes on student performance

• Stimulus for feedback discussion• Focal point for written feedback• A supplement to written feedback• Replacement for written feedback

Deakin University CRICOS Provider Code: 00113B

Presentation: how the information in the rubric is displayed

Deakin University CRICOS Provider Code: 00113B

Explanation: instructions or other additional information provided to users

• Written instructions• Discussion around rubric in class• Marking workshop• Video explanation

Deakin University CRICOS Provider Code: 00113B

How do you use rubrics?• Do you feel strongly about any of these

elements?• Discuss how you use rubrics in groups of

three or four• Structure your conversation around the

framework

Deakin University CRICOS Provider Code: 00113B

Scenarios – choose one, in groupsRubric advice• 1st year educational

theory course• 1000 students• Lecturer wants your

advice on rubric for 2000 word essay

Faculty rubric policy/procedures• Education faculty at a

large university have contracted you to write rubric policy/procedures

• What would you mandate?

Deakin University CRICOS Provider Code: 00113B

Questions/Comments?Pre-

19971998

20002002

20042006

20082010

20122014

0

1000

2000

3000

4000

5000

6000

7000

Mentions on Google ScholarMentions in books

@phillipdawson