Proving to improve: how Accounting uses double blind peer review to

assure TLOs

CUBEnet Forum

Canberra

13 Dec 2013

2

http://vimeo.com/channels/testingtogether/66041940

3

Scott Ray 4/4/09 http://www.flickr.com/photos/15075617@N05/3418528723

1.2 There are robust internal

processes for design and approval

of the course of study which take

account of external standards

and requirements, e.g. published

discipline standards …, and

comparable standards at other

higher education providers.

Accounting graduate

Knowledge

Application

JudgementCommunication

& Teamwork

Self management

Bachelor graduates are able

to justify and communicate

accounting advice and ideas

in straightforward

collaborative contexts

involving both accountants

and non-accountants

Master (entry)

diverse

Accounting graduate

Knowledge

Application

JudgementCommunication

& Teamwork

Self management

Bachelor graduates are

able to exercise judgementunder supervision to solve routine accounting problems in straightforward contexts using social, ethical, economic, regulatory and global perspectives

Masters minimal

emerging

/advanced

complex

Martin Morris 20/2/12 http://www.flickr.com/photos/martin55/6929721749/

5.5 The academic

standards intended to be

achieved by students and

the outcomes actually

achieved by students in

the course of study are

benchmarked against

similar accredited courses

of study.”

“external AoL”

7

Assess Enter Compare

Pre-F2F

F2F

Consensus Agree

Post-F2F

Apply

Calibrating the standard

Confirm

Calibration – Task validityWritten communication

Individual confidence pre-workshop

Individual results pre-workshop

• Min & max (n=26)

• Mean 1 SD

Group results at workshop

• Small groups (n=5)

• Consensus

91%

NA A

NA A

Calibration – UG student 1Written communication

Individual confidence pre-workshop

Individual results pre-workshop

• Min & max (n=26)

• Mean 1 SD

Group results at workshop

• Small groups (n=5)

• ConsensusNM M

NM M

78%

....S2, S3, S4, S5

External calibration &

review

Provider selects reviewers

Project selects standards & degree for

review

Provider selects tasks

Project generates random student

ID to submit

Provider submits de-identified data

Reviewers calibrate

3 anonymous reviewers submit (Project allocates

2 externals)

Provider uses results

(Project ensures consensus)

Feedback to submitting institution

Summary of results

• Using the results

• Next steps

Appendix 1 – detailed comments

1

Feedback to submitting institution

Your uni predicts that the BSc (BioMed) is likely to be selected by TEQSA as a sample degree for scrutiny in 2015. How would you demonstrate external AOL?

1. Rely on sample data reviewed as part of QVS

2. Select benchmarking partner dept to assess sample

assignments & exams in final UoS for 5% students in each

grade band

3. Pay an external academic to assess

4. Lobby ACDS to establish a national ‘double blind peer

review’ benchmarking process that includes calibration

5. Lobby CUBEnet to do DBPR benchmarking & calibration

6. Do nothing

Achievement Matters: External Peer Review of Accounting

Learning Standards

Australian Business Deans Council

Institute of Chartered Accountants in Australia

CPA Australia

Office for Learning and Teaching

Project team : Adelaide (B.Howieson), Deakin (K.Watty), RMIT

(B.O’Connell, P.de Lange), Sydney (M.Freeman), UWA (P.

Hancock), UWS (A.Abraham)

Participants from 17 providers

Websites

achievementmatters.com.au

disciplinestandards.pbworks.com

Disclaimer

The views expressed in this presentation do not

necessarily reflect the views of any of these stakeholders

Thank you