Assessing and Assuring Graduate Learning Outcomes

National ForaBrisbane Melbourne Sydney Adelaide

Perth

audio link to this presentation

AAGLO National Fora 2012

WELCOME●

Forum program● 9:15 Registration ● 9:30 Opening and brief overview of AAGLO Project● 9:45 Keynote address

Professor Trudy W. Banta● 10:45 Questions● 11:10 Morning tea● 11:30 Presentation of AAGLO interview findings● 12:00 Workshop

Response to issues raised (15 minutes - Trudy Banta)● 1:30pm - 2:15pm Lunch

Forum ObjectivesFor participants to engage with colleagues

in:● discussion of practice and issues in the assessment and assurance

of graduate learning outcomes in the Australian higher education context

● developing informed opinion to contribute to institutional decision-making at various levels

● forming collaborations for further investigation and innovation in this area.

●

The AAGLO PROJECT

● Funded in 2010 under the ALTC Strategic Priority Project Scheme to investigate● The types of assessment tasks most likely to

provide convincing evidence of student achievement of or progress towards graduate learning outcomes? and,

● The processes that best assure the quality of assessment of graduate learning outcomes.

●

● Project team:● Simon Barrie (The University of Sydney)● Clair Hughes (The University of Queensland)● Geoffrey Crisp (RMIT)● Anne Bennison – Project Manager (The University of Queensland)

● Timeline: Jan 2011 – August 2012● International reference group● Broad in scope and range of activities

Project website http://www.itl.usyd.edu.au/projects/aaglo/

Project activities and outcomes to date

Activities Outcomes

Situational analysis “Related projects” identified and documented, communication with project and institutional leaders

Literature review Summary papers1: The ALTC AAGLO project and the international standards agenda 2: Assurance of graduate learning outcomes through external review 3: Challenges of assessing Graduate Learning Outcomes (GLOs) in work-based contexts 4: Standardised testing of graduate Learning Outcomes in Higher Education 5: Approaches to the assurance of assessment quality6. Assessment policy issues in the effective assessment and assurance of GLOs

Endnote library

Consultation with reference groupVisits to international centres of excellenceConference roundtables

Participation in national debates

Response to government discussion paper on the Assessment of Generic Skills Co-authorship of “Mapping learning and teaching standards in Australian Higher education: An issues and options paper”

Interviews Findings

Keynote

Trudy Banta - pioneer in outcomes assessment● Professor in Higher Education ● Senior Advisor to the Chancellor for

Academic Planning and Evaluation at Indiana University - Purdue University (IUPUI)

● founding editor of “Assessment Update” ● numerous publications on outcomes

assessment.http://www.planning.iupui.edu/103.html

Questions

AAGLO INTERVIEW FINDINGS

audio link to discussion of project findings

AAGLO Interviews● Ethical approval ● Telephone interviews● Participants selected through LTAS project and in consultation with LTAS

scholars● 84 invitations to academics across 7 disciplines (Accounting/Business:

Chemistry: Drama and performance: Engineering: History: Law: Veterinary Science) representing LTAS demonstration clusters and range of university types and locations throughout Australia

● 48 interviews conducted of approximately one hour (2 partial)● broad coverage of assessment and assurance practice and issues● Nvivo software for analysis and storage of data.

The disciplines we selected were ...

We interviewed ......

● 30 male and 18 female academics● academics from 26 institutions● 15 Deans /Associate Deans● 12 with program-level responsibilities● 36 with single course responsibilities● 41 who taught in one or more courses ● 17 involved in disciplinary initiatives around assessment and standards such

as LTAS project ● 10 involved in other national projects● 3 LTAS Discipline Scholars ● 4 Quality Verification System (QVS) and 2 other external reviewers● 4 past or current members of disciplinary accreditation panels● several academics who had published in this area

Course levels

● L1 - 13● L2 - 6● L3 - 14● L4 and above - 7● Masters - 5

TASKS NOMINATED AS APPROPRIATE TO THE ASSESSMENT OF GLOS

Key assessment tasks by discipline

Business

Chemi

stry

Drama Engineering

History Law Vet Scienc

e

TOTAL

Critical review or essay

2 0 1 1 5 4 1 14

Examinations 0 0 0 0 0 1 4 5Oral presentation 5 1 1 3 0 1 0 11Performance 0 0 5 0 0 0 0 5Reflective piece 1 0 1 2 0 3 0 7Report

6 6 1 6 0 1 1 21

Tutorial and rehearsal activities

1 2 3 1 0 3 1 11

Work placement 1 0 0 0 0 2 0 3Working demonstration

0 0 0 3 0 0 0 3

Other 0 0 1 2 1 2 3 11 Multicomponent tasks

5 0 1 5 1 4 1 17

GLOs assessed using nominated tasks

Other task features

Active student role

Task relationship patterns within a course

Cumulative (a series of related tasks combined as a single product) – 9Linked – 15 (successful completions of a task indicated likelihood of success in following tasks)Repetitive -3 (same task repeated several times to develop expertise)Independent -16 (different tasks assessed different components of a course)

Effective task characteristics● Multiple, related stages● Aligned with course learning objectives – incorporation of

TLOs such as self-organisation, management, lifelong learning: reflect on social, cultural and ethical issues: apply local and international perspectives; plan ongoing personal and professional development.

● Blurred distinction between learning and assessment activities

● Activities and text types characteristic of profession● Authentic contexts, roles and audiences

● 12 real-life● 25 lifelike (definitional range)

● Careful group task design, management and grading● Active role that developed student capacity for self-

assessment and self-directed learning

HOW IS TASK QUALITY ASSURED?

Task quality assurance practicePre-implementation● Assessment policy● Other related policy (e.g. Quality

Assurance)● Mapping of program curriculum inputs

(25) or program assessment (5) ● Formal approval processes for new

and revised assessment by variously titled committees

● Course level (3) ● Program level (14)● Faculty or school level (26)● Institutional level (8) ● Multiple level (15)

● Some approval for examinations only● Informal only (5)

Post-implementation● Formal evaluation processes (24)

incorporating:● review of student satisfaction surveys● monitoring by boards of examiners or

other committees● audits and reviews● documentation and reporting of

responsive action by course and program coordinators and sometimes individual teaching staff.

● Student representation on faculty TL Committees (6)

● Response to student complaints (1) ● Informal only (6)

● ●

Assuring task quality

● Approval from a whole-of-program perspective● Approval for significant change as well as for new

assessment tasks● Effort spent prior to implementation to save effort after

implementation● Where multiple approval is required at least one level

provides feedback beyond policy compliance● Consequential review and evaluation procedures –

action required and reported● Institutional data collection and reporting support the

evaluation process● Inclusive – all have some level of responsibility for

assessment quality●

HOW IS THE QUALITY OF TASK JUDGEMENTS ASSURED?

The basis of judgements

● Course LOs based on institutional graduate attributes (28), personal experience (17) and accreditation requirements (12)

● Common practice to provide criteria with marks, criteria and standards rubrics or marking guidelines

● Links between wording of course LOs and assessment criteria often unclear

●

Assuring standardsPre-judgement - (calibration)Examples

● Workshop for staff to induct them into the standard expected for the award of different grades

● Project work is required at each level of the program with about 70 academics involved in the assessment process. As part of their induction they are provided with a training session during which everyone marks particular group reports from previous years and displays their mark on yellow paper on the reports around the room to enable them to compare their standards with those of others

● Preliminary marking of selected papers, discussion of the application of criteria and standards prior to marking of remainder of papers

● Much marking is undertaken by sessional staff. They are gathered together and the criteria are explained. The unit team pick out a small number of assignments randomly to mark and discuss. After achieving consensus these are used as exemplars to brief casual staff who will be doing the marking.

● Coordinator makes judgements of performance in collaboration with tutors.

● See also AAGLO Summary paper 5

Post-judgement – (consensus moderation)● No moderation rare and usually if

only single marker ● Moderation could be informal.

● The teams marking the assignment often sit in the same room to mark they don’t have to but normally do so as this is another opportunity for informal moderation.

● Consensus moderation most common approach (85 comments), e.g.

● discussion to reach agreement● double marking● random checks by coordinator

● Some instances (5) of normal distribution requirement with rescaling of ‘outliers’ or justification required

Sadler, D. R. (2012). Assuring comparability of achievement standards in higher education: From consensus moderation to calibration. Manuscript submitted for publication.

Assuring judgement quality

● Shared standards at program and course level● Effort spent to establish standards prior to

judgements to save effort after judgements have been made

● Criteria and standards basis for both assessment judgements and moderation

● Inclusive – all have some level of responsibility for assessment judgements including casual staff

● Resourcing to support effective calibration and moderation processes – rescaling cheaper but less effective as professional development

●

HOW IS STUDENT PROGRESS REPORTED ACROSS THE YEARS OF A PROGRAM?

Recording student GLO progress through a

program● Few examples of progressive recording of student GLO development ● Most common was aggregation of course grades in summary numerical

forms such as those required for progressive GPA calculation ● Some year level (horizontal) approaches● Mapping of inputs on assumption that coverage of GLOs in combination

with aligned assessment a logical proxy measure of progress. Challenged in institutions with standardised grade cut-offs such as 50% “Pass” grades.

● 3 reports of informal approaches with small student cohorts (e.g. team meetings)

● Reservations about ePortfolios effectiveness as practice inconsistent● Most reported monitoring student progress as a current priority – wait and

see attitude to possible TEQSA requirements

●

CAN YOU IDENTIFY EXAMPLES OF QUALITY IMPROVEMENT RESULTING FROM QUALITY ASSURANCE PRACTICES?

Quality improvement

Examples● Nomination of task role and audience for report

after participation in ‘Achievement Matters’ project● Lecturer feedback more challenging after

discussion and observation of feedback provided by colleagues

● Tutor provision of annotated samples of work to students to facilitate understanding of criteria and standards

All example attributed to quality assurance processes that encouraged and facilitated dialogue with colleagues

●