Implementing an Integrated System for Programmatic Assessment of the MBBS Program

A Program of Assessment for a Program of Study

Dr Tracey Papinczak

Quality assurance in education

“the totality of systems, resources and information devoted to maintaining and improving the quality and standards of teaching, scholarship and research, and of students learning experience”

Quality Assurance Agency in UK Higher Education

Quality assurance systemDoes the assessment– Align with best practice?

Have the following been undertaken?– Self-reflection – Examination of student feedback– Peer review– External review

1. Best practice in assessment:Individual instruments

EBP•Reliable •Valid •Objective•Acceptable•Constructively aligned•Equitable and fair•Cost-effective•Time-efficient

1. Principles: Best practice in assessment programs

• Clarify what are expected standards (criteria)• Facilitate deep learning

• Deliver high quality feedback• Positively motivate students

• Support development of learning communities• Involve students in decision making

• Deliver authentic assessment• Encourage reflection & self-assessment

• Be paced to support continuous learning• Have strong validity and reliability for all individual

assessment pieces

2. Self-reflection

Taking time

Reflecting in-

action and on-action

What is working?

What is not

working?

How can we

improve?

Where can we access

support?

3. Student feedback

Student feedback to the Evaluation Unit– Concerns about subjectivity of consultant assessment years 3 & 4– Lack of time to complete some written examinations– Excessive assessment in some rotations– In yrs 1 & 2, too little assessment

Course evaluations administered by TEDI– 2010 SET-C

• greater feedback on assessment needs to be provided – 2008 - 2009 iCEVAL

• LO to assessment match needs greater clarity• Assessment requirements and marking criteria need to be provided at

the start of the course• Assessment needs to be spread across the semester not concentrated

at the end

4. Peer review

4. Peer review

Examples:

• Peer review of examination papers in years 1 and 2

• Peer review of PBL tutors in year 1• Peer review of lecturers in Discipline

of General Practice

5. External Review

Australian Medical Council Review

Lack of centralised control over and management of assessment

Variable quality of assessmentRedundancies and gaps

Lack of examiner training

Planning for quality assurance

Quality Assurance of the Program of Assessment

Purposeful plan

Mapped to aims and

goals

Gaps identified

and addressed

Redundancies removed

Quality of each piece of assessment

EBP in assessment

Best choice

for KSA

Threshold learning outcomes

ALTC – Learning & Teaching Academic Standards project

Health, Medicine and Vet ScienceSix TL outcomes revised through

stakeholder consultationImplications for assessment within all

programs once TEQSA is underway (new Federal Government auditing agency within DEEWR)

Upon completion of their program of study, health graduates at professional entry level will be able to:

• Demonstrate professional behaviours • Assess individual and population health status, and where

necessary, formulate and implement management plans in consultation with patients/clients/carers

• Promote and optimise the health and welfare of patients/clients and populations

• Retrieve, critically evaluate, and apply evidence in the performance of health care activities

• Deliver safe and effective health care in collaboration with other health care professionals

• Reflect on current skills, knowledge and attitudes, and plan ongoing professional development activities.

Assessment in the MBBS Program

Threshold learning

outcomes (once TEQSA established)

Aims, goals and learning

outcomes

UQ Graduate attributes

Graduate attributes

In-depth knowledge & skills in the field of study• A comprehensive and well-founded knowledge in the field of study. • An understanding of how other disciplines relate to the field of study. • An international perspective on the field of study.

Effective Communication• The ability to collect, analyse and organise information and ideas and to

convey those ideas clearly and fluently, in both written and spoken forms. • The ability to interact effectively with others in order to work towards a

common outcome. • The ability to select and use the appropriate level, style and means of

communication. • The ability to engage effectively and appropriately with information and

communication technologies.

Graduate attributesIndependence and Creativity

• The ability to work and learn independently. • The ability to generate ideas and adapt innovatively to changing environments. • The ability to identify problems, create solutions, innovate and improve current

practices

Critical Judgement• The ability to define and analyse problems • The ability to apply critical reasoning to issues through independent thought and

informed judgement • The ability to evaluate opinions, make decisions and to reflect critically on the

justifications for decisions

Ethical and Social Understanding• An understanding of social and civic responsibility • An appreciation of the philosophical and social contexts of a discipline • A knowledge and respect of ethics and ethical standards in relation to a major

area of study • A knowledge of other cultures and times and an appreciation of cultural diversity.

3

7

35

Programmatic assessment

• Focussed on the whole of a program of study• Collaborative & consultative• Charts students progress over time• Involves multiple perspectives• Measures knowledge, skills & affective domains• Has a strong workplace focus• Based on principles of best practice in education• Measures previously underserved areas of assessment• Avoids overlap & over-assessment of commonly measured

KSAs

From the perspective that a program of study is more than the sum of each individual course, a program of assessment must be more than each individual measure.

The whole is greater than the sum of the parts

Research aimTo develop & to implement

a comprehensive, integrated assessment program which explicitly maps to the three aims, seven goals & 35 learning outcomes of the MBBS Program and the six threshold learning outcomes of the TEQSA

Educational approachDijkstra, Van der Vleuten & Schuwirth (2009)

– Six dimension framework

– Guides programmatic assessment

– Holistic focus

– Explicit mapping to program goals BUT

– Establishing a set of goals for an assessment program in medical education

MethodDijkstra dimension Dimension elements Details of Implementation

Goals Establishing a clear goal for the assessment program

Mapping of existing assessment items to aims, goals & learning outcomes;Identification of gaps, redundancies & assessment practices & instruments inconsistent with best practice;Consultation with stakeholders -assessment across all four years of the Program, potential barriers & other issues;Establishment of a shared goal for the assessment program

Supporting the program

Technical support for quality assessment & involvement of stakeholders

Professional development and support;Continued stakeholder input

Documenting the program

Assessment rules, blueprinting & planning;Plotting the assessment processes for clarity & accountability

Literature review to identify best evidence for assessment in higher education & medical education; Analysis of strategic plans & priorities at School, Faculty & University level;Development of a holistic plan for assessment

Program in action Collecting data, combining the data, reporting & decision-making

Implementation of the integrated assessment program

Improving the program Constant monitoring & application of relevant research

Evaluation & action research

Accounting for the program

Internal & external review; Accountability to stakeholders

UQ & AMC Review;Stakeholder feedback

Likely outcomes

• Creation of comprehensive map: learning goals, aims & outcomes matched to current assessment processes

• Establishment of holistic assessment program: sampling knowledge, skills & other key aspects of competence both summatively & formatively

• Development of spectrum of complementary assessment strategies: sampling student achievement in an iterative manner but with increasing sophistication

• Synthesis of well-informed, logical & highly defensible plan of assessment.

• Enhanced collaboration & understanding between all stakeholders

ConclusionThe opportunity exists to produce, over the next three years, an innovative program of assessment that best serves the needs of all stakeholders & sets an example of the high quality which can be achieved by one of Australia’s premier medical schools.

References

Dijkstra, J., Van der Vleuten, C. & Schuwirth, L. (2009). A new framework for designing programmes of assessment. Advances in Health Sciences Education, Advance online publication. Doi: 10.1007/s10459-009-9205-z.