Integrated Entry-to-Practice Assessment Strategiesclearweb.firstcloudit.com/2018_conference... · impact of commonly used assessments processes including MCQ’s, viva voce/oral exams,

Integrated Entry-to-Practice AssessmentStrategies

Philadelphia, PennsylvaniaSeptember 26 – 29, 2018

Council on Licensure, Enforcementand Regulation 2018 Annual Educational Conference 1

Integrated Entry-to-Practice Assessment Strategies


Glenys Wilkinson, Australian Pharmacy Council,

and

Diana Spizzirri, Ontario College of Pharmacists, Toronto, Canada

Annie Lee, Faculty of Pharmacy, University of Toronto, Canada




Objectives

• Describe the collaborative process between independentpharmacy organizations on common assessment goals

• Describe assessment strategies developed in Australia andOntario, Canada for entry to practice (licensure)

• Review the outcome of the process to develop theseassessment strategies and give examples of lessonslearned


Common Themes:

• Collaboration

• Implementation across different organizations

• Entry-to-practice competency assessment

• Evaluation of outcomes




The Australian Pharmacy Council Ltd (APC)

Is the independent accreditation authority for pharmacy educationand training. We:

• Accredit pharmacy schools and programs; intern trainingprograms; and accrediting agencies to accredit continuingprofessional development (CPD) activities

• Conduct examinations on behalf of the Pharmacy Board ofAustralia (PharmBA)

• Assess the qualifications and skills of overseas trained pharmacists

Australian Pharmacy Council

Registration requirements for pharmacists in Australia:

Graduation from an accredited approved degree program

Completion of 1824 hours of supervised practice

Successful completion of an accredited intern training program

Successful completion of registration examinations




Australian Pharmacy Council• Two registration examinations

– Intern Written Examination (IWE)

– National oral examination (Oral)

• The IWE is a competency-based 125 Multiple-Choice Questioncomputer-delivered exam

• The Oral examination is an assessment that requirescandidates to demonstrate that they can competently applytheir knowledge and skills

The Intern Year Assessment BlueprintA joint APC and PharmBA project that followed the revision andupdate of National Competency Standards Framework for Pharmacistsin Australia




The project

• To optimise the assessment of interns during their trainingyear against the newly developed competencies

• To ensure the appropriate assessment methodologydevelops a direct link between learning objectives andassessment methods

• To provide transparency and clarity for ITPs, preceptorsand interns regarding how competence is measured andby whom

Collaborative partners

• Australian Pharmacy Council (APC)

• Pharmacy Board of Australia (PharmBA)

• International Regulator colleagues – UK, USA, Canada,New Zealand, Ireland

• Other professions within Australia

• International Expert Reviewer – reviewed final documents




Governance – Steering Committee

ConsultationThree consultations were conducted during the project with:

Intern Training Providers

Examiners

Preceptors

Interns

International Expert Reviewer




The project outcomes1. Literature review of intern competency assessment (including

domestic and international processes and assessment methods) forInternational jurisdictions

2. Documentation and analysis of the current intern year assessmentstructures (internal use)

3. Development of an Assessment Blueprint for the intern year,mapped against the National Competency Standards Framework forPharmacists in Australia 2016

International literature review Identified strengths and concerns with applications of competency standards

Examined the use of Miller’s pyramid to guide the development of assessmentprocesses

Evaluated advantages, disadvantages, feasibility, validity, reliability, and educationalimpact of commonly used assessments processes including MCQ’s, viva voce/oralexams, OSCE’s and workplace based assessments

Concluded each method of assessment has its own advantages and disadvantages andno single method can adequately assess all aspects of clinical competence

Important to use multiple modes of assessment to ensure knowledge, skills andattitudes are demonstrated to a pre defined level

Criteria for good assessment described as validity, reproducibility, equivalence,feasibility, educational effect, catalytic effect and acceptability




Review of current intern year assessment

An internal review for APC and PharmBA purposes only

Qualitative review that considered the three component of internyear assessments as discrete activities

Recommended strategies to build on the strength of eachcomponent to enable integrated, holistic and comprehensiveassessments to ensure public safety and confidence in recentlyregistered pharmacists

Intern year blueprintAn assessment blueprint that utilized the following methodologies

• Multiple choice questions

• Oral assessment

• Objective structured clinical examinations (OSCE)

• Mini clinical evaluation exercise (mini-CEX)

• Case based discussion (CbD)

• Multisource feedback (MSF)

• In-training assessment (ITA)

• Portfolio




Ontario Pharmacy Patient Care Assessment Tool(OPPCAT)

Common pharmacy practice based assessment tool

Collaborative Partners

Ontario College of Pharmacists

Faculty of Pharmacy, University of Toronto

School of Pharmacy, University of Waterloo

Hospital Pharmacy Residency Forum of Ontario

Ontario College of PharmacistsRegulatory body for the practice of pharmacy in Ontario, Canada.

Registration requirements for pharmacists in Ontario:

Graduation from an approved accredited pharmacy degree program

Successful completion of Canadian pharmacy licensing exam

Successful completion of Ontario pharmaceutical jurisprudence exam

Successful completion of a practice based assessment (PACE)

Others (e.g. language proficiency, good standing…)




Ontario College of Pharmacists Practice Assessment of Competence at Entry (PACE)

Competency-based assessment in a pharmacy practice site by anappointed and trained College assessor

PACE is assessment then, if needed, individualized self-directeddevelopment with a coaching pharmacist to address any performanceareas

PACE Process




Collaboration

Ontario College of Pharmacists collaborated with other organizations

At the right time and place

Sharing of ideas and human resources

Similar goal and outcomes

Common need lead to moving forward quickly with less formalcollaboration

Working group established as natural evolution

More formality and structure introduced after tool implemented

Implementation in Programs• Advanced Pharmacy Practice Experience

(APPE) in PharmD program• PharmD for Pharmacists program

University ofToronto

University ofToronto

• Patient Care Rotations in 4th year PharmD• PharmD Bridging Program

University ofWaterloo

University ofWaterloo

• Patient Care Rotations in OntarioHospital PharmacyResidency ProgramsHospital PharmacyResidency Programs

• Practice Assessment of Competence at Entry(PACE) for pharmacists

Ontario College ofPharmacists

Ontario College ofPharmacists




OPPCAT Development TimelineSpring2014

Residencyprograms

anduniversities

Fall 2014Ontario

College ofPharmacists

(OCP)

Fall2015

v.5focusgroups

May 2016

U of Torontolaunch v.6

July2016

Residencylaunch

January2017

U ofWaterlooand OCPlaunch

January2017

SatisfactionSurvey

January2018

Dataanalysis/revision

OPPCAT

NAPRA Entry-to-Practice

Competencies

AFPCEducationalOutcomes

CanadianPharmacy

Residency BoardStandards

Literaturereview

Competencybased

frameworkSolo taxonomy

Five point Likertscale

Domains,elements andbehaviouraldescriptors

Companionglossary

Framework:

Standards:




OPPCAT – 5 Domains and 20 Elements

Patient Care(8)

Communication andEducation (4)

Professionalism (2)Professional

Collaboration (3)

PracticeManagement (3)

Domain

Element

BehaviouralDescriptor

Rating Scale




Behavioural Descriptors

Demonstrate progressionfrom levels 1 to 5

Each descriptor has 3 components:• Guidance/support required• Effectiveness/quality• Consistency

Guidance




OPPCAT Satisfaction Survey ResultsAnonymous on-line survey to assess the practicality, usability andsatisfaction with OPPCAT: Preceptors/Assessors (n=171), Students/Residents (n=69)

Survey Results – For Consideration

Adding ‘not applicable’ (some programs)

Too long and difficult to fill out tool (programs using fillable pdf)

Site freezes; hard to log on (OCP assessors)

Tweaking of some behaviour descriptors in Patient Care domain

Assessing specific knowledge of behaviours (residency preceptors)




Next StepsValidation Study (planned)

Data from all 4 programs to be considered for validation

1 – 2 cohorts of data available for the study

OPPCAT Updates and Training:

Revision of OPPCAT to v.7

Dissemination of OPPCAT to other programs in Canada

Collaborate to standardize OPPCAT training

OPPCAT Training Organizations have own OPPCAT training programs for

assessors/preceptors

Preceptor Development Working Group (2018) to collaborate

Taking inventory of resources used by each program

Evaluating a platform for delivering online programming

Reviewing video resources

Assessing need for resources and modules




Lessons Learned: Ontario

• Collaboration possible due to the shared commitment and ‘investment’

• Timing was important – i.e. right time and place

• Need patience and perseverance

– Different organizations need to ensure buy-in across all leadership teams

• Funding/financial support for work as it evolves

Lessons Learned: Ontario

• Online assessment tool

– Consider technology solution/platform across all organizations

• Standardization of tool through combined preceptor trainingand shared resources




Lessons Learned: APC• Collaboration challenging but rewarding

• Need authorising mandate from senior management

• First step in a change process i.e. Kotters ‘burning platform’

• Enables broad commitment to the final product

• Time consuming

• Requires focusing on the end goal but flexibility may be required

• ‘From the balcony’ principles of Adaptive Leadership (Heifer,Grashow &Linsky. 2009)

Lessons Learned: APC

• Elicits and highlights different ‘political’ considerations

• Engagement across organisational boundaries requires persistence,awareness of others and sensitivity

• Need to pay attention to the emotional side of collaboration andnegotiation

• Need effective communication – don’t assume




Speakers Contact Information

Glenys Wilkinson B.Soc Sci., MSW, M.App Sci.

[email protected]

Diana Spizzirri R.Ph., BSc.Phm., ACPR, [email protected]

Annie Lee R.Ph., BScPhm., MSc(T), [email protected]

Documents

Integrated Entry-to-Practice Assessment Strategiesclearweb.firstcloudit.com/2018_conference... · impact of commonly used assessments processes including MCQ’s, viva voce/oral exams,