Embed Size (px)
Citation preview
Ottawa Conference/CCME
C. Touchie, A. Gotzmann, C. Streefkerk,
A. De Champlain,
Medical Council of Canada
Putting the Pieces Together to Create a Defensible Blueprint:
A Subject Matter Expert Based Process
April 2014
Ottawa, Canada
1
Credentialing Examinations
• Validity:
– Associated with the interpretation assigned to
test scores NOT with the score or with the test
itself
– Kane’s validity argument:
• Scoring
• Generalization
• Extrapolation the content of the test is
matched appropriately to the
demands of practice
• Decision
2
Practice Analysis
• Practice Analysis Methods:
– Critical Incident Technique
– Task Inventory
– Professional Practice Model
• Professions: dynamic and social
2
Purpose of Blueprinting
• … is to assure the public that physicians
licensed to practice medicine have the
required knowledge, skills and attitudes
for safe and effective patient care.
• Only those who meet this standard are
qualified to enter professional practice
4
Decision points
• What we are assessing:
– Core foundational competencies expected of all physicians regardless of discipline for the purpose of fulfilling one of the requirements of licensure
• Decision point 1
– Assessments leading to a decision for entry into residency
• Decision point 2
– Assessments leading to a decision for entry into independent practice
5
Practice Analysis
• Factors to Consider
1. Information required
2. Best sources of information
3. Strategy for obtaining that information
4. Strategy to put it all together
6 Modified from M. Raymond, AERA 2014
1. Information Required
7
Focus of practice
descriptors
Level of
specificity
Attributes to be
measured
Current Issues Future of medical
education
Broad and
complex
Assessment needs
Supervising
PGY-1
Transition between
MD and residency
Entrustable
professional
activities
Supervisory needs
Incidence and
Prevalence
Common clinical
problems
Specific
presentations
(age, setting)
Frequency
National Survey MCC objectives Specific KSAs Importance of KSAs at
2 decision points
2. Sources of Information
8
Sources of Information
Current Issues Review of literature, medical educators
Supervising
PGY-1
New PGY-1s, supervisors
Incidence and
Prevalence
Billing data (CIHI, provincial sources), published
literature
National Survey Physicians: recent LMCC recipients, clinical
teachers
Informed public
3. Data Collection Strategy
9
Data Collection Strategy
Current Issues Review of literature, iterative review by experts
Supervising
PGY-1
Consensus technique/Survey
Incidence and
Prevalence
Review of existing data
National Survey Physicians: Survey
Informed public: Survey and focus groups
4. Strategy to pull it all together
Blueprint and Test Specifications
Defined 2 candidates Common Blueprint Test Specifications for
each decision point
Subject Matter Experts
12 Experts 3 day meeting
4 Reports
Current Issues PGY-1
supervision Incidence and
Prevalence National Survey
Who were the SMEs?
Blueprint
MRA Rep of Council
Central Examination Committee
Objectives Committee
Test Committees
RCPSC CFPC
UGME Deans
PGME Deans
University Rep of
Council
Results:
Proposed Common Blueprint
Dimensions of Care
Health Promotion and
Illness Prevention
Acute Chronic Psychosocial
Aspects
Ph
ysic
ian
Acti
vit
ies
Assessment/ Diagnosis
Management
Communication
Professional Behaviors
12
Dimensions of Care
Health Promotion and Illness Prevention
Acute Chronic Psychosocial
Aspects Row
Percent
Ph
ys
icia
n A
cti
vit
ies
Assessment/ Diagnosis 30±5
Management 20±5
Communication 30±5
Professional Behaviors 20±5
Column Percent 20±5 30±5 30±5 20±5 100
Assessments leading up to
Decision 1: Entry into Residency
13
Dimensions of Care
Health Promotion and Illness Prevention
Acute Chronic Psychosocial
Aspects Row
Percent
Ph
ys
icia
n A
cti
vit
ies
Assessment/ Diagnosis 25±5
Management 35±5
Communication 20±5
Professional Behaviors 20±5
Column Percent 20±5 25±5 35±5 20±5 100
Assessments leading up to
Decision 2: Entry into Independent Practice
14
Topics / Knowing
Processes / Doing
Processes/Doing
Textbook Topics
Academic Disciplines
Organ System
Clinical Problem / Reason for Visit
Medical Specialty
Cognitive Processes or Skills
Competency (ACGME, CanMEDS)
Physician Task or Skill
Conclusions
Defensible Blueprint:
• Define assessment purpose
• Choose information sources to inform appropriately
• Include judgments from stakeholders
Lessons learned:
• Careful planning
– Team
– Information gathered
– Process
16
