Upload
hugo-pullen
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Brian JollyCentre for Medical and Health Sciences Education
Monash University
Train the Trainer WorkshopOctober 31, 2003
Hong Kong International Consortium
Blueprinting
Hong Kong International Consortium
Objectives
• Describe basic ideas behind blueprinting
• Develop 2 examples
• Show how this does/does not articulate with IDEAL database
Hong Kong International Consortium
Principle
• The PURPOSE of the assessment will define the content, strategy, & technique used.
Hong Kong International Consortium
Face validity Content Construct Criterion Effects on learning
Aspects of Validity - Constructs
Hong Kong International Consortium
Face validity Content Construct Criterion Effects on learning
Aspects of Validity - Constructs
Hong Kong International Consortium
Face validity Content Construct Criterion Effects on learning
Aspects of Validity - Constructs
Hong Kong International Consortium
Define what is to be tested Clinical problems at some resolution
Tasks for each problem
Blueprint
Select test methods Authenticity appropriate to clinical tasks
Clinical tasks determine the method
Titrate against practical constraints
Content Steps
Hong Kong International Consortium
CompetenceCategories
CVS RS … GI ….. MSS Endo/Metab
Haem /Oncology
….
History Taking
Physical Exam
Tests and Procedures
…..
Management
Communication and Patient Education
Simple Blueprint
Hong Kong International Consortium
The Hospital Setting The Community Setting
Competence Categories
CVS RS … GI ….. CVS RS … GI
History Taking
Physical Examx x
Tests and Procedures x x
…..
Other Parameters - Hierarchical
Hong Kong International Consortium
Inpatient Outpatient/Ambulatory
CompetenceCategories
CVS RS … GI ….. CVS RS … GI
History Taking
Physical Examx x
Tests and Procedures x x
…..
Hierarchical - IDEAL® Related
Hong Kong International Consortium
The Child The Elderly
CompetenceCategories
CVS RS … GI ….. CVS RS … GI
History Takingx x
Physical Exam
Tests and Procedures
…..
Other Dimensions
Hong Kong International Consortium
The Child The Elderly
Cognitive /Affective
Categories
CVS RS … GI ….. CVS RS … GI
Recall x xReasoning
Affect x xSkill…..
Other Dimensions - Response
Hong Kong International Consortium
The Child The Elderly
Cognitive /Affective
Categories
CVS RS … GI ….. CVS RS … GI
Recall KFI KFI
Reasoning
Affect CI SP
Skill…..
Test – Response Matrix
Hong Kong International Consortium
Steps: A plausible test of Content Validity
Chart distribution of questions by level & topic
Compare this distribution with weighting of topics in curriculum or course documents
Note to establish content validity we need a demonstrated relationship
Jolly 2000, Medical Education,
Hong Kong International Consortium
Some researchers use judgments of clinicians or successful test-takers
Dimensions of blueprint not always easy to define c.f. the items
How clinical problems, tasks or other 'concrete examples' relate to content often taken for granted in blueprinting
Jolly 2000, Medical Education,
Steps: A plausible test of Content Validity
Hong Kong International Consortium
Summary
• Before selecting evaluation tools, need to create an evaluation “blueprint”
• The blueprint = matrix that outlines parameters for the assessment in 3 major areas: domains to be assessed, content & assessment instruments
Hong Kong International Consortium
• Blueprint matrix ensures balance across assessment formats - conversely
• Multiple methods usually required to achieve blueprint objectives
• Different assessment instruments allow balance of strengths and weaknesses of each method
Summary
Hong Kong International Consortium
Content
In Australia RACS has sole responsibility for goals, training delivery & standards See CANMEDS, Duties of a Doctor,
Investigate ways of addressing these issues
Get profession & public to assist
Responsibility for the ‘hard to measure’
RACS blueprint – are we a surgeon first & a specialist second?
Blueprinting Issues: e.g., RACS Assessment