Upload
meredith-wilcox
View
215
Download
0
Embed Size (px)
Citation preview
Assessing clinical judgment using the script concordance test: The
importance of using specialty-specific experts to develop the scoring key
Petrucci AM. MD1, Nouh T. MD1, Boutros M. MD1, Gagnon R. PhD3, Meterissian S. MD1, 2
Department of Surgery, McGill University 1
Center for Medical Education, McGill University 2
Faculty of Medicine, University of Montreal 3
Association for Surgical Education
Paper Session 1
March 22, 2012
Disclosures
The authors have no disclosures to declare
Current Assessment Tools
• MCQ
• Rich context MCQ
• Orals
• Short answer questions
• Long answer questions
The perfect assessment tool?
Graduating general surgery residents
Multiple choice examination Oral examination+
Institute of Applied Research in Health Sciences Educationhttp://www.script.md/portal/en/script_en.html
What is Reasoning?
SCT: from memorizing to reasoning
• New tool intended to assess mental processes in medical practice
• Can be used to evaluate a trainee’s approach to ill-defined problems including those encountered by expert surgeons
Example
-2 Contra-indicated or almost totally-1 Not useful or even detrimental 0 Nor less nor more useful+1 Useful+2 Necessary or absolutely necessary
The Scoring Grid (Modal Experts’ Choice)
-2 -1 0 +1 + 2
Number of experts choosing answer 0 0 5 4 1
Score 0 0 5/10 4/10 1/10
Transformed score 0 0 5/5 4/5 1/5
Credit per item 0 0 1 0.8 0.2
Nouh, T., et al. (2012). The script concordance test as a measure of clinicalreasoning: A national validation study. The American Journal of Surgery, In press
What have we done in the past?
Preliminary Study:McGill University
n= 36
Meterissian, S. et al. (2007). Is the script concordance test a valid instrument for assessment ofintraoperative decision-making skills? The American Journal of Surgery, 193, 248-251.
Pan Canadian Studyn= 202
Nouh, T., et al. (2012). The script concordance test as a measure of clinicalreasoning: A national validation study. The American Journal of Surgery, In press
Purpose
To determine whether using a specialty-specific scoring key would result in a progression of scores from R1 to R5, with the R5 residents having the highest scores and therefore improving the validity of the scoring key
Methods
Participants
n= 25McGill
n= 202(Nouh, T., et al, 2012)
n tot= 227
Specialty Specific Scoring Key
Colorectal
Hepatobiliary
Endocrine
Surgical Oncology
Thoracic
Trauma/ ACS
EXPERTS
Colorectal 1 1 1 1 1
Hepatobiliary 1 1 1 1 1
Endocrine1 1 1 1 1
Surgical Oncology 1 1 1 1 1
Thoracic 1 1 1 1 1
Trauma/ ACS1 1 1 1 1
EX
PE
RT
2
EX
PE
RT
3
EX
PE
RT
4
EX
PE
RT
5
EX
PE
RT
1
Creation of amalgamated expert exams
• Reliability: Cronbach alpha coefficient
• Analysis of item-to-total item correlation used to select the best items for the final analysis
• Construct validity tested with a one-way ANOVA with post-hoc comparisons test and planned contrasts
• All p values at alpha<5% were considered significant
Statistical Analysis
RESULTS
Final exam100-question testCronbach α = 0.81
153-question test Cronbach α = 0.67
Elimination of items with a negative item-to-total item correlation
Results
?
Scores rising with increasing level of residency: p< 0.001
Number of expert panel members matters!
Gagnon et al. Medical Education, 2005:
• The mean of residents’ scores increased with increasing panel size.
“…the number of points earned on an SCT is influenced by the number of members used in the panel”.
Mean Score Graph
Scores rising with increasing level of residency: p< 0.001
Juniors vs Seniors
Total N
R1/ R2 45.1 (7.6) 109
R3/ R4/ R5 50.4 (8.0) 118
The average score of junior residents proved to be significantly lower than the average score of senior residents: p< 0.001
Discussion
• The results of this study suggest that the scoring key should be developed through the use of specialty- specific experts
• This may be due to the increasing subspecialization of General Surgery
• This has important implications in the application of the SCT on a wider level
Limitations• We used only 5 amalgamated general
surgery experts for each surgical discipline instead of the recommended >10
Gagnon et al. Medical Education 2005
• The resident pool consisted mostly of the same residents from our last study with the addition of 25 new residents from our institution
Future implications…
National wide study using the specialty-specific expert scoring key.
The SCT might be ready for a national in-training formative examination.
Acknowledgements
• Dr. Sarkis Meterissian• Dr. Robert Gagnon• Dr. Thamer Nouh• Dr. Marylise Boutros
Thank you