Embed Size (px)
Citation preview
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations.
Author: Chan Kulatunga-Moruzi, Ph.D.
Date: September 24, 2011
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
2
Introduction: Communication skills
Organize ideas and information
Convey ideas and information effectively
Important skill for health care professional
Collaborative care
Globalization
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
3
Introduction: CanMeds Roles
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
4
Background: MCAT Verbal Reasoning
Research Indicates Predictive Validity
Violato & Donnon (2004)
MCAT-VR & MCCE Part 1 = 0.258
MCAT-VR & MCCE Part 2 = 0.241
Kulatunga-Moruzi & Norman (2002)
MCAT-VR & MCCE Part 2-CS = 0.43
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
5
Background: Verbal Reasoning Callahan, Hojat, Veloski, Erdmann, & Gonnella (2010)
MCAT-VR & USMLE Step1 = 0.21
MCAT & USMLE Step 2 = 0.32
MCAT & USMLE Step 3 = 0.29
Donnon, Paolucci & Violata (2007) Meta Analysis*
Basic science
Clerkship
USMLE Step 1, 2 & 3
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
6
Background: MCAT Writing Sample
Research Findings Mixed
Stephens & Reamy (2009)
MCAT-WR & Family Medicine OSCE = 0.25
Hojat et.al. (2000)
Clerkship: ratings, written exams
Residency: interpersonal skills, data gathering, socio-economic aspects of care, physician as an educator
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
7
Purpose:
Examine the reliability and concurrent validity of writing samples.
Writing samples obtained:
• on the same day as interview
• on-site
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
8
Purpose:
Acceptability of written MMI stations
Feasibility of written MMI stations
Relationship between verbal & written MMI stations on same construct
Reliability of written MMI stations with and without scoring rubric
Concurrent validity of written MMI stations
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
9
Methods:
60 applicants interviewed
Physician Assistant Education Program
12 MMI stations
10 Verbal MMI stations (10 minutes each)
2 Written MMI “stations” (20 minutes each)
in a computer lab
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
10
Methods:
Writing samples similar to MCAT writing samples.
Provide interpretation or explanation statement.
Offer example that illustrates the opposite point.
Explain how conflict between viewpoints might be resolved.
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
11
Methods: 1 Verbal & 1 Written station same construct:
Freedom of speech
Verbal station:
Clip of student protest of Ann Coulter speaking at UWO.
Would you support this speaker giving a presentation?
Written station:
In a country that fosters freedom of speech, the
expression of ideas should never be censored.
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
12
Methods:
Each station rated by one rater
All applicants rated by the same rater
10 Verbal MMI station
Raters provided with rubric
2 Written MMI stations – rated twice
Raters provided with rubric
Raters not provided rubric
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
13
Results:
Acceptability:
Exit survey data similar for written and verbal stations
Written stations took longer to score, not unreasonable
Feasibility:
Written stations posed no obstacles
Easily incorporated
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
14
Results:
Relationship between oral and written communication:
Correlation between verbal & written MMI stations on the same construct (freedom of speech): r = 0.60
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
15
Results:
Reliability of written MMI stations with & without rubric
• Reliability (ICC) without rubric: 0.70
• Reliability (ICC) with scoring rubric: 0.82
Reliability (ICC) 12 stations: 0.84
• Item total correlation without rubric: 0.40, 0.42
• Item total correlation with rubric: 0.62, 0.46
Item total correlations verbal stations: 0.18 – 0.65
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
16
Results:
Concurrent validity of written MMI stations
Correlation between the written MMI average and verbal MMI average: 0.80
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
17
Conclusions:
Written MMI stations are acceptable
Written MMI stations are feasible
Positive relationship between oral and written communication
Scoring rubrics improves reliability of assessment
Writing stations illustrate concurrent validity
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
18
Thank You
Acknowledgements
Physician Assistant Education Program
John Cunnington
Nancy Weller
Michelle MacDonald
McMaster University Computer Services Unit
Mahan Kulasegaram
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
19
Background:
Extra slides
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
20
Background:
Why might the predictive validity be low?
Appropriate outcome measure?
Reliability of outcome measure?
Reliability/validity of MCAT Writing Sample?
The Acceptability, Feasibility, Reliability and Concurrent Validity of Written MMI Stations
21
Background:
Although MCAT writing sample may be a weak predictor of future performance….
Generally good correlation between MCAT verbal reasoning and writing sample
indicating perhaps that they are tapping the same construct… communication
