Developing High Quality Developing High Quality Clinical Skills AssessmentsClinical Skills Assessments

University of North Carolina – Chapel University of North Carolina – Chapel Hill School of MedicineHill School of Medicine

November 10, 2011November 10, 2011

Ann Jobe, MD, MSNAnn Jobe, MD, MSNClinical Skills Evaluation Collaboration Clinical Skills Evaluation Collaboration

Philadelphia, Pennsylvania Philadelphia, Pennsylvania

The most important consideration in The most important consideration in developing a clinical skills developing a clinical skills assessment is to keep in focus the assessment is to keep in focus the purposepurpose of the assessment of the assessment

Formative or Summative?Formative or Summative?

Formative:Formative: to provide feedback for to provide feedback for improvement of performance, usually improvement of performance, usually midway through a course of study.midway through a course of study.

Summative:Summative: to distinguish between to distinguish between those who are competent, and those those who are competent, and those who aren’t, usually at the end of a who aren’t, usually at the end of a course of study.course of study.

Test Design Test Design Begin at the beginning…Begin at the beginning…

• What are you trying to assess?What are you trying to assess?

• What is the level of the examinee?What is the level of the examinee?

• How much time will the examinee have? How much time will the examinee have? Can they complete the task in the time Can they complete the task in the time allotted?allotted?

Test Design Test Design

• What are you trying to teach and What are you trying to teach and assess?assess?

Objectives for assessment – history taking Objectives for assessment – history taking skills, PE skills, communication skills, data skills, PE skills, communication skills, data interpretation skills?interpretation skills?

• What is the level of the examinee?What is the level of the examinee? Focus and length of checklists will be Focus and length of checklists will be

different for 1different for 1stst year medical students and year medical students and individuals seeking board certification individuals seeking board certification

USMLE Step 2 Clinical SkillsUSMLE Step 2 Clinical Skills

Purpose is to assess ability to:Purpose is to assess ability to: Gather information from patientsGather information from patients Correctly perform physical Correctly perform physical

examination maneuversexamination maneuvers Synthesize and communicate Synthesize and communicate

findings to patients and colleaguesfindings to patients and colleagues

Examinees are moving to supervised Examinees are moving to supervised patient care (PGY-1)patient care (PGY-1)

Development Of An Development Of An Examination BlueprintExamination Blueprint

A test blueprint defines the requirements A test blueprint defines the requirements for each examination, regardless of where for each examination, regardless of where or when it is administered. or when it is administered.

Commonly seen cases.Commonly seen cases.

Blueprint (continued)Blueprint (continued)

Identify the criteria used to define an Identify the criteria used to define an exam blueprintexam blueprint• Examination length Examination length • Case content Case content • Examinee tasks (e.g. history, physical, Examinee tasks (e.g. history, physical,

communication)communication)• Setting: inpatient or outpatient Setting: inpatient or outpatient • Patient gender, age Patient gender, age

Case SelectionCase Selection

Documentation of commonly seen Documentation of commonly seen casescases• if specialty focused assessment, cases if specialty focused assessment, cases

will be those common to that specialtywill be those common to that specialty Most common presenting complaints Most common presenting complaints

in clinical settingin clinical setting Health Department population Health Department population

statistics – especially for region, statistics – especially for region, state or local areastate or local area

Case Pool SizeCase Pool Size

Dependent on blueprint criteriaDependent on blueprint criteria Security concerns (more is better!)Security concerns (more is better!)

Case Content

Cardiovascular Respiratory

Gastrointestinal Musculoskeletal Constitutional Neurological Psychiatric Genitourinary Women’s health Other

Form

Patient age

Age less than 18 Age 18 – 44 Age 45 – 64 Age 65 +

Patient Gender

Male Female

Case Acuity

AcuteSubacute/Chronic

Blueprint Criteria Define Case NeedsBlueprint Criteria Define Case Needs

This information is provided to faculty This information is provided to faculty to guide their case developmentto guide their case development• Case content: GastrointestinalCase content: Gastrointestinal• Case acuity: AcuteCase acuity: Acute• Patient age: 18-44Patient age: 18-44• Patient gender: FemalePatient gender: Female

Complete a “Medically Relevant Case Complete a “Medically Relevant Case Details” sheetDetails” sheet

Case Development Committee Case Development Committee

Physicians (medical school faculty): Physicians (medical school faculty): content expertscontent experts

SP trainers: training expertsSP trainers: training experts SPs: portrayal expertsSPs: portrayal experts

Process evolved from experienceProcess evolved from experience

Usually break into several groups Usually break into several groups

Each group focuses on developing Each group focuses on developing several cases to fit blueprint needsseveral cases to fit blueprint needs

Case Development ProcessCase Development Process Chief Complaint Chief Complaint

““Medically Relevant” case details Medically Relevant” case details

Patient Personality ProfilePatient Personality Profile

Case checklistCase checklist

Examinee instructionsExaminee instructions

Patient note development Patient note development

Chief Complaint/ Chief Complaint/ Differential DiagnosesDifferential Diagnoses

Brainstorming processBrainstorming process List all of the possibilitiesList all of the possibilities Then narrow the choices as Then narrow the choices as

development proceedsdevelopment proceeds

Medically Relevant Case DetailsMedically Relevant Case Details Patient /Case NamePatient /Case Name RaceRace

• Any Any • African American African American • Caucasian Caucasian • Other _________________Other _________________

BMIBMI• AnyAny• Weight proportionate to heightWeight proportionate to height• Overweight: > 24Overweight: > 24• OtherOther

Medically Relevant Case DetailsMedically Relevant Case Details

GenderGender• Male Female EitherMale Female Either

Age RangeAge Range•   10 year age range – example 30 years old = 25-35 10 year age range – example 30 years old = 25-35

AcuityAcuity

• Acute Acute • Subacute/ChronicSubacute/Chronic

Category (Blueprint)Category (Blueprint) Exclusionary Medical conditions/scarsExclusionary Medical conditions/scars Differential DiagnosisDifferential Diagnosis

Medically Relevant Case DetailsMedically Relevant Case Details

Communication TasksCommunication Tasks•    Fostering the RelationshipFostering the Relationship• Gathering InformationGathering Information•    Providing InformationProviding Information•    Making DecisionsMaking Decisions• Supporting EmotionsSupporting Emotions

Overview of Case Stimuli for Communication Overview of Case Stimuli for Communication TasksTasks

Thread of EncounterThread of Encounter Primary MDPrimary MD

Personality ProfilePersonality Profile

Giving life and individuality to the Giving life and individuality to the patients who are being seen – more patients who are being seen – more like “real patients”like “real patients”

Not verbatim statements from a Not verbatim statements from a script – more general ideas of each script – more general ideas of each patient’s perspective patient’s perspective

Personality ProfilePersonality Profile

Who am I and how would I describe Who am I and how would I describe myself?myself?

Why am I here?Why am I here? What made me come in today?What made me come in today? What do I think is going on and why?What do I think is going on and why? What are my expectations for the What are my expectations for the

visit and how will I react if my visit and how will I react if my expectations are not met?expectations are not met?

Personality ProfilePersonality Profile What questions do I have?What questions do I have? What concerns/anxiety/fears do I What concerns/anxiety/fears do I

have?have? How has this illness impacted my How has this illness impacted my

life?life? What do I bring with me?What do I bring with me? What do I look like?What do I look like? How do I behave/what’s my How do I behave/what’s my

communication style (mood/attitude communication style (mood/attitude related to my illness)?related to my illness)?

Personality ProfilePersonality Profile

What is my level of trust of the What is my level of trust of the medical profession?medical profession?

What type of communication style do What type of communication style do I appreciate in a health care I appreciate in a health care provider? How will I respond to provider? How will I respond to different styles of communication? different styles of communication? How much information do I want the How much information do I want the doctor to provide?doctor to provide?

Personality ProfilePersonality Profile

What words won’t I understand What words won’t I understand (jargon)?(jargon)?

What are my own words I use to What are my own words I use to describe what is going on?describe what is going on?

How motivated am I to accept How motivated am I to accept medical advice?medical advice?

How do I feel about the use of How do I feel about the use of complementary/alternative complementary/alternative medicine?medicine?

Case Checklist Case Checklist Recording instrument Recording instrument Checklist length Checklist length One concept per itemOne concept per item Evidence-based items Evidence-based items Use lay language Use lay language Example: Onset Example: Onset (“I’ve been coughing for about a (“I’ve been coughing for about a

month.”)month.”)

Refining the ChecklistRefining the Checklist

Brainstorm a large number of itemsBrainstorm a large number of items Role playRole play Checklist revisedChecklist revised

• A – essentialA – essential• B – importantB – important• C – relevantC – relevant• F - deleteF - delete

Examinee InstructionsExaminee Instructions

NameName AgeAge Setting Setting Presentation problemPresentation problem Vital signsVital signs Examinee Tasks Examinee Tasks

• Refers to the skills that the Refers to the skills that the examination assessesexamination assesses

Role Play with SP Role Play with SP

A physician from another group A physician from another group “sees” the patient“sees” the patient

Following the encounter, the Following the encounter, the physician provides his/her differential physician provides his/her differential diagnoses (see if it matches what the diagnoses (see if it matches what the group selected): face validitygroup selected): face validity

Revise the checklist based on Revise the checklist based on observing the physicianobserving the physician

Role Play with SP Role Play with SP

This allows an SP to provide feedback This allows an SP to provide feedback on any difficulties that arise in on any difficulties that arise in portrayal of the case – especially if portrayal of the case – especially if an SP needs to repeat the case an SP needs to repeat the case several times in an assessment several times in an assessment sessionsession

Development of Patient NoteDevelopment of Patient Note

Key essentials and case-based Key essentials and case-based scoring guidelinesscoring guidelines

Questions?Questions?