Assessing Procedural Competencies

Assessing Procedural Assessing Procedural CompetenciesCompetencies

Amy S. Oxentenko, MD, FACP, FACGAmy S. Oxentenko, MD, FACP, FACGAssociate Professor of MedicineAssociate Professor of Medicine

Fellowship Director, Mayo Clinic-RochesterFellowship Director, Mayo Clinic-Rochester

Outline: Outline: Procedural CompetencyProcedural Competency

• The NumbersThe Numbers

• The SkillsThe Skills

• The FutureThe Future

Competent or Not?Competent or Not?

• 33rdrd year fellow year fellow

• 285 colonoscopies285 colonoscopies

• Cecal intubation 95%Cecal intubation 95%

• Adenoma detection 35%Adenoma detection 35%

• Withdrawal 10 minWithdrawal 10 min

• Patient tolerance 97% Patient tolerance 97% fair to excellentfair to excellent

• 33rdrd year fellow year fellow

• 427 colonoscopies427 colonoscopies

• Cecal intubation 69%Cecal intubation 69%

• Adenoma detection 12%Adenoma detection 12%

• Withdrawal 6 minWithdrawal 6 min

• Patient tolerance 85% Patient tolerance 85% fair to excellentfair to excellent

YESYES NONO

What Needs to be What Needs to be Measured?Measured?

CompetencyCompetency

Minimum Minimum Procedure Procedure

NumberNumber

Defined Defined level of level of abilityability

The NumbersThe Numbers

Minimum Cited NumbersMinimum Cited Numbers

• Flex sigsFlex sigs 25-3025-30

• EGDsEGDs 130130

• ColonsColons 140140

– Based on goal of cecal intubation > 90%Based on goal of cecal intubation > 90%– Others found competence at 275 casesOthers found competence at 275 cases11

– Others found competence at 500 casesOthers found competence at 500 cases22

11Using Sedlack dataUsing Sedlack data22Using Spier dataUsing Spier data

Cecal Intubation RateCecal Intubation Rate

Sedlack RE. Gastrointest Endosc 2011;74:355-66.

Spier BJ, et al. Gastrointest Endosc 2010;71:319-24.

Procedure Logs: Procedure Logs: Not Just Numbers Anymore!Not Just Numbers Anymore!

• ““A skilled preceptor must be available to A skilled preceptor must be available to teach and supervise the fellows in the teach and supervise the fellows in the performance and interpretation of performance and interpretation of procedures, which must be documented procedures, which must be documented in each fellow's record, including in each fellow's record, including indications, outcomes, diagnoses, and indications, outcomes, diagnoses, and supervisor(s).”supervisor(s).”

IV.A.6.d).(2) on Page 19, GI Program Requirements, “Tracked Changes” document

The SkillsThe Skills

Procedure LoggingProcedure Logging• ““Assessment of procedural competence should include Assessment of procedural competence should include

a formal evaluation process and NOT be based solely a formal evaluation process and NOT be based solely on a minimum number of procedures performed. on a minimum number of procedures performed.

• Each program must define criteria for competence for Each program must define criteria for competence for all required and elective procedures.all required and elective procedures.

• The record of evaluation must include the fellow’s The record of evaluation must include the fellow’s logbook or an equivalent method to demonstrate that logbook or an equivalent method to demonstrate that each fellow has achieved competence in the each fellow has achieved competence in the performance of required procedures.”performance of required procedures.”

V.A.1.a).(2) and 1.b).(1).(a) on Page 20-21, GI Requirements, “Tracked Changes” document

Multi-Society Evaluation FormMulti-Society Evaluation Form(MSEF)(MSEF)

• AASLD, ACG, AGA, ASGEAASLD, ACG, AGA, ASGE

• Part of the GI Core CurriculumPart of the GI Core Curriculum

• Third Edition, May 2007Third Edition, May 2007

http://www.asge.org/WorkArea/showcontent.aspx?id=3584http://www.asge.org/WorkArea/showcontent.aspx?id=3584

Any Downfalls of the MSEF?Any Downfalls of the MSEF?

• Lacks anchoring characteristics for all pointsLacks anchoring characteristics for all points

• Not validated for continuous assessmentNot validated for continuous assessment

• Grade inflation (our problem, not the form’s)Grade inflation (our problem, not the form’s)

• Compare graduates across programs?Compare graduates across programs?

• What constitutes competent?What constitutes competent?

Sedlack RE. Gastrointest Endosc 2010;72:1125-33.

Mayo Colonoscopy Skills Mayo Colonoscopy Skills Assessment Tool (MCSAT)Assessment Tool (MCSAT)

• 13-item survey13-item survey

• Staff completed on each colonStaff completed on each colon

• Took < 1 minute to completeTook < 1 minute to complete

• Embedded in MERGE databaseEmbedded in MERGE database–Allowed for recording of procedure Allowed for recording of procedure

# for fellow, fellow name, etc.# for fellow, fellow name, etc.

Sedlack RE. Gastrointest Endosc 2010;72:1125-33.

Overall Skill in ColonoscopyOverall Skill in Colonoscopy

Sedlack RE. Gastrointest Endosc 2011;74:355-66.

Other Date Other Date Retrieved Per Retrieved Per

TraineeTrainee

Barriers of the MCSAT or Barriers of the MCSAT or Similar Systems?Similar Systems?

• Many procedures performedMany procedures performed– Assessment needs to be quick/simpleAssessment needs to be quick/simple

• Differing procedures performedDiffering procedures performed– Similar models needed: EGD, PEGs, capsules, etcSimilar models needed: EGD, PEGs, capsules, etc

• Compliance with completionCompliance with completion– Too easy for staff to forget or not take the timeToo easy for staff to forget or not take the time

• Differing endoscopy database systemsDiffering endoscopy database systems– No communication across programsNo communication across programs

Sunset: MERGESunset: MERGEOctober 2012October 2012

Sunrise: ProVationSunrise: ProVationOctober 2012October 2012

ProVation Tracking: EGDProVation Tracking: EGD

ProVation Tracking: ColonsProVation Tracking: Colons

Pros/Cons of ProVationPros/Cons of ProVation• PROS:PROS:• Compliance with completionCompliance with completion

– Automatic pop-up on all fellow EGDs and colonsAutomatic pop-up on all fellow EGDs and colons– Staff cannot sign off until completeStaff cannot sign off until complete

• CONS:CONS:• Yet another databaseYet another database

– How long with it be around?How long with it be around?– Not everyone has itNot everyone has it– Dependent on others to add features desiredDependent on others to add features desired

• Detail desiredDetail desired– Has to fit into radio buttons, brief, succinctHas to fit into radio buttons, brief, succinct

The FutureThe Future

The Game Has Changed in The Game Has Changed in the Setting of NASthe Setting of NAS

• No longer a numbers gameNo longer a numbers game

• No longer a competency yes/no gameNo longer a competency yes/no game

• Now it is all about meeting milestones Now it is all about meeting milestones on the way to becoming competenton the way to becoming competent

http://www.acgme-nas.org/assets/pdf/NEJMfinal.pdf

A Blueprint for Milestones A Blueprint for Milestones or Competency?or Competency?

UnacceptableUnacceptable CompetentCompetent IdealIdeal

Adenoma Adenoma Detection RateDetection Rate

>20%>20%

Colonoscopy Colonoscopy Withdrawal TimeWithdrawal Time

> 6 min> 6 min

Cecal Intubation Cecal Intubation RateRate

> 95%> 95%

Complication RateComplication Rate < 1/200 bleed< 1/200 bleed

<1/1000 perf<1/1000 perf

Polyp Retrieval Polyp Retrieval RateRate

> 95% > 10 mm> 95% > 10 mm

> 80% < 10 mm> 80% < 10 mm

Patient TolerancePatient Tolerance > 90% fair to > 90% fair to excellentexcellent

A Blueprint for Milestones A Blueprint for Milestones or Competency?or Competency?

UnacceptableUnacceptable CompetentCompetent IdealIdeal

Adenoma Adenoma Detection RateDetection Rate

> 20%> 20% >20%>20%

Colonoscopy Colonoscopy Withdrawal TimeWithdrawal Time

7-15 min7-15 min > 6 min> 6 min

Cecal Intubation Cecal Intubation RateRate

> 90%> 90% > 95%> 95%

Complication RateComplication Rate < 1/200 bleed< 1/200 bleed

<1/1000 perf<1/1000 perf

Polyp Retrieval Polyp Retrieval RateRate

> 95% > 10 mm> 95% > 10 mm

> 80% < 10 mm> 80% < 10 mm

Patient TolerancePatient Tolerance > 90% fair to > 90% fair to excellentexcellent

A Blueprint for Milestones A Blueprint for Milestones or Competency?or Competency?

UnacceptableUnacceptable CompetentCompetent IdealIdeal

Adenoma Adenoma Detection RateDetection Rate

< 15%< 15% > 20%> 20% >20%>20%

Colonoscopy Colonoscopy Withdrawal TimeWithdrawal Time

> 20 min> 20 min 7-15 min7-15 min > 6 min> 6 min

Cecal Intubation Cecal Intubation RateRate

< 80%< 80% > 90%> 90% > 95%> 95%

Complication RateComplication Rate < 1/200 bleed< 1/200 bleed

<1/1000 perf<1/1000 perf

Polyp Retrieval Polyp Retrieval RateRate

> 95% > 10 mm> 95% > 10 mm

> 80% < 10 mm> 80% < 10 mm

Patient TolerancePatient Tolerance > 90% fair to > 90% fair to excellentexcellent

Milestones and NarrativesMilestones and Narratives

SummarySummary• Procedure numbers are not enoughProcedure numbers are not enough

– An anchor at which competency An anchor at which competency assessment should beginassessment should begin

• Procedure details are now neededProcedure details are now needed– Indication, findings, complicationsIndication, findings, complications

• Competency tracking is requiredCompetency tracking is required– Milestones will pave the way, and they Milestones will pave the way, and they

need to be carefully developedneed to be carefully developed

Thank you!Thank you!

oxentenko.amy@mayo.eduoxentenko.amy@mayo.edu