Web-Based Portfolios: A Powerful New Tool for Surgical Education

Web-Based Portfolios:Web-Based Portfolios: A Powerful New Tool for Surgical EducationA Powerful New Tool for Surgical Education

Catherine E. Lewis MD, Areti Tillou MD,Catherine E. Lewis MD, Areti Tillou MD,

Michael W. Yeh MD, Chi Quach, Michael W. Yeh MD, Chi Quach,

Jonathan R. Hiatt MD, O. Joe Hines MDJonathan R. Hiatt MD, O. Joe Hines MD

Department of Surgery Department of Surgery David Geffen School of Medicine at UCLADavid Geffen School of Medicine at UCLA

Professional CompetenceProfessional Competence

• Competency-based model for GME - 1998 Competency-based model for GME - 1998

• ““Habit of lifelong learning” Habit of lifelong learning” – – LeachLeach

• ““Reflection in daily practice for benefit of Reflection in daily practice for benefit of individual and community”individual and community”

– – Epstein & HundertEpstein & Hundert

• Surgical educators challenged to Surgical educators challenged to teach, assess and document competenceteach, assess and document competence

Portfolios:Portfolios:Multifunctional tool for competenceMultifunctional tool for competence• Documentation:Documentation:• Collection of resident’s workCollection of resident’s work

• Demonstrate learning, achievement, deficiencyDemonstrate learning, achievement, deficiency

• Assessment:Assessment:• Reliable and validReliable and valid

• Requires well-defined criteria & structureRequires well-defined criteria & structure

• Teaching and development:Teaching and development:• Guided by mentorsGuided by mentors

• Promotes self-assessment and growthPromotes self-assessment and growth

Portfolio ModelsPortfolio Models

ModelModel ContentContent Benefits & UsesBenefits & Uses

LearningLearning (Growth)(Growth)

FlexibleFlexible Self-assessmentSelf-assessmentMentoringMentoring Lifelong learningLifelong learning

Assessment Assessment (Showcase)(Showcase)

StructuredStructured Reliable & valid assessment Reliable & valid assessment Professional certificationProfessional certification

HybridHybrid BothBoth Use for both formative &Use for both formative & summative assessmentsummative assessment facilitated by contentfacilitated by content access privilegesaccess privileges

Specific AimsSpecific Aims

• Describe UCLA portfolio systemDescribe UCLA portfolio system

• Analyze effects of the portfolio systemAnalyze effects of the portfolio system

• Survey of residentsSurvey of residents

• Evaluations by facultyEvaluations by faculty

Portfolios in Surgical EducationPortfolios in Surgical Education

UCLA Portfolio SystemUCLA Portfolio System

• Learning modelLearning model

• Web-based, password-protectedWeb-based, password-protected

• 2 GB / user2 GB / user

• Developed by expert groupDeveloped by expert group

• Development cost $25,000Development cost $25,000

• Launched May 2007Launched May 2007


ContentContent

Confidential Personal Public

Letters of rec Presentations Presentations

Incid. reports Publications Readings

Discipline CV Goals / objs

Eval reports

Licensing


ContentContent





Eval reports

Licensing


ContentContent





Eval reports

Licensing


ContentContent





Eval reports

Licensing







Surgery Education PortfolioSurgery Education Portfolio


Resident Survey DesignResident Survey Design• ComprehensiveComprehensive

• VoluntaryVoluntary

• MeasuresMeasures• Understanding of systemUnderstanding of system

• AcceptanceAcceptance

• UseUse

• Perceived effect on competenciesPerceived effect on competencies

• Response rate 69%Response rate 69%


Resident Survey ResultsResident Survey Results


% Agree% Agree

Appreciate organizational functionAppreciate organizational function 100100

Purpose / functions explainedPurpose / functions explained 9595

View as important education toolView as important education tool 8787

Understand uses of systemUnderstand uses of system 8282

Aided development of Aided development of competencecompetence

4545

Faculty EvaluationsFaculty Evaluations

• Comparison pre- and post-portfoliosComparison pre- and post-portfolios• Global assessmentsGlobal assessments

• End-of-rotation End-of-rotation

• Competency basedCompetency based

• Capture, n:Capture, n:• Before implementation = 1488Before implementation = 1488

• After = 697After = 697


Improvement in All CompetenciesImprovement in All Competencies


3.7

3.8

3.9

4

4.1

4.2

4.3

4.4

4.5

Medical Medical KnowledgeKnowledge

Patient Patient CareCare

ProfessionalismProfessionalism Inter-Inter-personal personal SkillsSkills

Practice Practice Based Based LearningLearning

Systems Systems Based Based PracticePractice

PrePre

PostPost

Patient CarePatient Care


Pre Post p

Obtains complete H & P 4.18 4.24 0.04

Generates appropriate ddx 4.10 4.14 1.16

Develops appropriate rx plan 4.08 4.15 0.02

Provides appropriate patient care 4.24 4.25 0.38

Adequate operative prep of pts 4.18 4.23 0.07

Medical KnowledgeMedical Knowledge


Pre Post p

Fund of basic & clinical knowledge 4.07 4.12 0.07

Engages in ongoing learning 4.00 4.05 0.10

Prepares for operation 4.07 4.12 0.09

Understands risks and benefits 4.11 4.19 0.01

ProfessionalismProfessionalism


Pre Post p

Follows through on tasks 4.34 4.32 0.71

Demonstrates integrity 4.44 4.45 0.38

Well groomed, appropriate attire 4.43 4.44 0.38

Available to patients/family/faculty 4.37 4.39 0.28

Communicates w/ attending effectively 4.33 4.37 0.13

Knows limitations 4.25 4.31 0.05


Pre Post p

Demonstrates care & concern 4.35 4.38 0.18

Communicates effectively:

With patients 4.31 4.35 0.13

With other healthcare providers 4.27 4.32 0.08

Maintains medical record 4.28 4.33 0.07

Interpersonal and Communication SkillsInterpersonal and Communication Skills

Systems Based PracticeSystems Based Practice


Pre Post p

Provides cost-conscious care 4.14 4.17 0.21

Effectively uses system resources 4.20 4.23 0.20


Reflect Reflect criticallycritically

Aptitude Aptitude for changefor change

Teach Teach othersothers

Overall Overall PBLIPBLI

Practice Based Learning & ImprovementPractice Based Learning & Improvement

Portfolio Learning HelixPortfolio Learning Helix


Experiences, Experiences, achievements and achievements and

deficiencies recorded deficiencies recorded in portfolioin portfolio

Resident reflectsResident reflectsin order toin order to

learn and drawlearn and drawmeaningmeaning

New knowledgeNew knowledgeis tested on newis tested on new

experiences whichexperiences whichthen are recordedthen are recorded

PBLIPBLIMentoringMentoring

Keys to Portfolio SuccessKeys to Portfolio Success

• MentorshipMentorship

• Support and acceptance by usersSupport and acceptance by users

• Minimal time commitmentMinimal time commitment


Limitations of Present StudyLimitations of Present Study

• Early analysis after implementationEarly analysis after implementation

• Other improvements in residencyOther improvements in residency


ConclusionsConclusions

• Portfolios can aid in development Portfolios can aid in development and documentation of competenceand documentation of competence

• Favorable response by residentsFavorable response by residents

• Positive effect for all competenciesPositive effect for all competencies

• Valuable organizational toolValuable organizational tool

• Integral to lifelong learning andIntegral to lifelong learning and professional developmentprofessional development


Documents

Web-Based Portfolios: A Powerful New Tool for Surgical Education