Assessing Professional and Personal Development in Contemporary Graduate Education Graham D. Rowles, Monica M. Underwood, John F. Watkins and Heidi H

Assessing Professional Assessing Professional and Personal and Personal Development in Development in Contemporary Graduate Contemporary Graduate EducationEducation Graham D. Rowles, Monica M. Underwood, John Graham D. Rowles, Monica M. Underwood, John F. Watkins and Heidi H. EwenF. Watkins and Heidi H. EwenUniversity of Kentucky Graduate Center for GerontologyUniversity of Kentucky Graduate Center for Gerontology

Mary T. JohnsonMary T. JohnsonIndiana University School of MedicineIndiana University School of Medicine

Graham D. Rowles, Monica M. Underwood, John Graham D. Rowles, Monica M. Underwood, John F. Watkins and Heidi H. EwenF. Watkins and Heidi H. EwenUniversity of Kentucky Graduate Center for GerontologyUniversity of Kentucky Graduate Center for Gerontology

Mary T. JohnsonMary T. JohnsonIndiana University School of MedicineIndiana University School of Medicine

• ““Business as Usual” doesn’t satisfy Business as Usual” doesn’t satisfy the public expectation for the public expectation for compassionate professionals any compassionate professionals any longerlonger

• Learners are challenged to Learners are challenged to incorporate defined guiding principlesincorporate defined guiding principles

• Graduate programs are codifying Graduate programs are codifying these principles within formal student these principles within formal student assessment processesassessment processes

Our Sector of Education Our Sector of Education is Undergoing Major is Undergoing Major

ChangeChange

Old Social ContractOld Social Contract• Professionals were perceived as Professionals were perceived as

motivated by their service to othersmotivated by their service to others

• Society relied upon professional Society relied upon professional associations to set ethical and associations to set ethical and performance standards performance standards

• Colleagues can effectively Colleagues can effectively monitormonitor compliance to standards for practicing compliance to standards for practicing professionalsprofessionals

New PerceptionsNew Perceptions• The public is suspicious of professionals The public is suspicious of professionals

of all typesof all types

• Society relies on formal administrative Society relies on formal administrative rules and government regulations for rules and government regulations for control over professional behaviorscontrol over professional behaviors

• Litigation, lobbying, and attention to Litigation, lobbying, and attention to special interest groups have altered special interest groups have altered relationships with professionalsrelationships with professionals

Emerging ParadigmEmerging Paradigm• Peer awareness & assessment of Peer awareness & assessment of

self in relationship to othersself in relationship to others

• Continuing development at all levels Continuing development at all levels to enhance professional viability to enhance professional viability

• Programs that strive to inculcate Programs that strive to inculcate learners with professional standards learners with professional standards during trainingduring training

Professional Professional CharacteristicsCharacteristics

IU School of MedicineIU School of Medicine•5 Competencies Defined by 5 Competencies Defined by Medical Expertise Medical Expertise• 4 Competencies Related to4 Competencies Related to Personal Growth & Role Recognition Personal Growth & Role Recognition

UK Graduate Center for GerontologyUK Graduate Center for Gerontology•7 Dimensions of Intellectual 7 Dimensions of Intellectual & Professional Growth & Professional Growth•4 Dimensions of Personal Growth4 Dimensions of Personal Growth

Graduate Center forGerontology


From Admission to From Admission to Graduation and Graduation and BeyondBeyondAssessing Outcomes in Assessing Outcomes in Gerontology Doctoral Gerontology Doctoral EducationEducation

Graham D. Rowles, Ph.D.

Monica M. Underwood, B.S.

John F. Watkins, Ph.D.

Heidi H. Ewen, Ph.D.

University of Kentucky

Supported by University of Kentucky Quality Enhancement Program grant

CIP 30.1101; PCS 046

Graham D. Rowles, Ph.D.

Monica M. Underwood, B.S.

John F. Watkins, Ph.D.

Heidi H. Ewen, Ph.D.

University of Kentucky

Supported by University of Kentucky Quality Enhancement Program grant

CIP 30.1101; PCS 046 Graduate Center forGerontology

Doctoral Program DevelopmentDoctoral Program Development

19891989 University of Southern CaliforniaUniversity of Southern California19911991 University of Massachusetts, BostonUniversity of Massachusetts, Boston19941994 University of South FloridaUniversity of South Florida19971997 University of KentuckyUniversity of Kentucky19981998 University of KansasUniversity of Kansas20032003 University of Maryland Baltimore/ University of Maryland Baltimore/

University of Maryland Baltimore CountyUniversity of Maryland Baltimore County20042004 University of North TexasUniversity of North Texas20052005 Miami University of OhioMiami University of Ohio20052005 Purdue University (dual discipline Purdue University (dual discipline

degree) degree)


Diversity of EmphasisDiversity of Emphasis Interdisciplinary: theoretical and appliedInterdisciplinary: theoretical and applied

((University of Southern California)University of Southern California)Public Policy

((University of Massachusetts, Boston)University of Massachusetts, Boston)Health/Mental Health/Cognitive Aging/Public Health/Mental Health/Cognitive Aging/Public Policy Policy

((University of South Florida)University of South Florida) Interdisciplinary: Aging and HealthInterdisciplinary: Aging and Health

((University of Kentucky)University of Kentucky) Applied GerontologyApplied Gerontology

((University of North Texas)University of North Texas)Social and Behavioral GerontologySocial and Behavioral Gerontology

((University of Kansas)University of Kansas)(Miami University of Ohio)(Miami University of Ohio)

Epidemiology/Policy/PsychosocialEpidemiology/Policy/Psychosocial((University of Maryland Baltimore/ University of Maryland Baltimore/ University of Maryland Baltimore County)University of Maryland Baltimore County)

Dual Degree with DisciplineDual Degree with Discipline((Purdue UniversityPurdue University))


ContextContext

1. 1. Lack of evaluation strategies and tools Lack of evaluation strategies and tools for the for the dynamic assessment of student dynamic assessment of student progress progress through doctoral programs in through doctoral programs in gerontologygerontology

2. 2. Lack of information on gerontology Lack of information on gerontology doctoral doctoral program outcomesprogram outcomes

3. 3. Lack of mechanisms for assessing Lack of mechanisms for assessing overall overall doctoral program effectivenessdoctoral program effectiveness

Graduate Center for

Gerontology

The Quality Enhancement The Quality Enhancement ProjectProject

Objective

To develop a comprehensive tracking To develop a comprehensive tracking system and evaluation tool to provide system and evaluation tool to provide ongoing assessment of student progress and ongoing assessment of student progress and doctoral program effectiveness in moving doctoral program effectiveness in moving toward specified programmatic and personal toward specified programmatic and personal goals.goals.


The Quality Enhancement The Quality Enhancement ProjectProject

Elements

1. Comprehensive program data base and resource system.

2. Self rating assessment tool for students to dynamically monitor their own progress toward accomplishing specific professional and personal goals.

3. Protocol for faculty assessment of students.

4. System of interactive feedback among students and faculty. Graduate Center for

Gerontology

The Data SystemThe Data System Characteristics Characteristics

1.1. On-line system 24 hour accessOn-line system 24 hour access

2.2. Student profiles and portfolio Student profiles and portfolio (controlled by student)(controlled by student)

3.3. Assessment tool (completed by Assessment tool (completed by students with selective students with selective faculty access)faculty access)

4.4. Controlled Accessibility Controlled Accessibility and privacyand privacy


Graduate Center for

Gerontology

ScreenScreen

Sample:Sample:

Profile and Profile and Portfolio Portfolio PagesPages

ScreenScreen

Sample:Sample:

Profile and Profile and Portfolio Portfolio PagesPages

Dimensions of Dimensions of AssessmentAssessment

Professional Growth

Scholarly and Professional Independence

Critical and Integrative Thinking

Methodological Competence

Intellectual Risk Taking/Creativity

Gerontological Knowledge Base

Scholarly Communication

Identification with Gerontology


Dimensions of AssessmentDimensions of Assessment (Professional Growth)(Professional Growth)

1) Scholarly and Professional Independence

• I am able to express my own thoughts and ideas.

• I am able to conduct myself effectively in a professional environment.

Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree,

9=Don’t KnowGraduate Center for

Gerontology

Dimensions of AssessmentDimensions of Assessment(Professional Growth)(Professional Growth)

2) Critical and Integrative Thinking

• I am confident in my ability to critically assess research design and methods.

• I feel confident integrating insights and ideas from diverse areas of research and scholarship.

Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know Graduate Center for

Gerontology


3) Methodological Competence

• I am confident in my ability to formulate useful research questions.

• I am competent working in a clinical environment (i.e. patient contact).

• I am confident in my ability to analyze and interpret data.

Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know



4) Intellectual Risk Taking/Creativity

• I am confident expressing alternative points of view in scholarly and professional settings.

• I am confident in extending theory through my own research activities.




5) Gerontological Knowledge Base

• I feel well-grounded in theory.

• I am well-versed in the current issues and problems of aging.




6) Scholarly Communication

• I am confident in my ability to orally present my research ideas and findings.

• I am confident in my ability to express my research ideas or findings in writing.

Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor

Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know



7) Identification with Gerontology

On the following scale, indicate the extent to which you identify yourself as a gerontologist.

10 – Identify myself as a gerontologist

9

8

7

6

5 – Not sure of my scholarly identity

44

33

22

11

00 – I identify myself as a disciplinarian – I identify myself as a disciplinarian

(biologist, sociologist, psychologist)(biologist, sociologist, psychologist) Graduate Center forGerontology

Dimensions of Dimensions of AssessmentAssessment

Personal Growth

Involvement/Commitment

Emotional Wellbeing/Stress Management

Physical Health

Time ManagementGraduate Center for

Gerontology

Dimensions of AssessmentDimensions of Assessment(Personal Growth)(Personal Growth)

1) Involvement/Commitment

• Engagement/involvement in departmental seminars, colloquiums, meetings and events is important to my development.

• I have opportunities to participate in the development of the program.




2) Emotional Well-Being/Stress Management

• I can adjust to the rigors of the doctoral program.

• I can maintain a sense of self as a participant in this program.




3) Physical Health

• How is participating in the doctoral program affecting your health, if at all?

(1=very negatively, 2=negatively, 3=no effect, 4=positively5=very positively, 9=don’t know)

• How often do you experience symptoms of depression or anxiety?

(1=always, 2=often/frequently, 3=occasionally/infrequently, 4=never)

Graduate Center for


4) Time Management

• I maintain adequate levels of leisure and recreation.

• I effectively manage the many responsibilities and competing demands of the program.



Self Assessment Self Assessment ScheduleSchedule

1. Entry into program

2. End of first semester (completion of initial core)

3. End of first year

4. Completion of Integrative Seminar (usually following fourth semester)

5. Following qualifying examinations

6. Following doctoral dissertation defense

7. Two years post graduation Graduate Center forGerontology

Current StatusCurrent Status

1. Pilot tested with four students (a Sigma Phi Omega Initiative)

2. Students excited about potential for file sharing and collaborative work on projects

3. Students enthusiastic about vita building potential and flexibility.


Current StatusCurrent Status (cont.d) (cont.d)

4. Concerns about open faculty access to personal growth measures (particularly emotional wellbeing/stress management). Resolution: only advisor and Director of Graduate Studies (on a “need to know” basis) have access to personal growth measures.

5. System now fully on line as of September, 2006

6. Research using the system has begunGraduate Center for

Gerontology

Future DirectionsFuture Directions

1. Continued research on student progress, productivity and outcomes (IRB approved, initial analyses completed), and curriculum evaluation and refinement of program

2. Evaluation and refinement of system

3. Sharing of system with other doctoral programs to develop comprehensive

system of evaluation of progress in gerontological education.


Assessing Student Progress Assessing Student Progress in a New Medical Curriculum in a New Medical Curriculum Using Electronic PortfoliosUsing Electronic Portfolios

Mary T. Johnson, Ph.D. Mary T. Johnson, Ph.D. Indiana University School of MedicineIndiana University School of MedicineDirector of Problem Solving Competency Director of Problem Solving Competency

Medical CompetenciesMedical Competencies•Problem SolvingProblem Solving•Lifelong learningLifelong learning•Basic Clinical SkillsBasic Clinical Skills•Effective CommunicationEffective Communication•Scientific Basis of MedicineScientific Basis of Medicine

•Social and Community Contexts of Health CareSocial and Community Contexts of Health Care•Self-Care, Self-Awareness, & Personal GrowthSelf-Care, Self-Awareness, & Personal Growth•Moral Reasoning and Ethical JudgmentMoral Reasoning and Ethical Judgment•Professionalism & Role RecognitionProfessionalism & Role Recognition

• The time is ripe for Electronic The time is ripe for Electronic Portfolios in U.S. Medical EducationPortfolios in U.S. Medical Education

• European medical portfolio modelEuropean medical portfolio model

• ACGME Toolbox – Documentation of ACGME Toolbox – Documentation of learning for Residency Programslearning for Residency Programs

• US medical schools: educational needs US medical schools: educational needs similar to other professional disciplinessimilar to other professional disciplines

Portfolios in Portfolios in Medical EducationMedical Education

• Medical schools should share responsibility Medical schools should share responsibility with medical students for their learning with medical students for their learning outcomesoutcomes

• A better, more granular picture of how A better, more granular picture of how emerging professionals understand and emerging professionals understand and assume accountability for their own learning assume accountability for their own learning is neededis needed

• A vision should develop to facilitate student A vision should develop to facilitate student planning of learning choices and assembly planning of learning choices and assembly of evidence for their learning of evidence for their learning

School-Wide ConcernsSchool-Wide Concerns

The Indiana Statewide System for Medical Education

Electronic portfolios will be used soon to assess student

competency accomplishments

at all sitesSTATEWIDESTANDARDS

Integration of separate electronic systems for

competency assessment, course grading, and

verification of student clinical encounters

Student Perception:Student Perception:Where Is Competency Where Is Competency

Content?Content?

Medical Student Medical Student

Formative Learning Experiences Embedded in the Curriculum

Students Select Best Artifacts from Across Their Medical

Learning Experiences

Students Select Best Artifacts from Across Their Medical

Learning Experiences

Validated Artifacts

Validated Artifacts

Drivenby

CurriculumSelf-Selected

Examples

• Faculty Comments Imported• Grading Imported

• Cannot be ChangedAfter Selected

Medical Student Competency Portfolio

Medical Student Competency Portfolio

THE WORKING PORTFOLIO:An Archive of

Learner Artifacts

THE WORKING PORTFOLIO:An Archive of

Learner Artifacts

Student Competency Reflections

SCIENTIFIC BASIS OF MEDICINE – Feb 27, 2006 @ 2:45pm

“The ELISA Assay is a very common and important laboratory test. I have found the

knowledge that I gained from researching this test to be invaluable. I also appreciate

opportunities to present to a group so that I can practice my communication skills.”

Competency Content:Competency Content:Self Selected ExamplesSelf Selected Examples

What are tensions between What are tensions between portfolio uses, and how can they portfolio uses, and how can they be resolved?be resolved?

Faculty Must Engage Faculty Must Engage in Discussion About in Discussion About

Best PracticesBest Practices

– MetaphorMetaphor: : Narrative medicine - “our stories” Narrative medicine - “our stories” vsvsTraditional medicine - “just the Traditional medicine - “just the facts”facts”

– Locus of controlLocus of control: : Curriculum-Driven vs Curriculum-Driven vs Learner-Defined ContentLearner-Defined Content

– PurposePurpose: accountability vs wonder: accountability vs wonder

• Course Level Student AssessmentCourse Level Student Assessment

• Clerkship Level Student AssessmentClerkship Level Student Assessment

• Competency Level Student AssessmentCompetency Level Student Assessment

• Program EvaluationProgram Evaluation

• Individual Residency Advisement & Individual Residency Advisement & Dean’s Letter PreparationDean’s Letter Preparation

• Liason Committee on Medical Education Liason Committee on Medical Education (LCME) Accreditation Documentation(LCME) Accreditation Documentation

Potential Portfolio Potential Portfolio ApplicationsApplications

• AssurancesAssurances – licensure examinations, traditional – licensure examinations, traditional checkpoints for specific accountability (USMLE: checkpoints for specific accountability (USMLE: Step 1 before Year 3, Step 2 prior to graduation) Step 1 before Year 3, Step 2 prior to graduation)

• CertificationsCertifications – BLS, ACLS, CME, Match to – BLS, ACLS, CME, Match to Residency Program & USMLE Step 3Residency Program & USMLE Step 3

• BalanceBalance – Accomplishment at Three Levels of – Accomplishment at Three Levels of Medical Competency; Integration across DisciplinesMedical Competency; Integration across Disciplines

• Reflection ConnectionsReflection Connections – an additional – an additional demonstration of the compassionate, caring demonstration of the compassionate, caring dimension of medical training dimension of medical training

Graduating Competent Graduating Competent PhysiciansPhysicians

Concluding ObservationsConcluding Observations


•Evolving Central Assessment Evolving Central Assessment GoalGoal

•Balancing Benefits and Balancing Benefits and BurdensBurdens

•Compliance IssuesCompliance Issues

•Confidentiality and Control Confidentiality and Control IssuesIssues•Relating Academic and Relating Academic and Professional Growth to Life Professional Growth to Life as a Wholeas a Whole

•Evolving Central Assessment Evolving Central Assessment GoalGoal

•Balancing Benefits and Balancing Benefits and BurdensBurdens

•Compliance IssuesCompliance Issues

•Confidentiality and Control Confidentiality and Control IssuesIssues•Relating Academic and Relating Academic and Professional Growth to Life Professional Growth to Life as a Wholeas a Whole

For further information For further information contact:contact:

• Mary T. Johnson, Ph.D.Mary T. Johnson, Ph.D.

Associate Professor of Microbiology Associate Professor of Microbiology and Immunologyand Immunology

Indiana University School of Indiana University School of MedicineMedicine

Statewide Competency Director for Statewide Competency Director for Problem SolvingProblem Solving

[email protected]@iupui.edu

• Graham D. Rowles, Ph.D.Graham D. Rowles, Ph.D.

Director and ProfessorDirector and Professor

Graduate Center for GerontologyGraduate Center for Gerontology

University of KentuckyUniversity of Kentucky

[email protected]@uky.edu

• John F. Watkins, Ph.D.John F. Watkins, Ph.D.

Associate ProfessorAssociate Professor

Graduate Center for Graduate Center for GerontologyGerontology



• Heidi H. Ewen, Ph.D.Heidi H. Ewen, Ph.D.

Post-doctoral ScholarPost-doctoral Scholar

Graduate Center for Graduate Center for GerontologyGerontology



Documents

Assessing Professional and Personal Development in Contemporary Graduate Education Graham D. Rowles, Monica M. Underwood, John F. Watkins and Heidi H