Students becoming professionals: transparent assessment ... · 2016 RATINGS 2015 RATINGS ASSESSMENT...

CORECOMPETENCIES(CCs)FORLEADERSINBIOMEDICALRESEARCH

SampleRubric: Fullrubriccopiesavailable,onpaper.

2016RATINGS

2015RATINGS

ASSESSMENTMUSTBEBASEDONOUTCOMES:• Asstudentsareintegratedintoscience,iterativeandoutcome-

orientedfeedbackisimperative.² Formativeandsummativeassessmentsareneeded² Skill-basedperformanceassessmentdifferentiatestudent

progressiontodegree² Suchassessmentsvalidatehabitsthatarerootedinthe

behaviorsofsuccessfulstudentsandshouldbetransparenttoallstudents.

FACULTYDEVELOPMENTNEEDS:• Facultymustbetrainedinitially,andthendevelopedtoassess

appropriatelyandconsistently.

• Involvementoffacultyindevelopinginstitution-specificCoreCompetenciesiscriticalforunderstandingandcommitment.

STUDYDESIGN:• Thistypeofoutcomes-orientedsurveysinrepeatedmanner

withanalysisbenefitsdesign(andrevision)ofeffectiveprograms.

• Longitudinaldataarerequired,withrepeatedmeasuresduetothecomplexityofadmissions,studentvariability,andinter-raterreliability.

Studentsbecomingprofessionals:transparentassessmentusingCoreCompetenciesJulieDavisTurner,PattyFarrell-Cole,andStevenJ.Triezenberg

VanAndelInstituteGraduateSchool,333BostwickAve.,NE,GrandRapids,Michigan49503

Background&Rationale

REFERENCESAppreciationtotheseresourcesaswedevelopedtheframeworkforCoreCompetencies:1. Walker,G.E.,C.M.Golde,L.Jones,A.C.Bueschel,P.Hutchings.(2008)TheFormationofScholars(San

Francisco:Jossey-Bass).2. Lovitts,B.(2007)Makingtheimplicitexplicit:creatingperformanceexpectationsforthedissertation.

(Sterling,VA:StylusPress).3. NRC.(2001)Knowingwhatstudentsknow.(cited1/20/2017)

http://www.nap.edu/openbook.php?isbn=03090727274. CouncilofGraduateSchools,Assessment&ReviewofGraduateProg,1990,2005,2011.

http://cgsnet.org/assessment-and-review-graduate-programs

Evidenceofprogressionacrosscohorts• DomainsofResearch &Ethic/ProfessionalPracticeshowthelargest

increaseindevelopmentacrosscohorts.

• DomainsofKnowledge andCommunication demonstratelessevenlevelsofmastery,possiblyduetoindividualcohortattributes.

EDUCATIONALPHILOSOPHY• Interactive,problem-basedcurriculumgivesstudentsresponsibilityforlearning.• Proposal-drivenproblemsmeasurecontentmastery,creativity,andscientific

communication.• Likescience,optimallearningrequirescollaborationamongstudentsandfaculty.

WHATWELEARNED

MISSIONThemissionofVAIGSistotrainPhDscientiststobeleadingscholarsincellular,molecular,andgeneticbiologyofhumandisease.

Claim1:CCassessmentidentifiesadevelopmentalprogression.

Claim2:CCsdefinemasterybetweenthesisadviserandstudent.

Claim3:EvidencefromannualCCassessingdrivesprogramchanges.

CONCLUSIONS

• CoreCompetenciesarticulateastrongsetofexplicitexpectationsforstudentlearningoutcomesatVAIGS.

• Therubric,asadevelopmentalframework,aidsthesisadvisersinmonitoringandunderstandingtheirstudents’progressandfacilitatesstudentaccountabilitywithcommonexpectations.

• Therubricprovidesfacultywithlanguagetoconveyclearexpectationsandgraduatedratingsforobjectiverationale.

VAIGScohortsincreaseinmasteryoffourCoreCompetencyDomains(Knowledge,Research,Communications,Ethics&ProfessionalPractice),assessedbyThesisAdviser(2016averages;cohort=2-4students,n=18total)

Exceptional

Heightened

Advancing

Intermed.

Beginning

Exceptional

Heightened

Advancing

Intermed.

Beginning

cohort

Basedon“expectedoutcomes”(setpercohort):• By6th Year,allstudentshaveachievedHeightenedto

Exceptional(asanticipated– showninWHITE).• IsolatedareasofClinicalContext(Knowledge),Scientific

Ethics,orAuthorshipshowrepeatedlags.

ComprehensiveProgramReviewrecommendations(2016)employedthesepreliminarydatatoaddresslags:• Curricularmodificationsinwrittencommunicationand

grantsmanship• Revitalizedcoursein“translationalresearch”• Refreshercourseforscientificethics

Futurefacultycommitteemandateswillbeinformedby2017-18studydata.

Takentogether,CoreCompetencyoutcomesprovidesareal-time“dashboard”forprioritizingprogramchanges.

VAIGSCORECOMPETENCIESv2Thestudentcananddoes:

©VanAndelInstituteGraduateSchool2015,version2 CoreCompetencies-122915v2.docx

KNOWLEDGE Beginning Intermediate Advancing Heightened ExceptionalDescribekeyconceptsinbiomedicalscience

Describegeneralconceptswithgapsanderrors

Describegeneralconceptscorrectlyandclearly

Explainconceptswithexpandedhistoricalandcontextualdetail

Recognizeandapplykeyconceptsastheyappearinnewsettings

Evaluatenewchallengesorcontroversiesrelevanttoconcepts

Placecoreconceptsintherelevantclinicalcontext

Describethesisresearchanddiseaseassociationingeneralandlimitedterms

Describeclinicalcontextofmolecularmechanismsaspresentedbyothers

Connectclinicalissuesandmolecularmechanismsaccuratelyandbroadly

Recognizeandapplykeyconceptsastheyappearinnewclinicalsettings

Useclinicalinformationtoelicitgapsincollectiveknowledge

Knowscientificliteraturerelevanttotheresearcharea

Understandbiomedicaltextbookdescriptionsofkeyconcepts

Understandreviewarticledescriptionsofemergingconcepts

Understandandevaluateaprimaryresearcharticle

Comparecontradictorypublicationsinthedissertationresearcharea

Evaluateliteraturebeyondthedissertationresearcharea

RESEARCH Definesoundrationale/identifygapinknowledge

Performexperimentsasinstructedwithoutconsideringrationale

Understandrationaleprovidedbyothers

Userationaletojustifyprioritiesfordailywork

Buildexpandedrationaleforprojectandexperiments

Effectivelydefendgivenrationaleagainstobjectionsoralternatives

Frameappropriatehypothesis

Accepthypothesisfromothers Contributetoformationofhypotheses

Generateindependentbutincompletehypotheses

Independentlygeneratetestablehypothesescoupledwithspecificaims

Recognizealternativeandtestablehypotheses

Applycreativeandappropriateexperimentaldesign

Executeexperimentsdesignedbyothers

Collectandemployappropriateapproaches

Seekexpertadvice&gleannovelapproachesfromliterature

Criticallyevaluateandmodifyapproaches

Inventandapplyalternativestrategiesandtroubleshootspotentialoutcomes

Usecontrolsappropriately Performcontrolsasinstructedbyothers Usecontrolsformostapproaches Incorporatebothpositiveand

negativecontrolsconsistently

Applyrigorouscontrolsanduseoutcomestoredirectexperimentalpath

Analyzeunexpectedresultsandallownewdiscoveriestoemerge

Executeexperimentswithtechnicalskill

Receiveinstructionandpracticenewskillswithsupervision

Exhibitappropriateexperimentaltechniqueandseekadvicewhenexperimentsfail

Anticipateresultsandobtainreproducibleresults

Mastertechnicalunderstandingandtroubleshootexperimentalfailuresindependently

Generatehightechnicalproductivity,publication-qualityfigures;directothersintechnique

Demonstratecriticalanalysisandthinking

Acceptcurrentmodelsasaccurateandsufficient

Describeexperimentsthatgeneratedcurrentmodel

Observelimitsofcurrentmodels;identifygapsinknowledge

Comparecontraryresults,yieldingmultiplenewhypotheses

Defendpositionorhypothesis-drivenexperimentstoexpandthefieldofstudy

Integrateresultsintorelevantmodels

Acceptonlythoseresultsconsistentwithhypothesis.

Comparenewdatawithpriorresultsfromself,samelab,otherlabs

Elucidatelogicalflawsincurrentmodelsbasedonexperimentaldata

Reviseextantmodelsbasedonnewdatafromselforothers.

Createnewmodelsthatintegrateexperimentalfindingswithexternaldata;Critiquedataandanalysisofothers

COMMUNICATION Beginning Intermediate Advancing Heightened Exceptional

SpeakeffectivelyPlanandpresentshortoralsummariesinlabmeetingsormentormeetings

Planandpresentjournalcluborresearch-in-progresstalksforlaborcenterusingsimpleorpre-setfigures

Presentclear,well-organizedresearchandjournalclubtalksorposterswitheffectiveuseofschematicanddatafigures/conclusions

Presenteffectiveposterorplat-formpresentationsatconferences;handlequestionsconciselyandeffectively;managelogisticsanddisruptions

Elegantcommunicationinmultiplesettings(conferences,seminars,classroom,massmedia,adhoc).

Writeeffectively Writesentences,paragraphs,and Employstandardstructuresof Buildcoherentandlogicalargument Presentcomplexmodelsorrationale Presentcomplexmodelsorproposals

Cohortratingsdeterminecurricular“gaps”.• ThesisAdviserratingswereaveragedandcomparedto

“expectedoutcomes”foreachcohort• Thosecompetenciesinwhichratingslagareshown:

ValueofcomparingStudenttoThesisAdviserratings• Clarifiesmentorexpectationsiniterativemannerwithconsistentlanguage• Identifiesmiscommunicationsinthementor-studentrelationship• Providestangible,consistentbehaviorsthatallstudentscanworktowardcoordinately

Concordanceequals:average [(faculty scorex ) – (student scorex )]

across domainaverage [(faculty scorex ) – (student scorex )]

across domain

HistoryofCoreCompetenciesatVAIGS

18students 18students

VAIGSCoreCompetencies–Two-yearstudy

page2

©VanAndelInstituteGraduateSchool2017

Claim2.CoreCompetenciesdefinemasteryinformativemannerbetweenthesisadviserandstudent.

InordertodeterminetheeffectCore

Competencieshaveontheunderstandingof

masteryinscience,wecomparedthe

concordanceofscoresfromfacultytostudent.

Concordanceequals:

average[(facultyscorex)–(studentscorex)]acrossdomain

• Concordanceatzeroindicatesthatthestudentandfacultywereinagreement.

• Positiveconcordanceindicatesthatmean

facultyratingofstudentmasterywashigher

thanmeanstudetrating.

Figure2showshighestconcordanceoverallfourdomainsfor6thyearor2ndyearcohorts.

Discordanceishighestamong4thand5thyear

studentswherestudentratingsareconsistently

higherthanmentorratings.

Claim3.EvidencefromCoreCompetenciesannualassessmentdriveVAIGSprogramchanges.

ToidentifyCoreCompetencieswhere

VAIGS’programmaynotadequately

preparestudents,thesisadviserratings

wereusedtodeterminewhetherVAIGS

studentsexcelorlagindevelopmentof

specificcorecompetencies.

Thedataindicatethatby6thyear,all

studentshaveachievedHeightenedtoExceptionalratingsfortheResearchdomain.

However,furtherprofessionaldevelopment

isneededinEthicsandProfessionalPractice.

CONCLUSIONSCoreCompetenciesarticulateastrongsetof

explicitexpectationsforstudentlearning

outcomesinourgraduateprogram.The

rubric,asadevelopmentalframework,aids

thethesisadviserinmonitoringand

understandingtheirstudent’sprogressandfacilitatesaccountabilityforthestudent.

CONTACTSJulieDavisTurner,PhD PattyFarrell-Cole,PhD SteveTriezenberg,PhD

Co-organizer;AssociateDean Co-organizer;EvaluationSpecialist Dean,VAIGS234-5596;julie.turner@vai.org 234-5363;patty.farrellcole@vai.org 234-5708;steve.triezenberg@vai.org

Figure2.DisparitybetweenFacultyandStudentRatings.ConcordancewascalculatedforeachCoreCompetencywithineachdomainand

averagedoverthecohort(keyshownatright).

Table1.UsingCoreCompetenciesforProgramReviewbyevidenceofLags

Domain/CoreCompetencies 2ndyr

3rdyr

4thyr

5thyr

6thyr

Knowledge(3totalcompetencies)Describekeyconceptsinbiomedicalscience

Knowscientificliteraturerelevanttoresearcharea

Placecoreconceptsintherelevantclinicalcontext

Research(7totalcompetencies)Applycreative/appropriateexperimentalcontrols

Usecontrolsappropriately

Integrateresultsintorelevantmodels

Communications(3totalcompetencies)Speakseffectively

Writeseffectively

Ethics&ProfessionalPractice(6competencies)Managedatawithscientificintegrity

Engageinbestauthorshippractices

Addressethicalproblemsinscientificresearch

Complywithsafety&regulatorystandards

Displayappropriatelabcitizenship

Workcollegially&effectivelyasa

team/collaborator

LEGEND:LAGSin2015LAGSin2016 Lagsinboth2015and2016

KnowledgeResearch CommunicationEthics/

(3competencies)(7competencies)(3competencies)Prof.Practice

(6competencies)

Method:deidentified datacollection(groupedbycohort)• UsingSurveyMonkey© anduniqueidentifer codesforstudentsandmentors,VAIGSgathersanddeidentifies ratings,maintainingmentor-studentassociation.

Higherm

entor

Higherstud

ent

-1.2

-0.9

-0.6

-0.3

0.0

0.3

0.6

0.9

1.26thYear(n=2)

5thYear(n=4)

4thYear(n=3)

3rdYear(n=4)

2ndYear(n=4)

CommunicationsResearch Ethics&ProfessionalPracticeKnowledge

Concordance

POSTER#11

Whileearlyinthestudy,preliminarydatashow:• Lackofconcordance(in2015)necessitatedrelianceonmentorratings.• Cohort-specificattributesmayskewconcordance.• Improvedconcordanceoccurredin2016relativeto2015.

– Thisimprovementmaybeduetoenhancedrubricfamiliarity.

Growthovertime (limiteddisplay:only2longitudinaldatasets)

• Formostofdomains,mentorratingsincreasefrom2015to2016.• Ratingsarecomplicatedbyissueswithtimingandconsistency

5th –6th year

2nd –3rd year

-1.2

-0.9

-0.6

-0.3

0.0

0.3

0.6

0.9

1.26thYear(n=3)

5thYear(n=3)

4thYear(n=4)

3rdYear(n=4)

2ndYear(n=4)

Research(7Competencies)

Communica/ons(3Competencies)

Knowledge(3Competencies)

Ethics&ProfessionalPrac/ce(6Competencies)

Concordance

Higherm

entor

Higherstud

ent

Communicationü SpeakeffectivelyüWriteeffectivelyü Communicatetodiverseaudiences

Ethical&ProfessionalPractice

üManagedatawithscientificintegrityü Engageinbestauthorshippracticesü Addressethicalproblemsinscientificresearchü Complywithsafetyandregulatorystandardsinlabü DisplayappropriatelabcitizenshipüWorkcollegiallyandeffectivelyasateamor

collaborator

Summer2015

2010- 2014Intermittentusebyfaculty/students

February2016

December2016

RevisionofCCrubricbyfacultysubcommittee

1st CoreCompetencyAssessment

(100%compliance)

2nd CoreCompetencyAssessment

(100%compliance)

Knowledgeü Describekeyconceptsinbiomedicalscienceü Placecoreconceptsintherelevantclinicalcontextü Knowscientificliteraturerelevanttotheresearcharea

Researchskillsü Definesoundrationale/identifygapinknowledgeü Frameappropriatehypothesesü Applycreativeandappropriateexperimentaldesignü Usecontrolsappropriatelyü Executeexperimentswithtechnicalskillü Demonstratecriticalanalysisandthinkingü Integrateresultsintorelevantmodels