Assessment as a Process of Improvement€¢ Finally, value objectives usually describe beliefs about...

GuidetoAssessmentatGreenvilleUniversity Rev.January2017EricWatterson,DeanofAssessmentandInstitutionalResearch

NationalInstituteforLearningOutcomesAssessment.(2011).TransparencyFramework.Urbana,IL:UniversityofIllinoisandIndianaUniversity,NationalInstituteforLearningOutcomesAssessment(NILOA).Retrievedfrom:http://www.learningoutcomesassessment.org/TransparencyFramework.htm

AssessmentasaProcessofImprovementTheprocessofassessmentatGreenvilleUniversityisfacilitative—itpromotesacontinuousprocessimprovementthroughreview,revision,andtheimplementationofbestpracticeswithinacademicdepartmentsandotherfacetsofcampusoperation.Assessmentisthesystematicprocessofcollectingandanalyzingdataforthepurposeofimprovinglearning.Asa“communityoflearners”thisprocesstouchesthelivesofpersonsinallsegmentsofinstitutionaloperations.Assessmentisnottheevaluationofindividualprograms,individualfaculty,staff,orstudents.ItisaprocessthatprovidesinformationastohowwellprogramswithintheUniversitycarryouttheirpurposes.Faculty,administrators,staff,andstudentsparticipateinthisprocessofongoingimprovementinhowtheUniversityservesthepublicatlarge.Theassessmentprocessisviewedasformofscholarship,awayofinquiringastowhethertheprogramsoftheUniversityareactuallyproducingthedesiredoutcomesandgoals.Thecontentofeachassessmentplanisuniquetoeachprogramanddepartmentbuttheprocesswithineachareainvolvessimilarstepsinacycleofmeasurement,analysisandreporting,developingimprovements,implementingimprovements,andevaluatingtoseeifthedesiredchangehastakenplace.GreenvilleUniversityadoptedtheNILOATransparencyFrameworkin2014toassistoureffortsofcollectingandcommunicatingevidenceofourprocessofinstitutionalimprovementandtransformation.Ourwebsitedisplaysthismodelwithlinkstothecontentineachcomponent.http://www.greenville.edu/about/institutional_assessment/index.html

GuidetoAssessmentatGreenvilleUniversity

InstructionsforDevelopingaProgramAssessmentPlan

Theassessmentplanguidestheselectionandcreationofstudent-focusedassessmentactivitiesacrossthecurriculuminordertoimproveoureducationalprograms.Thisplan,incoordinationwithinstitutionalassessmentpolicies,shouldprovideguidancetodepartmentandprogramfacultyforselectingappropriatecourse-basedassessmentactivitiesthatalignwithlearningoutcomes.Overall,theassessmentplanshouldbecenteredon: A.outcomestobeachieved B.performancelevelstobeattainedrelatedtothoseoutcomes C.methodsforanalyzingtheperformanceresults D.describinghowtheresultswillbeusedforimprovement

Step1.Identifyprogramlearninggoals

“Learningiseasierwhenlearnersunderstandwhatgoaltheyaretryingtoachieve,thepurposeofachievingthegoal,andthespecificattributesofsuccess.”Chappuis,S.,&Stiggins,R.(2002).

Goalsaregeneralstatementsaboutwhatyouhopeyourstudentswillgainfromyourprogramormajor.Theyserveasthepurposetowardwhichspecificclassroomandotheractivitiesaredirectedanddonotnecessarilyneedtobemeasurable,perse.Programlearninggoalsaregenerallyaccomplishedduringalonger-termtimeframe.

• Whatwouldasuccessfulgraduateoftheprogramlookliketodayandinthefuture?• Whatarethemajoracademicgoalsstudentsshouldachieveuponcompletionoftheprogram?• Whatwouldasuccessfulstudentknowandbeabletodobytheendoftheprogram?

Eachprogramshoulddevelopapproximately3to5learninggoals.Departmentswithmultipleprogramsormajorswillneedtodeveloplearninggoalsthatcharacterizetheuniquenessofeachprogram.

Step2:Identifyprogramlearningobjectives

Objectivesarespecificstatementswithanactionverbandcontentreference.Thesespecificclassroom-basedandotherlearningactivitiesareintendedtohelpyoureachyourprogramlearninggoals.Theymustbemeasurableandhaveashort-tomedium-termtimeframeduringwhichtheyshouldbedemonstrated.Determinetheactivities,thendeterminewheretolocatetheminthecurriculum(seestep3).

ThethreemaintypesofobjectivesinuseatGreenvilleareknowledge,skill,andvalue.

GuidetoAssessmentatGreenvilleUniversity

• Whenwritingknowledgeobjectives,youaretryingtodefinethemainconcepts(e.g.,theoreticalprinciples;disciplineknowledge;etc.)thatstudentsmustknowwhentheygraduate.

• Whenwritingskillobjectives,youaretryingtodescribethelargerskills(e.g.,problemsolving;analysis;etc.)thatstudentshavegainedbythetimetheygraduate.

• Finally,valueobjectivesusuallydescribebeliefsaboutthenatureofafieldofstudyorsubjectmatter,perceptionsaboutinterdisciplinaryconnections,ethics,orexpressionofone’sChristianfaith.

LevelsofObjectivesObjectivescanalsoreflectdifferentlevelsoflearning

• MasteryObjectivesreflectminimumcompetencieswithinthediscipline–thoseskillsthatmustbemasteredbythestudentbeforemovingontothenextlevelofinstruction.Masteryobjectivestendtobeveryspecificandlimitedinscopeand,therefore,canoftenbearticulatedwithgreatspecificity(Palomba,et.al.,1999).

o Forexample,AllAccountingmajorsshouldbeableto:Balanceafinancialstatement,PrepareanExcelspreadsheet,Trackaccountsreceivable.

• DevelopmentalObjectivesreflectmorecomplex(orhigherorder)learning

outcomes–thoselearningtasksonwhichstudentscanbeexpectedtodemonstratevaryingdegreesofprogress.Notethatthesedevelopmentalobjectivesareoftenwritteninatwo-stageprocessinwhichthegeneralobjectiveisstatedalongwithasampleofspecificlearningoutcomes.

o Forexample,AccountingstudentsareexpectedtounderstandGenerallyAcceptedAccountingPractices(GAAP):ExplainGAAPinlayman’sterms,Discussdifferencesbetweenacceptedandnon-acceptedpractices,GiveexamplesofwhentouseandreferenceGAAP.

Inthiscase,theobjectiveforthestudentistounderstandGAAP.Whilesomestudentsmaydemonstrateeachofthelearningoutcomesassociatedwiththisobjective,somemayonlydemonstratetwo,andsomeonlyone.

AlistofsampleactionverbsbasedonBloom’staxonomyoflearningmaybefoundintheAppendixA.

Step3:AlignCourseswithProgramGoalsandObjectives

Onceaprogramhasclearlystatedlearningoutcomes,each courseinthecurriculumshouldbelinkedtoatleastoneprogramobjective.Somecourseswillbeassociatedwithmorethanoneobjective.

• UsetheLearningOutcomeAlignmentWorksheettodisplaytheselinkages.Examinetheworksheetforgaps.Objectivesthatarenotlinkedmayneedtobere-examinedforrelevance.Objectivesthatareover-usedmayneedtobere-examinedanddividedintosmaller,morespecificobjectives.SeeAppendixB.

GuidetoAssessmentatGreenvilleUniversity

• CreateaCurriculumMaptodisplayhowprogramobjectivesarestagedacrossthecurriculum.Programobjectivesneedatleast3touchpointsinthecurriculum,e.g.introduced,reinforced,demonstrated.SeeAppendixC.

Step4:Documentevidence

Programfacultyandstaffshouldmakedecisionstogetheraboutwhatcourseswillprovideevidencetodeterminehowwellstudentsaremeetinglearninggoalsandprogramobjectives.Iftheprogramincludesacapstonecourse,useitasoneofthecourses/learningexperiencestoselectasampleofstudentworkforassessment.Goodassessmentincludesamixofdirectandindirectevidence.

Direct Evidence - ClearandCompellingEvidenceofWhatStudentsAreLearning• Ratingsofstudentskillsbyfieldexperiencesupervisors• Scoresandpassratesonappropriatelicensure/certificationexamsorotherpublishedtests(e.g.,MajorFieldTests)thatassesskeylearningoutcomes• “Capstone”experiencessuchasresearchprojects,presentations,theses,dissertations,oraldefenses,exhibitions,orperformances,scoredusingarubric• Portfoliosofstudentwork• Scoregainsbetweenpre-andposttests(publishedorlocal)orwritingsamples• Studentreflectionsontheirvalues,attitudesandbeliefs,ifdevelopingthoseareintendedoutcomesoftheprogram

Indirect Evidence - EvidencethatStudentsAre“Probably”Learning,ButExactly WhatorHowMuchisLessClear

• Coursegrades*(seenoteonGrades,below)• Assignmentgrades,ifnotaccompaniedbyarubricorscoringguide• Admissionratesintograduateprogramsandgraduationrates• Placementratesofgraduatesintoappropriatecareerpositionsandstartingsalaries• Studentratingsoftheirknowledge,skillsandreflectionsonwhattheyhavelearnedintheprogram• Student/alumnisatisfactionwithlearning,collectedthroughsurveys,exitinterviews,orfocusgroups• Studentparticipationratesinfacultyresearch,publicationsandconferencepresentations• Honors,awards,andscholarshipsearnedbystudentsandalumni

*Grades&ProgramAssessmenta.WhyarecoursegradesinsufficientforProgramAssessment?•Coursegradesreflectwhatstudentshaveachievedinasinglecourse•Gradesareacompositeofastudent’sachievementofcourseoutcomes•Gradesreflecttheevaluationpractices,

policies,andcriteriaofindividualinstructors•Facultyteachingthesamecoursemayteachdifferentmaterial•Facultyteachingthesamecoursemayemphasizedifferentcourseobjectives

GuidetoAssessmentatGreenvilleUniversity

b.WhatgradesmightbeuseableforProgramAssessment?•Gradesonasingleitemorseriesofitemsfromanexamorquizdirectlylinkedtoaprogramobjectiveandthatallstudentsanswer•Gradesonanassignmentdirectlylinkedtoaprogramgoalandcompletedbyallstudents

•Gradesbasedonthesamegradingstandardsandcriteriaacrossallfacultyandcourseofferings(withconfirmedinter-raterreliability)•Gradessupportedbyorverifiedusingotherevidence•Gradesthatreflecttheconsensusofmultiplefaculty

*Suskie,L.(2009).Assessing student learning: A common sense guide (2nded.).SanFrancisco:Jossey-Bass

Step5:Conductassessmentsandusetheresultsforimprovement

Specificmetricstobeevaluated,aswellasbenchmarklevelsofperformancetobeachieved,shouldbeexpresslyidentifiedwithintheassessmentplan.Theperformanceresultsshouldbereviewed,analyzed,andsharedwithstakeholders.Keypointsforimprovementbasedontheseresultsshouldalsobeidentifiedandimprovementactionsshouldberecommended.Thoseactionscanthenbereviewed,approved,andimplemented.Thiscontinuousimprovementcyclecanthenberepeatedovertimecreatingahistoryofcontinuousimprovement.

Theevidencegathered(data)willbeusedtoimprovecourseofferingsintermsofcontent,deliverystrategies,learningexperiencesprovided,assignments,andlearningoutcomes.Theevidencecollectedfromindividualcoursesoragroupofcoursesinamajorwillbesharedwithprogramanddepartmentfacultytomakecourseimprovementsastheyofferthecoursesinsubsequentsemesters.Inaddition,datacanalsobeusedmid-semestertomakechangesand/orimprovementstocourseofferings.Asindicatedearlier,thegoalofthiseffortistoenhancethecurriculumtoimprovestudentlearning.

Step6:ClosetheLoop

Themissinglinkinmostassessmentplanningisadescriptionofhowtheresultingevidencewillbeusedforimprovement,whowillusetheevidence,andwhentheimprovementswillbeimplemented.Attheendofeachsemester,yourdepartmentsandprogramsshoulddiscussyourassessmentresultsandtheirimplicationsforpossibleimprovements.Describetheinstructionalimprovementsyouintenttomake,whenandhowtheywillbeimplemented,andthemethodforknowingiftheinitiativeissuccessful.SeetheAnnualAssessmentReport(AppendixD)forinstructionsondocumentingthisfinalstepofyourplan.

GuidetoAssessmentatGreenvilleUniversity

AppendixA:SampleActionVerbsSampleactionverbsforeachtypeoflearningobjective.AdaptedfromBloomsTaxonomy.Knowledge Comprehension Application Analysis Synthesis Evaluation(Remember) (Understand) (Apply) (Analyze) (Evaluate) (Create)CountDefineDescribeDrawLabelListMatchNameOutlinePointQuoteReadRecallReciteRecognizeRecordRepeatReproduceSelectStateWriteMemorizeArrangeDuplicateOrderRelateTabulate

AssociateClassifyComputeContrastConvertDefendDescribeDifferentiateDiscussDistinguishEstimateExplainExtendExtrapolateGeneralizeGiveexamplesInferIdentifyIndicateInterpretLocateParaphrasePredictReportRestateReviewRewriteTranslate

AddApplyCalculateChangeChooseClassifyCompleteComputeDemonstrateDiscoverDivideEmployExamineExperimentGraphInterpolateManipulateModifyOperatePerformPracticePrepareProduceRelateResearchOrganizeScheduleServiceShowSketchSolveSubtractTranslateTroubleshootUseUtilizeWrite

AnalyzeApplicationAppraiseBreakdownCalculateCategorizeCombineCompareConnectContrastCriticizeDesignDetectDiagramDifferentiateDiscriminateDistinguishExamineExperimentExplainInferOutlinePointoutQuestionRelateSelectSeparateSubdivideTestUtilize

ArrangeAssembleCategorizeCollectCombineCompileComposeConstructCreateDesignDevelopDeviseDriveExplainFormulateGeneralizeGenerateGroupIntegrateInventFormulateManageModifyOrderOrganizePlanPreparePrescribeProposeRearrangeReconstructRelateReorganizeReviseRewriteSetupSpecifySubstituteSummarizeTransform

AppraiseArbitrateArgueAssessAttachAwardChooseCompareConcludeContrastConvinceCoreCriticizeCritiqueDecideDefendDetermineDiscriminateEvaluateExplainGradeInterpretJudgeJustifyMeasurePredictPrioritizeRankRateRecommendRefereeRejectSelectSummarizeSupportTestValue

AppendixB.LearningOutcomeAlignmentWorksheetGreenvilleUniversity,Greenville,IL62246 Date:________________________

7

Department__________________________________________________Program______________________________________________ChairofDepartment________________________________________ProgramCoordinator___________________________________________DepartmentMissionStatement ProgramMissionStatement(includeamissionstatementforeachprograminthedepartment)DepartmentObjectives:QualitiesandcompetenciesexpectedingraduatesofthisdepartmentAtthecloseoftheirdegreestudentswillbeableto:1.2.3.4.5.expandlistasneeded

Program/MajorObjectives:Qualitiesandcompetenciesexpectedingraduatesfromthisprogram/majorAtthecloseoftheirdegreestudentswillbeableto:1.2.3.4.5.expandlistasneeded

AppendixB.LearningOutcomeAlignmentWorksheetGreenvilleUniversity,Greenville,IL62246 Date:________________________

8

StudentLearningOutcomes DepartmentObjective#

ProgramObjective#

CourseNumber

CourseObjective#

AssignmentinCourse AssessmentMethod

1.SeekTruthThroughCriticalInquiryandResearch1.1Practicecriticalself-awareness

1.2Understandourworldandcomprehendquantitativeandconceptualrelationships

1.3Thinkintegrativelytosolveproblems

1.4Applyskillsandsystematicreasoning

2.CollaborateandCommunicate2.1CommunicateandCooperate

2.2Valueothers

3.EngageCultureandBeCreative3.1Demonstrateculturalawareness

3.2Demonstratecreativityandappreciationforarts,beauty,andideas

4.DemonstrateFaithandLearninginAction4.1Recognizeworldviews

4.2ApplyChristianvalues

4.3RespondtoGod

4.4Maintainhealthyself-regardandagrowth-focusedlifestyle

Note:makesuresyllabiareloadedintoD2Lforallcourses.Usethecurrentsyllabustemplate.

Department&ProgramAssessmentPlans

AppendixC:CurriculumMapProgram:______________________________________________________________ Year:_________________

ProgramLearningOutcomesCourses/LearningOpportunities

(programoutcome1) (programoutcome2) (programoutcome3) (programoutcome4) (programoutcome5) (programoutcome6) Key:I=Introduced,R=Reinforced,D=Demonstrated*Thekeycanbemodifiedaccordingtoparticulardisciplinaryfeaturesandcurricularofferings.Forexample,programscanindicatewhethertheoutcomeisintendedtobemetatabasic(B),intermediate(I),oradvanced(A)level.Stepstofollowindevelopingacurriculummap:1.Listprogramlearningoutcomes.2.Listrequired/coreprogramcoursesandotherlearningopportunities(e.g.seniorresearch,seniorrecitals,etc.)notnecessarilyconnectedtoacourse3.Reviewsyllabitodeterminethealignmentbetweencoursecontent(assignments),courseobjectives,andprogramlearningoutcomes.4.Makeajudgmentregardingthelevelatwhichoutcomesareaddressedineachcourseandlearningopportunity(e.g.,Introduced,Reinforced,Demonstrated;Sample:

ProgramLearningOutcomesCourses/LearningOpportunities

101 202 210 212 304 310 315 412 4851 I R R R D D 2 I R R D 3 I R R D D4 I I R D D D5 I R R R R D

Key:I=IntroducedR=ReinforcedD=Demonstrated

Department&ProgramAssessmentPlans

AppendixD:AnnualReportFormInstructionsforDepartmentChairsandProgramCoordinators: Attheendofeachacademicyear,completethisworksheetandsendittothe appropriateyeardropboxinD2L.Keepacopyforyourownrecords.Developingtheyearendsummaryreport:1.BeginwithreviewingwhatyouhavediscussedinDepartmentaland/orProgrammeetingsorconversationswithyourdepartment/programfaculty. a.WhatdiscoveriesdidyoumakefromyourIDEAfeedback,orfrom comments/complaintsmadebystudentsthatcausedyoutomakechangesinhow youdeliveredorsequencedcontentinacourse? b.Werethereinnovationsinyourfieldthatinspiredyoutore-organizecontent withinacourseinordertoincludethenewinformation? c.Didyoudevelopanewwayoftestingstudentunderstandingofanareaof content?Howdidtheyrespond? d.Anyofthesetypesofresponsestostudent’sneedsandtheiractualperformance couldbeabeginningpointforyourreport.2.Outlinewhatyouchangedincurriculumdesign,testingstrategies,generatingdiscussion,developingpeerreviewtechniques,orwhateveradaptationyoumadetoimprovestudentlearning.3.Reportonwhatyoulearnedfromtheexperience.Ifyouhavenewstudentoutcomedata,summarizeit.Ifyouareintheprocessofgatheringthesedata,indicatethetimeframeinwhichitwillbeaccomplished.4.Describewhereyouwillgofromhere...whataretheimplicationsofwhatyouhavelearnedintermsoffuturechangesincurriculumand/orcoursesequencing?

ProgramAssessmentAnnualReportForm:ProgramImprovements

Program__________________________________________Semester_______________Coordinator__________________________________Date______________________FacultyParticipating:AreaforImprovementidentifiedthroughStudentOutcomeData:Facultyobservationsonthisarea:ProposedchangesinresponsetoStudentOutcomeData:TimelineforResults:CriteriaforEvaluation:

ProgramAssessmentAnnualReportForm:CourseImprovements

Program__________________________________________Semester_______________Coordinator__________________________________Date______________________FacultyParticipating:AreaforImprovementidentifiedthroughStudentOutcomeData:CourseNumber

CourseComponent

Whatdidyouchangewithinthecoursetorespondtotheproblem?

Whatresultswereobtained?Howdidstudentcompetenciesimproveasaresultofthechange?Willthiscomponentbecomeapermanentpartofthecurriculumofferingswithinthiscourse?WhatimplicationsdoesthisimprovementhavefortheProgramasawhole?