Essentials of Assessment

Essentials of AssessmentEric Hampton, Ph.D.Cindy Crowder, Ph.D.Assessment in ContextEvaluationA process of reaching a conclusion, judgment, or decision about an evaluation object. This involves judging the worth of something (Scriven, 1967).AssessmentProcedures and processes which identify, collect, and prepare data to serve evaluative needs (e.g. student outcomes, educational objectives, program objectives).MeasurementA process of systematically assigning numbers to measured attributes according to established rules.Attributes of High Quality Assessment (Stiggins, 1997)Clear TargetsClear achievement targets and knowing what you are after is a must.Focused PurposeKnow why the assessment is conducted and how results will be used.Proper MethodThe method of assessment must match the target.Sound SamplingA representative sampling of possible performances is gathered.Accuracy in AssessmentAssessment limits error and bias in measurement.

Potential Educational Assessment Targets (Adapted from Stiggins, 1997)KnowledgeReasoning/problem solvingSkillCreation of productsDispositions/attitudesPotential Assessment MethodsDirect Assessment MethodsStudent knowledge, skill or product is directly examined or observed. Student performance on the direct measure is compared against measurable performance criteria.Student work products assessmentClassroom assessment of knowledge/reasoningObservations of student skillsStandardized testsNational certification examsLocally developed testsJuried reviewSimulationsInternship evaluations based on learning outcomesPotential Assessment MethodsIndirect Assessment MethodsAssessment data gathered on reported perceptions of student learningStudent self-assessment of learningSurveys (alumni, employer)Exit InterviewsFocus groups

Assessment ParadigmsQuantitativeGathering assessment data in numeric form.Can be analyzed statistically.

QualitativeGathering assessment data in narrative form.Can provide rich detail.Matching Data to OutcomesDo not address a specific outcome with a global measure.Course grades are good reflections of overall performance in a class.Course grades are poor reflections of particular learning outcomes.Can be impacted by many factors not tied to the outcome (attendance, participation, etc.)Summative exam scores are good reflections of overall performance in a content area or construct of knowledge.Summative exams may not provide sufficient information about particular outcomes.Particular items from exams may provide a more direct measure of student performance on particular outcomes.Matching Data to OutcomesRubrics are useful in assessment of specific performance criteria for outcomes.Can be used in student papers, theses, presentations, portfolios, projects, etc.Review existing rubrics for a match with desired student learning outcomes.Revise when the coherency of the match between rubric and outcome can be improved.Analyze existing student work (course projects, papers, etc.) for match with desired student learning outcomes and develop new rubrics.Find those outcomes not adequately assessed by the first two steps and develop new student performances and corresponding rubrics.

Some Assessment GuidelinesAll assessment methods have advantages and disadvantages. There is no perfect method in the abstract.A good assessment has a strong match with the specific outcome to be assessed.A good assessment demonstrates validity for the purposes for which the data will be used.A good assessment is measured with accuracy and reliability.A good assessment is feasible in terms of resources (time, effort and money)

Use multiple methods of assessment and data gathering.Any one method of data collection carries with it its own strengths and its own weaknesses.Use of only one method leads to mono-method bias.Student learning or process outcomes should be approached from multiple aspects, utilizing multiple methods, and carried out by multiple individuals.Assessment/Evaluation StepsGenerate assessment questions based on student learning outcomes.Generate ideas on behaviors, skills, attitudes, performances, and dispositions which would provide data to answer these assessment questions.Compare questions with ideas, looking for congruence. If congruence is not perfect, consider whether ideas need to be added or removed, or assessment questions added or removed.Assess existing data sources.What is already being gathered?What would need to be developed?What can be gathered?Assessment/Evaluation StepsDevelop an assessment plan which collects data to provide answers to assessment questions.Identify how/when the data will be collected.Identify how/when the data will be analyzed.Identify how/when the data will be disseminated and to whom.Operationally define each measure.What behaviors, skills, etc. are targeted?Where will each be exhibited or measured?In what way will each be exhibited or measured?Develop measurement/evaluation/assessment instruments.Identify what differentiates successful and unsuccessful performance.Develop rubrics for assessment of performance.Assessment/Evaluation StepsInitial assessment of evaluation plan adequacy.Adjust plan/measures/metrics accordingly.Carry out assessment.Collect initial data.Analyze initial data.Evaluate analyzed data for adequacy in answering the assessment questions posed.Evaluate effectiveness of instruments.Evaluate effectiveness of analysis.Make revisions to the assessment plan if necessary.Assessment/Evaluation StepsEvaluate data in comparison to assessment questions.What are the strengths of student performance or program preparation?What are the weaknesses of student performance or program preparation?Disseminate assessment results.Consider gathering and listening to feedback from stakeholders on the usefulness/appropriateness of the findings.Continue data collection.Revise assessment plan as necessary to meet program/accreditation needs.Assessment at ISUStanding RequirementsMission StatementOutcomes LibraryCurriculum MapCommunication of OutcomesAssessment Plan2011-2012; 2012-2013; 2013-2014Assessment CycleAssessment FindingsAction Plan based on findingsStatus ReportAssessment at ISUTimelines2011-2012 Assessment CycleAssessment plan May 1, 2011.Assessment findings May 1, 2012.Action plan December 1, 2012.Status report May 1, 2013.2012-2013 Assessment CycleAssessment plan May 1, 2012.Assessment findings May 1, 2013.Action plan December 1, 2013.Status report May 1, 2014.

Aspects Involved in AssessmentMission StatementProgram Educational/Process Objectives (3-6)Student Learning Outcomes/Process Outcomes (3-6 for each objective)Student Learning Outcomes/Process Outcomes Aligned with Practices (Curriculum Mapping)Assessment Plan: which objectives/outcomes will be assessed when; methods; performance targets;Assessment: Collection, Analysis of EvidenceEvaluation: Interpretation of EvidenceAction Plan

Mission StatementLinks the function of the unit to the overall mission and strategic priorities of ISUIdentifies the programs purposeIdentifies the primary stakeholders (e.g., students)Formulating a mission statement:What is the primary function of the unit?What are the core activities?What should those whom you serve experience while/after interacting with your unit?

Outcomes LibraryLanguage of assessment:LEARNING OBJECTIVE: general knowledge/skill/ability a student should have at time of graduateLEARNING OUTCOME: Specific accomplishments to be achieved. What are you looking for in student performance to tell if they get it?Stated in the form of : + In general, aim for 3-6 OBJECTIVES and 3-6 OUTCOMES for each objective.

OBJECTIVE 1: Students will be able to conduct, analyze and interpret experiments, and apply experimental results to improve processesSLO 1.1: Students will develop and execute experiments to validate designs.SLO 1.2: Students will design and execute test plans as a part of system commissioning.

Objective 2: Students function effectively on teamsSLO 2.1: Student gathers information that relates to the teams topic.SLO 2.2: Student shares in the work of the team.SLO 2.3: Student listens to other teammates.(Could be measured using a rubric during observation and for peer evaluation.)

Curriculum MapEducational experiences (e.g., courses, internships) mapped to Objectives/OutcomesEnsure that experiences are present at appropriate levels to support student achievement of each outcomeCommunication toolFaculty identify gaps in curriculumSharing with students help them understand how their courses form a curriculum and support achievement of identified outcomes

Curriculum MapEducational Objective #1Student Learning Outcome 1.1Student Learning Outcome 1.2Student Learning Outcome1.3Student Learning Outcome1.4Courses and Learning ActivitiesEDU101 IIIEDU 115IEDU 207PEDU 242PPEDU 302 PPPEDU 499Capstone RRRRLegend: I = Introduced; P = Practiced; R = ReinforcedCommunication of OutcomesA description of how students and other stakeholders are informed about the programmatic learning outcomes.

Identify programmatic stakeholders.Identify methods for informing constituents about learning outcomes (e.g. program handbooks, syllabi, programmatic documents, web publishing)The best method of communication for students may not be the best method of communication for stakeholders.This is distinct from communication of assessment findings.Assessment PlanWhich outcomes are assessed?Include timeline not all outcomes need be assessed every year.How are the outcomes assessed?What methodology is employed to collect assessment data?Where will outcomes be assessed?What are the targets for student achievement?Who is responsible for carrying out each element of the assessment plan (identified by title)?Assessment CyclesDeveloping assessment cycles:Dont try to assess every outcome every year. Develop a timetable for assessment activities.Identify person(s) responsible for each assessment activity.Try to avoid random acts of assessment.Assessment Cycle TableStudent Learning Outcomes:2010-112011-122012-132013-142014-152015-16SLO 1.1XXSLO 1.2XXSLO 1.3XXSLO 2.1XXSLO 2.2XXSLO 2.3XXSLO 3.1XX

Assessment FindingsAggregate your dataAnalyze your dataAsk whether the target for achievement was met.If not, what are the recommendations for improvement?If you cannot tell from the collected data, revise your assessment plan.Provide supporting evidence (meeting minutes, etc.) of faculty discussions of the assessment findings and proposed improvements.Action PlanFocus on the findings.In light of the findings, what will the program do?Include a timetable for implementing this response.Evaluate and discuss the resources necessary to support the action plan.Identify individuals responsible for ensuring that implementation occurs.Assessment and Evaluation Cycles

DefinitionsFrom G. Rogers, ABET, Program Assessment of Student Learning: Keep it Simple

TermsDefinitionsProgram Educational ObjectivesProgram educational objects are broad statements that describe what graduates are expected to attain within a few years after graduation; early career (3-5 years) attributes that students will be able to demonstrate upon graduation.Student (Learning)OutcomesStudent outcomes describe what students are expected to know and be able to do by the time of graduation. These relate to the knowledge, skills, behaviors, and attitudes that students acquire as they progress through the program.Performance CriteriaSpecific, measurable statements identifying the performance(s) required to meet the outcomes; confirmable through evidence.AssessmentAssessment is one or more processes that identify, collect, and prepare data to evaluate the attainment of student outcomes and program educational objectives. Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the objective or outcomes being measured. Appropriate sampling methods may be used as part of an assessment process.EvaluationEvaluation is one or more processes for interpreting the data and evidence accumulated through assessment processes. Evaluation determines the extent to which student outcomes and program educational objectives are being attained. Evaluation results in decisions and actions regarding program improvement.Questions?An Assessment Plan TableObjective 1:Student Learning Outcome/ Process OutcomeCourses/Educational StrategiesAssessment Method(s)Source(s) of AssessmentTime of Data CollectionPerson(s) Responsible1.11.21.31.41.5