How to Develop and/or Update A Simple and Clear

Assessment Plan

Office of Academic Program Assessment

Spring 2013

Overview Introduction Definition of Basic Terms Differences Between Goals and Outcomes Criteria for Evaluating Goals and Outcomes Using Rubrics in PLOs The Next Step: Developing the Assessment

Plan

Handouts1. PowerPoint slides2. WASC Rubric for Assessing the Quality of

Academic Program Learning Outcomes3. Examples of Goals and Outcomes4. Curriculum map (Sociology)5. Rubric (BA Nutrition and Food)6. VALUE Rubric on Critical Thinking

Learning Goal“a high-level, very general statement of learning expected of graduates, aligned with the institution’s mission, vision, and values (more specific learning outcomes are derived from goals)”

(WASC Draft 2013 Handbook of Accreditation, p. 45)

Learning Outcome“a concise statement of what the student should know or be able to do. Well-articulated learning outcomes describe how a student can demonstrate the desired outcome; verbs such as “understand” or “appreciate” are avoided in favor of observable actions, e.g., “identify,” “analyze.” Learning outcomes can be formulated for different levels of aggregation and analysis. Student learning outcomes are commonly abbreviated as SLOs, course learning outcomes as CLOs, program learning outcomes as PLOs, and institution-level outcomes as ILOs.”

(WASC Draft 2013 Handbook of Accreditation, p. 48)

Assessment PlanAssessment—an ongoing, iterative process consisting of four basic steps: 1.defining learning outcomes2.choosing a method or approach and then using it to gather evidence of learning3.analyzing and interpreting the evidence4.using this information to improve student learning

(WASC Draft 2013 Handbook of Accreditation, p. 40)

Qualities of Goals(adapted from Graves, 2000)

General but not vague Transparent to all stakeholders (no jargon) Realistic Simple—usually not multifaceted Something the program addresses Stated in terms of the learner

Qualities of Outcomes(adapted from Graves, 2000)

More specific than goals Directly relate to goals

hierarchical relationship cause-effect relationship

Focus on what students learn and/or associated processes

More objectives than goals Stated in terms of the learner Transparent to all stakeholders

More Qualities of Outcomes(WASC Rubric for Assessing the Quality of

Academic Program Learning Outcomes) Complete & well-organized Both discipline-specific and institution-wide

outcomes Clearly distinguish between levels (e.g.,

undergraduate versus graduate) Have explicit criteria statements developed

by faculty (e.g., rubrics)

Example: BA General Communication

GOAL: Students will communicate effectively in a

variety of contextsSUB-GOALS:1. Demonstrate proficiency in oral

communication2. Demonstrate proficiency in written

communication

Example: BA General Communication

Learning Outcomes for Sub-Goal 1(Demonstrate proficiency in oral communication)1.determine presentation needs in different situations2.correctly use visual aids3.make appropriate language choices4.use proper structure5.effectively deliver presentations

Example: BA General Communication

Learning Outcomes for Sub-Goal 2 (Demonstrate proficiency in written communication)1.implement a variety of style sheets2.use thesis statements3.use appropriate organizational strategies4.apply transitions5.include appropriate evidentiary support material6.employ grammar conventions

The Importance of Verbs(Mager, 1975, cited in Brown, 1995)

Multiple Interpretations to know to understand to really understand to appreciate to fully appreciate to grasp the significance

of to enjoy to believe to have faith in

Fewer Interpretations to write to recite to identify to sort to solve to construct to build to compare to contrast

Criteria for Evaluating GoalsHave program goals been specified?1. Is each goal comprehensible? 2. Is each goal justified?

– represents institutional goals– represents what faculty and/or discipline feel to

be important3. Is each goal stated in terms of the learner? 4. Will all goals actually be addressed in the program? 5. Can all goals be achieved by students in the

program?

Criteria for Evaluating OutcomesHave program learning outcomes (PLOs) been developed?1. Does the program specify clear PLOs?2. Are there distinct differences in the level of

specificity between program goals and program learning outcomes?

3. How will achieving program learning outcomes help to reach program goals?

4. Are PLOs achievable within the curriculum?

Criteria for Evaluating Outcomes5. Have each of the following been considered in

developing PLOs?: – Do PLOs focus on the key knowledge, skills and

values students learn in the program? – Are the verbs in PLOs clear and unambiguous?

Do they describe the observable behaviors by which students will demonstrate learning?

– Are the criteria for performance on each PLO explicit and transparent? (Are there benchmarks?)

– How and where can you observe each PLO?– What expectations do you have for student

achievement for each PLO?

Rubrics for PLO Specification Especially for skills-based outcomes Rubrics allow faculty to agree on explicit criteria

statements that describe observable behaviors differentiate levels of performance

• e.g. , Cultural and global awareness/sensitivity including demonstrated understanding, respect and

• support of multiple perspectives from other disciplines, societies, individuals, groups, and cultures

Example Rubric

BA in Nutrition and FoodAwareness of the integration of the different concentrations in FACS and their importance to the relationships between humans and their diverse environments as individuals and groups as a whole.

Rubric Development Time-consuming Look at existing rubrics

e.g., VALUE rubrics can be used ... without change with revisions as a heuristic

Group Discussion Your program includes “critical thinking” as a

learning goal and has decided to assess it this year. You have not yet specified PLOs or criteria for this goal.

Look at the VALUE rubric1. How useful would this rubric be to your program

for establishing PLOs and explicit criteria?2. Choose one dimension. How would you need to

revise the rubric to represent how that dimension is realized in your program?

Debrief

1. What did you learn about the process of specifying PLOs and criteria from doing this?

2. What did you learn from trying to adapt the rubric?

The Next Step:Sustainable Assessment PlansGood Assessment Plans: Focus on the program (e.g., majors and

concentrations) rather than courses Have a limited number of goals and outcomes Are developed collaboratively Are ongoing Are manageable Use multiple methods of data collection Anticipate how results will be used and lead to

improvement

Comprehensive Assessment Planning

Include both content area knowledge and skills/competencies

Multi-year plan with 1-2 outcomes assessed each year

Direct and indirect methods Capstone student projects (direct) Student exit surveys: self-reporting on all

outcomes

Writing an effective assessment Plan

Outcome Assessed and/or Assessment Question

Method(s) to Collect EvidenceBrief description of what will be collected, how, and by whom. (Enter one method per row; add rows as needed.)

Method to Analyze/ Evaluate* Brief description of how the evidence will be analyzed or evaluated and by whom. (When applicable, include scoring criteria or rubric in an Appendix.)

Timeline & StatusList the semester/dates when the evidence will be collected and evaluated. 

Lead Team membersList the name(s) of those who will oversee collecting, analyzing, reporting, and using results. 

LO 1 Capstone Assignment (Direct)

Include sample size, scoring criteria, expected outcomes, etc.

LO 1 Exit Survey (indirect)

From UH-Manoa Assessment Template: http://http://manoa.hawaii.edu/assessment/resources/templates.htm

The Next Step:Sustainable Assessment Plans

What should be assessed??

IPP definition of ProgramsWhat appears on the diplomaUndergraduate majors/concentrationsGraduate majors/concentrations

Do we have to assess all of these separately??

The Next Step:Sustainable Assessment Plans

Issues• too many outcomes, not enough time• too many programs, not enough time

An Example: Biological Sciences Undergraduate programs 1,449 students (excluding pre-major) BA Biological Sciences BS Biological Sciences (7 concentrations) Common curriculum for all concentrations: two LD

and two UD coursesHow can we assess so many programs??

Too Many Programs—Not Enough Time

Learning Goals/OutcomesStudents will demonstrate the ability to

communicate in the Biological SciencesWritten CommunicationOral CommunicationInformation Literacy

Students will demonstrate the ability to apply the process of scienceCritical ThinkingQuantitative Reasoning

Skills & Competencies

Learning Goal 1: Students will demonstrate the ability to communicate in the Biological Sciences

Learning Goal 2: Students will demonstrate the ability to apply the process of science

LO’s/Courses

Written Communication

Oral Communication

Information Literacy

Critical Thinking

Quantitative Reasoning

Bio 1 I I I I IBio 2 I,D I,D I,D I,D  Bio 100 D D D D DBio 184       DBio 121       D  Bio 139 D     D DBio 160 D D D D DBio 188 M   M M MElectives Various Various Various Various Various

BA & BS-General: Curriculum Map

The problem

Shared courses: Bio 1, 2, 100, 184Other courses: problematic!Bio 188: Evolution

Required by 4 of 7 concentrationsWill capture 70% of students

Five Year Assessment PlanL.O. / Year 1 2 3 4 5

Written Communication

Bio 188/capstone

Oral Communication

Capstone

Information Literacy

Bio 188/capstone

Critical Thinking Bio 188/capstone

Quantitative Reasoning

Bio 188/capstone

Content Knowledge

Bio 188/capstone

Indirect self-assessment

Bio 188/capstone

Bio 188/capstone

Bio 188/capstone

Bio 188/capstone

Bio 188/capstone

For more information

• Please contact Office of Academic Program Assessment– Amy Liu (amyliuus@hotmail.com)– Julian Heather (jheather@csus.edu)– Shannon Datwyler (datwyler@csus.edu)– Elizabeth Strasser (strasser@csus.edu)