Designing Effective & Measureable Student Learning Outcomes

2015 Bonner Assessment Institute Kristin Norris, IUPUI

Goals for Today

• Build capacity to determine WHAT you need to assess, WHY, and how does that translate into outcomes

• Align student learning outcomes with course/program activities and assessment

Getting a pulse….

Getting a Pulse

• On a scale of 1-10, how confident are you that…. 1. Have developed measurable outcomes for your course/program? 2. Have created processes that provide good evidence of that learning outcome? 3. Are able to assess the student learning? 4. Based upon evidence, know what to do differently in your course/program in order to increase student

learning?

• Why are you/are you not confident? What is causing you to question?

Why Outcomes and Assessment Matter

•Commonly asked questions •Getting beyond ‘doing good’ •Intentional decision making •Strategic plan and civic mission of institutions

Importance of Good Questions

•What do you need to know and why? •Refining your questions

•Your question should dictate everything (your plan, your methods, your approach, your measures)

Rigor in Research (Assessment)

• Rigorous thinking • Hypothesis exploration (Starts with a good question) • Systematic approach • Builds upon prior research • Contributes to the field • Triangulation across inquiry • Creates useful, relevant knowledge

(Patton, 2012)

Schoen’s Reflective Practice

•Trait of a good professional •Reflection in-action •Attentive to new knowledge •Curious, test ideas, adjust practices •Revise plans and practices - why are you doing this?

Language

TERMS• Measure/Metrics

• Assessment

• Evaluation

• Research

NOTE: There are no clear or “singular” definitions utilized across community engagement (CE), which makes it even more important for us to have a clear understanding of how we define our work.

Measures/Metrics… it’s about the outputs; how can CE be measured or quantified?

Main Example in Higher Education: numbers or “measurements” surrounding

“performance indicators” Examples of Measurements:

• # and % of faculty/staff/students involved in CE • # of internships in the community • $$ economic impact of CE

Assessment…it’s about the outcomes; what are we trying to change or impact through CE?

Main Example in Higher Education: student learning outcomes

Examples of Assessment: • What do students learn through CE experiences? • How does CE influence students’ development? • How does CE bridge “us v. them” with the community? • What skills are developed through CE experiences?

Evaluation…it’s about the practice itself; is CE effective, efficient, of a certain quality, etc.?

Main Example in Higher Education: program and/or teacher evaluation

Example Evaluation Questions: • What is a high quality service-learning experience? • Are CE programs cost-effective? • How do CE programs meet their objectives (i.e. outcomes and outputs)?

Research…it’s about being more generalizable, per se

• Institutional Research • Higher Education Research • Disciplinary Research • Community-Engaged Research (CER) • Community-Based Research (CBR)

Examples of research: • Institutional: how does IUPUI do CE? • Higher Education: how does HE do CE? • Disciplinary: how does Geography do CE? • CBR: How does one get a classroom engaged in the community? • CER: How does one engage with the community in research?

Outputs, Outcomes, Impact

Inputs Outputs Outcomes Impacts

All the resources put into the project to enable the delivery of outputs

All the activities undertaken and products and services delivered

The changes, benefits, learning or other effects that result from the outputs

The effect of a project at a higher or broader level, in the longer term, after a range of outcomes have been achieved

Starting with Outcomes in Mind

What do you want to assess?

Where do your CE assessment interests lie?Institution

School

Department

Course

Project 1 Project 2

Program of Study

Course

Project

Center/Office

Program

Project 1

Project 2

1

2

3

4

Recommended Resource: http://ccoe.rbhs.rutgers.edu/forms/EffectiveUseofLearningObjectives.pdf

Outcome Statements Should…• Describe what students should be able to demonstrate, represent, or produce based on their

learning histories; • Rely on active verbs that identify what students should be able to demonstrate, represent, or

produce over time. (Maki, 2004) • Exist at different levels:

• Lesson/Activity • Program • School/Dept/Unit/Center (multiple programs) • Institutional

• ***Should be able to map your outcomes across all of these****

Why are student learning outcomes important? • At the course level

• Build a foundation for your course • Define your expectations • Describe how your course is relevant to the students’ personal, academic, and/or professional development • Keep you focused on what is important • Enable students to articulate what they are learning and have learned.

“Students will role play the problem-solving, communication, and teamwork skills necessary to effectively resolve real-world hotel management scenarios.”

Why are student learning outcomes important? • At the unit/lesson level

• Build a structure for a unit or session • Focus each unit or session • Give learners a clear picture of what to expect • Provide criteria for constructing assessments • Guide the selection of learning activities • Teach learners how to be successful

“Given a problem with two unknowns, students will, in a step-by-step fashion, describe how to solve the problem.”

Components of Effective Outcomes • The intended outcome clarifies what you are attempting to assess

• The intended outcome is measurable

• The intended outcome is useful and meaningful • One or more methods of assessment can be explicitly tied to the intended outcome

• The CBC Method (three parts) • Condition: Under what conditions? • Behavior: What should they be able to do? • Criterion: How (well) must it be done?

ActivityIdentify issues with the learning outcome and recommend changes

for improvement

What recommendations would you make for improving the learning outcome?

•Outcome 1: Students have an increased ability to identify societal issues and community needs.

•Outcome 2: Students have an increased understanding of opportunities for community involvement.

•Outcome 3: Students deepen their understanding of the value of civic engagement. •Outcome 4: Students have an increased motivation to remain civically engaged after [insert university].

•Outcome 5: Students heighten their cultural awareness and ability to work in diverse settings

Self-check Questions

• Is it clear what you are assessing? • Is the intended outcome measurable?

• Active verbs, active verbs, active verbs

• Is the intended outcome measuring something useful AND meaningful?

• How will this outcome be measured?

Process for Developing Outcomes

IUPUI’s Process

General Hints, Tips, and Lessons Learned

•Doesn’t happen in isolation •Iterative process •The process is just as important as the product – creates shared understanding and buy-in

•Overwhelming and potentially frustrating at first

Steps Taken at IUPUI

•Regularly scheduled meetings (monthly) – all staff •Bi-weekly meetings we Dir. of Assessment and Program Director •“Imagine you had $650,000 to develop civic-minded graduates. What would that look like”

•Post-it notes – 3-4 word statements •Group all of our ideas into categories

Example from IUPUI – Civic Identity

Definition: Students embrace their role as civic agents and as social trustees of knowledge working with others to create change in the community

Outcome: • Possess a sense of responsibility and commitment to use knowledge and skills gained through

college experiences to contribute to the greater good of society(social trustee) • Make informed and principled choices that positively impact society (civic agency) • Espouse and enact values, dispositions and beliefs that lead to positive societal change • Articulate effectively how being civically engaged has fostered and informed their civic

knowledge, skills, dispositions, and future intentions

Example from IUPUI – Civic Communication SkillsDefinition: Students apply communication strategies to be effective in a community context, showing the ability to express, listen, and adapt ideas and messages based on the diverse perspectives of others

Outcomes:

• Listen critically to effectively respond to complex situations • Value the diverse perspectives of multiple stakeholders • Demonstrate empathy towards others • Understand the process, outcomes, and active participation in civil dialogue by applying civic communication skills • Value reciprocity and consensus building • Engage in an inclusive decision-making process that maximizes strengths and leads to mutually beneficial goals

Assessment Tools and Strategies for Student Learning

Part 2

Goals for Part 2

•Data considerations: What do you need, does it already exist, how will you collect it

•How will you report your findings? Who needs to know? •Example tools and strategies

Context Matters When it Come to Data

Activity 1

Getting to know your inner OZ

When you think about the context for assessing community engaged activities at your campus [within your department, program, project, etc.], which character from the Wizard of Oz do you most identify with and why?

•Auntie Em/Uncle Henry •Cowardly Lion •Dorothy Gale •Flying Monkeys •Glinda the Good Witch •Munchkins

•Ozmites •Scarecrow •Tin Man •Toto •Wicked Witch of the West •Wizard

My/Our Allies [R=real; P=Potential]

Predators [R=real; P=Potential] Individuals, Groups,

Policies, Systems     

   

 

My/Our Strengths [WD=Well Developed; R=Room for Growth;

U=Untapped]

Local/Global Threats [R=Real;

P=Potential]   

ACTIVITY 2

Developing an Assessment Plan

Using the Matrix

Completing the Matrix

• Step 1: Insert your Learning Goals and Learning Outcome/Objective (columns 1 and 2)

• Step 2: Identify course/program activities (e.g., reflections, dialogue, readings, events, trainings) – which learning outcome do they align with?

• Note: If an activity doesn’t align with one of your learning outcomes, then why are you doing it?

• Note: What happens if you’ve identified a set of program outcomes yet none of your activities can be mapped to that outcome? You have a gap in your programming

Completing the Matrix (cont.)

• Step 3: Identify your sources of evidence (column 4)

• Brainstorm sources of evidence for the following examples: • Value the diverse perspectives of multiple stakeholders

• Stay abreast of contemporary issues and make informed decisions using multiple sources of information

KEY – Be intentional and strategic

Data

So you say you have data…..Questions to consider: • What data do you already have? • What types of data are you collecting on a regular basis? • How are you currently using your data? Are you using all of it? • internal purposes vs. external purposes • What are you still unable to do with your data? Why? • Who decides what you collect, how you ask the questions, and how it is used? • Do you need more data? If so, what data do you need? • What lens(es) are you using when you consider these questions

Sources of Evidence of Student Learning

• Surveys (QR Code vs. email, timing) • Capturing the discussion during reflection sessions • Focus Group • Observations • Wordle • Second-hand reflection (in the classroom the next day) • Pictures

SYSTEMATIC

see Whitley (2014) in MJCSL

Additional Considerations: Potential Factors Influencing Student Learning• Faculty interactions • Reflection • Dialogue across difference • Direct vs indirect service • Project vs hours • Student self-select partner or project vs assigned • Community partner involvement

Closing the Assessment Loop

• If you are not using the data you are collecting – STOP • Who needs this information and what do they care about?

What is the best way to communicate/tell your story? • Need to know vs Nice to know • Starts with a good question

Examples & Tools

IUPUI’s Civic-Minded Graduate

• Civic-Minded Graduate Scale (30-items)

• Civic-Minded Graduate Scale Short-Form (6-items)

• Civic-Minded Graduate Narrative Prompt

• Civic-Minded Graduate Rubric

• Civic-Minded Graduate Rubric 2.0 (see handout)

• AAC&U National Task Force

Already Existing Tools or Processes

•Faculty Annual Report •Campus Climate Survey •Alumni Survey •Faculty/Staff Survey •PULSE Surveys •Initiatives; Course Tags •Taxonomies