Assessment Reconsidered: Why direct measures are important and how to implement them

2014 NASPA Annual Conference * Baltimore, Maryland * March 15-19, 2014

Tuesday, March 18, 2014Stan Dura

Assessment ReconsideredWhy direct measures are important and

how to implement them


Agenda

• Introduction• Discuss the current landscape• Share rationale and examples• Discuss opportunities and implementation• Discuss implications• Q & A


Current Landscape

• We’re losing the trust of our constituents• Only 11% of business leaders & 14% of citizens

thought graduates gained the skills needed• Lumina Foundation, Gallup Report 2013

• Half of employers could not find graduates with the skills needed for entry level positions• Hart, 2013

• Student & institutional debt is skyrocketing, with blame being placed on auxiliary staff & services• Troop, 2013


Current Landscape

• We’re losing the trust of our constituents• Over 30 state governments had implemented or

were considering performance base funding for state schools in 2012• AASCU, 2013

• Experts are predicting the end of campuses with the advent of MOOCs & open-source knowledge• AASCU, 2013; Aoun, 2012


Current Landscape

• There’s evidence were not doing our job well• What schools think employers want differ

substantially from what employers say they want• Sternberg, 2013

• Students are not demonstrating gains in critical thinking and reasoning skills• Arum and Rokso, 2011


Current Landscape

• Evidence of skill is being demanded…• What do we offer?• Numbers of students attending & other counts• Satisfaction and Benchmarking• Self reports of what students say they learned• Theory and literature • Emulation of best practices


Current Landscape

• Current practices are inadequate for our time• We are tasked with ensuring students learn• And showing evidence that they’ve done so

• This requires a different approach to student learning


• Are we service providers or educators?• Do we design events or learning experiences?• Developmental theory or Learning Theory?• Who’s responsible & accountable for learning?• Is it equitable for some to get the benefit

although everyone pays?

Rationale


• Students spend around twice the amount of waking hours outside the classroom• That’s countless hours demonstrating:• Leadership and teamwork• Interpersonal successes and failures• Personal health and hygiene routines• Academic discipline• Problem solving

Rationale


• Why are we not capturing that?• UO, in one year, identified over 50 experiences and

70 outcomes to measure…directly.• Student Board members - teamwork• RA training – reflective thinking, job skills• Meditation class – heart rate and self report• Employees – professionalism, evaluation factors• Student leaders – budgets, reflection, job skills

Rationale


• We observe and evaluate students performing numerous skills and competencies• That’s evidence of:• Negotiating conflict and diverse interests• Managing emotions and behaviors• Personal health and hygiene routines• Performing job functions • Written and verbal communication

Rationale


Rationale

• Why are we not capturing that?• UO developed rubrics and knowledge based tests

for each of the outcomes• Our first step was to re-learn…

how to define outcomes


Rationale

• We had to set higher standards…

• Student ticket sales clerk = Multicultural Experience• “Student sales clerks must interact with

individuals from different backgrounds.”


Rationale

• We had to develop specific outcomes...

- As a result of ______________________, and- given ____________________________, the - student will __________________________- to a _____________ level of competency,- as measured by ________________.


Rationale


- As a result of completing conduct process, and- given ____________________________, the - student will __________________________- to a _____________ level of competency,- as measured by ________________.


Rationale


- As a result of completing conduct process, and- given different perspectives & reflection, the - student will __________________________- to a _____________ level of competency,- as measured by ________________.


Rationale


- As a result of completing conduct process, and- given different perspectives & reflection, the - student will reflect on & evaluate behaviors- to a _____________ level of competency,- as measured by ________________.


Rationale


- As a result of completing conduct process, and- given different perspectives & reflection, the - student will reflect on & evaluate behaviors- to a intermediate level of competency,- as measured by ________________.


Rationale


- As a result of completing conduct process, and- given different perspectives & reflection, the - student will reflect on & evaluate behaviors- to a intermediate level of competency,- as measured by the Conduct Reflection rubric.


Examples

We had to learn how to define behaviors- For example: Conduct reflection paper

- Define what effective reflection & evaluation look like.

- Define differences between levels of quality- We translated that into a rubric


ExamplesLearning Behavior

Beginning Intermediate Accomplished Exemplary

Taking Responsibility

for one’s actions

Resident has significant difficult recognizing and articulating a reasonable degree of responsibility.

They may deflect much of it or accept much of it

without question or critical thought.

Resident has some difficulty recognizing and articulating

a reasonable degree of responsibility. They may

deflect some of it or accept some of it without question

or critical thought.

Resident has little difficulty recognizing and articulating

a reasonable degree of responsibility. They do not deflect much if any and the responsibility they take is contextually reasonable and well-thought out.

Resident has no difficulty recognizing and articulating

a reasonable degree of responsibility. They

demonstrate no deflection at all and demonstrate

sound reasoning in terms of distinguishing their

degree of responsibility from others.

Recognizing one’s impact on

others

Resident has significant difficult recognizing the

impact of their behavior on others. They often attribute undue blame either upon

others or themselves.

Resident has some difficulty recognizing the impact of their behavior on others but can describe some

impact. They often minimize or exaggerate the seriousness of that impact.

Resident has little difficulty recognizing the impact of their behavior on others.

They often attribute reasonable degrees of

responsibility to themselves and others.

Resident has no difficulty recognizing the impact of their behavior on others.

They almost always attribute reasonable

degrees of responsibility to themselves and others.


ExamplesLearning Behavior Beginning Intermediate Advanced Exemplary

Contributes to Team Meeting

Shares ideas and opinions but does not advance the work of the group. May tend to push own ideas and withdraw when the group does not adopt their idea.

Shares ideas and opinions and sometimes moves the group forward by offering thoughtful new suggestions or ideas instead of repeating their position.

Often helps the group move forward by offering alternative solutions or courses of action that build on the ideas of others.

Consistently helps the group move forward by articulating merits of proposed ideas in a neutral non judgmental manner.

Engaging Team Members in Dialogue

Engages with and listens to the group but may not take turns and may interrupt.

Engages group by restatingthe views of other members and/or asking clarifying questions. Usually does so by taking turns with few interruptions of others.

Often engages group byconstructively building upon orsynthesizing the contributionsof others. Usually does so through taking turns and rarely interrupts others.

Consistently engages group by bothconstructively building upon and synthesizing thecontributions of others as well as noticing when someone is not participating and invitinghim/her to engage. Consistently takes turns and rarely interrupts.

Nurtures Constructive Team Climate

Generally supports a constructive group climate by treating other members respectfully. May demonsrate some instances of unconstructive or unsupportive behavior or attitude.

Often supports a constructive group climate by treating othermembers respectfully andconveying a positive attitude about the group and its work. Sometimes makes an effort to motivate other members.

Consistently supports a constructive groupclimate by treating othermembers respectfully,conveying a positive attitude about the group and its work, and often motivates other groupmembers.

Consistently supports a constructive groupclimate by treating othermembers respectfully,conveying a positive attitude about the group and its work, motivating other groupmembers, and providingassistance to group members when needed.

Negotiating Conflict Passively accepts alternate

viewpoints, ideas, and opinions.

Sometimes redirects focus toward common ground, toward task at hand (away from conflict). But generally doesn't overtly call attention to there being any conflict.

Identifies and acknowledgesconflict and stays engagedwith it. Frequently redirects focus toward common ground, toward task at hand.

Consistently addresses conflict directly and helps to manage/ resolve it in a way that strengthens overall group cohesiveness.


Examples

Different Types of Rubrics- Holistic

- Covers all aspects of the behavior in one row- Analytical

- Breaks the behavior down into specific parts- Most of the time you’ll use a hybrid of both


Opportunities

Student Leader behaviors

Roommate Agreements

Student Employee evaluations

Student Leader training

Student Employee training

Health Center appointments

Counseling sessions

Club / Org Budget requests

Service Learning blogs/reflections

Student employee communications

RA observationsFitness trainings

Career Counseling sessions


Opportunities

What opportunities are there at your institutions?


Opportunities

Let’s pick one1. Can we specify the behavior(s)?2. Can we describe how different levels of quality look?3. Is a holistic, analytical or hybrid rubric appropriate?

We’ll repeat this as time allows…


Examples

How do you capture it?- Electronically

- iRubrics, Campus Labs, Spreadsheets, etc.- Paper

- Grid on paper- Combo’s


Implications

What implications do you see?- Ideologically and culturally- Technologically- Training- ???


Thank You!!!

Stan Dura, MA, ABDDirector, Student Affairs Assessment & Research

University of Oregon514-326-8267 [email protected]

mailto:[email protected]


Education

Assessment Reconsidered: Why direct measures are important and how to implement them