U.S. Survey RevealsSmall Impact of

Outcomes Assessment on Student LearningPresented at the

31st EAIR Forum

Vilnius, Lithuania

24 August, 2009

By

Trudy W. BantaProfessor of Higher Education

andSenior Advisor to the Chancellor forAcademic Planning and Evaluation

Indiana University-Purdue University Indianapolis355 N. Lansing St., AO 140

Indianapolis, Indiana 46202-2896tbanta@ iupui.edu

http://www.planning.iupui.edu

Designing Effective Assessment:

Principles & Profiles of Good Practice

Trudy W. Banta

Elizabeth A. Jones

Karen E. Black

Jossey-Bass (Wiley) 2009

© TWBANTA-IUPUI

Profiles

Invited over 1000

Received 146

Selected 49 for use in full

Categorized all 146 and published

Web sites

© TWBANTA-IUPUI

Outline for Profiles

Background and Purpose

Methods over ? Years

Resources Required

Findings

Use of Findings

Impact of Using Findings

Success Factors

Web sites© TWBANTA-IUPUI

© TWBANTA-IUPUI

Plan

Implement

Evaluate

ImproveCulture of Evidence

~ Organization ~of

Principles & ProfilesPlanning

Implementing

Improving & Sustaining

- Building a Scholarship of Assessment

Banta & Associates

Jossey-Bass 2002

© TWBANTA-IUPUI

Types of Institutions Represented

Doctoral and Research Universities

Private 7

Public 16………..47% Master’s 13………..26% Baccalaureate 7…..……14% Associate 5………..10% ACE 1……..…..2%

© TWBANTA-IUPUI

Planning Profiles

Brigham Young University

Campus Wiki for degree learning outcomes

USMA at West Point

Interdisciplinary teams assess 10

mission-related goals for learners

Kennesaw State University

2008 CHEA Award for linking assessment

with planning, program review, faculty

development

© TWBANTA-IUPUI

USMA @ West Point6 Developmental Domains

1. Intellectual

2. Physical

3. Military

4. Social

5. Moral-ethical

6. Human spirit

© TWBANTA-IUPUI

USMA @ West PointIntellectual Domain

10 Goals (write, speak, think; engineering, math, info tech)

A. Stated learner outcomes

1. Standards

a. Rubrics developed by faculty

© TWBANTA-IUPUI

USMA @ West PointInterdisciplinary Goal Teams

use

• Curriculum-embedded direct measures of learning

• Student surveys (fr., sr.)• Graduate survey (3 years after)• Employer surveys• Employer focus groups

© TWBANTA-IUPUI

USMA @ West PointUse of Assessment Findings

Review of core curriculum

Changes in warranted areas:

History

English

Engineering

Information Technology

© TWBANTA-IUPUI

As a catalyst

for outcomes assessment

accreditation is powerful

regional

disciplinary

© TWBANTA-IUPUI

Engaging faculty

in taking a second look

at student work

for outcomes assessment purposes

may require extra pay

© TWBANTA-IUPUI

Standardized tests

of generic skills

(e.g., writing, critical thinking)

used by just 8%

always supplemented

© TWBANTA-IUPUI

© TWBANTA-IUPUI

At the University of Tennessee

CAAP

Academic Profile (now MAPP)

COMP (like CLA and withdrawn by 1990)

College BASE

© TWBANTA-IUPUI

In TN We Learned

1) No test measured 30% of gen ed skills

2) Tests of generic skills measure primarily prior learning

3) Reliability of value added = .1

4) Test scores give few clues to guide improvement actions

© TWBANTA-IUPUI

An Inconvenient Truth

.9 = the correlation between SAT

and CLA scores of institutions

thus

81% of the variance in institutions’

scores is due to prior learning

More use of RUBRICS

locally developed

VALUE from AAC&U

© TWBANTA-IUPUI

Better Ways to Demonstrate Accountability

Measures of Learning

Standardized tests in major fields

Internship performance

Senior projects

Electronic portfolios

External examiners

© TWBANTA-IUPUI

© TWBANTA-IUPUI

Student Electronic Portfolio• Students take responsibility for

demonstrating core skills• Unique individual skills and

achievements can be emphasized• Multi-media opportunities extend

possibilities• Metacognitive thinking is enhanced

through reflection on contents- Sharon J. Hamilton

IUPUI

Accountability Report85% achieve Outstanding ratings in

writing as defined . . .

78% are Outstanding in applying knowledge and skills in internships

75% are Outstanding in delivering an oral presentation

© TWBANTA-IUPUI

For External Credibility

Collaborate on rubrics

Use employers as examiners

Conduct process audits

© TWBANTA-IUPUI

© TWBANTA-IUPUI

Build Assessment into Valued Processes

1. Assessment of learning

2. Curriculum review and revision

3. Survey research

4. Program review

5. Scholarship of Teaching & Learning

6. Evaluation of initiatives

7. Faculty development

8. Promotion & tenure

9. Rewards and recognition

Impact of Using Assessment Findings

NOT

measured in

student learning gains

?WHY?

© TWBANTA-IUPUI

© TWBANTA-IUPUI

Group Assessment Has Failed toDemonstrate Institutional Accountability

• 42% less than 2 years old• Focus on improvement at unit level• Rare aggregation of data centrally• Too few faculty involved• HE scholars focused on K-12 assessment