Educating Ourselves through Program Assessment – Continuously Improving our Service to Students...

Educating Ourselves through Program Assessment –

Continuously Improving our Service to Students Dr. Ann Groves Lloyd

Associate DeanStudent Academic Affairs

College of Letters & ScienceUniversity of Wisconsin-Madison

Snapshot of Student Academic Affairs

Associate Dean

Deputy Assoc. Dean for Advising Deputy Assoc. Dean for Policy Deputy Assoc. Dean for Pathways

CCASCareer Services

FIGSLSACAAP

Assistant DeansFor Policy

American IndianAfrican AmericanSoutheast Asian

Chicano/aUndergrad Research Scholars

Summer Collegiate ExperienceWriting Fellows

Letters & Science Student Academic Affairs Demographics• Approximately 18 units• Size varies from 15 staff members to one• Some units have assigned populations• Some units have a campus-wide mission• College encompasses nearly 17,000

students• Familiarity and use of tracking and

evaluations tools varied widely

SAA – Vision for Assessment

• Each unit would assess program quality, productivity, need, and demand

• Continuously improve quality of programs offered by capturing voices of students and other constituents

• Enhance program planning and budgeting• Establish baseline data to explore longitudinal

trends and patterns• Satisfy institutional/state/national reporting and

review requirements

Charge to Assessment Committee

• Develop instruments to measure student perceptions and overall quality of student experience

• Develop standardized expectations for unit directors/coordinators

• Provide staff with a practical, helpful tool• Begin systematizing data collection to acquire

retention and longitudinal data• Pilot project in place for 2002-03 academic year

L&S SAA Assessment Committee

• Made up of a cross-section of staff in SAA

• Determined that a handbook would be the best route to go

• Developed timeline and content areas

• Split into teams to work on each section

Factors for Success

• Involved those who will implement and use the Handbook

• Process involved on-going negotiation and compromise

• Wanted to take the assessment picture down to very small and practical increments

• Took existing systems into account• Developed flexible, adaptable plan• Encourage risk-taking

Foundation

• Differentiates between tracking for annual reporting and collecting data for program improvements (Service Quality versus Student Learning Outcomes)

• Does not ask unit directors/coordinators to get into determining retention rates

• Next steps will be defining and measuring learning outcomes

Handbook Introduction

• Two-hour session with unit coordinators/directors in May 2002

• Associate Dean discussed her vision for assessment for our division of student academic affairs

• Assessment Team walked everyone through the contents of the Handbook

Handbook Contents

• Associate Dean’s vision for Assessment in Student Academic Affairs

• Ethics in assessment• Expectations for unit directors regarding

collecting usage statistics and program improvement data, developing annual reports, and conducting self-studies

• Primer on the basic tools of assessment• A resource section

Ethics in Evaluation

• Ensure ethical treatment of research participantsAvoid harmInformed consentFreedom from coercion

• Ensure ethical collection of data• Ensure ethical interpretation and

presentation of data

Primer on Statistical Concepts

• Includes brief discussion and cautions regarding interpretation of data

• Also addresses managing data, including issues of confidentiality and security

• Has changed significantly since evolution of on-line survey tools such as Zoomerang.com and Survey Monkey

On-Going Assessments

• Capturing student names/IDs in ISIS and/or stand-alone databases (discouraged)

• Collect demographic information about program participants

• Capture information about advising appointments• Establish systems to regularly assess student satisfaction

with services• Obtain feedback for program improvement• Systematically gather data to assess retention and

persistence (done at the divisional and college level)

Ways to Collect On-Going Data

• Phone calls• Emails• Advising contacts• Resource use• Workshop attendance• Events in general• Website hits• Number of students turned away from services

Annual Reports

• Each SAA unit will submit an assessment plan in September each year.

• Each unit will submit an annual report due in January each year.

• Responsibility of unit director/coordinator• Snapshot of previous year’s

accomplishments and a look toward future initiatives and challenges

Contents of Annual Report

• Table of Contents

• Executive Summary

• Mission Statement and Core Values

• Strategic Plan if applicable

• Overview of past year

• Future goals and initiatives

Self Studies

• Ten-year cycle for self studies

• Dean’s office has determined expectations for self studies

• Additional details are outlined in the handbook

Assessment 101

• Selecting Assessment Tools

• Statistical Concepts

• Interpretation

• Managing Data

• Surveys

• Interviews

Selecting Assessment Tools

• Quantitative

• Quantitative

• Advantages and disadvantages of each

• Encourages multiple measures and methods

Surveys

• Gathering demographic information

• Types of questions

• Ways to gather survey data

• Tips

• Survey Kit by Arlene Fink (Sage Publications)

Interviews

• Role of human subjects committee

• FAQ prior to interviews

• Focus groups versus in-depth individual interviews

• Analyzing interviews

Resource Section

• List of campus resources• List of web-based resources• Books on assessment• Article by Upcraft & Schuh from About Campus

(1998)• AAHE – 9 Principles of Good Practice for

Assessing Student Learning• Handbook can be found at:

http://wiscinfo.doit.wisc.edu/obpa/Assessment/SAA_Assessment.htm

Where are we now?

• Got the tracking and evaluation systems established in all units

• Have had two sets of assessment plans and annual reports turned in

• Obtained feedback from unit directors/coordinators and began updating handbook

• SAA has spend the past nine months going through its own assessment and evaluation process

Final Thoughts

• Assessment can be intimidating, but knowing where you are is essential to determining where you want to go!

• Just pick someplace to begin – don’t get overwhelmed by the big picture

• Develop a culture that embraces continuous improvement and inclusion of student voices in the planning

Stretch and Take a Break!

Discuss with your colleagues your biggest challenge regarding

assessment

Self-Evaluation of Assessment Practices

• Tool for evaluating “where you’re at” regarding assessment in your unit/department

• Identify strengths upon which to build

• Identify aspects to improve or enhance

• Catalyst for important conversation and action

Composite GraphUnit Self-Evaluation of Assessment Processes

UW-Madison

A Role Model 4

Very Effective 3

Just Beginning 1

Not Happening 0

DRAFT 12.11.02

7 8

Link

ed to

Dec

isio

nM

akin

g

6

Con

tinuo

us

5

Res

ourc

es

4

Impl

emen

ted

Reg

ular

ly

3

Pro

cess

es p

art o

fA

sses

smen

t Pla

n

2

Eva

luat

ion

Pro

cess

es E

xist

1

BasicallyEffective 2