Assessing the Work of Higher Education: Institutional Effectiveness and Student Learning

Dr. Jo Allen, Senior Vice President & ProvostWidener University

Middle States Commission on Higher Education, April 2008

Overview of Presentation

Operational Terms Drivers of assessment Assessment of institutional effectiveness Assessment of student learning outcomes Blending assessments Benefits and cautions Questions and concerns

Assessment: An Operational Definition

Assessment is the process of asking and answering questions that seek to align our stated intentions with documentable realities. As such, in higher education, it deals with courses, programs, policies, procedures, and operations.

Evaluation: An Operational Definition

Evaluation focuses on individual performance in the sense of job completion and quality, typically resulting in merit raises, plans for future improvement, or—in less satisfying cases—probation and possibly firing.

Assessment vs. Evaluation

Assessment focuses on the work to be done, the outcomes, and the impact on others—not on the individuals doing the work.

Evaluation focuses on the work of the individuals—their contributions, effectiveness, creativity, responsibility, engagement, or whatever factors the organization deems most desirable.

Assessment vs. Evaluation

Assessment focuses on the work to be done, the outcomes, and the impact on others—not on the individuals doing the work.

Evaluation focuses on the work of the individuals—their contributions, effectiveness, creativity, responsibility, engagement, or whatever factors the organization deems most desirable.

Assessment of Institutional Effectiveness vs. Student Learning

Institutional effectiveness = the results of operational processes, policies, duties and sites—and their success in working together—to support the management of the academy

Student learning = the results of curricular and co-curricular experiences designed to provide students with knowledge and skills

What or who is driving assessment?

Accreditors… charged with determining the reputable from

non-reputable institutions and programs charged with checking on practices that

affect the viability and sustainability of the institution and its offerings

represent disciplinary and institutional interests

Assessment drivers (cont’d.)

The public: “Ivory Tower,” liberal bias, ratings/rankings Legislators: responsive to citizens’ concerns about quality, costs,

biases….or? Prospective faculty: Quality and meaningful contributions to students’

lives Prospective parents: real learning and preparation for careers Prospective students: How will I measure up? And what kind of job

can I get when I graduate? Funding agencies/foundations: evidence of commitment to learning

and knowledge and evidence of [prior] success

Higher Education Realities

Competitive nature of higher education– National rankings– Institutional research and data– Marketing– Niche markets

Tuition Costs Consumer attitudes of students: learning

outcomes and institutional effectiveness

Matters of Institutional Quality

Can we justify costs/prices of attendance? Can we verify the quality of our educational offerings

in measurable terms? Can we verify the effectiveness of operational

contributors to a sustainable educational experience? Can we use data and other findings to improve the

quality of our educational and operational offerings? Can we use those findings to align resources

(financial, staff, curricular, co-curricular) to enhance desired outcomes?

Sites of Institutional Effectiveness

Processes [existence and transparency]– Enrollment: Admissions, financial aid, registration– Curricular: Advising, progress toward degree completion– Budgeting: operations/salaries; capital; bond ratings and

ratios; endowment management; benefits; etc.– Planning: strategic planning, compact planning, curricular

planning, etc.– Judicial: education/training, communication, sanctions, etc.– Residence Life: housing selection, training for RAs, conflict

resolution/mediation,

Sites of Institutional Effectiveness

Units/Offices of operations– Advancement– Admissions, Bursar, Registrar– Center for Advising, Academic Support, etc.– Campus Safety– Maintenance– IT– Institutional Research– Athletics

The Assessment Cycle: Key Questions for Institutional Effectiveness

What services, programs, or benefits should our offices provide?

For what purposes or with what intended results? What evidence do we have that they provide these

outcomes? How can we use information to improve or celebrate

successes? Do the improvements we make work?

What are we looking for?

EXAMPLES of evidence: Our admission of students for whom our institution is

the first choice has risen 30%. 95% of students report satisfaction with the housing

selection process. 5 faculty committees participated in the last planning

cycle. Overall, faculty, staff, and students report feeling

safe on campus, following the new Campus Safety Improvement initiatives.

Where do we seek improvement [and what evidence will help us]?

We need to raise the number of students who choose our institution as their first choice to 95% by 2010.

All faculty committees will be invited to participate in the next planning cycle.

Students (39%) still report feeling unsafe in the mezzanine of the University Gallery. We will …..

The Iterative Assessment Cycle for Institutional Effectiveness

Mission/Purposes

Objectives/Goals

Outcomes

Implement Methods to

Gather Evidence

Gather Evidence

Interpret Evidence

Make decisions to improve programs, services, or benefits;

contribute to institutional experience; inform institutional

decision-making, planning,

budgeting, policy, public accountability

What qualities point to institutional effectiveness?

A well-articulated set of processes for critical functions

A clear line of responsibility and accountability for critical functions

An alignment of the importance of the function and sufficient resources (staff, budget, training, etc.) to support the function

Evidence of institution-wide knowledge of those critical functions, processes, and lines of responsibility

What kinds of evidence points to institutional effectiveness?

Well-managed budgets Accreditation and governmental compliance Clearly defined and supported shared governance

(board, president, administration, faculty, staff, and students)

Communication pathways and strategies [transparency]

Consensus on mission, strategic plan, goals, priorities, etc.

Student (and other constituencies’) satisfaction

How do we measure institutional effectiveness?

Tangible evidence: Audited budget statements, handbooks, enrollment data, institutional data

Records/reports of activities and/or compliance Self-studies pointing to documented evidence Surveys of satisfaction, usage, attitudes, confidence,

etc. Disciplinary accreditation reports

The Assessment Cycle: Key Questions for Student Learning

What should our students know or be able to do by the time they graduate?

What evidence do we have that they know and can do these things?

How can we use information to improve or celebrate successes?

Do the improvements we make work?

The Iterative Assessment Cycle

Mission/Purposes

Objectives/Goals

Outcomes

Implement Methods to

Gather Evidence

Gather Evidence

Interpret Evidence

Make decisions to improve programs; enhance student learning and development;

inform institutional decision-making, planning,

budgeting, policy, public accountability

Student Learning Assessment: What should students know or be able to

demonstrate by the time they graduate?

Civic engagement Diversity appreciation Communication skills Professional responsibility Ethics Critical thinking Collaborative learning Leadership Mathematical or Quantitative

competence

Technological competence Scientific competence Research skills Cultural competence Interdisciplinary competence Civic responsibility Global competence Economic/financial

competence Social justice

What might our sources of evidence be?

Essays/Theses Portfolios (faculty or external readers evaluated) Quizzes Oral presentations Homework assignments Lab experiments Tests Journal entries Projects Demonstrations

What are we looking for?

Evidence of students’ skill level (basic competency to mastery)– based on faculty-articulated standards of quality

and judgments– applied to all students’ work evenly– indicative of aggregate evaluations of

performance or knowledge– informative for course or program improvements

Can we use the same processes and strategies to assess both arenas?

Measuring learning versus effectiveness, efficiency, and/or satisfaction

– BEYOND ANECDOTAL INTO EVIDENCE

Methods of testing, projects, demonstrations versus surveys, records, reports

– QUALIFY OR QUANTIFY THE OUTCOMES

Use of results (revisions versus training)

– MODIFY WHAT YOU DO TO AFFECT OUTCOMES

What is similar?

A commitment to doing the very best job possible under whatever conditions exist

A commitment to recognizing ways that altering those conditions can affect the outcomes

A commitment to recognizing that altering the outcomes can affect the conditions

Ultimately….

We hold ourselves and our colleagues accountable for articulating the intentions of our work and then measuring the realities, resulting in designing and implementing strategies for improvement over time.

How are we doing? How can we do better?

COMMITMENT

CONSISTENCYCULTURE

CARE

QUESTIONS?

Comments?