Improving Undergraduate Education through the Assessment of Student Learning

SACS-COC Institute on Quality Enhancement and Accreditation

July 30, 2006

Peter T. Ewell

National Center for Higher Education Management Systems

Looking Back: Origins of the Assessment Movement Two Decades Ago

• Undergraduate Reform Reports of 1985-86

• Internal Stimulus: Call for More Coherent Teaching/Learning Approaches and Information for Improvement

• External Stimulus: Stakeholder Demands for Information on “Return on Investment”

• Tensions in Motive and Message Ever Since

Why Didn’t Assessment Go Away?

• Pressure to Produce Evidence of Student Learning Outcomes Never Let Up

• By Early 1990s, Accreditors Replace States as Primary External Stimulus to Get Started

• Intermittent Federal Interest in Assessment as an Element of National Accountability

• But Resulting Faculty Ambivalence About a Process Seen as “External” and “Administrative”

Looking Back: What’s Been Accomplished?

• Assessment Is for the Most Part Perceived as Inevitable and Legitimate

• Vast Majority of Institutions Have Statements of Learning Outcomes (General and Programmatic)

• A “Semi-Profession” of Folks Involved in Assessment

• Steadily Growing Sophistication with Respect to Methods of Gathering Evidence

Looking Back: What Hasn’t Happened?

• Authentic Integration of Assessment into Faculty Cultures and Behaviors

• Assessment Activities Still Largely “Added On” to the Curriculum Instead of Being Embedded In It

• Systematic and Widespread Use of Assessment Results for Institutional and Curricular Improvement

• Proactive and Sincere Institutional Engagement with Accrediting Organizations Around Topics of Assessment

Assessment as a “Perpetual Movement?”

• Most Social Movements…– Fade as Fads, or – Go Away Because Core Ideas are Mainstreamed

• The “Assessment Movement” Has Done Neither...– External Requirements Keep it Alive– But External Pressures also Constrain Faculty Buy-In and

Meaningful Institutional Use

• Moving Beyond a “Perpetual Movement” Will Demand...– Making Assessment Real to Faculty by Connecting It to the

Actual Practice of Teaching and Learning– Re-Focusing Accountability on the Authentic Student

Abilities that Society Says it Needs

Some Prominent Changes in Higher Education’s Operating Environment

• Doing More with Less- Need for Curricula that are Effective and Efficient- Need Information About Curricular Functioning to Enable

Effective Action

• Changes in Instructional Delivery- Competencies and “Deep Learning”- Student Engagement and Role of Technology

• New Views of Accountability- “Quality”— From Inputs/Processes to Results- Stakeholder Voices — Students and Employers

Challenges to the Academy: The Internal Dimension

• Changing Paradigm of Teaching and Learning

• Resulting Changes in Academic Roles, Behaviors and Structures– Nature and Role of “Faculty” in Instruction– Patterns of Student Enrollment

• [Technology as a “Wild Card”]

A Changing Paradigm of Teaching and Learning

• From “Faculty Teaching” to “Students Learning”

• Students “Make Their Own Paths” through Multiple Learning Opportunities

• Explicit Designs for Learning Based on Research

Changing Paradigm: Possible Lines of Response

• Demonstrated Achievement Becomes Paramount, not “Seat Time”

• Students “Teach” One Another

• Individual (and Asynchronous) Paths

• Technology Seen as Opportunity to Re-Think [not as a “Solution”]

Some Implications for Student Assessment Processes

• Assessments Reinforce Common Standards for Learning Across Curricula and Classes

• “Seamless” Assessments Become an Integral Part of Curriculum and Pedagogy

• Assessments Emphasize Connections and Longitudinal Development, not Just Attainment

Assessment Approaches: A Resulting Shift in Emphasis

• Accountability-Based: Assessments Added onto Instruction to “Check Up” on the System in the Aggregate

• Scholarship and Improvement: Assessments Built into the System to Simultaneously Assure Standards and Provide Feedback on Collective Performance

Kinds of Information Needed

• Alignment of Key Learning Outcomes Across Units, Sequences, and Courses

• Match Between Curricular Design, Delivery, and Student Experience

• Match Between Instruction and Needs of Diverse “Student Bodies”

• Effectiveness of Particular Innovations and Interventions

Changing Nature of the Faculty Role

• Dis-aggregation of Instructional Roles

• More Things for Faculty to Do

• New Potential Career Patterns and Paths

New Faculty Roles: Possible Lines of Response

• Emphasize Peer Support and Collaboration in Faculty Development

• Recognize and Regularize Alternative Career Paths

• Recognize “Mentorship” as the One Thing You Can’t Responsibly Outsource

Kinds of Information Needed

• Capturing and Re-Aggregating Data About Discrete Instructional Functions

• Accounting Technology-Based Costs

• Tracking Faculty “Assets”

Changing Patterns of Student Attendance

• Increased Levels of Multi-Institutional Attendance

• Increased Complexity in Course-Taking Behavior within Institutions

• Greater (and Unpredictable) Time Lapses Between Instructional Encounters

Changing Patterns of Attendance: Possible Lines of Response

• Coherence Based on Common Practices [and Outcomes], not Common Content

• Stress Ways for Students to Exploit and Reflect on Their Own Experiences

• Establish Clear Transition Points at which to Assess Student Mastery of Key Concepts

Kinds of Information Needed

• Relationships Between Particular Institutional Experiences and Particular Outcomes

• Tracking Student Learning Styles and Individual Paths of Development

• “Episode-based” [as opposed to time-based] Data Structures

Challenges to the Academy: The External Dimension

• Increased Accountability

• Changing Expectations Regarding What Students Know and Can Do

• The “New Competition”

What Are States Doing?

• Forces Influencing State Approaches– Decreased Agency Capacity Due to Funding Cuts

– Momentum of K-12 Reform (NCLB)

– Political Uncertainty and Instability

• Types of State Policy Responses– P-16 Alignment

– Performance Measures and “Report Cards”

– Achievement Testing (Driven by K-12)

What’s Shaping Accreditation?

• Forces Influencing Accreditors– Pressure for Specific Performance from Federal

Government

– Demands from Institutions to “Add Value”

– New Models from Other Sectors and Abroad

• “New Looks” in Accreditation– Focus on Outcomes and Effectiveness

– Presenting Evidence [e.g. “Portfolios”]

– Review Approaches [e.g. “Academic Audits”]

– Connection to Institutional Planning [e.g. QEP]

Learning Outcomes: What Employers Expect

• Higher-Order “Literacies” as Well as Specific Skills

• Framed in Terms of “Practice” [not “Knowledge”]

• As Much About Attitudes as Academics [“Soft Skills”]

External Forces: Some Resulting Influences on the Academy

• Increased Emphasis on Credentialing

• Modularity and Acceleration to Increase Accessibility

• Pressure to Respond to “Students as Customers”

• Accountability “Superstructures” that Divert Attention and Information Resources

Kinds of Information Needed

• Outcomes and Performance Measures

• Data on What Experiences/Services Students Can Expect

• Peer Comparisons [Increasingly Outside the Academy] and Comparative Performance

• Needs and Satisfaction of External Stakeholders

Attributes of a Meaningful “Culture of Evidence”

• Shared Recognition That Many (But Not All) Things Are Knowable

• An Accessible Store of Information About Organizational Condition and Performance

• An Attitude Toward Problem-Solving that Minimizes “Finger-Pointing”

• Clear Follow-Through On Decisions Made and Why They Were Taken

Cultures of Evidence: Success Factors

• Visible Metaphor of Scholarship

• Beginning with Real Problems and Processes, not with “Method”

• Consistent Messages from Leadership

• Periodically “Re-Socializing” the Community

• “Closing the Loop” with Action

Cultures of Evidence: Inhibiting Factors

• Either Excessive or Non-Existent Consequences

• Alien Language and “Management Culture”

• Excessive Complexity

• Burnout and “Committee Fatigue”

• Changing the Rules

Meaningful Assessment is More About Mindset than Method

• Questions About Learning are not Just Matters of Opinion– What Information Might We Collect?– What Might We Expect to Find?– What Difference Would Finding Out Make?

• Assessment is About Improving Practice– What are We Trying to Fix?– How Good is Good Enough?– What Changes are Implied?

The Bottom Line

• For Internal Management, “Seat of the Pants” Decision-making is no Longer Sufficient– Information Used Openly, Consistently, and Continuously to

Inform Academic Decisions

• For External Constituencies, “Trust Me” is no Longer Sufficient– Need Clear, Understandable Evidence of Student Academic

Attainment

• Be Vigilant about the Information You Choose and the Signals it Sends– Make Sure that What You Measure is What You Value– Harness the Accreditation Process to Make it Happen!