CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation

CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation

2013 CRLA ConferenceThe Boston Park Plaza Hotel

November 9, 2013Karen S. Agee, Ph.D.

Reading and Learning Coordinator EmeritaUniversity of Northern Iowa

[email protected]


On a scale of 1 (clueless) to 10 (guru), how knowledgeable are you of student learning

outcomes (SLO) assessment strategies?

1 2 3 4 5 6 7 8 9 10


Developing SLO Expertise:Do our programs and services

help students to succeed?


Self-assessment of student learning outcomes is now

required by accrediting agencies for all programs.


Higher Learning Commission Requirement

4.B. The institution demonstrates acommitment to educational achievement and improvement through ongoing assessment ofstudent learning.

(continued. . . )


Higher Learning Commission Requirement, continued:

1. The institution has clearly stated goals for student learning and effective processes for assessment of student learning and achievement of learning goals.



2. The institution assesses achievement of the learning outcomes that it claims for its curricular and co-curricular programs.



3. The institution uses the information gained from assessment to improve student learning.



4. The institution’s processes and methodologies to assess student learning reflect good practice, including the substantial participation of faculty and other instructional staff members.


Postsecondary institutions seek reaccreditation every 10 years but are expected to show long-

term data and evidence.


The Higher Learning Commissionwants every institution to develop

a “culture of assessment.”

Are we concerned about/with

student learning

outcomes assessment?

Stages of Concern Related to Change

Stage 0, Awareness:“I am not concerned about SLO

assessment.”Stage 1, Informational

“I would like to know more.”Stage 2, Personal

“How will SLO assessment affect me?”Stage 3, Management

“I seem to be spending all my time assessing student learning.”

Stages of Concern Related to Change (cont.)

Stage 4, Consequence“How will my assessment strategies

improve student learning?”Stage 5, Collaboration

“How can I relate what I am doing to what others are doing?”

Stage 6, Refocusing“I have ideas about some even better

assessment strategies!” Adapted from Hord, Rutherford, Huling-Austin, and Hall, 1987, Taking Charge of Change


Assessing student learning outcomes is a necessary and fascinating aspect of evaluation,


. . . and the CAS standards for learning assistance programs provide an outline of learning outcomes to consider.


Is it even possible to assess student learning in a learning center ?

How can tutoring center staff document gains in student learning ?


Our thesis:Nobody on campus can better assess student learning outcomes (SLO) than we can.


Do we share these premises? 1. Program evaluation is necessary.2. Student learning assessment is

an important part of program evaluation.

Program Evaluation

• New to higher education in the 1960’s• Difficult for learning centers in the 1970’s• Rare for developmental education programs before

1980’s• After publication of A Nation at Risk, considered

essential for the improvement of teaching and learning“[T]he educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a Nation and a people.”“If an unfriendly foreign power had attempted to impose on America the mediocre educational performance that exists today, we might well have viewed it as an act of war.”


Assessments in program evaluation?

• Assessment of the need for the program• Assessment of program design• Assessment of how the program is being

implemented• Assessment of the program's outcome or impact• Assessment of the program's cost and efficiency

Assessment Exercise #1~ volunteers needed ~


What is assessment?

Assessment is measurement against a standard.


“A learning assistance center is any place where learners, learner data, and learning facilitators are interwoven into a sequential, cybernetic, individualized, people-oriented system to service all students (learners) and faculty (learning facilitators) of any institution for whom learning by its students is important” (Christ, 1971, p. 39).


A learning center must then by definition

be a goal-focused program that “steers” by feedback.

Program Evaluation Measuresper Hunter Boylan, 1981

• Student grade point averages

• Gain scores from pretest to posttest

• Ratings of student satisfaction with program services

• Student retention

Limitations of Typical Evaluation Measures: GPAs

• Many variables may affect course grades and grade point averages

• Difficult to measure impact of any specific activity

• Stepwise regression incorporating all factors of GPA requires sophisticated statistical skills

Limitations of Typical Evaluation Measures: Gain Scores

• Interactive effect of pretesting

• Invalidity of commercial, norm-referenced test for assessing student gain

• Potential difference between gain score and actual behavior

Limitations of Typical Evaluation Measures: Student Satisfaction

• Questions constructed to elicit favorable responses

• Bias from survey setting or conditions• Bias from student attitude and unrelated

experiences• Not always differentiated by student usage• Satisfaction not synonymous with skill

Limitations of Typical Evaluation Measures: Student Retention

• Even more generalized than GPAs

• Confounded by transient issues causing dropping out temporarily

• Lower retention sometimes associated with program success

Program Evaluation

Meanwhile, back in the 1970’s, student affairs professionals decided to write standards for student services programs.

CAS Standards

Council for the Advancement of Standards in Higher Education (CAS)

www.cas.edu• Founded in 1979• Promotes standards in student affairs• Creates book of professional standards and

guidelines • Develops self-assessment guides


Overview of CAS Standards• General Standards (2011), www.cas.edu• 43 functional area standards available for

purchase as digital documents or printed books

• Learning Assistance Program Standards and Guidelines and LAP Self-Assessment Guide available for $35 at www.cas.edu Standards store

CAS Standards

Which standards appropriate for us?

Learning Assistance Programs (LAP)

TRiO and Other Educational Opportunity Programs (TOEOP)

CAS Standards

Learning Assistance Programs (LAP)

Approved by CAS in 1986 with “expert” input from CRLA and NADE

and revised in 1996 and 2007

CAS Standards

Ultimate purpose of CAS?

To foster and enhance student learning, development, achievement,

and citizenship.

CAS Standards: 12 Parts

1. Mission2. Program3. Organization and Leadership4. Human Resources5. Ethics6. Law, Policy, and Governance

CAS Standards: 12 Sections

7. Diversity, Equity, and Access8. Institutional and External Relations9. Financial Resources10.Technology11.Facilities and Equipment12.Assessment and Evaluation

CAS Standards: Program

CAS Learning and Developmental Outcomes (2008)Six domains, 28 dimensions

• Programs and services must assess relevant and desirable student learning and development outcomes and provide evidence of their impact on student learning and development.

• Programs must articulate how they contribute to or support students’ learning and development in the domains not specifically assessed.

CAS LAP Standards

Let’s look now at the

LAP standards (pp. 5-10 of handout)

and the Learning and

Developmental Outcomes

(separate handout)

Orthodox Methods of Assessing SLOs

Direct methods• Students demonstrate learning via some form

of standardized test focusing on aspects of student learning– Examples: CAAP, CLA, ETS Proficiency Profile

(formerly MAPP), GRE subject tests, PRAXIS exams

Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment

across the institution. Sterling, VA: Stylus.


Direct methods• Students demonstrate learning via some form

of standardized test focusing on aspects of student learning– Examples: CAAP, CLA, ETS Proficiency Profile (formerly

MAPP), GRE subject tests, PRAXIS exams

. . . but couldn’t we assess even more directly?


Indirect methods• Students report perceptions of their learning

and the educational environment that supports that learning– Examples: CSEQ, NSSE, NSLLP

Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across

the institution. Sterling, VA: Stylus.


Indirect methods• Students report perceptions of their learning

and the educational environment that supports that learning– Examples: CSEQ, NSSE, NSLLP

Couldn’t students report their perceptions even more powerfully?


Performance-based methods• Students represent learning in response to

assignments or projects that are embedded in their educational experiences

Examples: – Successful solution of a problem not previously understood– Demonstrated writing skill in a senior honors paper

Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution.

Sterling, VA: Stylus.


Performance-based methods• Students represent learning in response to

assignments or projects that are embedded in their educational experiences

Examples: – Successful solution of a problem not previously understood– Demonstrated writing skill in a senior honors paper

Couldn’t we assess students’ performance?

CAS Self-Assessment Outline

1. Review/revise mission2. Identify program goals and learning goals3. Align major programs and services with goals4. Specify desired outcomes • Program and operation outcomes • Learning and development outcomes

5. Map learning and development goals to CAS learning domains and dimensions

6. Determine assessment approach





Specify Desired Outcomes

Program and Operation

Learning and Development

Write SLO Statements:

• Specify who, under what conditions, what, to what extent, how, why

• Consider immediate learning and longer-term outcomes






Map learning and development goals to CAS learning domains and dimensions


1. Review/revise mission2. Identify program goals and learning goals3. Align major programs and services with goals4. Specify desired outcomes • Program and operation goals • Learning and development goals



Enrich Program Evaluation With SLO Assessment

What student learning outcomes do you want to assess?

Any volunteers?


Keeling & Associates’ Format for Writing Student Learning Outcomes

Someone who does ___ will be able to do ___ to a degree, and it

is demonstrated by their doing ____, which can be shown to ___ who requires it for

____.


In learning assistance programs, we seek productive behavior change as evidence of learning.

• Students who complete Speed Reading will report an increase in assignment reading of at least 50%, as demonstrated by pre- and post-course surveys.

• Students who complete Speed Reading will be able to read twice

as efficiently, as demonstrated by doubling reading efficiency score on pre/post DRTs.


In learning assistance programs, we seek productive behavior change as evidence of learning.

• Participants in an In One Ear and Out the Pen workshop will be able to list and describe the four basic steps of Quiz Notes in their own words, as demonstrated by reporting these steps orally to the presenter at the end of the workshop.

• Participants in an In One Ear and Out the Pen workshop will be able to use Quiz

Notes to prepare for an exam in a lecture course as demonstrated by bringing completed notes to an Ask-a-Tutor session and answering at least 75% of Quiz Notes questions correctly without peeking when quizzed by a Peer Instructor.


It’s time to look at the assessment samples handout.

Three Assessment Approaches for LAPs

• Dorothy Williams’ Measures of Metacognitive Development

• UNI Reading & Learning Center’s Knowledge-Skills-Attitudes

• May and Harris’ Scale: Where Are You Now?

Dorothy Williams’ Measures of Metacognitive Development

www.lsche.net/resources/lrnr_asses/assess_slo_metacgntv_rubric.htm

• SLOs and instruments developed for metacognitive development, levels of critical thinking, locus of control, and use of learning styles

• Rubrics developed for metacognitive motivation, acquisition, retention, and performance

• SLOs and instruments used in general tutoring, writing center, math center, reading center, academic skills center

• SLOs and instruments used for both appointment-based and drop-in services of these offices

UNI Reading & Learning Center’s Knowledge-Skills-Attitudes

• Developed from UNI’s Qualities of an Educated Person project by Karen Agee and Peer Instructors

• K-S-A used for Ask-a-Tutor appointments and Coordinator consultations

• K-S-A assessment made collaboratively with student whenever possible

• End-of-session assessment technique• Growth and change in Knowledge-Skills-Attitudes


Knowledge• Understand specific subject knowledge (1)• Synthesize multiple readings, text and notes, or diverse

sources (2)• Synthesize course content with lived experience (3)• See implications of new knowledge (4)• Seek new information to solve problems (4)• Think critically (5)• Rethink previous assumptions (6)• Reason effectively (7)• Formulate creative approaches (8)


Skills• Accurately self-appraise skills, abilities, and needs (9)• Develop relationships with faculty, staff, and students

(13)• Seek and offer help appropriately (14) • Work collaboratively with others (15)• Set and pursues goals (21)• Communicate effectively (22)• Demonstrate technological competence (23)• Manage time effectively (24)


Attitudes• Demonstrate ethics and integrity (11)• Demonstrate the joy of learning (12)• Explore issues of purpose, meaning, courage,

and hope (12)• Treat others with respect (13)• Appreciate diversity and difference (17)• Demonstrate social responsibility (18)

May and Harris’ Scale: Where Are You Now?

• Developed to assess problems and progress in tutoring sessions

• Used by tutors and student clients to make progress visible

May and Harris’ Scale: Where Are You Now?

Setting Goals for a Tutoring SessionCRLA Handbook for Training Peer Tutors and Mentors

Mark S. May, Clayton State University, & Jacqueline Harris, Ball State University

“As soon as [session] goals have been set, the tutor can ask, ‘On a scale of 1-10, how far along are you to reaching this goal?’ This simple exercise sets the stage for the end of the session, when the tutor can ask the same question again. It gives both the tutor and student the opportunity to provide each other with immediate feedback and positive reinforcement. It also helps define short- and longer-term goals for the client and establishes a possible framework for the next session.”

1 2 3 4 5 6 7 8 9 10

Mission Statements

• Delta College (Michigan):The mission of the Teaching/Learning Center is to support students to become successful independent learners. This support from professional staff, peer tutors, other students, or by way of T/LC services to faculty includes determination of needs; individual, group, and distance facilitation; and services needed to promote lifelong learning and the mission of Delta College.

Mission Statements

Liberty University (Lynchburg, VA):The primary purpose of the Bruckner Learning Center is to provide University-wide academic support services for all students and faculty in general and special needs students in particular.

Mission Statements

Owensboro Community and Technical College (Kentucky):The Teaching and Learning Center's Mission is:

to promote student success by sharing our knowledge of the learning process with students, faculty, administration, and staff in a collaborative partnership for the enhancement of learning; and

to promote student success by providing academic support services to faculty.

Mission Statements

Paradise Valley Community College (Phoenix, AZ):

The mission of the Learning Support Center is to support learning and help everyone achieve their potential. We provide hope and encouragement in a friendly, safe environment.

Mission Statements

University of Texas (Austin):UT Learning Center: Our mission is to empower students in their pursuit of lifelong learning within a safe and inclusive learning environment.

Mission Statements

Virginia Wesleyan College (Norfolk):

The Learning Center coordinates tutorial, placement, and academic support services at Virginia Wesleyan College. Academic support services include transitional advising during Summer Orientation and Winter Session, academic counseling for provisional students and those on academic probation, and disability services. Our mission is to give students the resources they need to become successful independent learners while maintaining an open channel of communication with faculty. We supplement faculty instruction by providing peer and professional tutoring, appropriate resource materials to supplement course work, study skills workshops, and alternate testing facilities.

Mission Statements

Southwestern Michigan College:

The Teaching & Learning Centers of SMC are dedicated to creating an environment where students, staff, faculty and community members may seek support to help achieve success in both their academic and personal lives.

What is a rubric?

A rubric is a scoring scale utilized to measure a student’s or staff

member’s performance against a predetermined set of criteria.

What does a rubric do?

A rubric

divides a desired outcome into its component parts

identifies what are acceptable and unacceptable degrees of performance for a specific outcome

• These component parts serve as criterion points,

and

the rubric provides explanations of appropriate degrees of performance for each criterion


Back to our first question:

On a scale of 1 (clueless) to 10 (guru), how knowledgeable are you of student learning

outcomes (SLO) assessment strategies?

1 2 3 4 5 6 7 8 9 10

Enrich Program Evaluation With Student Learning Outcomes Assessment

ReferencesBoylan, H. R. (1981). Program evaluation: Issues, needs, and realities. In C. C. Walvekar

(Ed.), Assessment of learning assistance services (pp. 3-16). San Francisco, CA: Jossey-Bass.

Christ, F. (1971). Systems for learning assistance: Learners, learning facilitators, and learning centers. In F. L. Christ (Ed.), Interdisciplinary aspects of reading instruction: Proceedings of the Fourth Annual Conference of the Western College Reading Association (pp. 32-41). Los Angeles, CA: WCRA.

Hord, S. M., Rutherford, W. L., Huling-Austin, L., & Hall, G. E. (1987). Taking charge of change. Alexandria, VA: ASCD.

Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.

Karen S. Agee, Ph.D., served the College Reading and Learning Association (CRLA) as secretary, president-elect (conference chair), president, past president (coordinator of state, region, and chapter leaders and special interest group leaders) and executive assistant to the Board. Karen currently represents CRLA on the Council of Learning Assistance and Developmental Education Associations (CLADEA) and on the Board of Directors of the Council for the Advancement of Standards in Higher Education (CAS). She serves on the editorial board of the Journal of Developmental Education, The Learning Assistance Review, and the Journal of College Reading and Learning. She has received CRLA’s Robert Griffin Award for Long and Outstanding Service and the CRLA Board’s Special Recognition Award, UNI’s Exemplary Service Award, and the Iowa Board of Regents’ Award for Staff Excellence. Karen coordinated the Reading and Learning Center of the learning center at the University of Northern Iowa 1984-2009. She had originally intended to teach Latin and Greek but was sidetracked by interest in the reading comprehension difficulties of her elementary and college students in Indiana and New Mexico. In 2012 Karen co-edited with Russ Hodges the Handbook for Training Peer Tutors and Mentors (Cengage Learning). She previously collaborated with Dr. Hodges on the program management chapter in Handbook of College Reading and Study Strategy Research (2nd ed., edited by Rona F. Flippo and David C. Caverly, 2008, Routledge).

Documents

CAS Assessment: Using Student Learning Outcomes to Transform Program Evaluation