1 Understanding Student Learning Outcomes and Assessment Bob Barr Andrew LaManque Rob Johnstone Adapted from material prepared by Fred Trapp, Long Beach

1

Understanding Student Learning Outcomes and

Assessment

Bob BarrAndrew LaManque

Rob Johnstone

Adapted from material prepared by Fred Trapp, Long Beach CC,

the RP Group, and James O.Nichols, University of Mississippi

Breakout Session, District Opening Day, September 22, 2005

2

Session Outcomes

Describe a multi-step process for conducting learning outcomes assessment and name the purpose of each step.

Distinguish among levels of assessment work.

3

Session Outcomes

Critique a student learning outcome statement.

Describe two means of assessment and match them to an outcome statement.

4

What Is Student Learning Outcomes Assessment?

An ongoing process of systematically, gathering, analyzing, and interpreting evidence aimed at understanding and improving student learning.

Involves making learning expectations explicit and public.

Involves setting appropriate criteria and high standards for learning quality.

5

Why Do Assessment?

The Reason – Improve student learning and institutional support services.

The Effort – Continuous reflection on the teaching-learning process

An Occasion to Our Display Work – Regional or program accreditation.

6

Accrediting Commission for Community & Junior Colleges

(ACCJC)

Standard II: Student Learning Programs and Services

“The institution engages in ongoing, systematic evaluation and integrated planning to assure currency and measure achievement of its stated student learning outcomes for courses, degrees, certificates, and general education and vocational programs.”

James Nichols, Department Guide & Record Book 7

The Assessment Process

Student Learning

Assessment Activities

3. Means of Assessment

and Criteria for Success

2. Intended Learning

Outcomes

5. Use of Results

4. Assessment Results

1. Institutional Mission & overall institutional learning outcomes

8

Assessment Process and Key Questions

1. Link of GE and program to mission & institutional outcomes; course to program outcomes.

Where does the ______ program fit into the mission and learning outcomes of the college?

2. Specify intended learning outcomesWhat learning comes of completing the course,

program, or degree?

9


3. Identify means of assessment and criteria for success

How shall we discover what students have learned? How good is good enough?

4. Assemble the results and make meaning from them

How well did they do relative to expectations?

10


5. Using the resultsWhat shall we do with these results?

Document, document, documentSuggestion…

Department/program plan Program review

11

Levels of SLO Assessment Work

College-wide General education or core

competencies Instructional Program or pre-major

or Student Service Program Course or sections of a commonly

taught course

12

Step 1. Linking to Mission -Expanded Statement of

Purpose

Mission of the college Broad statements of defining purposes and

outcomes GE, program, course purpose statements

Links the GE package and program to the mission of the college

Links the course to GE package or program goals Cornerstone of self-assessment (& self-

study) How well are you accomplishing your purposes?

13

Step 1. Linking to Mission -Linking the Three Levels

College or GE Outcome Students will be able to speak and write

effectively. Academic Program Outcome (Psychology)

Students can speak and write effectively using the principles and concepts of the discipline.

Course Outcome (Psych Measurement) Students can prepare a written summary and

interpretation of standardized test results.

14

Traditional Achievement Outcomes Success rates Retention or persistence rates Number and rates of graduates or transfers

Student Learning Outcomes What faculty intend students to know

(cognitive), think or believe (affective) or be able to do (behavioral) when they have completed

Degree (general education) Field of concentration (program) Course

Two Types of Educational Outcomes

(Institutional/Program Performance)

15

Step 2. Specify Intended Learning Outcomes

Ideally, the curriculum and courses are designed by working backwards from the ultimate outcomes.

What roles will program graduates hold and what must they be able to do “out there” in the real world or in upper division work?

How will the outcomes be demonstrated “in here?”

What skills and concepts need to be learned?

16

Step 2. Specify Intended Learning Outcomes

Helpful hints Use concrete, action verbs

See Bloom’s Taxonomy of Educational Objectives Anticipate the “deliverable”

What is it that students must do to convince you that they are where you want them to be with respect to learning?

Enabling vs. terminal Target key concepts or change points in the

program Major aspects of learning central to the discipline

17

Example: Specify Intended GE Learning Outcomes

Be exposed to good writing examples. (in need of repair)

Upon completion of a degree, the student will be able to…

Write effectively

Compose a range of written products designed for different audiences and purposes.

18

Example: Specify Intended Psychology Program Learning

Outcomes

Hear and read about the scientific method of inquiry as used in the social sciences. (in need of repair)

Upon completion of the program, the student will be able to…

Use critical thinking, skeptical inquiry, and, when possible, the scientific approach to solve problems related to behavior and mental processes..


Expanded Statement of Institutional

Purpose

Identity Statement:

Your Community College is an open-admission, community-based, comprehensive college.

Mission Statement:

Serve persons of all ages in preparing for entry and careers in a variety of fields, such as automotive technology.

Program Intended Educational Outcomes:1. Graduates of the Automotive Technology Program will be successfully employed in the field. (achievement outcome, not a SLO)

2. Graduates of the Automotive Technology Program will be technically proficient.

3. Employers of Automotive Technology Program graduates will rate them competent based on the education received in the program and would employ others.

Example: Auto Tech Program Intended Student Learning

Outcomes

20

Step 2. Specify Intended Student Services Program Learning

Outcomes

Know how to be a more prepared student. (in need of repair)

Upon completion of the EOP&S summer transition program, the student will be able to…

Develop and use a personal schedule that incorporates regular study time, regular study location and study groups to match their class schedule.

21

Characteristics of Intended Student Learning Outcomes

Student-focused, not teacher focused Focus on learning resulting from an

activity, not the activity itself Focus on important, non-trivial

aspects of learning Focus on skills & abilities central to

the discipline, based on professional standards of excellence

22

Characteristics of Intended Student Learning Outcomes

General enough to capture important learning but clear and specific enough to be measurable

Focus on aspects of learning that will develop and endure but that can be assessed in some form now

23

Aligning Outcomes,Teaching-Learning and Assessment

Outcome statements

Methods of teaching/learning

Assumptions about teaching/learning

Methodsof assessing

Assumptions about methods

What do you expect students to know, understand, be able to do as a result of your teaching?

What methods develop/ foster your primary outcomes?

What assumptions underlie your methods?

What methods do you believe provide evidence of desired student outcomes?

What assumptions underlie your belief that each method will elicit students’ demonstration of your desired outcomes?

Step 3. Some Means of Assessment

Embedded Narrative Assignments Case study Portfolio (paper or electronic) Capstone project Senior exit essay

Performances Critical situation scenario Debate Demonstration Oral presentation Online monitoring

Tests Standardized test Homegrown test In class objective test In class essay

Student Centered Exit interview Focus group Self-rating of progress

Archival Library use, on-line activity Transcript analysis Transfer records Stakeholders Perspective

Alumni, employee, employer surveys

Experience of faculty who inherit our students

We’ve always done this, but now systematically to improve student learning rather than just to assess the student and assign a grade.

25

Step 3. Means of Assessment Levels of assessment

College (General Education, Core Abilities)

Program (Deg, cert, or course sequence) Course

Types of evidence Direct Indirect and supportive

26

Step 3. Means of Assessment Direct evidence of student learning

outcomes (think: in the aggregate) Passage of license, certification or subject

area exams Results of general education exams Capstone projects or performances Employer or internship supervisor ratings of

student performance Class assignments, projects, etc. Departmental exam results But, NOT grades.

27

Step 3. Means of Assessment Indirect evidence of student

learning outcomes (think: in the aggregate) Student self-reflective reports Surveys of perception

Alumni, graduating or current students Focus group interviews

28

Step 3. Means of Assessment

“Targets” for Assessment Cognitive outcomes (knowledge,

thinking disciplines) Attitudinal outcomes (values, beliefs) Behavioral/performance outcomes

(use of knowledge or values) or Knowledge, Skills, and Abilities

(KSAs)

29

Grading is not Assessment Evaluation of individual students =

assessment Focus is individual not groups of students A summative, not formative act Objectivity of single evaluator vs. group Generally not accepted as direct evidence Uses of the grading process

Agreed upon course exam or part of exam Row and column model for assignments

30

Grading and SLO Assessment(Embedded Assignment)

Criteria Tim Jane Mary Joe Dave Average

Spelling 3 4 1 2 3 2.6

Grammar 2 5 3 2 5 3.4

Punctuation 4 5 2 3 4 3.6

Structure 3 2 3 5 3 3.8

Total 13 17 10 12 15

Student Grade C A D C B

Total down the column for individual grading. Analyze across the row for assessment of intended outcomes from the group.

James Nichols

31

Step 3. Means of Assessment - Course Exams

Agreed upon course exam questions Monitor student responses to

identify topics of weakness Item analysis to discern what groups

are having difficulty with selected items

Revisit teaching approach to the topic

32

Step 3. Means of Assessment - Embedded Assignments

Embedded narrative class assignments Evaluated for the grade mark Later a sample is re-inspected to

Assess program accomplishment against expected outcomes

Explore the learning process Agreed upon performance criteria

The fruits of campus dialogue Expert judgment of the faculty

Inter rater reliability

33

Step 3. Means of Assessment -Rubrics

Key Questions What essential elements must be

present in student work to ensure high quality?

What does student work look like at each level of mastery?

34

Step 3. Means of Assessment -Elements of A Useful Rubric

Dimensions/criteria of rating (the rows) Levels of mastery (the columns) e.g.,

Beginner Developed Accomplished

Commentary describing features of work at each level of mastery (the cells) Build from strengths and weaknesses teachers

see in student work over the years

35

Step 3. Means of Assessment - Portfolios

A collection of student work Selective or all inclusive Displays what students can do Best if student is required to reflect upon

and evaluate their own work and learning Caution in program use

Require the same work product from all students at pre-determined points in time.

36


Attitudinal Learning Surveys of students, graduates,

alumni Affirmation of accomplishments Indirect measure

Exam or task to explore attitudes and apply values

37


Behavioral and Performance Student Learning Outcomes Specific motor skills Computer applications use Writing talent Computational skills Speaking ability

38


Behavioral and Performance Case studies Internships Simulations Projects Classroom work in a capstone

course

39

Step 3. Means of Assessment - Final Performances

Most frequently used means Portfolio review Public performance Juried competition

Drawn from “embedded” assignments that require synthesis and application of learning

40

Step 3. Criteria for Success

Identify the criteria before assessment is done

How well “ought” the completers (on average) perform?

Role of dialogue, professional judgment and collective responsibility

41

Step 3. Criteria for Success Primary student performance areas

Minimum overall score, rating, response expected if program is functioning at an acceptable level

Sub areas of student performance More detailed minimum sub-scale or

performance which faculty need to review to ascertain likely cause and make improvement

42

Program Intended Educational Outcomes:


3. Employers of the Automotive Technology Program graduates will be rate them competent based on the education received in the program.

Means of Program Assessment and Criteria for Success:

2a. At the close of their final term, 90% of the graduates will be able to identify and correct within a given period of time all of the mechanical problems in five test cars that have been "prepared" for the students by Automotive Technology Program faculty. No single automotive malfunction will fail to be identified and corrected by more than 20% of students.

2b. 80% of Automotive Technology Program graduates will pass the National Automotive Test. On no subscale will students average missing 30% or more of the items.

3. 80% of the respondents to an Employer Survey conducted every 3 years by the college will respond that they find graduates competent and would employ others.

Example: Criteria for Success - Automotive Technology

Program

43

Step 4. Assessing, Assembling Results and Making Meaning of

Them

Assessing and Assembling Results Logistics of doing assessment Recording results

Making meaning The creative tension between “ought”

and “is”

44


2a. At the close of their final term, 90% of the graduates will be able to identify and correct within a given period of time all of the mechanical problems in five test cars that have been "prepared" for the students by Automotive Technology Program faculty. No single automotive malfunction will fail to be identified and corrected by more than 20% of students.

2b. 80% of Automotive Technology Program graduates will pass the National Automotive Test. On no subscale will students average missing 30% or more of the items.

Summary of Data Collected:

2a. 79% overall success rate. Electrical system malfunction undetected by 34% of students.

2b. Pass rate on National Automotive Test was 83%; however, on “hydraulic theory” subscale students missed an average of 34% of questions.

Example: Recording ResultsGraduates of the Automotive Technology Program will be technically proficient.

45

Step 5. Documenting and Using the Results - “Closing the Loop”

Fitting the response to the results Faculty adjust what we teach Faculty adjust how we teach Faculty adjust the means of

assessment Faculty adjust where we put the bar

of success Document

see J. Nichols, Department Guide & Record Book 46


2a. 79% overall success rate. Electrical system malfunction undetected by 34% of students.

2b. Pass rate on National Automotive Test was 83%; however, on “hydraulic theory” subscale students missed an average of 34% of questions.

3. 90% reported willingness to employ graduates, but only 50% of body shops.

Use of Results:

2a. Expanded electrical trouble-shooting component of AT 202 to include automotive electrical systems.

2b. Modified means of teaching hydraulic theory during AT 102 (Basic Auto Systems).

3. Added body shop representative to Advisory Committee and are reviewing curriculum to determine if separate program is needed.

Example: Auto Tech Program(Using Results & Documenting Those

Actions)

47

The Bottom Line, A Call To Learning

A college is an organization composed of learners

(End)



Purpose

Mission Statement:

Your Community College is an open-admission, community-based, comprehensive college.

Goal Statement:

Serve persons of allages in preparing for job entry and careers in automotive technology.


1. Graduates of the Automotive Technology Program will be successfully employed in the field.(achievement, not Student Learning Outcome)


3. Employers of the Automotive Technology Program graduates will rate them competent based on the education received in the program.


1a. 50% of the responding graduates of the Automotive Technology Program will report employment in the field on the Graduating Student Survey administered at the time of program completion.1b. 80% of the graduates of the Automotive Technology Program will report employment in the field on the recent Alumni Survey administered one year after graduation.

2a. At the close of their final term, 90% of the graduates will be able to identify and correct within a given period of time all of the mechanical problems in five test cars that have been "prepared" for the students by Automotive Technology Program faculty. No single automotive malfunction will fail to be identified and corrected by more than 20% of students.2b. 80% of Automotive Technology Program graduates will pass the National Automotive Test. On no subscale will students average missing 30% or more of the items.

3. 80% of the respondents to an Employer Survey conducted every 3 years by the college will respond that they find the graduates competent and would employ others .


1a. 73% reported employment.

1b. 81% reported employment one year after graduation.

2a. 79% overall success rate. Electrical system malfunction undetected by 34% of students.2b. Pass rate on National Automotive Test was 83%; however, on “hydraulic theory” subscale students missed an average of 34% of questions.

3. 90% reported willingness to employ graduates, but only 50% of body shops.

Use of Results:

1a. Revised criteria for success to 70%.

1b. No action necessary at this time, however, will continue to monitor.

2a. Expanded electrical trouble-shooting component of AT 202 to include automotive electrical systems.

2b. Modified means of teaching hydraulic theory during AT 102 (Basic Auto Systems).

3. Added body shop representative to Advisory Committee and are reviewing curriculum to determine if separate program is needed.

Automotive Technology Program

James Nichols 49


Purpose

Mission Statement: … an open-admission, community-based, comprehensive college designed to provide inexpensive, quality educational opportunities (transfer, career/technical and continuing education)…

Goal Statement:Serve traditional students seeking the first two years of instruction leading to a bachelor’s degree


1. Students transferring will find courses taken fully accepted as prerequisites for junior and senior level courses at four-year colleges. (not a SLO)

2. After one year of adjustment to the four-year college, the grades of students transferring will be similar to those of students who initially enrolled at four-year colleges.

3. Graduates transferring to a four-year college as a full-time student will complete degrees at almost the same rate as those students originally enrolling at the four-year college. (not a SLO)


1. Each year one of the college’s six academic departments will contact their counterparts at the three four-year colleges where most students transfer and all courses designed to support the transfer will be found fully accepted as prerequisites by the four-year institutions.

2. Analysis of data concerning the grades of students transferring to four-year colleges will indicate that the differences between the average of transfer students’ GPAs and that of native students is statistically insignificant one year after enrollment at the four-year college.

3. Analysis of data received from each of Your CC three primary transfer student destinations will indicate that the difference in the average number of semesters to the BA degree completion of full-time transfer students from Your CC and students originally enrolling in the four-year college is statistically insignificant.


1. Problem noted in introductory accounting where microcomputer technology is now used at two primary transfer institutions.

2. Overall GPA of transfers found comparable to native students, but significantly less in math classes.

3. Degree completing time of Your CC transfer students found to be virtually identical to native students.

Use of Results:

1. Expansion of microcomputer use in introductory accounting problems at Your CC is underway.

2. Math 107 (college algebra) is being strengthened to better relate with calculus at four-year institutions..

3. No change necessary.

Transfer Program, Your CC(CC with no majors and few transfers)

Documents

1 Understanding Student Learning Outcomes and Assessment Bob Barr Andrew LaManque Rob Johnstone Adapted from material prepared by Fred Trapp, Long Beach