1

Developing and Assessing General Education Learning Outcomes: A

Collaborative Commitment across The Institution

Workshop at Miami Dade College

November 21, 2005

Peggy Maki

PeggyMaki@aol.com

2

Workshop Foci

Building a Culture of Evidence across The Institution

Grounding Assessment of GE in Teaching and Learning

Collaboratively Developing Learning Outcome Statements-- Claims about Student Learning

3

Validating Learning Outcome Statements through Maps and Inventories of Educational Practice

Designing or Selecting Valid Assessment Methods that Align with Students’ Educational Experiences

Developing Standards and Criteria of

Judgment

4

• Analyzing and Interpreting Results of Student Work

• Closing the Inquiry Loop

5

Mission/Purposes

Learning Outcome Statements

How well dostudents achieve

our outcomes?

Gather Evidence

Interpret Evidence

Enhance teaching/ learning;

inform institutional decision-

making, planning, budgeting

6

Your Learning Outcomes

Articulate some GE learning outcome statements that align with what and how students learn in your programs and services

Map GE outcome statements to assure students have diverse and multiple opportunities to learn

Identify some direct and indirect assessment methods to capture student learning

7

Develop some standards and criteria of judgment to score student work

Identify when and where to assess and how to collect evidence of student learning

Identify when and who will assess evidence of student learning

8

Identify possible times across the institution when colleagues can come together to interpret results and reach consensus about ways to improve student learning

After implementing changes, identify when you will reassess the efficacy of changes.

9

PeersPeers

Spiritual Leaders

Spiritual Leaders

Lab Assistants

Lab Assistants

Graduate Students;Teaching

Assistants

Graduate Students;Teaching

Assistants

Support Services

Staff

Support Services

Staff

Librarians and

Resource Staff

Librarians and

Resource Staff Advisory

BoardMembers

AdvisoryBoardMembers

AthleticCoaches

AthleticCoaches

AcademicAdvisers

AcademicAdvisers

Intern Advisors

Intern Advisors

MentorsMentors

Student Services Staff

Student Services Staff

TutorsTutors

FacultyFaculty

Learning

Learning

Building a Culture of Evidence

10

R.W. Emerson, “Intellect,” Essays (1841)

“How can we speak of the action of the mind under any divisions, as of its knowledge, of its ethics, of its works, and so forth, since it melts will into perception, knowledge into act? Each becomes the other. Itself alone is. Its vision is not like the vision of the eye, but is union with the things known.”

11

How do you learn?

List several strategies you use to learn:

_________________________________________________________________________________________________________________________________________________________________________

12

Grounding Assessment of GE in Teaching and Learning

Learning is a complex process of interpretation-not a linear process

Learners create meaning as opposed to receive meaning

Knowledge is socially constructed (importance of peer-to-peer interaction)

National Research Council. Knowing What Students Know, 2001.

13

Learning involves creating relationships between short-term and long-term memory

Transfer of new knowledge into different contexts is important to deepen understanding

Practice in various contexts creates expertise

14

People learn differently—prefer certain ways of learning

Deep learning occurs over time—transference

Meta-cognitive processes are a significant means of reinforcing learning (thinking about one’s thinking and ways of knowing)

15

Integration of learning and development over time….

Cognitive

AffectivePsychomotor

16

Specific Questions that Guide Assessment

What do you expect your students to know and be able to do by the end of their education at your institution?

What do the curricula and other educational experiences “add up to?”

What do you do in your classes or in your programs or services to promote the kinds of learning or development that the institution seeks?

17

Questions (con’d)

Which students benefit from various classroom teaching strategies or educational experiences?

What educational processes are responsible for the intended student outcomes the institution seeks?

How can you help students make connections between classroom learning and experiences outside of the classroom?

18

Questions, con’d:

What pedagogies/educational experiences develop knowledge, abilities, habits of mind, ways of knowing/problem solving, and dispositions?

How are the curriculum and co-curriculum designed to develop knowledge, abilities, habits of mind, ways of knowing, and dispositions?

19

How do you intentionally build upon what each of you teaches or fosters to achieve programmatic and institutional objectives—contexts for learning?

What methods of assessment capture desired student learning--methods that align with pedagogy, content, curricular and instructional design?

20

Common Categories of GE Learning

Writing

Speaking

Quantitative Reasoning

Problem solving, critical thinking

21

Leadership

Lifelong learning

Ethical awareness; social responsibility

Team work

Global perspectives; multiple perspectives

22

Mesa Community College Categories (AZ)

Written and oral communication Critical thinking/problem solving Numeracy Arts and humanities Scientific inquiry Information literacy Cultural diversity

23

Inventory from MDC’s Student Services Last Friday:

Writing Speaking Reading Comprehension Critical thinking/problem solving Quantitative reasoning/problem solving Technology Application of knowledge Proficiency in a chosen field

24

Cultural literacy Globalism Teamwork/solo work Self-initiative/independence Social responsibility Ethical awareness Leadership Ability to adapt to environments/changes

25

Categories under which students learn and develop

List several categories under which you believe students learn or develop as a result of MDC’s GE program? ________________________________________________________________________________________________________________________________________________________________________________________________

26

Inventory Based on Nov. 21 Cross-Campus Group Work

Writing Speaking Listening Quantitative reasoning, including ability to

assess and evaluate Critical thinking Ethical awareness

personal/social responsibility, including cultural dimensions

27

Environmental ethics Computer/information literacy Cultural Literacies Problem solving Problem posing Financial responsibility Workforce skills Knowledge about self, others, community,

world

28

Leadership Active learning (self) Teamwork Ability to link across the curriculum and

experiences Time management Global perspectives/diversity Cultural sensitivity

Interpersonal skills Adaptability

29

Scientific thinking/methods Appreciation of the arts, including a global

perspective Life skills

30

Collaboratively Developing Learning Outcome Statements

Learning outcome statements describe what students should be able to demonstrate, represent, or produce based on how and what they learn at the institution through multiple, varied, and intentional learning opportunities.

31

Rely on active verbs, such as create, compose, calculate, develop, build, evaluate, translate, etc., that target what we expect students to be able to demonstrate

Emerge from what we value and how we teach or students learn; that is, they emerge from our educational practices and are developed through consensus

Are written for a course, program, service, or the institution

32

Can be mapped to the curriculum and co-curriculum

Can be assessed quantitatively or qualitatively

33

Levels of Learning Outcome Statements

Institution-level Outcome Statements, including GE

Department-,Program-, Certificate-level Outcome Statements

Course/Service/Educational Experience Outcome Statements

34

Distinguishing between Objectives and Outcomes

Objectives state overarching expectations such as–

Students will develop effective oral

communication skills.

OR

Students will understand different

economic principles.

35

Mesa Outcomes under Arts and Humanities

Demonstrate knowledge of human creations Demonstrate an awareness that different

contexts or world views produce different human creations

Demonstrate an understanding and awareness of the impact that a piece has on the relationship and perspective of the audience

Demonstrate an ability to evaluate human creations

36

Capital Community College (CT)

Communicate effectively

Reason scientifically and or quantitatively

Think critically

Develop a global perspective

(See handout)

37

Ethics—Students should be able to…

Identify and analyze real world ethical problems or dilemmas, and identify those affected by the dilemma.

Describe and analyze the complexity and importance of choices that are available to the decision-makers concerned with this dilemma

38

Articulate and acknowledge their own deeply held beliefs and assumptions as part of a conscious value system

Describe and analyze their own and others’ perceptions and ethical frameworks for decision-making

Consider and use multiple choices, beliefs, and diverse ethical frameworks when making decisions to respond to ethical dilemmas or problems.

California State University Monterey Bay: University Learning Requirements, 2002

39

Example from ACRL Literate student evaluates information and its

sources critically and incorporates selected information into his or her knowledge and value system.

ONE OUTCOME: Student examines and compares information from various sources in order to evaluate reliability, validity,accuracy, timeliness, and point of view or bias.

40

Quantitative Literate Graduates according to MAA Should be Able to:

1. Interpret mathematical models such as formulas, graphs, tables, and schematics, and draw inferences from them.

2. Represent mathematical information symbolically, visually, numerically, and verbally.

3. Use arithmetical, algebraic, geometric, and statistical methods to solve problems.

41

4. Estimate and check answers to mathematical problems in order to determine reasonableness, identify alternatives, and select optimal results.

5. Recognize that mathematical and statistical methods have limits.

(http://www.ma.org/pubs/books/qrs.html)

The Mathematics Association of America (Quantitative Reasoning for College Graduates: A Complement to the Standards, 1996). See also AMATYC draft, 2006.

42

Writing

See NCTA Guidelines

See WPA Outcomes in attachments for outcomes at the end of the first year of writing

43

Ways to Articulate Outcomes

Adapt from professional organizations

Derive from mission of institution/program/department/service

Derive from students’ work

44

Derive from ethnographic process

Derive from exercise focused on listing one or two outcomes “you attend to”

Consult taxonomies

45

Bloom’s Taxonomy—cognitive, psychomotor, affective

Webb’s Taxonomy—depth of knowledge

Shulman’s Taxonomy—table of learning

Taxonomies That May Help You Develop Outcome Statements

46

Depth of Knowledge (Webb)

Recall and recognition

Processing skills and concepts

Strategic thinking

Extended thinking (complex reasoning, planning, design)

47

Dimensions of Knowledge

Facts

Procedures—series of step-by-step actions and decisions that result in the achievement of a task

Processes—flow of events or activities that describe the big picture

48

Concepts—class of items, words, or ideas

known by a common name

Principles—guidelines, rules, parameters

Metacognitive—knowledge of one’s own cognition

49

Shulman’s Taxonomy

Engagement (active learning)

Knowledge and understanding

Performance, practice, or action (act in and on the world)

Reflection and critique (cease action to discover or “make progress”)

50

Judgment and design—consider context—even restraints

Commitment and Identity—move inward and connect outward

http:///www.carnegiefoundation.org/elibrary/docs/printable/making_differences.htm

51

Exercise: Write one or two GE learning outcome statements under a category of learning

__________________________________

__________________________________

___________________________________

52

Exercise:

How well do your learning outcome statements meet the criteria for well-written outcome statements (see handout)?

53

Validating Learning Outcome Statements through Maps and Inventories of Practice

• Reveal how we translate outcomes into educational practices offering students multiple and diverse opportunities to learn

• Help us to identify appropriate times to assess those outcomes

• Identify gaps in learning or opportunities to practice

54

Help students understand our expectations of them

Place ownership of learning on students

Enable them to develop their own maps or learning chronologies

55

Collaborative Development of A Curricular-Co-Curricular Map

I,R,E Course Course Educational

experience

Outcome 1:

Outcome 2:

Outcome 3:

Outcome 4:

56

Inventories of Educational Practice

Provide in-depth information about how students learn along the continuum of their studies

Identify the range of educational practices and assessment experiences that contribute to learning outcomes (See handouts)

57

Exercise: How will you use maps and inventories?

Discuss how you will go about the process of developing a curricular or curricular-co-curricular map and how you will label peoples’ entries

Discuss how you might use inventories of educational practices

58

Designing or Selecting Valid Assessment Methods that Align with Students’ educational Experiences

“Every assessment is also based on a set of beliefs about the kinds of tasks or situations that will prompt students to say, do, or create something that demonstrates important knowledge and skills. The tasks to which students are asked to respond on an assessment are not arbitrary.“

National Research Council. Knowing what students know: The science and design of educational assessment . Washington, D.C.: National Academy Press, 2001, p. 47.

59

Design or Select Assessment Methods that Prompt Students to:

Transfer, integrate, apply, synthesize

Value interdependence among courses and experiences

Re-use and re-configure what they have learned (even to re-position their understanding)

Self-reflect on their emerging learning

60

For example, do students

Apply business principles to a student-run organization?

Apply principles of effective writing to a proposal for an independent study or project?

Explore multiple perspectives in solving a campus issue or problem?

Self-reflect on principles underlying their actions or decisions?

61

Assumptions UnderlyingTeaching

Actual Practices

Assumptions UnderlyingAssessment Tasks

Actual Tasks

62

Inference Drawing

Validity of the Method

63

What Tasks Elicit Learning You Desire?

Tasks that require students to select among possible answers (multiple choice test)?

Tasks that require students to construct answers (students’ problem-solving and thinking abilities)?

Question: Consider the contexts for each of these kinds of tasks in your work

64

When Do You Seek Evidence?

Formative—along the way?

For example, to ascertain progress

or development

Summative—at the end?

For example, to ascertain mastery level of achievement

65

Direct Methods of Assessment

Focus on how students represent or demonstrate their learning (meaning making)

Align with students’ learning and assessment experiences

Align with curricular-and co-curricular design verified through mapping

66

Invite collaboration in design (faculty, students, tutors?)

67

Standardized Instruments Psychometric approach—values quantitative

methods of interpretation

History of validity and reliability

Quick and easy adoption and efficient scoring

One possible source of evidence of learning

68

Do Not Usually Provide

Evidence of strategies, processes, ways of knowing, understanding, and behaving that students draw upon to represent learning

Evidence of complex and diverse ways in which humans construct and generate meaning

Highly useful results that relate to pedagogy, curricular design, sets of educational practices

69

Authentic, Performance-based Methods

Focus on integrated learning

Directly align with students’ learning and previous assessment experiences

Provide opportunity for students to generate responses as opposed to selecting responses

Provide opportunity for students to reflect on their performance

70

Do Not Provide

Immediate reliability and validity (unless there has been a history of use)

Usually do not provide easy scoring unless closed-ended questions are used.

71

Direct Methods across Students’ Learning Chronology

On-line tools

Critical events

Assemblage of learning objects

Virtual learning environments or situations (including chatrooms and resource rooms)

72

Scenarios

Storyboards

Self-directed group projects

Magic box

Personal and annotated websites

73

Log book or journal tasks that explore an issue over time

Event analysis

Video clips

Case studies over time as students move through courses and educational experiences

74

Externally or internally juried reviewed projects

Oral defense

E-portfolio

Aristotle’s finger exercises

Interpreting visual material or data

75

Representation –concept mapping or problem solving (3-D)

Practice of Artists’ Machetes

Mining data

Students’ drawings and models—perceptual enhances understanding, analysis. and analytical ability

76

Chronological tasks that prompt students to stretch over time

Draw on knowledge/understanding to solve problem in a different context

Problems with solutions: Are there other solutions?

Team-based projects

Self-reflections

77

Magnify or reduce to seek wider implications and relationships (a la Lewis Thomas)

Professional/disciplinary practices

Embedded assignments

78

Performance on national licensure examinations

Locally developed tests

79

Indirect Methods of AssessmentThan Can Be Combined with Direct Methods

Programs or Courses selected by students

Focus groups (representative of the population)

Interviews (representative of the population)

Surveys

80

Other Sources of Information that May Be Useful in Your Interpretation

CSSE results

Grades

Participation rates or persistence in support services

81

Course-taking patterns

Students’ majors

Transcript analyses or audits (co-curricular transcript?)

82

Exercise:

Using the handout, determine the degree of alignment of the direct and indirect methods you may use to asses your outcome statements.

83

Developing Standards and Criteria of Judgment

A set of criteria that identifies the expected characteristics of a text and the levels of achievement along those characteristics. Scoring rubrics are criterion-referenced, providing a means to assess the multiple dimensions of student learning.

Are collaboratively designed based on how and what students learn (based on curricular-co-curricular coherence)

84

Are aligned with ways in which students have received feedback

(students’ learning histories)

Students use them to develop work and to understand how their work meets standards (can provide a running record of achievement).

85

Raters use them to derive patterns of student achievement to identify strengths and weaknesses

AnalyticHolistic

86

Interpretation through Scoring Rubrics

Criteria descriptors (ways of thinking, knowing or behaving represented in work)

CreativitySelf-reflectionOriginalityIntegrationAnalysisDisciplinary logic

87

Criteria descriptors (traits of the performance, work, text)

CoherenceAccuracy or precisionClarityStructure

88

Performance descriptors (describe well students execute each criterion or trait along a continuum of score levels):

Exemplary—Commendable– Satisfactory- Unsatisfactory

Excellent—Good—Needs Improvement—Unacceptable

Expert—Practitioner—Apprentice--Novice

89

Development of Scoring Rubrics

Emerging work in professional and disciplinary organizations

Research on learning (from novice to expert)

Student work

90

Interviews with students

Experience observing students’ development

91

Consider the following guidelines as you develop a scoring rubric for one or more of our outcomes

Identify the purpose of the rubric—for student feedback, for justifying a grade, for program-level understanding about student learning

Identify the overall format—analytic or holistic?

92

Identify the full range of criteria you will assess with indicators for these criteria

Identify the performance descriptors; within each cell identify leveled performance

93

Pilot-testing the Scoring Rubric

Apply to student work to assure you have identified all the dimensions with no overlap

Schedule inter-rater reliability times:

-independent scoring

-comparison of scoring

-reconciliation of responses

-repeat cycle

94

Analyzing and Interpreting Results

Seek patterns against criteria and cohorts

Build in institutional level and program

level discourse

Tell the story that explains the results—

triangulate with other data, such as

CSSE or participation rates

95

Be able to aggregate and disaggregate data to guide focused interpretation

Collectively determine what you wish to change

96

Examples of Changes:

Increased attention to weaving experiences across the institution, a program, or a department to improve student achievement

Changes in advising based on assessment results

Closer monitoring of student achievement--tracking

97

Faculty and staff development to learn how to integrate experiences that contribute to improved student learning

Changes in pedagogy and curricular and co-curricular design

Development of modules to assist learning; use of technology; self-paced learning, supplemental learning

98

Implement agreed upon changes

Re-assess to determine efficacy of changes

Focus on collective effort—what we do and how we do it

Closing the Inquiry Loop to Learn

99

Structures

Assessment Committees at the institution and department or program levels

Development of task forces to assume responsibilities

100

Collective community dialogue designed to build on institutional

learning

Collective community contributions about ways to adapt, revise, or innovate practices to improve student learning

Collective community interpretations of results

Working group that collects and analyzes students’ work or responses

Working group that selects or designs assessment methods to capture learning over time

Working group that develops outcome

statements to guide cycles of inquiry

Working Group that identifiescollective

expectations for learning

Assessment Committee

101

Communication: Collaborative Interpretation

Disciplinary work groups

Cross-disciplinary work groups

Formal opportunities to share program-level findings at the institution-level; opportunities to share institution-level findings at the program-level

102

Communication: Decision-making Bodies

Planning (short- and long-term planning)

Budgeting

Decision-making

Allocation of Resources

103

Human, Financial, Technological Support Grad students or part-time support to assist

with development of methods or research on methods, collection or analysis

Analysis of results

Faculty and staff development or resources to support efforts

Development of technology to house results or to draw from existing data

104

Exercise:

Describe the structures, processes,

decisions, and channels and forms of

communication that currently exist at

MDC, as well as your ideas for

deepening the commitment to

assessment (see handout).

105

“What and how students learn depends to a major extent on how they think they will be assessed.”

John Biggs, Teaching for Quality Learning at University: What The Student Does. Society for Research into Higher Education & Open University Press, 1999, p. 141.

106

Works Cited

Biggs, J. (1999). Teaching for Quality Learning at University: What The Student Does. Society for Research into Higher Education & Open University Press, 1999, p. 141.

Maki, P. (2004). Assessing for Learning: Building a Sustainable Commitment Across the Institution. Sterling, VA: Stylus Publishing, LLC, and the American Association for Higher Education.

National Research Council. (2001). Knowing What Students Know: The Science and Design of Educational Assessment. Washington, D.C.: National Academy Press