Assessing Integrated Learning

Maureen Pettitt, Ph.D.Skagit Valley College

Beth Hartsoch, M.A.Western Washington University

PNAIRP – Oct ‘07

Session Overview Learning Communities at SVC Established Approaches

Student Writing, Course Satisfaction Survey & CCSSE

SVC/WWU Project WELS Survey & Focus Groups

Themes from the Data New Studies in Progress

Interdisciplinary Learning at SVC

Interdisciplinary courses are a means of delivering instruction and fostering student learning

A response to curricular issues: Faculty felt that students did not see connections between and among

disciplines, and needed to engage subjects more fully, to see

education as a dynamic and interconnected process of exploration and discovery

CCSSE

Student Writing

Student Satisfaction Survey

Past Approaches:Multiple Assessment Methods

1. Student Writing Study

Faculty/IR team developed two-year research project to assess student attainment of overarching General Education learning outcomes

Student Writing Study: GoalsAssess students' ability to: Apply a variety of concepts/texts/contexts

and perspectives to solving problems, thinking about issues

Connect one’s own life experience, ideas and abilities with those that others bring

Understand and value the learning process for oneself and for others

Connect to external, lifelong social issues

Student Writing Study: Method

Identified courses being taught in Learning Communities, stand-alone, and DE that could be “matched” over a two-year period

Faculty agreed to participate and give students course credit

Developed a set of questions based on Gen Ed goals and faculty input

Students surveyed beginning-, mid-, and end-course

End of Course Questions1. Have your learning expectations been met?2. Have you learned things that you hadn't

anticipated? If so, please describe.3. What do you think are the most important

aspects of your experiences in this course that account for your learning?

4. What have you learned in this course that will matter to you five years from now?

Student Writing Study: Findings

Students’ responses regarding valuing learning, connecting classroom learning with the world, and connecting with other were fairly similar in both stand-alone and LC courses.

Students in LC courses were far more likely to cite applying or appreciating a variety of perspectives to problem-solving (55% versus 15%)

Student Writing Study: Student Comments

“After taking this course I feel that I can make connections to various things, such as history, influences, people, and culture. This course taught me the value of making connections and things from my own perspective.”

Student Writing Study: Student Comments

“By combining course topics you get the ‘bigger picture’ and are able to sort of apply what we are learning better. By applying a subject or topic to another subject or topic you have to comprehend what you are learning and apply it to other things.”

Student Writing Study: Comparisons

Comparing student responses in the stand-alone courses with students in collaborative courses—same courses with the same instructor(s): Students in stand-alone courses were

less likely to write about these connections or about learning, and

tended to focus on personal growth, liking faculty, etc.

2. Quarterly Surveys Faculty-developed pen-and-paper survey Scale from “strongly disagree” to

“strongly agree” Faculty are provided their course results

and comments, plus the cumulative for the quarter for all courses

Several years ago, we created a version that could be scanned

Learning CommunitySurvey Questions 1. "This learning community has been

a valuable educational experience."

2. "I believe it was probably more valuable to have taken these classes together than it would have been to take them separately."

Mean Responses to Value Questions (N = 5156)

4 .1

3 .93 .8

3 .6

3 .33 .43 .53 .63 .73 .83 .9

44 .1

V a lu a b le M o re V a lu e T o g e t h e r

L C C o m p L C

Linked & Federated Composition LCs

(N = 2780)

4 .0

3 .63 .7

3 .4

3 .0

3 .2

3 .4

3 .6

3 .8

4 .0

V a lu a b le M o re V a lu e T o g e t h e r

L in k e d F e d e ra t e d

Student Comments LC: “This class has been fun and not

sucky at all. I think I have learned a lot.” Link: “I did not learn too much other

than how to explore more in depth, how to write more.”

LC: “If they had been separate, I would have known the what and where, but not the why, and the why is always the most important question.”

3. Community College Survey of Student Engagement

Administered the CCSSE in 2003, 2005, and 2007

Went to this survey because it most closely matched the institution’s focus on student learning and engagement

The LC question was a bonus extra!

CCSSE LC Question Stem: “Which of the following have you

done, are you doing, or do you plan to do while attending this college?”

Category: “Organized learning communities (linked courses/study groups led by faculty or counselors)”

Response Categories: I have done I plan to do I have not done nor plan to do

Method Used T-test for independent

samples to examine differences in effort and engagement between students who had taken Learning Communities and those who had not done nor planned to do.

Results are only for the latest survey administration (Spring 2007)

  Learning Community

Sig.Variable HaveTaken

(n = 173 )

Not Taken

(n = 232)Asked questions in classor contributed to classdiscussions

3.00 3.00 1.000

Made a classpresentation

2.43 2.27 .069

Prompt: “In your experiences at this college during the current school year, about how often have you done each of the following?”Responses range from 1 (“Never”) to 4 (“Very Often”)Responses range from 1 (“Never”) to 4 (“Very Often”)

Findings

  Learning CommunitySig.Variable Have

Taken(n = 173 )

Not Taken

(n = 232)Prepared two or more drafts of a paper or assignment before turning it in

2.96 2.45 .000

Worked on a paper or project that required integrating ideas or information from various sources

3.14 2.75.000

Worked with other students on projects during class

2.88 2.69 .021

Worked with classmates outside of class to prepare class assignments

2.49 2.10 .000

Discussed ideas from your readings or classes with others outside of class (students, family members, co-workers, etc.)

2.87 2.69 .038

Learning Community

Sig.Variable Have Taken

(n = 173 )

Not Taken

(n = 232)Worked harder than you thought you could to meet an instructor’s standards or expectations

2.78 2.52 .003

Used email to communicate with an instructor

3.10 2.67 .000

Discussed grades or assignments with an instructor

2.76 2.58 .039

Discussed ideas from your readings or classes with instructors outside of class

2.11 1.88 .012

Worked with instructors on activities other than coursework

1.74 1.45 .000

The advantage of using multiple methods….

……you you get to get to do a do a “Reality “Reality Check” Check” on a on a periodic periodic basisbasis

What we didn’t have….Perceptions of students after they transferred from Skagit

Collaboration with Western Washington University’s Office of Survey Research WELS (Western Educational

Longitudinal Study) Data Focus Groups

WELS Transfer Study Process began in 2001; baseline data Fall

2003 Survey items related to students’

expectations, pre-WWU experiences, college preparedness, etc.

Three separate surveys of incoming transfer students administered on-line, with phone follow-up Prior to starting classes After the first quarter After finishing three quarters

Results for SVC/Non-SVC Transfers

Percent responding “5 or

more times”Not SVC (n = 507 )

SVC(n = 45)

Talked to a professor outside of class 54.6% 66.7%Personally contributed to a classroom discussion

76.3% 80.0%

Volunteered on or off campus outside of coursework

20.1% 33.3%

Participated in a club or activity 16.8% 24.4%Wrote a substantial scientific or other research paper of 5 pages or more

17.9% 17.8%

Wrote any other type of substantial paper of 5 pages or more

25.6% 4.4%

Focus Groups WWU generated list; SVC did

transcript analysis to identify student course taking, esp. Learning Communities and English Links

Identified two groups for interviews:1) a mix of none, one, or two or more LCs taken at SVC2) two or more LCs taken at SVC

Questions for all SVC transfers

What aspects of your learning experiences at Skagit Valley College have helped you in your courses at Western Washington?

If you could change anything about your learning experiences at Skagit Valley College so that you would be better prepared to succeed at Western, what would that be?

Can you talk a little bit about how Learning Community courses have helped or hindered your academic progress at Western?

If you took an English course that was linked with another course, can you describe how that learning experience has been useful or not in your studies at Western?

Additional Group Questions For the first group:

Of you who have taken more than one Learning Community or English Link, how did taking a second or third learning experience of this kind impact the development of your skills or knowledge?

For the second group: How did taking the second or third learning

experience of this kind impact the development of your skills or knowledge?

Helpful in Transition to WWU

Students say the following aspects have helped: interdisciplinary learning, analytical/critical thinking, research and writing, group work, one on one interaction with instructors, and course content in specific courses

Additional helpful aspects noted: completion of GURs, knowing people who have attended Western, and the financial benefit of attending SVC

Interdisciplinary Learning Students say that the interdisciplinary

aspect of learning communities helps to make classes more interesting and fun at SVC.

Exposure to interdisciplinary learning at SVC helps students to feel more confident in their classes at Western. Students say they have learned to make connections and to apply what they’ve learned to another topic or to the real world.

English Links Most students say English links were

useful in their studies at SVC and in transferring to Western—because English links challenged students to think and write analytically, outside of the box.

Students say English was more interesting because it was linked to a content area which provided something to write about. They felt better prepared for researching and writing analytical papers at Western.

English Links A few students say their English

link was not useful in their studies at SVC. They say the focus was more on grammar and punctuation, as it was in high school.

Research and Writing While some students say that Skagit

prepared them to think critically and write analytical papers (discussed earlier), other students say they were not prepared for writing analytical papers at Western.

Research and writing experiences in English links and learning communities at SVC helped to prepare students for research and writing assignments at Western.

Other Student Outcomes Analytical/critical thinking

The emphasis on critical thinking at SVC helps students in their classes at Western because they learned how to analyze information, make connections, and synthesize major ideas.

Group work Group work in links and learning

communities helped to prepare students for group work assignments at Western.

Challenge & Workload An interesting side note is students’

perceptions about the challenge and workload involved in learning communities and links: Some students say that learning communities

and links are more challenging because of the level of analyzing involved. For some students, there is also a greater workload than expected.

Other students say there is not a greater workload.

Faculty Coordination When instructors do not coordinate well

together, the classes seem separate—not linked. Students find that the lack of coordination between some instructors means lack of integration between courses that are supposed to be linked.

Learning communities work best for students when the instructors coordinate well together, and particularly when instructors are in the classroom teaching together.

Changes Students Would Make

Students say the biggest issue for them is academic advising—because advising directly impacts students’ learning experiences at SVC as well as their preparation for Western.

Students also say they would finish a DTA to avoid difficulties in transferring, make sure they had learning experiences in a large lecture class, and do something about disruptive students.

Future WELS Analyses Comparisons

SVC degree vs. no degree (or ranked by number of LCs taken)

SVC vs. other CCs LC comparison between CCs SVC vs. other CC vs. non-transfer

WWU third-year students

Future WELS Analyses Surveys

Baseline – administered prior to start of Fall qtr Focus = experiences at previous college &

expectations at WWU Transitions – administered at the end of Fall qtr

Focus = first quarter WWU experiences Spring Follow-Up – administered at the end of

third quarter Focus = first year experiences at WWU

Analyses Descriptives and regression

GPA Analysis – LCs Taken

0

0 .5

1

1 .52

2 .5

3

3 .5

E n t e r in g G P A Q P A a f t e r 2 Q t r s

N o L CO n e L CT w o + L C s

There were no significant differences between groups on either variable.

GPA Analysis - Degree

0

0 .5

11 .5

2

2 .5

3

3 .5

E n t e r in g G P A G P A a f t e r 2 Q t r s

N o S V C D e g re eS V C D e g re e

There were no significant difference between groups on entering GPA but there was for GPA after two quarters (p=.000)

N = 60 Entering GPA

GPA After 2

Qtrs

Sig.

Degree Completed No 3.15 2.54 .000 Yes 3.19 3.14LCs Completed None 3.32 2.43 One 2.87 2.77 .362 Two or more 3.24 2.98

Convergence of the Data Findings from surveys, focus

groups, and student writing strongly suggest that taking Learning Communities results in higher levels of effort and engagement collaborate with peers interaction with faculty

Convergence of the Data The WWU data about transfer

students also suggest that students who complete Learning Communities at Skagit or the degree perform better than those who do not

However, we may need to look at the kinds and level of writing students are required to do in their courses

It’s all about perceptions…

Latest SVC Efforts…. Learning Communities Assessment

Project (Washington Center for the Improvement of Undergraduate Education) Protocol developed at Harvard (Project Zero) Rubric being tested

Developmental Learning Communities Project SVC project to research student success

based on developmental course delivery methods, including counseling-enhanced

Thanks for coming!