Mixed Methods Presentation

7/29/2019 Mixed Methods Presentation

1/50

Using Mixed Methods to Assess the Efficacy of a

First-Year Experience Course and Program

Kevin Coughlin, Dean, Institutional Research, Planning, and Effectiveness

Eileen DeLuca, Dean, College and Career Readiness


2/50

Edison State College

Open access, baccalaureate-granting state college. Five-county service district comprises three counties

along Floridas southwestern Gulf coast, and two inland

counties.


3/50

Annual Enrollment Data

Table 5.2Five-Year Unduplicated Headcount by Ethnicity, District

2007-08 2008-09 2009-10 2010-11 2011-12*

African American 1,810 2,122 2,686 2,839 2,757

Asian/Pacific Islander 387 427 468 502 484

Caucasian 11,881 13,226 14,598 14,519 13,906

Hispanic 2,959 3,565 4,163 4,112 3,906

Native American 42 54 73 79 63

Two or More Races --- --- 303 1,018 1,494

Not Reported 449 354 999 1,541 1,425

Total 17,528 19,748 23,290 24,610 24,035

Year-to-Year % Change 12.7% 17.9% 5.7% -2.3%

Five-Year % Change 37.1%

*2011-12 data include end of term summer/fall data and beginning of term spring semester data.


4/50

Achieving the Dream Risk Factor Data

Source: Edison State College Institutional Research, Planning and Effectiveness (2012). Focal Point.

Five risk factors from the July 2009Achieving the Dream report (July/August 2009, Data

Notes) were applied to Edison State College FTIC students.


5/50

Achieving the Dream Risk Factor Data

Source: Edison State College Institutional Research, Planning and Effectiveness (2012). Focal Point.


6/50


7/50

Development of Quality Enhancement Plan (QEP)

August 2008-College initiated engagement in internal and

external research to identify the QEP topic that would have

the greatest impact on student learning.

Fall 2009-College embarked upon a nine-month, externally

guided self-study using the Foundations of Excellence in the

First College Year (FOE) assessment model.

April 2009-QEP focus emerged as a unique version of a first-

year experience (FYE) course with a focus on developing

critical thinking and success strategies among students.


8/50

Research Base for QEP

Various models of first-year seminars have been shown tohave a significant impact on students in terms ofretention, persistence, student satisfaction, and academicperformance. Retention (Potts & Schultz, 2008; Miller, Janz & Chen, 2007;

Ryan & Glenn, 2004; Derby & Smith, 2004; Jamelske, 2008) Persistence (Lang, 2007; Porter & Swing, 2006; Stovall, 2000)

Student satisfaction and engagement (Summerlee &Murray, 2010; OGara, Karp, & Hughes, 2009; Engberg &Mayhew, 2007)

Academic performance for both academically under-prepared and well-prepared students (Potts & Schultz,2008; Jamelske, 2008; Lang, 2007, Stovall, 2000)


9/50

QEP Goal and Program Outcomes

Primary Goal

Enable first-time-in-college students to become self-reliantlearners imbued with critical thinking skills.

Student Outcomes

Facilitate an increase in student retention rates, rates ofpersistence, and graduation rates.

Foster increased rates of student satisfaction and studentengagement.

Faculty and Staff Outcomes

Apply newly obtained knowledge to their practices to promotecritical thinking and enhance the likelihood of success forfirst-year students.

.


10/50

Student Learning Outcomes based on Frameworks

Critical Thinking

Apply the intellectual traits, standards, andelements of reasoning in the context of theirpersonal and academic lives

Demonstrate intellectual rigor and problem-solving skills by analyzing and evaluatinginformation, generating ideas, and resolvingissues

Explore how background experiences impacttheir values and assumptions and explain howthey influence personal relationships


11/50


Applied Learning

Enhance their awareness of the larger diverse

community both inside and outside EdisonState College

Apply one or more skills learned in the FYE

course to other academic endeavors


12/50


Relevancy

Construct a plan for a successful path into andthrough completion of a degree or certificate

Evaluate student-to-student and student-to-faculty interactions, and reflect on theirrelationship to academic, career, and socialdevelopment

Reflect on the General Education competencies

at Edison State College and articulate theirapplication to academic and career goals


13/50


Success Strategies

Develop strategies for effective written andverbal communication, use of technology,listening, reading, critical thinking andreasoning

Demonstrate independence and self-efficacy

through effective personal management, use of

college resources and the development ofpositive relationships with peers, staff and

faculty


14/50

Cornerstone Experience Requirement Timeline

Initial

Implementation

Beginning Spring 2012

All FTIC degree-seeking students testing into 2 or more

developmental courses

YEAR 1

2012 - 2013

All FTIC degree-seeking students testing into 2 or more

developmental courses

YEAR 2

2013 2014

All FTIC degree-seeking students testing into any number

of developmental courses

YEAR 3

2014 - 2015

All FTIC degree-seeking students testing into any number

of developmental courses

YEAR 42015 2016 All FTIC degree-seeking students

YEAR 5

2016 - 2017

All FTIC degree-seeking students


15/50

Direct Measures

Department/

Unit

Outcome

Measurement

Method/Assessment Tool

Outcome-Specific Goal

(Performance Expectation)

Critical

Thinking

Student

Learning

Outcomes

Results of the Critical Thinking Journal

assessment scored with the Critical

Thinking Rubric

By the end of the Spring 2012 semester, 70%

of students who complete the course will

achieve a 3 (accomplished) or higher on all

relevant aspects of the rubric).

Final Essay Assignment scored withCritical Thinking Rubric

By the end of the Spring 2012 semester, 70%of students who complete the course will

achieve a 3 (accomplished) or higher on all

relevant aspects of the rubric.

Scores on the California Critical

Thinking Disposition Inventory

After completing the Cornerstone

Experience course, students will have

significant improvement in the following

Critical Thinking Dispositions: Truth Seeking,

Open Mindedness, Analyticity, Systematicity,

Inquisitiveness, Confidence in Judgment,

Maturity in Judgment.


16/50

Direct Measures

Department/

Unit

Outcome

Measurement




Success Skills

Student

Learning

Outcomes

Scores on the Smarter

Measure Learning Readiness

Indicator life factors,

personal attribute,

technology knowledge, and

technical competencyitems.

After completing the Cornerstone Experience course,

students will have significant improvement in the

following indicators: Personal Attributes, Life Factors,

Technology Knowledge and Technology Competency.

Success Strategies Group

Presentation rubric

By the end of the Spring 2012 semester, 70% of

students that complete the course will achieve a 3

(accomplished) or higher on all relevant aspects of

the rubric.

Qualitative data from Final

Essay assignment used to

Develop a Success Strategies

Rubric

An analysis of randomly selected essays will yield

categories and concepts that can be used to identify

success strategies that are most salient among

students. This qualitative assessment will inform the

development of a survey to be employed in

subsequent terms.


17/50

Indirect Measures

Department/

Unit

Outcome

Measurement

Method/Assessment

Tool



Student

retention,

persistence,

and

graduation

rates.

Within course

completion rate (derived

from course grade

distributions)

Once fully implemented, students will successfully

complete the Cornerstone Experience at a rate of 85%

with a C or better.

Term-to-term retention

reports (derived from

the Banner Student

Information System)

Using AY 2011-12 baseline data, term-to-term retention

will increase by 5% each year.

Year-to-year retention

reports (derived from

the Banner Student

Information System)

Using AY 2011-12 baseline data, year-to-year retention

will increase by 3% each year.


18/50

Indirect Measures

Department/

Unit

Outcome

Measurement

Method/Assessment

Tool



Student

retention,

persistence,

and

graduation

rates.

Cohort graduation

reports derived

through the Banner

Student Information

System

This analysis will use the cohort graduation rate

associated with students that entered ESC as FTIC

during AY 10-11 (baseline).

Cohorts from AY 11-12 and AY 12-13 who graduate

within 150% of the expected time required will

increase by 10% Cohorts from AY 13-14 and AY 14-15 who graduate

within 150% of the expected time required will

increase by 10%

Cohort from AY 15-16 who graduate within 150% of

the expected time required will increase by 10%

Course Outcome items

from SIR II: 29, 30, 31,

32, 33 and Student

Effort and Involvement

items: 34, 35 and 36

Beginning AY 2012-13, faculty results for these items will

meet or exceed the comparative mean for four-year

institutions.


19/50

Indirect Measures

Department/Unit

Outcome

Measurement




Student

satisfaction andstudent

engagement.

Engaged Learning items from the

SENSE: 19a, 19b, 19e, 19g, 19h, 19i,19j, 19k, 19l, 19m, 19n, 19o, 19q, 20d2,

20f2, and 20h2

Beginning AY 2012-13, there will be a 5%

increase in the Engaged Learningbenchmark over the previous years results.

Student-Faculty interactions items

from CCSSE: 4k, 4l, 4m, 4n, 4o, and 4q


increase in the Student-Faculty interactions

benchmark over the previous years results.Faculty/Student Interaction items

from SIR II and a subset of Active and

Collaborative Learning items from

CCSSE (4f, 4g, 4h, and 4r)

Beginning AY 2012-13, faculty results for

these items will meet or exceed the

comparative mean for four-year

institutions.

Qualitative data from focus group

responses

QEP Assessment committee will

analyze levels of students satisfaction

and engagement through a series of

focus group discussions. Concepts

and categories derived through this

analysis will inform program and

curricular enhancements.


20/50

Indirect Measures

Department/Unit

Outcome

Measurement




Faculty applicationof professional

development to

promote critical

thinking and

enhance the

likelihood ofsuccess for first-

year students.

Academic Challenge items fromCCSSE: 4p, 5b, 5c, 5d, 5e, 5f, 6a, 6c, 7,

9a (Fall 2012)

Beginning AY 2012-13, there will be a5% increase in the Academic Challenge

benchmark over the previous years

results.

Professional Development Surveys Following completion of the

professional development modules,80% of trained faculty will report using

critical thinking and first-year student

success strategies as measured on

Likert scale items.

SIR II Communication items: 6, 7, 8, 9and 10

Beginning AY 2012-13, faculty resultsfor these items will meet or exceed

the comparative mean for four-year

institutions.


21/50

Indirect Measures

Department/Unit

Outcome

Measurement




Staff application

of professional

development to

promote critical

thinking and

enhance thelikelihood of

success for first-

year students.

Professional Development Surveys Following completion of the professional

development modules, 80% of trained staff

and administrators applying critical

thinking and first-year student success

strategies as measured on Likert scale

items.

SENSE items from A Plan and a

Pathway to Success category:

18d, 18g, 18e, 18f, and 18h


increase in A Plan and Pathway to Success

benchmark over the previous years

results.


22/50

Highlights of Quantitative Measures-Direct

Rubric Design and Data Collection

Critical Thinking Journal Studies

California Critical Thinking DispositionInventory (CCTDI) Pre-Post

SmarterMeasure Learning Readiness Indicator

Pre-Post


23/50


Common Course Assignments

Critical Thinking Journal

Group Project: Success Strategies

Final Essay

Edison GPS (Go Picture Scribe)


24/50



25/50

Critical Thinking Journal

Actual Results Use of Results

The students achievement of each dimension (Clarity,

Accuracy, Relevance, Significance, and Logic) of the rubricwas measured on a 4-point scale.

Overall means for each dimension:

Clarity: 2.73 (64.59% received 3 or higher)

Accuracy: 2.94 (80.73% received 3 or higher)

Relevance: 3.03 (85.37% received 3 or higher)

Significance: 2.92 (75.79% received 3 or higher)

Logic: 3.00 (82.70% received 3 or higher)

The goal for Accuracy, Relevance, Significance, and Logic

were met. The number of students receiving a 3 orbetter

for Clarity fell short of the stated goal (-.5.41%) with Clarity

being the dimension with the lowest of the overall means.

Students will receive rubric scores

and feedback for each entry, ratherthen at mid-point and final. Faculty

provide increased feedback on clarity

and use of Standard English. Faculty

encourage usage of Writing Centers.

Use of the Lee Campus Academic

Success and College Prep Center labs

has become more fluid. Students

with writing needs receive assistance

in either lab.


26/50

Rubric Standardization

1. Please comment on how you believe the Critical Thinking Rubric worked for

you in scoring essays today.

2. Looking at the levels or performance on the rubric, are any too similar? (e.g.,

Is it difficult to distinguish between a score of 4 and a score of 3?) Explain.

3. Examine the five criteria listed. Is there any redundancy? Do you believe

you believe you may be scoring students more than once on the same

criterion?

4. In what ways would you change the Rubric for ease of use? Use the attachedform to be specific.

5. Do you have any other comments or suggestions about the Critical Thinking

Rubric as a tool for scoring journal entries?


27/50

Inter-Rater Correlations


28/50


29/50

CCTDI Pre-test/Post-test


30/50

SmarterMeasure Pre-test/Post-test


31/50

Highlights of Quantitative Measures-Indirect

Term-to-Term Retention


32/50


Table 1

Term to Term Retention by Base Fall TermNot Retained

Following Term

Retained

Following Term

Totals

Fall 2011 Frequency 194 535 729

Percent Overall 14.22 39.22 53.45

Row Percent 26.61 73.39

Column Percent 54.04 53.23

Fall 2012 Frequency 165 470 635

Percent Overall 12.10 34.46 46.55



Total Frequency 359 1005 1364

Percent 26.32 73.68 100.00

X

2

(1, N = 1364) = 0.069, p < 0.793


33/50


Table 2

Fall 2012 Term to Term Retention by Participation in SLS 1515Not Retained

Following Term

Retained

Following Term

Totals

Not in Frequency 57 104 161

SLS 1515 Percent Overall 8.98 16.38 25.35



Enrolled in Frequency 108 366 474

SLS 1515 Percent Overall 17.01 57.64 74.65



Total Frequency 165 470 635

Percent 25.98 74.02 100.00

X2(1, N = 635) = 9.95, p < 0.002


34/50

Highlights of Qualitative Measures-Direct

Final Essay Assignment Grounded Theory

Survey Development


35/50

Thematic Coding Final Essay Prompt: Reflect upon the main themes that emerged from all the

class topics and discussions and extract those ideas that you found most influentialor important. Then create a new plan for achieving success in college and forestablishing positive relationships with peers, staff and faculty.

Random sample of 33% of sections.

Three-raters used a modified version of Grounded Theory to engage in opencoding of essays followed by selective coding.

Essays were reviewed until saturation occurred.

Codes were grouped into Concepts and Categories.

Categories served as framework for development of a Success Strategies Survey.

Grounded Theory Method adapted from Charmaz, K. (2008) in Hess-Biber, S.N. and Leavy, P. (Eds.) Handbookof Emergent Methods. The Guilford Press.


36/50

Concepts-Listing, Describing, Applying Categories

Communication/Listening Skills

Communication Strategies

Diversity Knowledge

Small Group Communication Skills

Relationship Building

Personality Inventories

Campus Engagement

Goal-Setting

Goal Attainment StrategiesTime Management Skills

Organizational Skills

Persistence

Critical Thinking Skills

Cognitive StrategiesCourse Success Strategies

Learning Styles

Use of College Resources


37/50


38/50


39/50

Please indicate the level of improvement you have made in the following areas as a result of completing the


40/50

Please indicate the level of improvement you have made in the following areas as a result of completing the

SLS 1515 Cornerstone Experience course.

Answer OptionsI didn't do thismuch before

taking this

class and nowI still don't

I didn't do this

much beforetaking this

class and now

I haveimproved a

little

I didn't do this

much beforetaking this class

and now I haveimproved a

great deal

I have alwaysdone well at

this and

haven'tchanged

I have alwaysdone well at

this, but I have

made someimprovement

Response Count

Arriving to class on time.

2.5%(1)

5.0% (2) 5.0% (2)46.0%

(18)42.5% (17) 40

Attending class.2.5% (1)2.5% (1)10% (4)40.0% (16)46.0% (18) 40

Reviewing the course schedule.

0% (0) 7.7% (3) 10.3% (4) 33.3% (13) 48.7% (19) 39

Using a calendar and/or lists to make

sure assignments are completed on time.

7.5% (3)2.5% (1)17.5% (7) 25.0% (10) 47.6% (19) 40

Working on large projects incrementally

(little by little, not waiting until the last

minute).

7.5%(3)

10.0% (4)22.5% (9)10.0% (4) 60.0% (20) 40

SLS 1515 Cornerstone Experience course.

Answer Options

I didn't do thismuch beforetaking thisclass andnow I stilldon't

I didn't dothis muchbefore takingthis classand now Ihaveimproved alittle

I didn't dothis muchbeforetaking thisclass andnow I haveimproved agreat deal

I havealwaysdone well atthis andhaven'tchanged

I have alwaysdone well atthis, but I havemade someimprovement

ResponseCount

Arrivingtoclassontime.

2.5%

(1)

5.0%

(2)

5.0%

(2)

46.0%(18)

42.5%(17)

40

Attendingclass.

2.5%

(1)

2.5%

(1)

10%

(4)

40.0%(16) 46.0%(18) 40Reviewingthecourse

schedule.

0%

(0)

7.7%

(3)

10.3%

(4)

33.3%

(13)

48.7%(19) 39Usingacalendarand/or

liststomakesureassignmentsare

completedontime.

7.5%(3) 2.5%(1) 17.5%(7) 25.0%(10) 47.6%(19) 40Workingonlarge

projectsincrementally

(littlebylittle,notwaiting

untilthelastminute).

7.5%

(3)

10.0%

(4)

22.5%

(9)

10.0%

(4)

60.0%(20) 40


41/50

Please indicate the level of improvement you have made in the following areas as a result of completing theSLS 1515 Cornerstone Experience course.

Answer Options

I didn't do thismuch beforetaking thisclass andnow I stilldon't

I didn't dothis muchbefore takingthis class andnow I haveimproved alittle

I didn't dothis muchbeforetaking thisclass andnow I haveimproved agreat deal

I havealwaysdone well atthis andhaven'tchanged

I have alwaysdone well atthis, but I havemade someimprovement

ResponseCount

Usingsmallgroup

communicationskills.

2.5%(1)

10.0%(4)

35.0%(14) 17.5%(7) 36.0%(14) 40Participatingandasking

questionswhen

appropriate.

7.5%(3)

12.5%(5)

22.5%(9)

25.0%.(10)

32.6%(13) 40

Formingarelationship

withotherstudents.

2.5%(1)

17.5%(7)

20.0%(8)

27.5%(11)

32.6%(13) 40

Meetingwiththe

professoroutsideof

classforhelp.

23.1%(9)

20.5%(8)

10.3%(4)

15.4%(6)

30.8%(12) 39

Thinkingcriticallyabout

textsandlectures.

5.1%(2)

5.1%(2)

36.9%(14) 23.1%(9) 30.8%(12) 39


42/50

Highlights of Qualitative Measures-Indirect

Focus Groups

Hi hli ht f Q lit ti M I di t


43/50

Highlights of Qualitative Measures-Indirect

Focus Group: Student Satisfaction and Engagement

Focus group leader #1 will welcome the students, ask each

to tell their name, describe the purpose of the session, and

pose the questions to the group. At the end of the session,s/he will briefly summarize the main points and ask if anyone

has anything they would like to add or amend.


44/50

1. What attracted you to Edison State College? (Why did you

choose Edison State College?)

2. Tell us about your experience in SLS 1515 (CornerstoneExperience) course.

3. Which aspects of the SLS 1515 (Cornerstone Experience)

course are you most satisfied with?

4. In what ways did the SLS 1515 (Cornerstone Experience)

course help you achieve your academic or career goals?

5. Tell us about the Peer Architect (peer mentor) assigned to

your class. (Follow-up: How did the Peer Architect help you

achieve your academic or career goals?)


45/50

6. Which aspects of the SLS 1515(Cornerstone Experience) course

are you least satisfied with?

7. Describe the campus-based activities you participated in as a

result of taking the SLS 1515 course.

8. Outside of attending classes, tell us about any college activities

that you participated in this term.

9. Outside of attending classes, what type of campus-based

activities would you be interested in participating in?

10. What is the most effective way for the college to encourage

your participation in campus activities?


46/50

Focus Group Themes

Engaging through the Passport Assignment

Requesting more sports and/or intramural activities

Receiving communication about and timing of campus events

Not seeing the value of the Critical Thinking Exam

Interacting with professor and other students

Requesting more technology training


47/50

QUESTIONS?


48/50

Contact

Kevin Coughlin

Dean, Institutional Research, Planning and Effectiveness

[email protected] DeLuca

Dean, College and Career Readiness and QEP Director

[email protected]

http://www.edison.edu/cornerstone
http://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstonehttp://www.edison.edu/cornerstone


49/50

References Derby, D. & Smith, T. (2004). An orientation course and community college

retention. Community College Journal of Research and Practice. 28, pp. 763-773.

Engberg, M.E. & Mayhew, M.J. (2007). The influence of first-year success courses onstudent learning and democratic outcomes. Journal of College StudentDevelopment, 48(3), 241-1280010081.

Jamelske, E. (2008). Measuring the impact of a university first-year experienceprogram on student GPA and retention. Higher Education, 57, 373-391. DOI10.1007/s10734-008-9161-1.

Lang, D. (2007). The Impact of a first-year experience course on the academicperformance, persistence, and graduation rates of first-semester college students ata public research university. Journal of the First-Year Experience & Students in

Transition, 2007, vol. 19, no. 1 pp. 9-25.

Miller, J.W., Janz, J.C. & Chen, C. (2007). The retention impact of a first-year seminaron students with varying pre-college academic performance. Journal of the First-Year Experience and Students in Transition 19(1). 47-62.


50/50

References continued OGara, L. Karp, M.M. & Hughes, K. (2009). Student success courses in the community college:

An exploratory study of student perspectives. Community College Review. vol. 36, n. 3.

pp.195-218.

Porter, S. R. & Swing, R.L. (2006). Understanding how first-year seminars affect persistence.Research in Higher Education. vol 47, no. 1.pp. 89-109.

Potts, G. & Schultz, B. (2008). The Freshman seminar and academic success of at-risk students.

College Student Journal. pp. 647-658.

Ryan, M.P & Glenn, P.A. (2004). What do first-year students need most: Strategies instructionor academic socialization? Journal of College Reading and Learning, 34 (2), Spring 2004. pp. 4-28.

Stovall, M. (2000). Using success courses for promoting persistence and completion. New

Directions for Community Colleges. No. 112, Winter 2000. pp.45-54.

Summerlee, A. & Murray, J. (2010). The impact of enquiry-based learning on academicperformance and student engagement. Canadian Journal of Higher Education. vol. 40, no. 2 pp.78-94.

Documents

Mixed Methods Presentation