Student Engagement and Mission Student Engagement and Mission Effectiveness: Effectiveness: Looking for Links between Looking for Links between

Outcomes and Institutional GoalsOutcomes and Institutional Goals

Ellen M. Boylan, Ph.D.Association for Institutional Research

2005 Forum

San Diego, California • May 29 – June 1

22

Learning Environment Institutional

Mission, Goals and Objectives

Student Engagement

HIGHER EDUCATION

ENVIRONMENT

Catholic colleges

BackgroundBackground

Sadlak (1986) asserts that the institutional mission statement influences student achievement…it describes the desired academic and social development skills to be possessed at end of the educational experience.

How does an institution measure the effect of its mission on students’ educational experience? Can it be measured? Do students’ perceptions of mission differ by institution?

33

Research QuestionsResearch Questions

1. Do student ratings of engagement and learning environment differ by student perceptions of institutional mission?

(a)    Are the Research Mission Questions reliable and valid measures of student engagement and perception of learning environment?

(b)    What is the relationship between the Research Mission Questions and NSSE benchmark scores for Consortium schools?

2. Do students and administrators share the same perceptions of institutional mission?

44

Study DesignStudy Design Select the National Survey of Student Engagement (NSSE) 2004 instrument (The College Student Report) to test

first year and senior students

Assemble an NSSE 2004 Catholic College

Consortium for mission research

Develop Research Mission Questions

Administer the NSSE 2004 with added Research Mission Questions to Consortium students

Administer a sub-set of Research Mission Questions to Consortium college administrators

Analyze results: factor analysis, correlation and regression, and t-test analysis of means

Marywood University Institutional Review Board Approval 2004-041

55

Sample Mission StatementSample Mission Statement Concepts common to the Consortium Colleges’ mission Concepts common to the Consortium Colleges’ mission

statements were drawn out, reduced, and formulated into 20 statements were drawn out, reduced, and formulated into 20 Research Mission QuestionsResearch Mission Questions

A Catholic university sponsored by the Congregation of the Sisters, Servants of the Immaculate Heart of Mary, Marywood University roots itself in the principle of justice and a belief that education empowers people. Enacting its ideals, Marywood offers students a welcoming and supportive community that encourages men and women of all backgrounds to shape their lives as leaders in serviceleaders in service to others. Proud of its liberal arts tradition and host of professional disciplinesprofessional disciplines, Marywood challenges students to broaden their understanding of global issues and to make decisions based on spiritual, spiritual, ethical, and religious valuesethical, and religious values. Marywood calls upon students to seek their full potential and invites all to engage in a lifelong process of learninglearning. Witnessing the efficacy of teaching and scholarship, Marywood educates students to live responsibly in a diverse and diverse and interdependent world.interdependent world.

66

Professional Development

PLeadership

L

AcademicExcellence

A

Community/Service

C

Ethical/Spiritual

DevelopmentE

DiversityD

The joy of learning is faithfully nurtured...power of learning.

Higher morals, and social conscience are valuable

develop...personal competencies

Collaborativeenvironment

Commitment to build a just and peaceful society

Develop moral character

Promotes leadership to local and boarded communities

Promotes service to local and boarded communities

Emphasizes on the growth of the human person

development of leadership

community of learning

spirit of servicepromotes just and ethical relations; values

diverse ages, backgrounds, and cultures

preparation for responsible living and meaningful work

Responsible academic freedom;provides intellectual preparation

Promote moral, spiritual, religious vales to students; moral prep

Imagination and spirit

intellect and spirit

academicexcellence

Service and leadershipcommunity

Imagination and Spirit

Prepare forleadership

Develop intellectually

Lives of serviceDevelop spirit & heart; values-based education

Prepare forprofessional careers

Intellect and spirit

commitment to serving others through a liberal arts education

Service opportunities in the community

diversity

professional disciplines

education empowers

calls upon students to reach their full potential…lifelong learning

leaders in servicespiritual, ethical, and religious values

live responsibly in a diverse and interdependent world

Career preparationResponsibility for decisions

Intellectual inquiryCommitment to service

personal growth diversity

Mission Statement Concepts - NSSE 2004 Catholic College Consortium

77

AdministrationAdministration

The NSSE Survey with attached Research Mission Questions was administered to 3,827 first-year and senior students at 14 Catholic Colleges in the United States in Spring 2004.

88

Institution DemographicsInstitution Demographics

Gender of students 2 single-sex institutions Percent female – 61 to 85

Size of Institutions Large - 1 Small - 2 Medium - 12

Location of Institutions Urban – 5 Suburban - 9

99

Characteristics of Characteristics of InstitutionsInstitutions

GenderGender

TOTAL %

INSTITUTION M F A 24 76 B 20 80 C 27 73 D 15 85 E 46 54 F 23 77 G 3 97 H 33 67 I 29 71 J 19 81 K 39 61 L 27 73 M 18 82 N 0 100

Total 26 74

1010

I nstitution

% Roman Catholic

Urban/ Suburban

Small

G 64 S

N 42 U

Medium

A 36 U B 51 S C 47 U D 51 S E 60 S F 35 S I 93 S J 61 S K 68 S L 36 S M 42 U

Large

H 54 U

Characteristics of Characteristics of InstitutionsInstitutions

Size and locationSize and location

1111

Research Question 1a.Research Question 1a.

Are the Research Mission Questions reliable and

valid measures of student engagement and perception of

learning environment?

1212

Reliability and factor analysisReliability and factor analysis

Reliability analysis of Research Mission Questions (20 items). Three items removed. Mission Perception Inventory (MPI) is produced (17 items). Factor analysis of MPI conducted. Four subscales produced: Sense of Mission Respect for Diversity

Individual Actions

Religious Practice

1313

Mission

Perception

Inventory(MPI)

(17 items)

Senseof

Mission(8 items)

Respect for

Diversity(5 items)

IndividualActions(2 items)

ReligiousPractice(2 items)

Research

Mission

Questions

(20 items)

Derivation of Mission Perception Inventory Derivation of Mission Perception Inventory (MPI) and four subscales(MPI) and four subscales

from Research Mission Questionsfrom Research Mission Questions

1414

Sense of Mission ScaleSense of Mission Scale (8 items (8 items = .867) = .867)

The mission is widely understood by students

Social and personal development is important

Ethical and spiritual development is important

This institution’s religious heritage is evident

Professors here discuss the ethical implications of what is being studied

This institution offers opportunities for volunteering and community service developing leadership skills students to strengthen their religious

commitment

1515

Respect for Diversity ScaleRespect for Diversity Scale (5 items (5 items = = .854) .854)

The faculty, staff, and students here respect different religions respect different races, cultures

Students free to express their individual spirituality

Different sexual orientations are accepted

The environment encourages appreciation of diversity

1616

Rotated Factor Matrix

1Sense ofMission

2Respect

for Diversity

3IndividualActions

4ReligiousPractice

Social and personal development of students is an important part of the mission at this institution.

.719 .266 .129 .049

Ethical and spiritual development of students is an important part of the mission at this institution.

.710 .181 .066 .169

This institution offers opportunities for: Volunteering and community service

.674 .141 .149 .030

Developing leadership skills. .656 .271 .160 .000

At this institution, there are opportunities for students to strengthen their religious commitment.

.604 .296 .029 .131

The heritage of the founding religious community… is evident here. .601 .201 .032 .078

The mission of this institution is widely understood by students. .515 .189 .089 .053

The professors at this institution discuss the ethical implications of what is being studied.

.477 .329 .111 .141

The faculty, staff, and students here are respectful of people of different religions. .290 .786 .029 .064

The faculty, staff, and students here are respectful of people of different races and cultures.

.361 .747 .012 .104

Students feel free to express their individual spirituality here. .289 .656 .064 .051

People of different sexual orientations are accepted socially here. .119 .621 .070 -.081

The environment here encourages students to develop an appreciation of diversity.

.293 .616 .067 -.005

How important is it to you that you accomplish the following: Volunteering in community service.

.185 .043 .693 .184

Influencing social values. .106 .081 .649 .118

Within the past week… participated in a religious service? .103 .001 .111 .660

Within the past week…spent time in private prayer or meditation? .099 .024 .153 .619

1717

Research Question 1b.Research Question 1b.

What is the relationship between the Research Mission Questions

and NSSE benchmark scores for Consortium schools?

1818

NSSE Benchmark ScalesNSSE Benchmark Scales

Level of Academic Challenge

Active and Collaborative Learning

Enriching Educational Experiences

Supportive Campus Environment

Student-Faculty Interactions

1919

Scale N

Items Cronbach

MPI 17 .880

Sense of Mission 8 .867

Respect for Diversity 5 .854

NSSE

Level of Academic Challenge (LAC) 11 .717

Active & Collaborative Learning (ACL) 7 .743

Enriching Educational Experiences (EEE) 11 .611

Supportive Campus Environment (SCE) 6 .761

Student Faculty Interaction (SFI) 6 .719

NSSE Beliefs & Values (NSSE/BV) 12 .807

Reliability of Scales Consortium data

2020

Scale MPI

Sense of

Mission

Respect for

Diversity LAC1 ACL2 EEE3 SCE4 SFI5

MPI

-Sense of Mission .768

-Respect for Diversity .712 .581

NSSE

1Level of Academic Challenge .308 .274 .187

2Active & Collaborative Learning .297 .224 .167 .481

3Enriching Educational Experiences .329 .305 .150 .412 .489 4Supportive Campus Environment .507 .526 .462 .272 .264 .277 5Student-Faculty Interactions .307 .282 .173 .460 .584 .549 .331

NSSE/Beliefs & Values .611 .551 .401 .455 .481 .659 .541 .509

Significant Correlations by Consortium Significant Correlations by Consortium

ScaleScale

***All correlations are significant at p<.001 level, 2-tailed, df=3,825.

2121

Institution Scale Correlations by LevelInstitution Scale Correlations by Level MPI Scales with NSSE ScalesMPI Scales with NSSE Scales::

Correlation is significant at the * p<.05 level (2-tailed), df=12; **p <.02 level (2-tailed), df=12;

***p<.01 level, 2- tailed, df=12; ****p<.001 level, 2-tailed, df=12,

Scale LAC ACL EEE SCE SFI

NSSE/ BV

Mission

First .21 .09 .63**

.84**** .31 .83***

Senior .01 -.01 .61** .64 .34 .82***

Diversity

First .10 .25 .66** .28 .70*** .50

Senior -.28 .10 -.32 .06 -.43 .04

MPI

First .19 .19 .61** .85**** .31 .84****

Senior -.18 .04 .36 .59* .02 .85****

2222

Research Question 1Research Question 1

Do student ratings of engagement and learning environment

differ by student perceptions of institutional mission?

2323

Procedure for regression analysisProcedure for regression analysis

Conduct regression analysis using Sense of Mission and Respect for Diversity as dependent variables, entering all NSSE scales as a group.

Check regression coefficients for significance.

Repeat analysis employing stepwise regression, entering scale(s) with significant beta as indicated.

2424

Standardized Regression Coefficients: Standardized Regression Coefficients: Sense of Mission and Student Class Level by NSSE Sense of Mission and Student Class Level by NSSE

ScalesScales Scale

β

t

p FirstYear .103 4.51 .000 Level of Academic Challenge

Senior .074 3.07 .002 FirstYear .010 .39 N.S. Active & Collaborative

Learning Senior -.046 -1.82 N.S. FirstYear .111 4.75 .000 Enriching Educational

Experiences Senior .163 6.44 .000 FirstYear .464 21.85 .000 Supportive Campus

Environment Senior .452 21.04 .000 FirstYear -.012 -.48 N.S. Student-Faculty Interactions

Senior .066 2.43 N.S.

2525

Model Summary – Sense of Mission

Regression analysis

Change Statistics

R Square Change F Change df1 df2

Sig. F Change

First Year Supportive Campus Environment .276 711.00 1 1866 .000

Supportive Campus Environment + Level of Academic Challenge + Enriching Educational Experiences

.027 35.76 2 1864 .000

Senior Supportive Campus Environment .278 660.11 1 1711 .000 Supportive Campus Environment

+ Enriching Educational Experiences + Level of Academic Challenge

.044 55.01 2 1709 .000

2626

Standardized Regression Coefficients: Standardized Regression Coefficients: Respect for Diversity and Student Class Level by NSSE Respect for Diversity and Student Class Level by NSSE

Scales Scales

Scale

β

t

p FirstYear .070 2.91 .004 Level of Academic Challenge

Senior .049 1.90 .058 FirstYear .027 1.00 N.S Active & Collaborative

Learning Senior .045 1.62 N.S FirstYear .017 .69 N.S. Enriching Educational

Experiences Senior -.019 -.70 N.S. FirstYear .436 19.43 .000 Supportive Campus

Environment Senior .442 19.04 .000 FirstYear -.013 -.47 N.S. Student-Faculty Interactions

Senior -.016 -.53 N.S.

2727

Regression analysis

Model Summary – Respect for Diversity

Change Statistics

R Square Change F Change df1 df2

Sig. F Change

First year Supportive Campus Environment .215 511.46 1 1862 .000

Supportive Campus Environment + Level of Academic Challenge .006 14.85 1 1861 .000

Senior Supportive Campus Environment .209 451.25 1 1710 .000 Supportive Campus Environment

+ Level of Academic Challenge .003 5.96 1 1709 .015

2828

Procedure forProcedure for post hoc post hoc analysis analysis

Institution scale correlations suggest differences between institutions may exist.

Explore this by comparing means on Sense of Mission and Respect for Diversity by level and institution.

2929

Post hocPost hoc - Mean Sense of Mission scores - Mean Sense of Mission scores

by institution and student levelby institution and student level

Sense of Mission

3.6

3.8

4.0

4.2

4.4

4.6

4.8

A B C D E F G H I J K L M N

Institution FY Senior

Mean

3030

Post hocPost hoc - Mean Respect for Diversity scores - Mean Respect for Diversity scoresby institution and student levelby institution and student level

Respect for Diversity

3.2

3.4

3.6

3.8

4.0

4.2

4.4

A B C D E F G H I J K L M N

Institution FY Senior

FY Mean

Sr Mean

3131

Research Question 2 Research Question 2

Do students and administrators share

the same perceptions of institutional mission?

3232

Student and administrator means Student and administrator means by scale and itemby scale and item

FYN=1868

SRN=1705

Admin.N=60 T* Sig.*

SENSEOF MISSION 4.03 4.01 4.28 3.36 .001

RESPECT FOR DIVERSITY

4.01 3.95 3.95 0.05 N.S.

RELIGIOUS PRACTICE 1.51 1.53 1.79 4.71 .000

*Comparison of senior and administrator means only

3333

Findings and ConclusionsFindings and Conclusions

Reliability analysis and factor analysis confirmed that the Mission Perception Inventory (MPI) is a reliable and valid measure of student perception of institutional mission ( = .85-.88).

Factor analysis of the MPI resulted in four reliable subscales, Sense of Mission, Respect for Diversity, Individual Actions, and Religious Practice.

The four subscales produced indicate the MPI reflects these concepts which are included in Consortium college mission statements.

The high reliability of subscales indicates that the MPI can be employed with confidence as measures of student mission perception in subsequent studies.

3434

Findings and ConclusionsFindings and Conclusions

The MPI, Sense of Mission, and Respect for Diversity scales correlate with the NSSE benchmark scales, particularly Supportive Campus Environment (SCE).

Respect for Diversity was significantly related (for first year students only) to SCE, Enriching Educational Experiences (EEE) and Student-Faculty Interactions (SFI) scales.

3535

Findings and ConclusionsFindings and Conclusions

Supportive Campus Environment (SCE), followed by Enriching Educational Experiences (EEE) and Level of Academic Challenge (LAC), are most closely related to Sense of Mission.

SCE contributed about 21% of the variance to the Respect for Diversity scale and was significant for both first year and senior students.

3636

Findings and ConclusionsFindings and Conclusions Post hocPost hoc

The institutions showing higher senior than first year scores on the Sense of Mission scale are medium in size, selective.

Institutions showing higher senior means compared to first year, have low to moderate percentages of Roman Catholic students, and/or are about 75% female.

The two schools with the lowest senior mean scores for Respect for Diversity have the largest percentage of students self-identifying as Roman Catholic.

3737

Findings and ConclusionsFindings and Conclusions Research Question 2Research Question 2

The mean score differences between administrators and seniors is significant for nine items.

Administrators perceiveSense of Mission conceptsand behaviors as more widelyexpressed on campus than students perceive they are.

Not unexpectedly, mean scores for administrators and students differed significantly on the Religious Practice scale.

3838

ConclusionsConclusions

Factor analysis and analysis of reliability indicate the Mission Perception Inventory (MPI) has extremely high reliability and four correlated, independent and strong factors.

There is a significant relationship between students’ Sense of Mission and Respect for Diversity and the Supportive Campus Environment they experience; and to a lesser extent, the academic challenges and enriching educational experiences provided them by higher education.

Sense of Mission is rated significantly higher by college administrators than students; however, Respect for Diversity ratings are not significantly different.

3939

For further information…

Ellen M. Boylan, Ph.D.Planning and Institutional Research

Marywood University570.348.6203

instres@marywood.edu