Does Orientation Promote Learning and Development?
An Assessment Plan Unearths the Answers.
Megan France, Steve Grande, andSara Finney
James Madison University
An overview Welcome The Orientation Program at JMU Our Assessment Plan Data Collection Results Use of results to make informed decisions about
program changes Q & A
James Madison University Public; four-year Shenandoah Valley, Virginia 3,800 first year students Primarily residential Center for Assessment and Research Studies
JMU Orientation One-day summer program: Summer
Springboard Part of a three-phase experience
1. One Book
2. Summer Springboard
3. August Orientation
Components for students and family members
Orientation Overview University welcome
~45 minutes; University President Focus on high expectations, student support, sense of community
Small group discussions led by peer mentors ~35 min; Personalizing university welcome message
Information on majors and unique academic programs ~60 min; Meetings between students and faculty
Advising and course registration ~2 hrs; Small group and 1-on-1 with advisor
Take care of university “business” and experience campus ~2.5 hrs; Staff and OPAs guide student through pre-enrollment
activities and campus
Activities1. Interaction with peers
2. Receive orientation materials
3. Experience the university culture
4. Inspiring speakers
5. Meet and interact with faculty and advisers
6. Interact with current JMU student mentors
Assessment Plan Development One year to develop goals and objectives
Labor intensive Be realistic Focus on a few goals
Finding or developing appropriate outcome measures
Data collection plan On-going process
As a result of Summer Springboard... Orientation Goal 1: Increase perceived cohesion to
JMU Objective 1: Significantly increase sense of belonging Objective 2: Significantly increase feeling of morale
Orientation Goal 2: Assist in social integration to JMU Objective 1: Significantly increase college self-efficacy
Orientation Goal 3: Value a diverse campus and increase openness to diversity and challenge Objective 1: Significantly increase universal-diverse orientation
Learning Goals?
Perceived Cohesion Measures the extent to which an individual
experiences and reports a sense of belonging and feelings of morale towards a group (JMU) Two subscales:
3 items measure Sense of Belonging “I see myself as part of the JMU community.”
3 items measure Feelings of Morale “I am enthusiastic about JMU.”
Scale of 1 (strongly disagree) to 9 (strongly agree)
Bollen and Hoyle (1990)
Goals and Objectives: Perceived Cohesion
ObjectiveObjective ActivityActivity MeasuresMeasures
Sense of Belonging
University Welcome and university
business
Perceived Cohesion Scale
(PCS)
Feelings of Morale
Peer mentor meetings
Perceived Cohesion Scale
(PCS)
As a result of Summer Springboard... Orientation Goal 1: Increase perceived cohesion to JMU
Objective 1: Significantly increase sense of belonging Objective 2: Significantly increase feeling of morale
Orientation Goal 2: Assist in social integration to JMU Objective 1: Significantly increase college self-efficacy
Orientation Goal 3: Value a diverse campus and increase openness to diversity and challenge Objective 1: Significantly increase universal-diverse orientation
Learning Goals?
College Self-Efficacy Inventory Measures self-efficacy or the confidence a student has
with respect to a range of college-related behaviors Three subscales:
4 items measure Roommate Efficacy “Divide chores with others you live with.”
7 items measure Course Efficacy “Take good class notes.”
9 items measure Social/Social Integration “Ask a professor a question.”
Scale of 1 (no confidence at all) to 10 (complete confidence)
Solberg, O'Brien, Villareal, Kennel, and Davis (1993)
Goals and Objectives: College Self-Efficacy
ObjectiveObjective ActivityActivity MeasuresMeasures
College Self-Efficacy
Faculty, adviser, and student staff
interactions
College Self-Efficacy Inventory
(CSEI)
As a result of Summer Springboard... Orientation Goal 1: Increase perceived cohesion to JMU
Objective 1: Significantly increase sense of belonging Objective 2: Significantly increase feeling of morale
Orientation Goal 2: Assist in social integration to JMU Objective 1: Significantly increase college self-efficacy
Orientation Goal 3: Value a diverse campus and increase openness to diversity and challenge Objective 1: Significantly increase universal-diverse
orientation
Learning Goals?
Miville-Guzman UniversalityDiversity Scale – Short Form Measures Universal-Diverse Orientation, which refers to an
awareness and acceptance of both similarities and differences among people Three subscales:
5 items measure Diversity of Contact “I am interested in learning about the many cultures that have
existed in this world.” 5 items measure Relativistic Appreciation
“Persons with disabilities can teach me things that I could not learn elsewhere.”
4 items measure Sense of Connection “Getting to know someone of another race is generally an
uncomfortable experience for me.” (R) Scale of 1 (strongly disagree) to 6 (strongly agree)
Miville (1992)
Goals and Objectives: Universal-Diverse Orientation*
ObjectiveObjective ActivityActivity MeasuresMeasures
Universal-Diverse Orientation
Role modeling and interaction
Miville-Guzman Universality-
Diverse Scale
(MGUDS)
Goals and Objectives
ObjectiveObjective ActivityActivity MeasuresMeasuresSense of Belonging University Welcome
and university business
Perceived Cohesion Scale
Feelings of Morale Peer mentor meetings
Perceived Cohesion Scale
College Self-Efficacy Faculty, adviser, and student staff interactions
College Self-Efficacy Inventory
Universal-Diverse Orientation
Role modeling and interaction
Miville-Guzman Universality-Diverse
Scale
Data Collection Who?
All first-year incoming students (not including transfer students)
What? First Year Student Survey Summer Springboard Survey
Both surveys included all three measures (PCS, CSEI, MGUDS)
Where? Surveys are on-line (i.e.,websurveyor)
Data Collection
When? First Year Student Survey is distributed early May and
needs to be completed prior to Summer Springboard Summer Springboard Survey is distributed via e-mail
after the students attend Orientation programming
Why this Design? To assess growth: The First Year Student Survey
serves as a pre-test to Summer Springboard (post-test)
There is no control population
Data Collection Issues Missing data
Assessment was not mandatory, but is expected 3,748 first year students 3,259 students completed the First Year Student Survey 1,909 completed both the First Year Student Survey and the Summer
Springboard Survey Students disregard of directions
249 students completed the post-test prior to the pre-test and their data could not be used (sample reduced to 1,660 students)
Invalid data 38 students from 1,660 were removed due to obvious response sets
Total sample was 1,622 out of 3,748 43.3%
Results: Explanation of Analyses Significance tests
Paired samples t-test used for analyses Compare mean of scores on pre-test with mean of scores on
post-test using the same students
Effect sizes Measure of practical significance
Important to investigate due to large sample
Cohen’s d Examined change in the metric of the variable Examined if students were more likely to reach
practical “cutoff” score after Summer Springboard.
Analyses: Perceived Cohesion Scale
Freshman Survey
(Pre)
Summer Springboard
(Post)
Mean SD Mean SDMean
Difference p d
Sense of Belonging 21.30 4.88 22.47 4.39 1.17 <.001 .25
Feelings of Morale 22.55 4.95 23.43 4.34 0.89 <.001 .18
Note. N = 1622. Scores range from 3 to 27 for both subscales.
Results: Perceived Cohesion
Students practically and significantly increased in sense of belonging and feelings of morale for JMU after attending Summer Springboard, but small effect size
What level of perceived cohesion would we like to see for all students after Summer Springboard?
Overall level of Sense of Belonging
Three items, 1 (strongly disagree) to 9 (strongly agree); range of total = 3 to 27 Wanted the majority of students to score above the
midpoint of scale at end of Springboard 18 or higher
Sense of Belonging at pre-test 76.7% (1,244 students out of 1,622)
Sense of Belonging at post-test 85% (1,379 students out of 1,622)
Overall level of Feelings of Morale
Three items, 1 (strongly disagree) to 9 (strongly agree); range of total = 3 to 27 Wanted the majority of students to score above the
midpoint of scale at end of Springboard 18 or higher
Feelings of Morale at pre-test 87.5% (1,419 students out of 1,622)
Feelings of Morale at post-test 90.3% (1,465 students out of 1,622)
Analyses: College Self-Efficacy Inventory
Freshman Survey
Summer Springboard
Mean SD Mean SDMean
Difference p d
Course Efficacy a 54.15 9.61 53.72 9.60 -0.43 .02 -.06
Roommate Efficacy b 33.90 4.83 33.58 4.95 -0.32 <.001 -.08
Social Efficacy c 67.43 13.38 68.60 12.90 1.16 <.001 .13
Note. N = 1619. a = scores range from 7 to 70, b = scores range from 4 to 40,c = scores range from 9 to 90
Results: College Self-Efficacy
Students practically and significantly increased only for social efficacy items, but again small effect size
What level of efficacy would we like all students to score after attending Summer Springboard with respect to interacting with faculty/staff?
Overall levels of social self-efficacy on CSEI
Five items; 1 (no confidence) to 10 (complete confidence); range of total = 5 to 50 Wanted the majority of students to score above the
midpoint of scale at end of Springboard 30 or above
Social efficacy at pre-test 78.9% (1,277 students out of 1,619)
Social efficacy at post-test 82.3% (1,333 students out of 1,619)
Analyses: Miville-Guzman Universality-Diversity Scale
Freshman Survey
Summer Springboard
Mean SD Mean SDMean
Difference p d
Diversity of Contact a 17.58 5.24 17.79 5.38 .21 .01 .06
Relativistic Appreciation a 22.55 4.13 22.46 4.34 -.08 .20 -.02
Sense of Connection b 20.92 3.92 20.73 4.04 -.19 .02 -.05
Note. N = 1,620. a = scores range from 5 to 30. b = scores range from 4 to 24.
Results: MGUDS
Students did not significantly increase in Diversity of Contact, Relativistic Appreciation, or Sense of Connection after attending Summer Springboard
What does this mean for Orientation at JMU? Goal 1: Increase perceived cohesion to JMU
Objective 1: Supported Objective 2: Supported
Goal 2: Assist in social integration to JMU Objective 1: Partially supported
Goal 3: Learn the value of a diverse campus Objective 1: Not supported
Use of Results: Perceived Cohesion Issues to consider for future assessment
Ceiling effect Students are scoring very high overall on this measure
Solution Change directions Find a more sensitive measure for perceived cohesion
2006: The following 6 items concern how you feel about JMU. Please read each item carefully and indicate the extent to which you agree with the statements…
2007: The following 6 items are asking how you feel about JMU. Not all students feel the same way or are expected to feel the same way. Please read each item carefully and respond according to how you feel at this current moment, not how you hope to feel as a first year student…
Use of Results: College Self-Efficacy
Issues to consider for future assessment Why didn’t students increase in roommate or course
efficacy? There is no programming (treatment); No need to assess
constructs
Minor change in social self-efficacy Investigating new outcome measures focused on confidence
to interact with faculty and staff Kicking around the idea of assessing help-seeking
What about the M-GUDS? As a result of our findings we realize that
Openness to Diversity is not substantially addressed during Summer Springboard, nor will it be.
Although increased Universal-Diverse Orientation was our goal, we realized that we first needed to help students get connected to JMU and meet faculty, mentors and peers before we could fully address openness to diversity.
Future Assessment for 2007
Clearer focus regarding goals of program Collecting PCS and CSEI No longer assess MGUDS during Summer
Springboard
Investigate ways to increase student participation in assessment endeavors
Revisiting goals and objectives
Thank you Questions?
Megan France ([email protected]) Steve Grande ([email protected]) Sara Finney ([email protected])
Thanks to Andrew Jones for helping with data analysis and shaping this proposal.
Perceived Cohesion1. I feel a sense of belonging to JMU.
2. I feel that I am a member of the JMU community.
3. I see myself as part of the JMU community.
4. I am enthusiastic about JMU.
5. I am happy to be at JMU.
6. JMU is one of the best schools in the nation.
CSEI1. Make new friends at college.
2. Divide chores with others you live with.
3. Talk to university staff.
4. Manage time effectively.
5. Ask a question in class.
6. Participate in class discussions.
7. Get a date when you want one.
8. Research a term paper.
9. Do well on your exams.
10. Join a student organization.
11. Talk to your professors.
12. Join an intramural sports team.
13. Ask a professor a question.
14. Take good class notes.
15. Get along with others you live with.
16. Divide space in your residence.
17. Understand your textbooks.
18. Keep up to date with your schoolwork.
19. Write course papers.
20. Socialize with others you live with.
MGUDS It is very important that a friend
agrees with me on most issues. I would like to join an organization
that emphasizes getting to know people from different countries.
In getting to know someone, I like knowing both how he/she differs from me and is similar to me.
Persons with disabilities can teach me things that I could not learn elsewhere.
I would like to go to dances that feature music from other countries.
Knowing about the different experiences of other people helps me understand my own problems better.
I often listen to the music of other cultures.
Getting to know someone of another race is generally an uncomfortable experience for me.
It's really hard for me to feel close to a person from another race.
I am interested in learning about the many cultures that have existed in this world.
I can best understand someone after I get to know how he/she is both similar and different from me.
I attend events where I might get to know people from different racial backgrounds.
I am only at ease with people of my own race.
Knowing how a person differs from me greatly enhances our friendship.
I often feel irritated by persons of a different race.
References Bollen, K.A. & Hoyle, R.H. (1990). Perceived cohesion: A
conceptual and empirical examination. Social Forces, 69(2), 479-504.
Solberg, V. S., O’Brien, K., Villareal, P., Kennel, R., & Davis, B. (1993). Self-efficacy and Hispanic college students: validation of the College Self-Efficacy Instrument. Hispanic Journal of Behavioral Sciences, 15, 80-95.
Miville, M.L., Gelso, C.J., Pannu, R., Liu, W., Holloway, P. & Fuertes, J.N. (1999). Appreciating similarities and valuing differences: Miville-Guzman Universality-Diversity Scale. Journal of Counseling Psychology, 46, 291-307.