Upload
kristen-brooks
View
25
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Evaluation Group Project
Citation preview
Running head: EVALUATION PLAN FOR BRIDGE TO LOYOLA 1
Evaluation Plan for Bridge to Loyola
Marlena Yang & Awatif “Ruth” Elias
Loyola University Chicago
EVALUATION PLAN FOR BRIDGE TO LOYOLA
2
Table of Contents
OVERVIEW OF BRIDGE PROGRAM 3 Introduction of Institution and Bridge to Loyola 3 History of Bridge to Loyola 4 Evaluation Purpose 6 Introduction of Evaluation Description 7 Bridge to Loyola’s Summer Program 8 Stakeholders 12 Logic Model 13 Assumptions and External Factors 15 Previous Evaluation 17
QUANTITATIVE APPROACH 17 Survey Description 18 Implementation and Administration 19 Statistical Analysis 20 Data Presentation 20
QUALITATIVE APPROACH 21 Participants 22 Focus Group Procedure & Implementation 22 Qualitative Data Analysis 24 Presentation Results 25
BUDGET 26 TIMELINE 26 NEXT STEPS 26 REFERENCES 28 APPENDICES 29
Appendix A: Logic Model 30 Appendix B: Survey Construct Map 31 Appendix C: Survey 32 Appendix D: Focus Group Email 36 Appendix E: Consent Form 37 Appendix F: Focus Group Protocol Script 38 Appendix G: Budget 40 Appendix H: Focus Group Sign-up Sheet 41 Appendix I: Timeline 42
EVALUATION PLAN FOR BRIDGE TO LOYOLA
3
Overview of Bridge Program
Postsecondary institutions enroll approximately 40% to 60% students who are in need of
additional support and further transition from high school to college (NCES, 2012). Universities
have instituted summer bridge programs in an effort to support these students in adjusting to
college life (Sablan, 2010). Summer bridge programs vary from one college or university to
another depending on the student population they serve. Bridge programs are implemented for
students needing additional support such as first-generation, low-income students,
underrepresented minority students or those who test as remedial (Kallison & Stader,
2012). Students identified to participate in these summer bridge programs either have low high
school grade point averages (GPAs), low scores in standardized tests such as the American
College Testing (ACT), the Scholastic Aptitude Test (SAT), and those majoring in a field that is
of focus for the program like science, technology, engineering and math (STEM) (Kezar, 2000,
as cited in Sablan 2010).
Bridge programs are intended to help students adjust to college socially and academically
by offering activities that include tutoring, study skills, advising, academic instruction and
general information about navigating college (Gullatt & Jan, 2003). We hope that by conducting
a process-based evaluation plan for a summer bridge program at Loyola University Chicago, we
will gain insights about the program’s effectiveness in the delivery of activities (workshops,
academic course, group service project, group extracurricular activities, & study hall) it offers its
students.
Introduction of Institution and Bridge to Loyola
Loyola University Chicago (LUC) is a private Catholic Jesuit institution founded in 1870
and is located in Chicago, Illinois. Loyola University Chicago is the largest Jesuit University in
EVALUATION PLAN FOR BRIDGE TO LOYOLA
4
the country. It has a total enrollment of 15,957 students with 10,168 undergraduate
students. This institution has six campuses: Lakeshore campus, Water Tower campus, Water
Tower campus, Health Sciences campus located in Maywood, John Felice Rome Center in Italy,
Beijing campus in China, and Ho Chi Minh City campus in Vietnam. Loyola’s mission focuses
on a diverse community that seeks God in all things and works to expand knowledge in the
service of humanity through learning, justice and faith.
The Bridge to Loyola program is a one-year (summer, fall, and spring), academic support
program providing services that include an intensive three-week summer program, academic
workshops, specialized academic advising and seminar class in both the fall and spring
semesters. The focus of this evaluation will be the summer portion of Bridge, which is a three-
week residential program all Bridge students are required to take before being fully admitted to
Loyola. Further details and information will be provided in later sections.
History of Bridge to Loyola
The Bridge Program at Loyola has taken different forms since its creation. The first
Bridge Program, the Learning Enrichment for Academic Progress (LEAP) was created in 1984.
LEAP started out as a summer bridge program that took place between one to two weeks during
the summer season. LEAP admitted students who had not met the admission criteria based on
ACT scores a chance to be fully admitted to Loyola once they participated in the summer
program. Participants in LEAP were required to attend lectures in which they were quizzed on
material covered including workshops that addressed academic and social aspects of being a
Loyola student. LEAP was later extended to a full academic year to continue supporting
students in their transition to college. Students took a university seminar, a three-hour credit
EVALUATION PLAN FOR BRIDGE TO LOYOLA
5
course in the fall, as well as a spring UNIV 102, a one-hour credit course to further build upon
their academic skills and adjustment to Loyola.
Success Through Educational Preparation (STEP) is another separate Bridge Program
from LEAP that has been offered at this institution. This program offered conditional admission
to students who did not meet Loyola’s admission requirement based on their high school grade
point averages (GPAs). STEP sessions often took place over a six-week period. Unlike LEAP,
STEP students were graded on attending study workshops, social activities, and on the academic
course. Students had to obtain a passing grade in the academic course in order to be admitted to
Loyola. However, in 2010, LEAP and STEP were combined to become the new Bridge to
Loyola Program due to reasons about funding and staff.
The Office of First and Second Year Advising (FSYA) operate the new Bridge to Loyola
program. Currently, Bridge to Loyola serves students who do not meet Loyola’s ACT/SAT
score requirement, but have the potential to succeed based on their high school grades. Bridge
serves students from all diverse backgrounds that fall under this certain category. It is designed
to help students who either lack motivation, have personal circumstances that inhibit their
academic performance, as well as students who are highly motivated, but come from
academically less rigorous high schools and therefore need additional support. Bridge to Loyola
was initiated to serve this student population at this specific institution. As Garcia (1991) stated,
a university or college may choose to require first-generation students, low-income students,
students of color, or students who have lower than average admissions tests and high school
GPAs to participate in a summer bridge program to raise the chances that these applicants will be
retained.
EVALUATION PLAN FOR BRIDGE TO LOYOLA
6
Evaluation Purpose
For the purpose of our evaluation plan, we will solely focus on the process of Bridge to
Loyola’s summer program versus the program as a whole, which includes the summer program
as well as a fall and spring portion. This process-oriented approach will guide us in describing
how effective Bridge to Loyola Summer Program services are delivered (Fitzpatrick, Sanders, &
Worthen, 2011). Our evaluation will be formative since its main purpose is to provide
information for program improvement (Fitzpatrick et al., 2011).
Ultimately, the overarching question we hope to address through the evaluation process is
how effectively is Bridge to Loyola’s summer program delivering its program content through
the services (academic seminars, summer course, community based learning project, study
groups, community programs, and living on campus) it offers? By effective, we want to measure
how useful the services are in meeting the needs of the students. For each of the services, we are
eager to learn if the content is applicable to the students’ personal and academic
needs. Discovering the effectiveness of each service will help the stakeholders tailor the
program content when necessary. Essentially, this evaluation plan is meant to be the starting
point for a potential process-oriented approach evaluation plan. Some overriding questions we
hope to answer after the evaluation is complete are as follows:
• Academic seminars: How effective stakeholders are in facilitating the content of the
following workshops: Learning styles, time management, and study and test preparation
skills.
• Community based service learning: Is there a reflective component post-volunteer
experience? If so, how are students expanding their knowledge?
EVALUATION PLAN FOR BRIDGE TO LOYOLA
7
• Academic course: How are students’ retaining content material within the intense three-
weeks of the program?
• Community programs: how are the activities designed, delivered and in what type of
environment and whether they are inclusive
Introduction of Evaluation Description
Given what we hope to tap into, our evaluation plan is intended to be a formative
evaluation. This type of evaluation will help to uncover ways to better deliver a program
(Wholey, Hatry, & Newcomer, 2010). By doing so, the evaluation approach will focus on the
process more so than the outcome, which former evaluation plans have generated in the past. By
referencing our evaluation question: “how effective is Bridge to Loyola’s summer program in
delivering its six program service components (academic success seminars, three-credit
academic course (CIEP 111), community based learning project, study hall sessions, community
programming, and live on campus experience) it offers?
By effective, we want to understand and fully grasp if the program is delivered according
to their learning outcomes and summer program benefits. We believe this process approach will
help us accomplish most of our program objectives. Since the focus is on the process, we hope
to gather information from students’ perspectives, observations of student engagement and make
alterations based on the information gathered to improve subsequent services. The process
approach is important because it will give insights about what is happening to students’
experiences while the services are being delivered, as well as inform if it is accomplishing the
intended outcomes set out in the program content.
Like every evaluation conducted, our program has potential strengths and weaknesses.
EVALUATION PLAN FOR BRIDGE TO LOYOLA
8
An area of strength for our process evaluation approach is it allows us to gather information
through observations and collect feedback from students, which will be used to make changes
while the summer program is still in session. This approach gives us the opportunity to focus
both on the program content, which has often been paid less attention to and on students. We are
also able to have insight on what parts of the program work well and which do not.
One potential weakness is overlooking student perspectives in shaping the program
services. The more varying perspectives present, the more they contribute in the creation of its
program services. However, since our main focus is centered on program content, we can easily
miss that student perspective. In addition, as evaluators, we do not have a direct relationship
with Bridge, therefore we might assume we know what Bridge needs but as is not true, what we
think they need might not align with their actual needs. We acknowledge that our subjective
perspectives might influence our evaluation.
Bridge to Loyola’s Summer Program
As mentioned in the introduction, the Bridge to Loyola program is a one-year, academic
support program providing services that include an intensive three-week summer program,
academic workshops, specialized academic advising and seminar class in both the fall and spring
semesters. However, the focus of this evaluation will be on the summer program, which is a
three-week residential program that all Bridge students are required to take and pass with a C- or
higher in order to be fully admitted to Loyola. In this section, we highlight the details of Bridge
to Loyola’s summer program in introducing the benefits, learning outcomes, and program
specifics.
The entirety of Bridge to Loyola’s summer program is through six service components:
academic success seminars, a three-credit academic course (CIEP 111), community building
EVALUATION PLAN FOR BRIDGE TO LOYOLA
9
programs, study hall sessions, and a community based learning project. It is through these areas
that Bridge students benefit personally, socially, and academically. The program is intended to
create a well-rounded college experience of what its like to be college student. It is designed to
mirror a typical first-year student schedule with classes, study time, and co-curricular activities.
In addition, students get to live on campus in one of the university residence halls with an
assigned roommate. In regards to the program learning outcomes, the overarching goal is to
develop self-efficacy and personal growth for students to be successful while transitioning to
their first year in college (Hill & Teetsov, 2014).
Bridge to Loyola strives to align their learning outcomes through the six service
components aforementioned. In doing so, the program kicks off with the mandatory orientation
program, which all incoming first-year students are required to attend. All Bridge to Loyola
students will arrive to campus ready to settle in the residence hall for three weeks. Loyola’s
first-year orientation program runs for two consecutive days welcoming students and their
families to the Loyola community. Soon after orientation is complete, the official Bridge to
Loyola program begins immediately with a warm welcome reception. After the welcome
ceremony, students’ depart directly to Loyola University’s Recreation and Ecology Center
(LUREC) for a weekend retreat filled with team-building activities, opportunities for self-
reflection, and workshops. In respective order, we will describe six of the service components
that make up the summer Bridge program: academic success seminar, academic course (CIEP
111), community based learning project, community programming, study hall, and living on
campus.
EVALUATION PLAN FOR BRIDGE TO LOYOLA
10
Academic Success Seminars
These are academic strategy workshops provided for Bridge students throughout the
summer. The stakeholders and campus partners facilitate all workshops with themes that include
reading and note-taking strategies, learning styles, time management, and studying and test
preparation skills. Each workshop session takes place for an hour during the weekday and is
mandatory for all students to attend.
CIEP 111 Academic Course
Bridge students are required to take a three-credit introductory course titled Sex and
Gender in Anthropological and Psychological Perspective. This course is intended to provide
students with a scientific foundation upon which anthropological and psychological concepts can
be built. The following are examples of a few of the learning goals presented by Grauer and
Leon (2013), which students will be introduced and expected to understand:
• Processes of human development
• Biological mechanisms of evolution
• How scientific conclusions and theories are developed
Community Based Learning Project
The stakeholders work in conjunction with community partners in finding service
learning opportunities for all Bridge students. This is the student’s opportunity to branch outside
of Loyola’s campus and explore different parts of Chicago. Students volunteer every Saturday
morning for a couple hours during the three weeks of the program.
Community Programming
The peer mentors and the graduate intern spearhead the community programs. These
programs are social activities intended to a build a sense of community. The activities are what
EVALUATION PLAN FOR BRIDGE TO LOYOLA
11
students look forward to after a long and intensive week of academics. The following are
examples of weekend programs from this past summer: target run, on-campus scavenger hunt,
movie night, Navy Pier fireworks, arts and crafts, pajama party karaoke night, and study break
game night. Each activity is intended for students to build community and develop relationships
with their peers.
Study Halls
Each night Sunday through Thursday, there is a study hall session from 6:30 p.m. to 8:30
p.m. Each study session is mandatory and expected for all student participants to take part in.
Students are provided with academic support and are surrounded by peers creating a supportive
and encouraging learning environment.
Living on Campus
Students get to first hand experience of what its like to live on a college campus. As part
of the three-week summer program, students live on campus in one of Loyola’s residence halls.
The peer mentors serve as resident assistants and the graduate intern’s role is similar to an
assistant residence director.
Post Bridge to Loyola Summer Program
By the end of the program, students who have successfully completed the summer
program will continue with Bridge to Loyola in the fall and spring completing the one-year
program. Bridge students will take a one-credit First-Year Seminar course (UNIV 101), which is
a university requirement for all first-year students. The primary stakeholders (discussed in the
next section) teach the course as well as academically advise Bridge to Loyola students. In the
spring, students will then take a CIEP 112 Skills Strategies course also instructed by the primary
stakeholders.
EVALUATION PLAN FOR BRIDGE TO LOYOLA
12
Stakeholders
There are a number of campus partners, community partners, as well as professional
faculty and staff who contribute to the operation of Bridge. In this section we will introduce the
primary stakeholders and other campus partners who help make this program possible and
function properly.
The primary stakeholders for Bridge to Loyola are academic advisors China Hill and
Natasha Teetsov from the Office of First and Second Year Advising (FSYA). Once the
Admissions Office (another stakeholder) determines who and how many students are selected
into Bridge, China and Natasha are given student information to coordinate the program that will
accommodate the selected group of students. Interestingly, China and Natasha do not participate
in the selection process, which is an important perspective they hope to gain for the upcoming
Bridge program. With that said, each summer is a different summer to look forward to, as the
number of students will determine the dynamic of the program. Last year, there were
approximately 100 student participants and 36 this past year. However, Natasha has indicated
that for the following summers, the numbers will remain around 40 students (Natasha Teetsov,
Personal Communication, November 18, 2014). A graduate intern, supervised by Natasha, also
works closely with the primary stakeholders in developing and executing programs before,
during, and after the summer portion of Bridge. The intern works directly in supervising,
training, and advising peer mentors in preparation for the students’ arrival and all social
programs. Aside from the primary stakeholders, a graduate intern, and the admissions office,
faculty members from the Department of Psychology and the Department of Anthropology are
also involved in the three-week summer program. Both faculty members co-instruct the Bridge
to Loyola summer CIEP 111 three-credit hour academic seminar course.
EVALUATION PLAN FOR BRIDGE TO LOYOLA
13
In terms of individuals and groups who are directly and indirectly affected by the
program, students are the ones personally and academically benefitting as the program is solely
created to support them. Meanwhile, we recognize the Loyola community, Bridge parents, and
community partners are groups who indirectly derive benefit based on the improvement and
success of the program. Bridge is aimed to amplify students’ academic performance, social
integration, as well as their ability to persist throughout their educational journey. The Loyola
community will gain students who will most likely know how to navigate campus and reach out
for support when needed. Retention rate is intended to improve, which is a good representation
of the university. Parents of Bridge students become more aware of Loyola resources and will
encourage their students to take advantage of those opportunities. The growth of their students is
in and of itself an affective benefit. As part of the service-learning component of the program,
community partners will gain volunteers to support their organization. All in all, individuals,
groups, and even society, are positively affected in some way.
Logic Model
A logic model is a visual representation that is useful for conceptualizing, planning and
communicating with stakeholders about our planned work and intended outcomes about the
Bridge to Loyola Summer Program (Kellogg, 2004). We will be utilizing a logic model (see
Appendix A) to illustrate inputs, outputs and outcomes. Inputs represent the activities and
resources the Bridge to Loyola summer session invests in. People tend to be the main focus of
inputs (Kellogg, 2004). The summer portion of the Bridge Program includes one staff from the
undergraduate admissions office that is responsible for recruiting students for this program. Two
academic advisors from FYSA are responsible for meeting and advising students, while two
faculty members, one from the Psychology Department and another from the Anthropology
EVALUATION PLAN FOR BRIDGE TO LOYOLA
14
Department work collaboratively to create a syllabus for the seminar course to be taught by them
during the summer. Lastly, one graduate intern is available to assist staff in any way needed, and
four peer mentors are present as resources for incoming students in any capacity they need such
as navigating campus, and asking relevant questions.
Additionally, campus partnerships are important in contributing to the success of the
program. The summer session teams up with the university orientation team, academic services
(i.e., tutoring), and residence life since Bridge requires its students to live in campus housing for
the duration of the program. Time is also an important asset spent training staff, reserving
locations where sessions will take place, as well as purchasing and gathering materials. These
materials include bus passes, pens, papers, day-to-day agenda of events and meals.
The activities offered to incoming first-year Loyola Bridge students encompass two main areas:
academic support and programming. Academic support is comprised of academic advising, a
summer course, and workshops about time management, test preparations, and reading
strategies. In programming, Bridge offers activities to help students build community by
participating in the welcome reception, retreat, exploring Chicago and taking part in a service-
learning project.
The short-term changes we expect will result after the completion of each particular
session or sessions. The first two outcomes can be grouped to represent study skills acquired
from attending academic workshops. We hope students can discover different learning styles
and apply them accordingly to the study material and establish effective study habits. The
following three can be grouped to represent a general understanding and application of those
concepts to their own lives. For example, we hope students are able to learn by navigating LUC
and taking initiative in exploring the city of Chicago, enhancing their self-efficacy for academics
EVALUATION PLAN FOR BRIDGE TO LOYOLA
15
such as posing questions, and gaining an awareness of college socially and academically. Short-
term changes are intended to lead to medium-term outcomes, which will take place after the
completion of the three-week residential summer program.
Some of the results of the medium-term outcomes include students’ confidence in
reaching out for support when needed, familiarization with where resources are located,
established connections and relationships with peers, and a smoother acclimation to college
during the first week of classes. Students will also likely apply the academic strategies they
learned from summer to their fall classes. The long-term outcomes take place during a student’s
enrollment at Loyola and beyond graduation. We hope students will be able to grow personally
and professionally by developing confidence and self-efficacy, as well as discerning their career
choices. We also hope students are able to take agency of their own learning via asking questions
in class, or establishing study groups with peers to better understand academic content while still
seeking support if needed. We expect them to be motivated to continue their education until
graduation and pursue their career aspirations. After participating in the program, we anticipate
them possibly giving back to the program by becoming mentors, sharing Bridge experiences with
prospective students, or in any other way they find fitting.
Assumptions and External Factors
We also took assumptions and external factors into consideration when developing our
logic model. We considered four assumption areas and four external factors that have a close
relationship with one another.
First, we assume the summer program outcomes set up first-year students for academic,
social and personal success after completion. To describe what we mean by this, Bridge to
Loyola is a well-rounded program aimed to prepare students to have a smooth transition to
EVALUATION PLAN FOR BRIDGE TO LOYOLA
16
college, which simply explains what the academic, social, and personal success refers to. This
also puts them at an advantage in navigating campus before the start of school. Next, we assume
first-year students are prepared for engagement and self-reflection, as the program is set-up with
opportunities to do so. Third, we assume campus partners are fully equipped and prepared to
facilitate academic and leadership workshops. Primary stakeholders work collaboratively with
campus partners to present on materials in which students will benefit from. Not only are
students exposed and introduced to opportunities at Loyola, but campus partners get to promote
their programs and through that, active student participation. Finally, our last assumption is that
the residence hall is a comfortable environment to build community. As mentioned earlier,
students are required to live on campus as part of their participation in Bridge to Loyola. Again,
going back to the well-rounded college experience addressed in our first assumption. We
recognize these assumptions connect with our established external factors. With that regard,
demographics, location, student expectations, and different student academic experiences are
identified external factors.
Students arrive to the program with unique experiences and different backgrounds. There
is a diversity of students in terms of social class, racial/ethnic identity, personalities, and first-
generation students. Not only that, but students may enter Bridge to Loyola’s summer program
from a rural area where they are not familiar with Chicago’s fast-paced environment and the
multicultural communities within Chicago. We also have to recognize that students may have
certain expectations prior to starting Bridge based on what they may have heard from peers who
went through the program. Overall, assumptions and external factors are important parts of our
logic model to highlight in our evaluation plan.
EVALUATION PLAN FOR BRIDGE TO LOYOLA
17
Previous Evaluation
In regards to previous evaluation, there has not been formal evaluation completed on the
Bridge to Loyola program. No assessment has been created on college statistical data besides
anecdotal data (China Hill, personal communication, September 26, 2014). Anecdotal data is
recorded to take note on successes among the students as well as information about the
circumstances under which students are not persisting. Prior to Natasha and China’s
involvement with Bridge, the former Assistant Director for FSYA worked with two graduate
students in the Higher Education program at Loyola to create an evaluation plan for Bridge to
Loyola. Although we wish to culminate a plan as well, our evaluation approach is different as
we aim to focus on the process of the summer program in terms of the effectiveness of program
delivery, which we describe in the next section. Meanwhile, the previous evaluation plan was an
outcome-based approach solely focusing on if students were successful and transformative by the
end of the program. Given no formal evaluation has been created, academic achievement data
has been collected and analyzed for retention and academic success levels. In 2010, Bridge to
Loyola program saw a 93% retention rate, which was the highest in recent years (“Bridge to
Loyola,” 2011)
Quantitative Approach
For the quantitative part of the evaluation plan, we will study student participants in the
Bridge to Loyola Summer Program. Since we plan to sample the entire student population of
approximately 40 students, the approach we intend to use is the census-sampling frame (Wholey,
et al., 2010). We will not use a comparison group because we currently recognize no relevance
for it. The program tends to make changes in terms of the recruitment process, therefore,
producing fluctuating numbers of student participants each year. For instance, during the
EVALUATION PLAN FOR BRIDGE TO LOYOLA
18
summer of 2013, the primary stakeholders had to coordinate the Bridge to Loyola summer
program for 130 students. However, summer of 2014 had 36 student participants. For the
upcoming year, Natasha emphasized that the student numbers will remain around 40.
We are anticipating a 100% student response rate, as there will be time during the
program for students to complete the paper-based survey. Cross-sectional design is the type of
research design we plan to use. It makes sense to utilize this design since our approach is
process-based and because we are distributing the survey at a single point in time for the entire
student population and a variety of program elements. Although a cross-sectional design is
necessary for our approach, it does reveal limitations. There is a higher chance for data-entry
error to occur. To prevent that high chance, we will keep the paper copy of the surveys and
double check responses.
Survey Description
Our survey (see Appendix B) has a total of eleven main questions with sub-questions for
the first seven. The last four questions (Q8 through Q11) are general demographic questions,
which are best asked at the end of the survey. These questions include racial/ethnic identity,
emphasis on parental academic background, campus familiarity, and geographic residence. Each
of these different types provides us with important context to collect when analyzing the data
results. It helps to simply get a general idea where students are coming from.
We sorted the variety of questions into three different categories: personal, academic,
and social. As indicated in our survey introduction and our evaluation program description, there
are six imperative service components of the summer program process we wish to
assess: academic success seminars, academic course (CIEP 111), community programming,
community based learning project, study hall sessions, and live on campus experience. We
EVALUATION PLAN FOR BRIDGE TO LOYOLA
19
decided to break up the questions into three categorical areas because we recognized a theme in
the delivery of each of these service components. The method of breaking up the sections into
three will help students fully grasp the intentions of the Bridge to Loyola’s summer program,
which is to create an academic, social, and personal growth experience.
Implementation and Administration
After formulating questions for our survey, we will use pilot testing to help us identify
which questions are confusing or need rewording, to determine the time needed to complete the
survey as well as organizational issues that emerge (Wholey et al., 2010). We will work with our
primary stakeholders Natasha and China, the primary academic advisors for Bridge to Loyola to
reach out to former Bridge participants who are still current Loyola students since the Bridge
Program only lasts for one year. We will be utilizing dual outreach to seek feedback from Bridge
alumni participants through e-mail and in person contact. We encourage all students interested in
helping us out to come look at our survey questions during business hours from 9-5pm before the
spring 2015 semester ends. We also understand that not everyone will be able to come in person,
therefore, those who would like to participate but are unable to come in person will be e-mailed
our quantitative survey and they can respond with recommendations, gaps, and suggestions.
Having multiple students take a look at our survey will give us many perspectives of how they
understand the survey and if the questions are relevant.
Natasha and China will be administering our paper-based survey after a mandatory
workshop event at the beginning of the third week. Since we are expecting about forty students
to complete the survey, it will be feasible to manually enter the data into a database. We will not
have an incentive to allow equal treatment for all participants. Since the academic workshops are
EVALUATION PLAN FOR BRIDGE TO LOYOLA
20
a requirement, we assume almost all students will be present and will be able to complete the
survey in the allotted 10-15 minutes timeframe, or however long they take.
Statistical Analysis
For the statistical analyses in assessing the process of Bridge to Loyola Summer Program,
we will use simple descriptive statistics to answer our evaluation questions, which describes the
characteristics of these variables in terms of their central tendency and their dispersion (Schuh,
2009). This method will help us to determine areas of strengths and weaknesses within the
program when looking at the traditional tendencies of mean, median, and mode. Then we will
employ inferential analyses to see whether there are significant differences between demographic
groups or whether particular items on the survey relate to each other.
Furthermore, we will examine results across demographic segments of race and gender
using t-test to identify mean differences on the Likert-scale questions. In examining these
segments, we can interpret whether gender differences exist as well as if different racial groups
experience the Bridge Program services differently. Even though we realize that results will be
useful for the Bridge to Loyola Summer Program, delving into the specific gender and racial
demographics can help us understand how students differ in those aspects and how to implement
changes in those particular areas.
Data Presentation
We will present our data using tables and charts. By using tables, we will describe what
students thought about the delivery of the program in terms of their academic, social, personal,
and live-in experiences. Creating tables will help us measure the extent to which students agree
with the effectiveness of particular program components. After that, we will be using a histogram
and a pie chart. The histogram would be more useful in seeing what activities garner more
EVALUATION PLAN FOR BRIDGE TO LOYOLA
21
student interest and by how much. The pie chart will be more useful in presenting demographics
such as race, gender, and type of activities (academic, social and personal).
Qualitative Approach
Qualitative research can utilize different forms such as transcripts, notes from interviews,
observational field notes, videos, and e-mails to gain a deeper understanding of a subject matter
(Wholey et al., 2010). For the qualitative approach of our evaluation, we will use a focus group
approach to help us better understand our evaluation question, and to obtain a richer narrative of
students experiences (Creswell, 2009). Our evaluation question aims to find out how effective
Bridge to Loyola Summer program is in delivering its program content through the following
services it offers: workshops, summer course, service learning, study groups, and social
activities. Schuh (2009) states that focus groups can help evaluators when they “need more
information to help shed light on quantitative data already collected” (p. 69). Conducting a
quantitative survey provided important results, but only general knowledge of responses to those
questions. Thus, carrying out a focus group will enable us to build upon questions from the
quantitative survey by probing for clarity by inquiring why and how events happened.
One of the reasons we chose to employ the focus group approach as opposed to other
methodologies is to elicit ideas, attitudes, feelings or perceptions regarding the process of content
delivery and student learning as perceived through students’ own perspectives. In addition, this
approach enables us to see ideas emerge from the group (Schuh, 2009). For example, when one
participant shares an idea or experience, it is likely that other students will recall similar
experiences they identify with and expand on them, thus generating common themes. Finally,
focus groups provide specific feedback and insights about their experiences.
EVALUATION PLAN FOR BRIDGE TO LOYOLA
22
Participants
Bridge students who completed the Bridge to Loyola Summer Program and decided to
attend Loyola will be the only ones asked to participate in focus groups. Since the summer
Bridge program consists of about thirty-six students, we will be conducting two separate focus
groups of six students each. “This number provides for enough conversation while yet allowing
ample time for individuals to express their opinions” (Schuh, 2009, p. 91). Even though we only
need twelve students overall, we will have a Google sign up sheet (see appendix H) with fifteen
spots in case not everyone shows up. Each focus group will be divided into two by ensuring that
there is variation between gender and racial demographics.
The focus groups will be a follow up to the quantitative survey response. The focus group
will serve to illuminate some of the survey findings. The quantitative survey will be completed
during the summer while the focus group will take place at the beginning of fall semester in
2015. An e-mail (see appendix D) will be sent out to all students who have completed the Bridge
to Loyola Summer Program and have chosen to enroll in Loyola.
Focus Group Procedure & Implementation
The primary stakeholders, Natasha and China, and the graduate intern, will alternate
moderating and observing (assistant moderator) the intended two focus groups. For instance, the
first focus group will be led by Natasha and observed by the graduate intern. The following
focus group will be observed by the graduate intern and moderated by China. The primary
stakeholders want to provide their graduate intern an opportunity to make the most of their
internship experience. Therefore, the graduate intern will take the lead in moderating the second
group while Natasha assist. With available space in the Office of First and Second Year
EVALUATION PLAN FOR BRIDGE TO LOYOLA
23
Advising, where the primary stakeholders are located, one of the conference rooms will be
reserved for each focus group.
Bridge to Loyola summer program begins at the end of July and finishes early August.
Once the program is complete, students have a few weeks before the start of the semester to
determine if they wish to continue their college journey at Loyola. There are also students who
may not have passed the summer course, which as a result, does not admit to Loyola that
academic semester. Students who decide to attend Loyola will be the ones invited to participate
in the focus group. The first two days (Tuesday and Wednesday) after Labor Day is when the
focus groups will occur, with one happening on each day. In an effort to accommodate student
schedules and because a food and drink incentive will be provided, the times will occur during
lunch and dinner for approximately ninety minutes total (30 minutes for lunch/dinner, 60 minutes
for focus group). Below is an example of the focus group schedule:
• Focus Group #1 (Tuesday, 12pm – 1:30pm), Sullivan Center Room 290
o Lunch will begin at noon
o Focus Group begins promptly at 12:30pm, Sullivan Center Room 290
• Focus Group #2 (Wednesday, 5:30pm – 7pm)
o Dinner will begin at 5:30 pm
o Focus Group begins promptly at 6pm
The focus group is intended to be small and intimate for a rich conversation. Our hope is
to have at least six students per session with the anticipation of an audiotape present in the space.
However, we recognize that some students may be less candid and uncomfortable if they know
they are being audio recorded. With that, we will emphasize the confidentiality part as well as
the importance and reason behind the decision to audiotape the focus group. We want to ensure
EVALUATION PLAN FOR BRIDGE TO LOYOLA
24
we do not miss any relevant information that will allow us to improve our program. The
assistant moderator will also being taking notes to pick up on non-verbal messages and thematic
topics, which is necessary when analyzing the data.
Our protocol (see appendix F) demonstrates the entirety of the focus group process from
start to finish. We formatted the protocol in a way that would allow individuals who have no
affiliation with the program to lead the discussion. The dynamic we wish to create is a
comfortable and friendly space. With that, we included a short low-energy icebreaker after
announcing the ground rules. We also indicated the final ground rule as to “have fun!” to go
along with nature of the space we aim to get at.
Qualitative Data Analysis
Once the focus groups are completed, the audiotapes will be transcribed by one of the
primary stakeholders. Since the purpose is to learn the delivery effectiveness (process) of the
program components in Bridge to Loyola’s summer program, we have decided not to develop a
coding scheme prior to the interview. Instead, we are relying on an emergent approach to
develop codes, which are themes emerged as the data is being reviewed (Whole et al., 2010). In
doing so, the methods of categorization used will be descriptive and pattern as we are seeking to
discover recognizable themes and trending patterns from the data. To help with this process,
word repetition and pawing are two coding techniques necessary for this approach.
We have decided not to employ member checking due to the lack of scheduling time.
Fall semester for the primary stakeholders is a busy time in the office with teaching and
academic advising. However, testing for inter-rater reliability will be in done to ensure both
primary stakeholders are consistent with the results from coding. Since member checking is not
considered, information pertaining to personal biases will be incorporated, especially since data
EVALUATION PLAN FOR BRIDGE TO LOYOLA
25
analysis is derived only from the interpretation of the primary stakeholders. Given the number of
data collection findings obtained from survey results, observations, and focus groups, process of
triangulation is essential when finalizing the conclusion for ensuring trustworthiness and
credibility (Schuh, 2009). Triangulation affirms and validates the final results. It is a process of
comparing and contrasting collected data across the qualitative and quantitative approaches and
recognizing that there are potential errors to encounter.
There are certainly benefits and limitations for our intended approach. A limitation, as
recognized earlier, is not incorporating member checking. This incorporation is important to
consider, as it is a valuable process that will ensure personal biases do not filter participants’
narratives in ways that do not reflect student’s personal thoughts and stories shared in the focus
groups. Another limitation is not being intentional about selecting who is partaking in the focus
group. Our pool of students is a randomized selection process, mainly to accommodate our
student’s availability. However, reliance on inter-rater reliability and the process of triangulation
is how we hope to counter both limitations addressed.
Presentation Results
For the final report, our results will be presented using tables, which will include direct
quotes in the table. For instance, since we are assessing the effectiveness of five service
components (academic workshops, study halls, academic course, extracurricular activities, and
service project), the first column will entail a row for each of these component areas. The
second column will have the themes derived from each area. The third column will list direct
quotes pertaining to each service component.
EVALUATION PLAN FOR BRIDGE TO LOYOLA
26
Budget
We will utilize the FSYA and LUC libraries to minimize the cost for our budget (see
appendix G). Our incentive consists of food (pizza, salad, and water), which will be the only
costs. Other than that, we plan to use the FSYA conference room to conduct focus group
interviews, as well as using the FSYA office for printing and storing data using its USB drive
and storing it in a locked drawer. The personnel from the FSYA office will be the ones
facilitating, transcribing, and observing the focus group interview.
Timeline
A brief 2015 timeline has been created (see Appendix H) to project when the evaluation
plan will come to life. A meeting with the primary stakeholders should begin in May when the
logic model is developed and presented. After that initial meeting, the following month should
be when the quantitative survey questions are generated and pilot testing will take place. The
very end of July and early August is when the survey is distributed to students. Data analysis of
the survey occurs within the same month the focus group email invitation is sent. By September,
after Labor Day weekend, the conduction of the focus groups should happen as well as
transcription. The entire month of October should be reserved for transcription and data analysis
to prepare for November when the final report is complete and ready to present.
Next Steps
It is completely up to primary stakeholders, Natasha and China, to carry out
improvements from data collection and if they choose to share the final report with campus
partners and students. We recommend the primary stakeholders share, as the campus partners all
play an important role in the process of the Bridge to Loyola program. In addition, we
encourage a conduction of the same evaluation plan for the next few years to check progression,
EVALUATION PLAN FOR BRIDGE TO LOYOLA
27
which would allow the opportunity to compare and contrast results. Stress management is a
crucial component primary stakeholders should take into account next time they conduct a
similar evaluation. Most of the students in Bridge Programs are those in need of additional
academic and transitional support. The material for this particular summer program is a lot and
packed in such a short time. We hope these stakeholders are able to identify how students
manage stress in a brief, three-week intensive Bridge to Loyola Summer Program.
EVALUATION PLAN FOR BRIDGE TO LOYOLA
28
References
“Bridge to Loyola” (2011). Bridge to Loyola [Annual Report]. First and Second Year Advising,
Loyola University Chicago, Chicago, IL.
Creswell, J.W. (2014). Research design: Qualitative, quantitative, and mixed method
approaches (4th Edition). Thousand Oaks, CA: Sage Publications.
Fitzpatrick, J., Sanders, J., & Worthen, B. (2011). Program evaluation: Alternative approaches
and practical guidelines (4th Ed.) New York: Longman.
Garcia, P. (1991). Summer bridge: Improving retention rates for underprepared students.
Journal of the Freshman Year Experience, 3(2), 91-105.
Grauer, A., & Leon, S. (2013). Sex and gender in anthropological and psychological perspective
[Class handout]. Department of Anthropology and Psychology, Loyola University
Chicago, Chicago, IL.
Gullatt, Y., & Jan, W. (2003). How do pre-collegiate academic outreach program impact college-
going among underrepresented students? Boston, MA: Pathways to College Netwo
Clearinghouse.
Hill, C., & Teetsov, N. (2014). Bridge to Loyola [Class handout]. First and Second Year
Advising, Loyola University Chicago, Chicago, IL.
Kallison, Jr, M. J., &Stader, L.D. (2012). Effectiveness of Summer Bridge Programs in
Enhancing College Readiness. Community College Journal of Research and Practice,
36,340-357.
Sablan, R.J. (2014). The challenge of summer Bridge Programs. American Behavioral Scientist
Schuh, J.H. & Associates (2009). Assessment methods for student affairs. San Francisco, CA:
Jossey-Bass.
EVALUATION PLAN FOR BRIDGE TO LOYOLA
29
Wholey, J.S., Hatry, H.P., & Newcomer, K.E. (Eds.). (2010). Handbook of practical program
evaluation (Third Edition). San Francisco, CA: Jossey-Bass.