Embed Size (px)
Citation preview
This article was downloaded by: [Colorado College]On: 18 November 2014, At: 19:07Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK
Tertiary Education and ManagementPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/rtem20
Assessing undergraduate education from graduatingseniors' perspective: Peer institutions provide thecontextAnne Marie Delaney aa Academic Affairs , Babson College , Babson Park, MA, 02457–0310, USA E-mail:Published online: 17 May 2010.
To cite this article: Anne Marie Delaney (2001) Assessing undergraduate education from graduating seniors' perspective: Peerinstitutions provide the context, Tertiary Education and Management, 7:3, 255-276, DOI: 10.1080/13583883.2001.9967057
To link to this article: http://dx.doi.org/10.1080/13583883.2001.9967057
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of theContent. Any opinions and views expressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon andshould be independently verified with primary sources of information. Taylor and Francis shall not be liable forany losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use ofthe Content.
This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
ANNE MARIE DELANEY
ASSESSING UNDERGRADUATE EDUCATION FROMGRADUATING SENIORS' PERSPECTIVE: PEER INSTITUTIONS
PROVIDE THE CONTEXT
ABSTRACT. This paper presents a comparative research study as a model for assessingundergraduate education. This evaluation model used Factor Analysis to create scales,tested the reliability of these scales, and used Analysis of Variance to compare theprimary institution with five types of comparable institutions. The value of this statistical/methodological evaluation model was shown. Results are based on Class of 1998 surveyresponses from 244 graduating seniors at the primary institution and 1481 students at 39peer institutions. Comparative analyses identified business courses, faculty, career prepara-tion and impact on students' technological skill as strengths and students' knowledge ofarts and foreign languages and campus social life as areas for improvement at the primaryinstitution. Recommendations advocated publicising students' superior satisfaction withcourses, faculty and career preparation; using these results as baseline data for futurecurriculum evaluation; and continuing efforts to enhance campus social life.
INTRODUCTION
PurposeThe purpose of this paper is to present the development and use of anevaluation model for assessing the perceived impact of undergraduateeducation on students' intellectual, personal, social and moral develop-ment. The paper is based on a comparative research study using seniorsurvey data for Class of 1998 graduating seniors. Results for the primaryinstitution are compared with data for peer institutions classified infive different categories. The paper also demonstrates how student feed-back can be translated into recommendations for programme and policydevelopment. The original study was designed to address the followingquestions. Compared with business major graduates at peer institutions:
— how satisfied are graduates of the primary institution with theacademic programme, student services and campus social life?
— to what extent do graduates of the primary institution think theirundergraduate education enhanced their knowledge and abilities in
Tertiary Education and Management 7: 255-276,2001.© 2001 Kluwer Academic Publishers. Printed in the Netherlands.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
256 ANNE MARIE DELANEY
various domains including: critical thinking, quantitative analyses,foreign languages and appreciation of the arts?
— overall, how satisfied are graduates of the primary institution withtheir education?; and
— if they had the chance to relive their college experience, wouldgraduates of the primary institution choose the same institution again?
Context for the studyThis study was sponsored by and conducted for Babson College, a private,business college located in Wellesley, Massachusetts, USA. Founded in1919, the College is committed to being an internationally recognised leaderin management education by educating students to be innovative leaderscapable of initiating, implementing and managing change. Field experi-ence is a major part of a Babson education, and the College collaborateswith domestic and international companies to create valuable internshipsfor students. Babson College enrolls approximately 1,600 undergraduatestudents from most of the 50 states, many US territories and some 70 othercountries.
REVIEW OF THE LITERATURE
The present study reflects two significant developments that have occurredin the field of assessment during the last two decades: first, a movementfrom normative based to empirically based approaches (Ewell 1988) and,second, an increased use of inter-institutional data sharing for assess-ment. As Trainer (1996) observed in his discussion on inter-institutionaldata exchange, recent trends in higher education - the growing popularityof benchmarking, performance indicators and comparative rankings -indicate that the need for comparative data will only increase. Further, abroad range of constituencies - accrediting agencies, trustees, legislatorsand the public are urging higher education leaders to assess their strengthsand weaknesses relative to their peers.
It is important to note that external and internal constituencies differ intheir perception of the primary purpose of assessment. The former groupconcentrates primarily on accountability while the latter focuses more onimproving curricula, instruction and student services. What unites the twogroups is the shared interest in finding out if college is helping students.
A major challenge for researchers is to develop assessment strategiesresponsive to both constituencies. Such strategies simultaneously provideevidence of accountability and useful information to guide improvements
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
ASSESSING UNDERGRADUATE EDUCATION 2 5 7
in academic programmes and student services. By achieving the dual goalsof accountability and improvement, assessment will strengthen highereducation both in terms of public perception and student growth. As Banta(1993) observed, assessment has a powerful potential for improving highereducation. "If it should realize that potential, one might argue at some pointin the twenty-first century that assessment has made a greater difference forstudents than any other single influence in the history of higher education"(p. 375).
To ensure success in assessment studies, experts recommend the useof multiple measures. Robert Pace (1979) observed that, by using multiplemeasures, an institution may be able to recognise more precisely where -and where not - to focus their monetary and reform efforts. Another keyprinciple is to concentrate on the students' voice as the most importantvoice in assessment. Ewell (1983) proposes that by using student satis-faction assessment results to shape institutional reform, institutions arevalidating the importance of the students' 'voice'. Echoing this theme,Lingrell (1992) observes that, since the ultimate beneficiary of assessmentis the student, it only seems appropriate to ask the student what he or shethinks.
Lingrell (1992) also presents the senior survey as an ideal vehicle forfocusing on the students' voice. He notes that,
. . . measurement during the senior year provides important information about previousyears in college, as well as about student perceptions about his/her preparation for lifeafter college Without the use of a senior survey, universities miss the chance to gainvaluable information about its students. Implementation of a senior survey program wouldgive faculty and administrators access to information that is essential in the evaluation ofacademic programs and student services, (p. 18)
Lingrell (1992) further suggests that senior survey assessment informationcan be used to: improve instruction, recognise outstanding programmesand teaching, improve retention, improve admissions, and prepare foraccreditation review. Annual senior surveys may serve as a catalyst towardinstitutional reform and improvement.
Two major areas appropriately addressed through a senior survey arestudent satisfaction and student perceptions of what they have learned andgained through their college experience. In their 1997 National StudentSatisfaction Report, Shreiner and Juillerat (1997) describe student satis-faction studies as ". . . self-examinations that enable institutions to measuretheir students' satisfaction with a wide range of college experiences"(p. 1). Furthermore, they note the importance of an educational institution'sserious consideration of students' satisfaction results as a way to recognisethe institutional strengths as well as areas in need of improvement.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
258 ANNE MARIE DELANEY
Shreiner and Juillerat (1997) argue that annual assessments provideregular feedback to an institution's internal and external constituents onthe effectiveness of all campus programmes and services. In addition,they can assist institutions in establishing annual benchmarking of theirstudent population; tracking the impact of new initiatives on student satis-faction; identifying new areas for further improvement; tracking studentexpectations as they progress through class levels; and identifying currentstrengths for recruitment activities.
With regard to the second major assessment area, perceived learningand benefit gained from college, Pascarella and Terenzini (1991) claim thatthe breadth of change is the most striking phenomenon in how collegeaffects students. "Students not only make statistically significant gainsin factual knowledge and in a range of general cognitive and intellectualskills; they also change on a broad array of value, attitudinal, psychosocial,and moral dimensions" (p. 557).
Astin (1996) agrees that affective concerns as well as cognitiveoutcomes must be addressed in assessment studies. He advises thatassessments should address the importance of citizenship, character andsocial responsibility. Astin (1993) also identifies the following aspects ofpersonal development pertinent to assessment: cultural awareness, leader-ship, social activism, cultural/artistic interest, and developing a meaningfulphilosophy of life. Kuh (1993) broadens the domain for assessment toinclude the following five dimensions: personal competence, cognitivecomplexity, knowledge and academic skills, practical competence andaltruism and estheticism. Kuh (1995) also proposes that out-of-classexperiences, that provide opportunities to develop leadership abilities andinterpersonal skills, should be addressed in assessment.
The present study follows many of the recommendations from theliterature. The study is based on an annual senior survey. Analyses incor-porate comparative data from peer institutions. Substantively, the studygives priority to the students' voice and focuses on student satisfaction andperceived growth in various areas - academic, personal, social and moral.
METHODOLOGY
Evaluation modelThe statistical and evaluation model developed in this study involved thefollowing steps:
1. Obtain student questionnaire data from the primary and comparablepeer institutions.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
ASSESSING UNDERGRADUATE EDUCATION 2 5 9
2. Use Factor Analysis to develop scales.3. Obtain reliability indices for the scales.4. Revise scales until the reliability is satisfactory.5. Score scales by averaging responses for items included in their
respective scales.6. Classify the comparative institutions into meaningful types.7. Conduct Analyses of Variance to compare the primary and peer
institutions.8. Present results clearly in tables and graphs so policy makers can
understand them.9. Formulate meaningful recommendations based on the results.
Data sourceThe study is based on survey responses of 244 graduating seniors fromthe primary institution and approximately 1,500 students from a diversegroup of 39 peer colleges and universities in the United States. Basedon institutional type and the national ranking schema of US News andWorld Report, the peer institutions were classified in the following fivecategories: first tier colleges; first tier universities; first and second tiercolleges; second, third, and fourth tier colleges; and a diverse tier group.
Methods of analysisThe primary method of analysis employed in the research was Analysis ofVariance with planned comparisons. This technique allows for a statisticalcomparison of the mean for the primary institution with the means for theinstitutional peer groups. Analyses were conducted with individual surveyitems as well as with computed scale scores that reflect a respondent'saverage score on a set of items.
Development of scalesFactor Analysis was conducted to establish construct validation, that is, toidentify the unidimensional or multidimensional constructs underlying thesatisfaction and assessment related items in the questionnaire. CommonFactor Analysis - or the principal axis factor method - was employed.An oblique method of rotation, oblimin was used to rotate the Factors.This method was chosen since it assumes that the factors are correlated.Factors were selected that explained a substantial amount of variance andmost items were chosen with factor loadings of 0.4 or higher. Scaleswere created by combining similar items into one measure. Prior to usingthe scales in the analysis, alpha reliability coefficients were computedto determine the internal consistency of the scales. Reliability analysis
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
260 ANNE MARIE DELANEY
TABLEI
Description of the College Satisfaction Scales
Scales
Quality of course instruction
Academic and faculty support
Student services
Diversity
Statistical properties
Mean
3.15
3.17
3.09
3.00
Standard
deviation
0.43
0.42
0.33
0.67
Reliability
0.84
0.81
0.83
0.86
No. of
items
57
13
2
Range of
responses
low-high
1-5
1-5
1-5
1-5
confirmed the internal consistency of two sets of scales: four collegesatisfaction scales and four assessment scales.
College satisfaction scalesTable I presents the names and statistical properties of the four satisfactionscales. Survey items comprising the scales are presented in the Appendix.As reflected in the alpha coefficients, the reliability of the college satis-faction scales is very high, from 0.81 to 0.86. The quality of courseinstruction scale mean represents students' average rating for courses inthe humanities and the arts, science and maths, social science, engineeringand business. The academic and faculty support scale represents students'average rating on different aspects of their academic programme, includingadvising, availability and attitude of the faculty, availability of courses,internships and off-campus study. The student services scale representsstudents' satisfaction with food, recreation, housing, computer and finan-cial aid services. The diversity scale represents students' mean rating forthe colleges' ethnic/racial diversity and climate for minorities.
Assessment scalesThe four assessment scales summarise students' perception of the impactof their undergraduate education on various knowledge and abilitydomains. Table II presents the names and statistical properties for thesescales which represent the following dimensions: relationship skills, crit-ical thinking, social/moral awareness and understanding science. Thereliability of these scales ranges from moderate to high; alpha reliabilitycoefficients for three of the four scales exceed 0.80 and the reliability is
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
ASSESSING UNDERGRADUATE EDUCATION
TABLE II
Description of the Assessment Scales
261
Scales Statistical properties
Mean Standard Reliability No. of Range of
deviation items responses
low-high
Relationship skills 3.35 0.51
Critical thinking 3.33 0.53
Social/moral awareness 2.41 0.74
Understanding science 1.96 0.74 0.82 2 1-4
0.710.83
0.820.82
45
32
0.71 for the relationship skills scale. Items comprising these scales arepresented in the Appendix.
The relationship skills scale represents students' mean rating on theperceived impact of students' undergraduate education on their abilityto lead and supervise, relate well to different peoples, be an effectiveteam member, and communicate well orally. The critical thinking scalereflects students' perception of the extent to which their undergraduateeducation enhanced their abilities to acquire new skills and knowledge;think analytically and logically; formulate creative, original ideas; eval-uate alternatives; and plan and execute complex projects. The social/moralawareness scale represents students' perception of the effect of theirundergraduate education on their ability to develop awareness of socialproblems; put problems in perspective; and identify moral and ethicalissues. The scale on understanding science refers to the perceived impacton the ability to understand the process and evaluate the role of science.
RESULTS
Satisfaction with college experiences — item level analyses
This section presents student satisfaction results based on item levelanalyses. Table EQ documents mean levels of satisfaction with variousaspects of the academic programme, career preparation, and campus lifefor graduates of the primary institution and business major graduates atpeer institutions.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
262 ANNE MARIE DELANEY
TABLEm
Differences between Primary and Peer Institution Graduates in Mean Level of Satisfac-tion
College experiences Primary Peer institutions (a)
Quality of business instruction
Faculty availability
Faculty attitude
Extra-curricular activities
Internships
Academic advising
Personal counseling
Religious/spiritual life
Availability of courses
Campus social life
3.66
3.51
3.47
3.22
3.11
3.02
3.00
2.88
2.84
2.51
1
3.31**
3.47
3.37
3.19
3.00
2.70
2.75
2.97
3.00
2.81
2
3.51***
3.30***
3.29***
2.86***
2.83***
2.32***
2.65***
3.18***
2.98*
3.15***
3
3.53**
3.52
3.43
3.13
3.10
3.01
3.00
2.89
2.90
2.56
4
3.38***
3.48
3.50
3.20
2.94
3.02
3.02
2.98
3.08**
2.84***
5
3.45***
3.40
3.40
3.13
2.87***
2.88
2.95
3.07**
3.16***
2.86***
Notes: Means are based on a scale ranging from 1 'very dissatisfied' to 4 'very satisfied',(a) 1 First Tier Colleges; 2 First Tier Universities; 3 First and Second Tier Colleges; 4Second, Third, and Fourth Tier Colleges; 5 Diverse Tiers.
Satisfaction with the academic programmeWith regard to the academic programme, graduates of the primary insti-tution report a significantly higher level of satisfaction with the quality ofbusiness instruction compared with all peer institutional groups. They alsoreport very high levels of satisfaction with the faculty - specifically facultyavailability and faculty attitude. On these items, the primary institutionmeans are significantly higher than the means for first tier universities.On other aspects of the academic programme, graduates of the primaryinstitution express a moderate level of satisfaction with academic advisingand a somewhat lower level of satisfaction for availability of courses.
Satisfaction with other college experiencesRegarding career preparation, graduates of the primary institution reporta superior level of satisfaction with internships - significantly exceedingthe means for first tier universities and the diverse tier group. With respectto campus life, graduates of the primary institution are similar to most oftheir counterparts at the peer institutions in being 'generally satisfied' withpersonal counseling. In contrast, they are significantly less satisfied withcampus social life.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
ASSESSING UNDERGRADUATE EDUCATION 263
Quality of Business Instruction
jI
3.60
3.403-2°3.00
Institutions
'Primary Institution " " Peer Institutions |
3.20 nM 3.10 -" 3.00 -« 2.902 2.80 •
S 2.702.60
Availability of Courses
" - - * *
1 2 3Institutions
|~™~™*Primary Institution ™ "
4
Peer Institutions j
5
3.20 1u 3.10 -= 3.003 290BS 2.80c 2.70 ig 2.60-S 2^0-Ä 2.40-
2.30
Campus Social Life
Institutions
"Primary Institution" " Peer Institutions
Figure 1. Differences between Primary and Peer Institution Graduates in Mean Level of
Statisfaction on Selected Satisfaction Survey Items.
Figure 1 graphically illustrates the differences between the means forthe primary institution and the peer institutional group means on threesatisfaction items: quality of business instruction, availability of coursesand campus social life. As illustrated, on quality of business instruction,the primary institution's mean consistently surpasses the peer institutionalgroup means, but on availability of courses and campus social life, themean consistently falls below the peer group means.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
264 ANNE MARIE DELANEY
TABLEIV
Differences between Primary and Peer Institution Graduates on Satisfaction Scales
College experiences Primary Peer institutions (a)
1
Quality of course instruction 3.15 3.18 3.14 3.20 3.26 3.19Academic and faculty support 3.17 3.18 2.97*** 3.20 3.21 3.15Student services 3.09 2.89** 2.82*** 3.00** 2.74*** 2.98***Diversity 3.00 2.54*** 2.69*** 2.74*** 2.98 2.97
Notes: Group 3 includes primary institution data. **p < 0.01; ***p < 0.001.(a) 1 First Tier Colleges; 2 First Tier Universities; 3 First and Second Tier Colleges; 4Second, Third, and Fourth Tier Colleges; 5 Diverse Tiers.
Satisfaction with college experiences: scale level analyses
College satisfaction scalesTable IV presents mean scores for the primary institution and the institu-tional peer groups on four satisfaction scales: quality of course instruction,academic and faculty support, student services and diversity.
As shown, similar to business major graduates at the peer institutions,graduates of the primary institution report a moderate mean level of satis-faction of 3.15 on the quality of course instruction scale. Also similar tomost peer groups, but significantly higher than that reported by first tieruniversities, they report a moderate, mean level of satisfaction of 3.17on the academic and faculty support scale. This scale represents students'average rating on several aspects of their academic programme, includingadvising, availability and attitude of the faculty, availability of courses,independent study, internships and off-campus study.
On the student services scale, graduates of the primary institution reporta mean of 3.09, significantly surpassing the means for all peer institu-tional groups. This result confirms students' relatively superior level ofsatisfaction with the College's services including food, recreation, housing,computer and financial aid. Finally, the primary institution's mean of 3.00on the diversity scale surpasses the means for all the peer institutionalgroups and is significantly higher that the means for groups 1,2 and 3.
Figure 2 graphically illustrates the relationship between the primaryinstitution and peer institutional group means on two of the satisfactionscales. As shown, on student services, the primary institution mean consist-ently surpasses the means for all peer institutional groups. The pattern ismore variable with respect to the academic and faculty support scale. Asindicated by the intersecting lines, the primary institution mean is higher
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
ASSESSING UNDERGRADUATE EDUCATION 265
3.20
3.10-
= 2.90-
1 2-80 -
2.70-
2.60-
2.50
Student Services
s '
Institutions
•Primary Institution ™ • Peer Institutions [
3.30-]
3.20-
3-10 •
3.00
2.90
Academic and Faculty Support
Institutions
"Primary Institution ~ ™ Peer Institutions
Figure 2. Differences between Primary and Peer Institution Graduates on SelectedSatisfaction Scales.
than the means for groups 2 and 5, but slightly lower than the means forgroups 1, 3 and 4.
PERCEIVED EDUCATIONAL IMPACT: ITEM LEVEL ANALYSES
Senior Survey respondents evaluated their undergraduate education interms of its impact on the development of their knowledge and abilitiesin various domains. On a scale ranging from 1 'not at all' to 4 'greatly',
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
266 ANNE MARIE DELANEY
TABLEV
Differences between Primary and Peer Institution Graduates in the Perceived Effectof Undergraduate Education on Abilities and Knowledge
Abilities/knowledge
Use computers
Use quantitative tools
Write effectively
Appreciate arts
Learn Foreign Language
Primary
3.55
3.18
3.07
1.90
1.56
Peer institutions (a)
1
3.10***
3.11
3.21
2.68***
2.21***
2
3.33***
3.15
3.06
2.44***
2.02***
3
3.39**
3.14
3.14
2.25***
1.90***
4
3.08***
2.91***
3.12
2.70***
2.08***
5
3.35***
3.11
3.01
2.35***
2.00***
Notes: Group 3 includes primary institution data. **p < 0.01; ***p < 0.001.(a) 1 First Tier Colleges; 2 First Tier Universities; 3 First and Second Tier Colleges;4 Second, Third, and Fourth Tier Colleges; 5 Diverse Tiers.
they rated the extent to which their undergraduate programme enhancedsuch capacities as using computers, using quantitative tools and learninga foreign language. Table V presents mean ratings for graduates of theprimary institution and business major graduates at peer institutions onsurvey items related to specific abilities and knowledge.
As shown, the primary institution graduates report the highest meanrating for the impact of their undergraduate education on their ability to usecomputers; the mean rating of 3.55 is significantly higher than the meanratings for all peer institutions. Compared with business major graduatesat the peer institutions, primary institution graduates also report a superiormean rating on using quantitative tools; the mean rating of 3.18 exceedsthe means for all peer institution groups and is significantly higher than themean for group 4.
With regard to other abilities, business major graduates at all of theinstitutions generally report a modest impact of their undergraduate educa-tion on their writing ability. Graduates of the primary institution alsoreport a minimal impact of their undergraduate education on their ability toappreciate the arts and learn a new foreign language; further, the primaryinstitution means of 1.90 and 1.56 respectively are significantly lower thanall the peer group means.
Figure 3 graphically displays the differences between the primaryinstitution means and the peer institutional group means on three of theassessment survey items. As shown, the primary institution mean surpassesall peer group means on using computers, but falls below all the peer groupmeans on appreciating the arts and learning a foreign language.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
ASSESSING UNDERGRADUATE EDUCATION 267
3.60-,
a, 3.50
I 3M "« 3.30 •
8 3.20 -S 3.10
Use Computers
«•
1
- » "*»•• * *
V
2 3 4 5Institutions
"""•"Primary Institution ™ ~ Peer Institutions
Primary Institution ** ~ Peer Institutions
Figure 3. Differences between Primary and Peer Institution Graduates on SelectedAssessment Survey Items.
PERCEIVED IMPACT OF UNDERGRADUATE EDUCATION: SCALE
LEVEL ANALYSES
Table VI presents differences in the perceived effect of undergraduateeducation between primary institution graduates and business major gradu-ates at peer institutions in terms of the four assessment scales created forthis study. As shown, primary institution graduates report the highest mean
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
268 ANNE MARIE DELANEY
TABLE VI
Differences between Primary and Peer Institution Graduates in the Perceived Effectof Undergraduate Education on Assessment Scales
Abilities/knowledge Primary Peer institutions (a)
1
Relationship skills 3.35 2.90*** 3.12*** 3.18*** 3.12*** 3.13***
Critical thinking 3.33 3.24 3.19*** 3.23 3.16** 3.21**
Social/Moral Awareness 2.41 2.83*** 2.68*** 2.55* 2.75*** 2.66***
Understanding Science 1.96 2.27** 2.19*** 2.13** 2.35*** 2.24***
Notes: Group 3 includes primary institution data. *p < 0.05; **p < 0.01; ***p <0.001.(a) 1 First Tier Colleges; 2 First Tier Universities; 3 First and Second Tier Colleges;4 Second, Third, and Fourth Tier Colleges; 5 Diverse Tiers.
on the relationship skills scale, which represents the perceived impact onthe ability to lead and supervise, relate well to different peoples, be aneffective team member, and communicate well orally. The mean of 3.35significantly surpasses all peer group means.
As shown, on the critical thinking scale, primary institution graduatesalso report a higher mean rating compared with business graduates at allof the peer group institutions. The scale reflects students' perception of theextent to which their undergraduate education enhanced their abilities toacquire new skills and knowledge; think analytically and logically; formu-late creative, original ideas; evaluate alternatives; and plan and executecomplex projects. The primary institution mean of 3.33 surpasses all of thepeer group means and is significantly higher than the norm for group 2, thefirst tier universities, and groups 4 and 5.
The pattern reverses with the next two scales - social/moral awarenessand understanding science. On both scales the primary institution meanfalls considerably below the means reported by business major graduatesat the peer institutions. As indicated by a mean of 2.41, graduates of theprimary institution generally perceive that their education has had only'a little' effect on their social/moral awareness; this effect is significantlylower than that of business major graduates at the five peer group insti-tutions. Finally, primary institution graduates perceive even less effecton understanding science; the mean of 1.96, reflecting less than 'a little'effect, falls significantly below all peer group means.
Figure 4 graphically illustrates the differences between the primaryinstitution means and institutional peer group means on two of the assess-
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
ASSESSING UNDERGRADUATE EDUCATION 269
3.40-
3.30-
| 3.20 •
« 3.10 •c
1 3 0 °2.90-
Relationship Skills
1
- - - " * • - - -S - - - - - -
2 3 4 5
Institutions
Primary Institution - - Peer Institutions |
2.90 -i
2.80-
of 2.70 -
1 2.60-I 2.50 -
S 2.40 -
2.30-
2.20 -
Social/Moral Awareness
1 2 3 4
Institutions
|""™^Primary Institution ™ " Peer Institutions
* -
5
Figure 4. Differences between Primary and Peer Institution Graduates on Selected
Assessment Scales.
ment scales. As illustrated, the primary institution mean exceeds all peergroup means on the relationship skills scale, but falls below all peer groupmeans on the social/moral awareness scale.
STUDENTS' OVERALL EVALUATION OF THE COLLEGE EXPERIENCE
In this study, students' overall evaluation of their college experience isbased on their responses to the following questions: 'Overall, how satisfiedhave you been with your undergraduate education at this institution?' and
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
270 ANNE MAME DELANEY
TABLE VII
Differences between Primary and Peer Institution Graduates inOverall Level of Satisfaction
Primary institution
3.14
Peer institutions (a)
1 2 3
3.23 3.18 3.20
4
3.07
5
3.14
Notes: Group 3 includes primary institution data,(a) 1 First Tier Colleges; 2 First Tier Universities; 3 First andSecond Tier Colleges; 4 Second, Third, and Fourth Tier Colleges;5 Diverse Tiers.
3.30 nBS
e
I 3'2° "g 3.10
3.00 •
1 2 3Institutions
•"•^"Primary Institution • •
4 5
Peer Institutions J
Figure 5. Differences between Primary and Peer Institution Graduates in Overall Levelof Satisfaction.
'If you had the chance to relive your college experience, would you chooseto attend this same institution again?'
College satisfactionTable VU presents mean levels of overall satisfaction for the primary insti-tution graduates and business major graduates at peer institutions. Thesemeans are based on a response scale from 1 'very dissatisfied' to 4 'verysatisfied' with 3 indicating 'generally satisfied'. As shown, the mean of3.14 indicates that graduates of the primary institution report they aregenerally satisfied with their undergraduate education.
Figure 5 graphically illustrates that the primary institution mean levelof satisfaction is the same as that of group 5 and relatively similar to themeans for the other peer groups. Analysis of Variance revealed that thedifferences between the primary institution mean and the peer group meanswere not statistically significant.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
ASSESSING UNDERGRADUATE EDUCATION 271
TABLE VIII
Differences between Primary and Peer Institution Graduates inStudents' Re-evaluation of their College Choice
Primary institution
3.57
Peer institutions (a)
1 2 3
3.80 3.96*** 3.62
4
3.36
5
3.70
Notes: Group 3 includes primary institution data. ***p < 0.001.(a) 1 First Tier Colleges; 2 First Tier Universities; 3 First and SecondTier Colleges; 4 Second, Third, and Fourth Tier Colleges; 5 DiverseTiers.
= 3.70
I 3-50
a 3.30-S 3.10S 2.90-
2.703
Institutions
"Primary Institution" • Peer Institutions
Figure 6. Differences between Primary and Peer Institution Graduates in Students'Re-evaluation of their College Choice.
Re-evaluation of college choiceResponses to the question, 'If you had the chance to relive your collegeexperience, would you choose the same institution again?' provide anothermeasure of graduates' overall evaluation of their undergraduate experi-ence. Responses to this question are based on a scale ranging from 1'definitely not' to 5 'definitely would' with 3 indicating 'maybe' and 4'probably would'. Results are presented in Table VIII. While the primaryinstitution mean is somewhat lower than the mean for the four groups,the difference is statistically significant only for group 2, the first tieruniversities.
As reflected in Figure 6, the mean differences between the primaryinstitution and the peer institutions are the largest with groups 1 and 2- the first tier institutions. As might be expected, more students from firsttier colleges and universities indicate they would make the same choiceagain.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
272 ANNE MARIE DELANEY
STRATEGIC POLICY RECOMMENDATIONS FOR THE PRIMARY
INSTITUTION
Based on the research findings, the study concluded with a set of stra-tegic policy recommendations designed to further enhance the College'sstrengths and identify areas for improvement, particularly in the academicprogram and student life areas. Illustrative recommendations follow.
Strengths
1. Public relations and admission recruitment efforts should accentuatestudents' very high level of satisfaction with business courses and thefaculty. Stress that students' satisfaction is superior to that reported bybusiness major graduates at first tier colleges and universities.
2. In college publications and admission literature, promote the excellentopportunities for career preparation. Feature students' superior levelof satisfaction with internship opportunities.
3. Promote the undergraduate programme's success in preparing gradu-ates for the technological and quantitative challenges of a career inmanagement. Disseminate the finding that graduating seniors reportthe highest rating for the programme's effectiveness in developingstudents' technological and quantitative skills.
4. In marketing and recruitment strategies, emphasise the value of theundergraduate programme's focus on team-based learning. Publicisethe fact that, compared with top tier college graduates, graduatingseniors report the highest rating for the effectiveness of their educationin developing relationship skills which are crucial to career success.
5. In admission publications and recruitment efforts, feature the highquality of student services. In describing campus life, report thefinding that students' satisfaction with student services surpasses thatreported by students at the first tier colleges and universities.
Areas for improvement
6. Review divisional course offerings with particular emphasis on thesequence and scheduling of elective course offerings. In addition,conduct focus group discussions with students from each of the fourundergraduate years to learn more about students' interests and needs.
7. These results should be used as baseline data in future evaluation of thenew undergraduate curriculum. Such evaluation should investigate theextent to which student perception changes regarding the effectivenessof courses in the social and natural sciences and the humanities sincethe implementation of the new curriculum.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
ASSESSING UNDERGRADUATE EDUCATION 2 7 3
8. Continue efforts to enhance the quality of campus social life so thatthe level of student satisfaction will be similar to that found with theacademic programme. Conduct focus group research and extensiveinterviews to understand the issue of quality of social life.
Utilisation and dissemination of the findings
Dissemination of the results from this research involved the distributionof the complete report with an executive summary to members of thePresident's Cabinet, including the Undergraduate School Dean as wellas other Deans and Directors of selected departments, such as CollegeMarketing and Development. The undergraduate admission and collegemarketing departments utilised the results of the study to publicise theperceived strengths of the college: the high quality business education;faculty dedication to teaching and availability to students; superior tech-nological education; and training and exceptional internship opportunities.Results are also being used in the College's preparation for accreditation.Internally, Academic Affairs Administrators and the Undergraduate Deanused the study in a formative evaluation to identify strengths and areas inneed of improvement in the academic and student life programmes.
DISCUSSION
This study demonstrates the relevance and value of an evaluation modelusing a senior survey as a vehicle for focusing on the students' voice inassessment. The research also demonstrates the enhanced power derivedfrom using peer institutional data for comparative norms. Developingdifferent types of comparable institutions increases the power of theevaluation model. Such data provide a benchmark against which theprimary institution can measure its performance. Comparative findingsfrom this study are particularly meaningful since the peers share thesame academic major and are preparing for similar careers in the field ofbusiness.
This research also illustrates how students' voices can contribute effec-tively to assessment. By expressing different levels of satisfaction andperceived growth, students identify potential programme strengths andareas in need of improvement. For example, results from this study provideclear evidence of a high level of satisfaction with the quality of theacademic programme. Compared with all of the peer groups, the primaryinstitution students rated the College significantly higher on the quality
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
274 ANNE MARIE DELANEY
of business instruction. Further, compared with students at first tier univer-sities, students at the primary institution offered significantly higher ratingson several other aspects of the academic programme, including facultyavailability, faculty attitude and academic advising. In contrast, comparedwith three of the five peer groups, students at the primary institutionreported significantly lower ratings on campus social life, thus identifyingstudent social life as an area in need of improvement.
In terms of perceived growth, the data show that graduates of theprimary institution rated their educational programme significantly higheron the ability to use computers. This finding reflects the College'sextraordinary investment in technology and technological education andsuggests that the programme is effectively preparing graduates for thehigh level of technological competence required for success in business. Incontrast, the data reveal that graduates of the primary institution rated theireducational programme significantly lower in terms of perceived growthin the ability to appreciate the arts and learn a foreign language. The latterfindings provide crucial information to guide efforts that might enhancethe programme's effectiveness in developing broadly educated businessleaders, capable of appreciating culture and functioning effectively intoday's global economy.
This study illustrates the value for institutions and researchers ofdata sharing through consortia. The primary institution benefits fromthe substantive findings derived from comparative analysis with peergroup data. Researchers benefit from the opportunity to share differentmethodological approaches to analysis and assessment.
The most important conclusion is the demonstrated value of a system-atic evaluation model. The major components of this model include usingFactor Analysis and reliability analyses to develop reliable and meaningfulscales; obtaining comparable institutional data; classifying institutions intomeaningful types; and using Analysis of Variance to compare institutionson the computed scales. Results are clearly communicated with relevantgraphs and tables. Meaningful recommendations are formulated based onthe results. When possible, comparisons are made with previous years'results.
ACKNOWLEDGEMENT
The author gratefully acknowledges the reviewers and Dr Ronald L.Nuttall for their review and critique of this article.
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
ASSESSING UNDERGRADUATE EDUCATION 275
APPENDIX: QUESTIONNAIRE ITEMS COMPRISING THE SCALES
College Satisfaction ScalesQ13. How satisfied are you with each of the following services oraspects of the college?
Quality of Course InstructionHumanities and the ArtsScience and MathsSocial SciencesEngineeringBusiness
Academic and Faculty SupportAcademic AdvisingFaculty AvailabilityFaculty AttitudeAvailability of CoursesIndependent StudyInternshipsOff Campus Study
Student ServicesFood ServicesRecreation/AthleticsComputer ServicesStudent HousingLibrary ResourcesClassroom FacilitiesLaboratory FacilitiesStudent Health ServicesStudent Center/UnionFinancial Aid ServicesBusiness OfficeStudent Voice in PoliciesStudent Government
DiversityEthnic/Racial DiversityClimate for Minorities
(Response Scale: 1 'very dissatisfied' to 4 'very satisfied')
Assessment ScalesQ14. Please indicate the extent to which each capacity was enhanced by yourundergraduate experiences.
Relationship SkillsLead and SuperviseRelate Well to Different PeoplesBe an Effective Team MemberCommunicate Well Orally
Academic and Faculty SupportAcquire New Skills and KnowledgeThink Analytically and LogicallyFormulate Creative/Original IdeasEvaluate AlternativesPlan and Execute Complex Projects
Social/Moral AwarenessDevelop Awareness of Social ProblemsPut Problems in PerspectiveIdentify Moral and Ethical Issues
Understanding ScienceUnderstand the Process of ScienceEvaluate the Role of Science
(Response Scale: 1 'not at all' to 4 'greatly')
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
276 ANNE MARIE DELANEY
REFERENCES
Astin, A.W. (1993). What Matters in College, Liberal Education 79(4), 4-15.Astin, A.W. (1996). Involvement in Learning Revisited: Lessons We Have Learned,
Journal of College Student Development 37(2), 123-134.Banta, T.W. (1993). Making a Difference. San Francisco: Jossey-Bass.Ewell, P. (1983). Information on Student Outcomes: How to Get It and How to Use It.
Boulder, Colorado: NCHEMS.Ewell, P. (1988). Outcomes, Assessment, and Academic Improvement: In Search of Usable
Knowledge. In J.C. Smart (ed.), Higher Education: Handbook of Theory and Research,Vol. IV. New York: Agathon Press, 53-108.
Kuh, G.D. (1993). In Their Words: What Students Learn Outside the Classroom, AmericanEducational Research Journal 30(2), 277-304.
Kuh, G.D. (1995). The other Curriculum: Out-of-Class Experiences Associated withStudent Learning and Personal Development, Journal of Higher Education 66(2),123-155.
Lingrell, Scot A. (1992). Student Outcomes Assessment: The Senior Survey (Report No.HE 025 950). Bowling Green, OH: Bowling Green State University. (ERIC DocumentReproduction Service No. ED 351 897)
Pace, C.R. (1979). Measuring Outcomes of College: Fifty Years of Findings and Recom-mendations for the Future. San Francisco: Jossey-Bass.
Pascarella, E.T. & Terenzini, P.T. (1991). How College Affects Students. San Francisco:Jossey-Bass.
Schreiner, L. & Juillerat, S. (1997). National Student Satisfaction Report, 1997 (Report No.HE 031 453). Iowa City, IA: USA Group Noel-Levitz. (ERIC Document ReproductionService No. ED 421 931)
Trainer, J.F. (1996). To Share and Share Alike: The Basic Ground Rules for Inter-Institutional Data Sharing. In J.F. Volkwein (Series Ed.) & J.F. Trainer (Vol. Ed.),Inter-Institutional Data Exchange: When to Do It, What to Look for, and How to MakeIt Work, Vol. 89. San Francisco: Jossey-Bass, 5-13.
Babson CollegeAcademic AffairsBabson Park, MA 02457-0310USAE-mail: [email protected]
Dow
nloa
ded
by [
Col
orad
o C
olle
ge]
at 1
9:07
18
Nov
embe
r 20
14
