Embed Size (px)
Citation preview
DOCUMENT RESUME
ED 359 854 HE 025 748
AUTHOR Goldstein, BernardTITLE Student Outcomes Assessment in the California State
University. Report of the California State UniversityAdvisory Committee on Student Outcomes Assessment.
INSTITUTION California State Univ. and Colleges, Long Beach.Office of the Chancellor.
PUB DATE Dec 89NOTE 47p.; This document is part of a collection produced
under the auspices of the California State UniversityInstitute for Teaching and Learning. The CSU/ITL,created in 1988, facilitates a 20-campus systemwidenetwork of faculty affiliates in response to thedemand for improved teaching and learning in thecollege classroom. Cover title varies slightly.
AVAILABLE FROM CSU Academic Publications Program, 400 Golden Shore,Suite 132, Long Beach, CA 90802 ($9).
PUB TYPE Reports Evaluative/Feasibility (142)
EDRS PRICE MF01/PCO2 Plus Postage.DESCRIPTORS *College Outcomes Assessment; Educational Change;
*Educational Policy; Educational Trends; EnrollmentTrends; Government School Relationship; HigherEducation; Outcomes of Education; Policy Formation;School Community Relationship; SocioculturalPatterns; *State Universities
IDENTIFIERS California; California State University; *CaliforniaState University Inst for Teach Learn; *CollegeTeaching and Learning Collection
ABSTRACTThis publication presents the findings and
conclusions as well as policy recommendations for a California StateUniversity (CSU) policy on student outcomes assessment. The reportwas formulated based on 16 meetings to study and discuss assessmentissues, to review material from the campuses and to provide guidanceto CSU representatives serving on an assessment study group advisoryto the California Postsecondary Commission. Chapter 1, "CSUInvolvement with Student Outcomes Assessment" defines outcomesassessment, describes the undergraduate education reform movement,conferences, assessment projects and committee work on the issue, andstate government interest. Chapter 2, "Factors Contributing toInterest in Student Outcomes Assessment" offers statements by leadersillustrating concerns about assessment and description of actions bygovernmental and professional agencies. Chapter 3, "GuidingPrinciples for Assessing Student Outcomes in the CSU," describes thereasons for adopting the twelve principles that guided response toexternal agencies and proposal of recommendations. Chapter 4,"Recommendations for a California State University Assessment Policy"contains 15 recommendations that seek to balance various objectivesand responsibilities. Three appendixes contain current assessmentpractices at CSU, a summary of responses to a survey, and abibliography of over 50 items. (JB)
'FORMA STATE UNIVERSITY
"PERMISSION TO REPRODUCE THISMATERIAL HAS BEEN GRANTED BY
California StateUniversity
TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC).-
tir
U.S. DEPARTMENT OF EDUCATIONOffice of Educafionat Research and improvement
EDUCATION..L RESOURCES INFORMATIONCENTER {ERIC)
Tens document has been reproduced asrece,+ecl from the person or orgarfizafionobcnafin90
C ?Amor changes have been made to unpre,rereproduction quality
Potnts of view or OPntOns staled .n th.s document do not neCessartiy represent offtc.atOERI postfion or poltcy -01
A REPORT TO THE CHANCELLOR'
from theAdvisory Committee on Student Outcomes Assessment
BEST CO AYWN
DECEMBER 1989
CSU Institute for Teaching and Learning
and
ERIC Clearinghouse on Higher Education
The California State University Institute for Teaching and
Learning (CSU/ITL) facilitates a 20-campus network of teaching
and learning programs in the CSU system. ERIC/HE has entered
into an agreement with CSU/ITL to process documents produced
by the system and create a mini-collectionwithin the ERIC
database.
Major objectives of this initiative are as follows:
increase awareness of the work of the CSU Institute
for Teaching and Learning;
increase access to the work of CSU/ITL affiliates;
begin to build a subset of information on teaching and
learning that supports The National Teaching and
Learning Forum (NTLF), ERIC/HE's newsletter;
encourage use of the ERIC system by CSU/ITL member
affiliates and the NTLF readership; and
test a model for'. collaborationbetween ERIC/HE and a
major higher education system.
All CSU/ITL ERIC RIE citations are tagged with the following
identifiersappearing in the IDEN:Field:
College Teaching and Learning Collection; and
California State University for Teaching and Learning.
All CSU/ITL citations carry the following statement in the
Note Field:
This document is part of a collection produced under the
auspices of the California State University Institute for
Teaching and Learning. The CSU/ITL, created in 1988,
facilitates a 20-campus systemwide network of faculty
affiliates in response to the demand for improved teaching
and learning in the college classroom.
STUDENT OUTCOMES ASSESSMENT INTHE CALIFORNIA STATE UNIVERSITY
Report of the California State UniversityAdvisory Committee on Student Outcomes Assessment
Bernard GoldsteinChair
December 1989
Academic Affairs, Plans and ProgramsOffice of the ChancellorThe California State University40() Golden ShoreLong Beach, CAlifornia 90802
CONTENTS
INTRODUCTION: Committee Letter to the Chancellor
CHAPTER 1. CSU Involvement with Student OutcomesAssessment
CHAPTER 2.
CHAPTER 3.
CHAPTER 4.
APPENDICES:
A:
Factors Contributing to Interest inStudent Outcomes Assessment
Guiding Principles for Assessing StudentOutcomes in the CSU
Recommendations for a California StateUniversity Assessment Policy
v
1
5
10
14
Current Assessment Practices in the CSU 19
B: Summary of Campus Responses to Surveyon AB 2016 25
C: Bibliography 37
IJ
THE CALIFORNIA STATE UNIVERSITYIAKERSFIELD CHICO DOMINGUEZ HILLS FRESNO FULLERTON HAYWARD
SACRAMENTO SAN REILNARDL40 SAN DIEGO SAN FRANCISCO SAN JOSE
OFFICE OF THE CHANCELLOR
(213) 590-
HUMROLDT
SAN LUIS OBISPO
LONG PEACH
SAN MARCOS
LOS ANGELES NORIIIRLDGE POMONA
SONOMA STANLSIAUS
August 21, 1989
Dr. W. Ann ReynoldsChancellorThe California State University400 Golden ShoreLong Beach, California 90802
Dear Chancellor Reynolds:
Enclosed please find the final report of the California StateUniversity Advisory Committee on Student Outcomes Assessment.This report summarizes the Committee's findings and conclusionsand makes recommendations for a California State Universitypolicy on student outcomes assessment.
The Committee met sixteen times over a period of more than ayear to study and discuss assessment issues, to review materialreceived from the campuses and to provide guidance to CSUrepresentatives serving on an assessment study group advisoryto the California Postsecondary Commission.
Our approach has been to engage all campus constituencies inwide ranging discussions of ideas, concerns and points ofcontention related to outcomes assessment. Campus views weresolicited and carefully reviewed and synthesized by theCommittee. The experience and recommendations of facultyinvolved in experimental assessment programs and projects wereconsidered. The several tension points (e.g. workload issues,availability of resources, definitions and values ofassessment) were vigorously debated in a systemwide conferenceon assessment.
Preliminary drafts of guiding principles for assessment in theCSU were distributed to Campus Senates, the California StateStudent Association, the systemwide Academic Senate, thePresidents and the Vice Presidents. Draft recommendations werediscussed with campus faculty and administrators at regionalmeetings in northern and southern California. At each step ofour deliberations, we received and incorporated manysuggestions for improvements in the guidelines andrecommendations.
0
440 GOLDEN SHORE, LONG BEACH, CALIFORNIA 901102-4275 INFORMATION: (213) S110 -SS46 TELEFAX: (213) 590-5749
Chancellor W. Ann ReynoldsPage Two
The recommendations in this report reflect Committee members'consensus on the best approach to student outcomes assessmentfor The California State University. We believe they areconsistent with what has been found to be most effective andbeneficial in assessment programs implemented around thecountry. We are hopeful that they will provide useful guidanceto campuses and serve to encourage faculty to seize the agendaon assessment.
The Committee-wishes to express its appreciation to all on thecampuses, in the Office of the Chancellor, and outside of CSUwho have assisted in its work. We are particularly indebted tothe individual committee members for the time and talent theyhave committed to a task that has at times been exhilarating,and at times plodding. We have gained valuable insights intoassessment and stand ready to discuss our recommendations atyour request. We are particularly concerned that discussionsabout assessment on the campus continue. We hope thatconsultation on this report will promote such discussion.
Sincerely yours,
Dr. Tomas ArciniegaPresidentCalifornia State University,
Bakersfield
Dr. y White LoewyProfessor of PsychologySan Francisco State University
rank W. Youngiate Dean
Academic Affairs, PlansOffice of the Chancellor
. Margaret BluePast PresidentThe CSU Alumni AssociationProfessor, Political ScienceCalifornia State University.
Dominguez Hills
Dr Kenneth NybergPr fessor of Sociology
alifornia State University,Bakersfield
Dr. Mary lliams urger Mr. J RichardsonVice President for Academic Affairs The California State StudentCalifornia State University, Association
Sacramento Office of the Chancellor
,1/4,4 a/ar464/-G4 Q11.1.44301fDr. Sally Loyd Casanova Ms. Allison WeberDean. Academic Affairs, Plans Chair, The California StateOffice of the Chancellor Student Association
Humboldt State University
Dr. Bernie Goldstein. Committee ChairDirector, Research & DevelopmentSan Francisco State University
CHAPTER ONE
CSU INVOLVEMENT WITHSTUDENT OUTCOMES ASSESSMENT
In November 1987 Chancellor Reynoldsestablished an Advisory Committee on StudentOutcomes Assessment and charged its memberswith: studying student outcomes assessment andadvising the Chancellor on policies relatedthereto; coordinating responses to the CaliforniaPostsecondary Education Corn mission inconnection with its study of outcomesassessment; and submitting a report andrecommendations for directions the CSU shouldtake with regard to outcomes assessment.
This report seeks to:
1. Identify the external and internal factors thatcontribute to the interest in outcomesassessment;
2. Interpret the implications of emergingassessment programs for the CSU;
3. Summarize important actions and activitiesrelated to outcomes assessment that haveoccurred within the CSU at the system level;and
4. Recommend specific assessment policies,strategies and practices the Committeebelieves are appropriate and of significantpotential benefit to the CSU.
Definition of Student OutcomesAssessment
A major difficulty in discussing "studentoutcomes assessment" is the lack of a commondefinition. It has become, in the words of TerryW. Hartle (p. 4), "a catch-all phrase that refers toa wide range of efforts to improve educationalquality."For all of its vagueness, however, arapidly expanding inventory of institutionalpractices and critical studies clearly distinguishesstudent outcomes assessment from narrowerapplications of the term "assessment" used priorto the emergence of national discussion in 1985.
For purposes of this report, the AdvisoryCommittee has adopted the definition of studentoutcomes assessment formulated by Carol M.Boyer and Peter T. Ewell in their analysis ofassessment in undergraduate education undertakenfor the Education Commission of the States:
Any process of gathering concrete evidenceabout the impact and functioning ofundergraduate education. The term can applyto processes that provide information aboutindividual students, about curricula orprograms, about institutions or about entiresystems of institutions. The termencompasses a range of procedures, includingtesting, survey methods, performancemeasures or feedback to individual students,resulting in both quantitative and qualitativeinformation.(Carol M. Boyer and Peter T. Ewell. "State-Based Approaches to Assessment inUndergraduate Education: A Glossary andSelected References." Denver: EducationCommission of the States, March 1988.)
Student outcomes assessment differs fromconventional testing practices, program reviewand accreditation processes in an importantrespect. It seeks to employ what can bediscovered about actual current/former studentlearning, skills, attitudes, behaviors andopinions--if possible over time--in illuminatingand evaluating the effectiveness of the educationalprograms in helping students achieve the goalsand objectives for those programs. Outcomesassessment attempts to provide, whether at thelevel of the individual course, program orinstitution, information to answer the question:What do you expect of your students and how doyou know if they are meeting your expectations?
CSU and the Undergraduate EducationReform Movement
The CSU, in cooperation with the CaliforniaPostsecondary Education Commission, hostedthe western regional dissemination conference ofthe report on Involvement in Learning in May1985. In response to the call for reform andimprovement voiced at this conference and to thestream of studies critical of higher education, theCSU Academic Senate undertook a year-longstudy of undergraduate education in the CSU.That study concluded that the CSU should accordgreater priority to research on teaching andlearning and recommended establishment of asystemwide Teacher-Scholar Institute.
2
Through their recommendations onassessment, some national studies, includingInvolvement in Learning and Integrity in theCollege Curriculum, conferred instant celebrityon a small number of colleges and universitiesthat had developed programs to evaluate thequality of educational programs by looking atoutput rather than input measures. During thesummer of 1985, Chancellor's staff and AcademicSenate leaders agreed to investigate theassessment models that were attracting nationalattention.
The First CSU Assessment Conference
In consultation with campus administratorsand the Academic Senate, "ProgramDevelopment," with strong emphasis on theevaluation of outcomes, was adopted as acategory for Academic Program Improvementgrants in 1986-87. To inform the developmentof the grant proposals, and to stimulate debate onoutcomes assessment by affected campusconstituencies, a systemwide conference was heldin October 1986. Its primary purpose was toidentify the essential characteristics of emergingassessment theories and practices and the contextsout of which they grew. Faculty andadministrators associated with state assessmentmodels in Tennessee, New Jersey, Missouri, andFlorida discussed the reasons and processesleading to the adoption of these programs, therationales for the program designs, and the costsand effects of their implementation. Theconference proceedings were published in theNew Directions for Higher Education series byJossey-Bass under the title, Student OutcomesAssessment: What Institutions Stand to Gain(edited and introduced by Dr. Diane Halpem).
An informal survey of the campus teams atthe end of the academic year indicated that follow-up activities, ranging from formal reports to theestablishment of university assessmentcommittees, had occurred on almost every CSUcampus.
Experimental Assessment Projects
As of spring 1989, eleven outcomesassessment projects on ten CSU campuses havebeen supported through grants from the AcademicProgram Improvement Fund. The directors ofthe 1986-87 projects were selected to makepresentations at the Third National AssessmentConference in Chicago in the summer of 1988.With their appearance, a California presence wasestablished in what had been a movement
dominated largely by eastern institutions. CSUfaculty and administrators presented a paneldiscussion on the development of outcomesassessment policy and programs in the CSU atthe 1989 Conference of the AmericanAssociation of Higher Education. Facultyassociated with assessment activities on CSUcampuses were on the program of the FourthNational Assessment Conference in June 1989.
Interest in Sacramento
School reform legislation in 1 wake of thepublication of A Nation at P K (1983) hadfocused the attention of Califor lawmakers onthe conditions of education in the state, and onthe performance of its public institutions. Theestablishment, in 1984, of a blue ribboncommission to study the community colleges(SB 2064, Stiem) and to review the CaliforniaMaster Plan for Higher Education (SB 1570,Nielsen) signaled the intent to scrutinizepostsecondary education with equal thoroughness.
State legislators' keen interest in theimplications of the assessment movement foreducational improvement in California becameevidf-. in the spring of 1986. In March,Assemblyman Tom Hayden, Chairman of theSubcommittee on Higher Education, publishedBeyond the Master Plan: A New Vision forHigher Education in California, a call forCalifornia to adopt features of outcomesassessment programs developed in other states.Since 1986 Assemblyman Hayden has authoredseveral bills proposing the establishment ofmandatory outcomes assessment programs. In1987, Senator Marion Bergeson introducedlegislation to require standardized testing ofcandidates for teaching credentials.
The Advisory Committee on StudentOutcomes Assessment
By mid-1987, assessment had become amajor educational and public policy issue inCalifornia. Conferences on assessment weresponsored by the California Assembly and theCalifornia Community Colleges. In the CSU,the Academic Senate devoted a session of its fallretreat to the topic and appointed a former Senatechair as special advisor on assessment issues.The California State Student Associationagendized debates on outcomes assessment overthe course of academic year 1987-88 and passed aresolution supporting outcomes assessment inJanuary 1988. CSU faculty and administratorsserved on a CPEC advisory committee to make
recommendations regarding outcomes assessmentto the California Legislature. In 1987, theWestern Association of Schools and Collegesrevised its standards for accreditation to requireevidence of program effectiveness in institutionalself studies.
In order to monitor and interpret thesedevelopments, and to coordinate CSU responsesto them, Chancellor Reynolds appointed anAdvisory Committee on Student OutcomesAssessment. Its members includedrepresentatives from the Academic Senate (3), theCalifornia State Student Association (2), campusadministration (2), the alumni association (1).and Chancellor's Office staff (2). Since its initialmeeting in December 1988, the Committee hascompleted the tasks described below.
1. Survey of Campus Views on Assessment
Committee members reviewed the literatureon outcomes assessment, deliberated theissues and evidence available to them, andsought to clarify their implications for theCSU. Based on this study, and on responsesto a survey of campus opinion about therecommendations contained in Assembly Bill2016 (Coded Memorandum APPS 87-31), theCommittee drafted a set of guiding principles.Chapter three of this report sketches theconsiderations that led to their formulation.The principles were circulated widely amongfaculty and administrators for review andcomment. The statement of principles, thusevolved, served to guide the Committee inresponding to requests for advice from theCalifornia Postsecondary EducationCommission and in shaping recommendationsfor system policy on outcomes assessment.
2. Response to the Legislature
Supplementary language to the Budget Act of1987 required the CSU to report to theLegislature progress made "toward adoptionand implementation of comprehensiveoutcomes assessment mechanisms forevaluating student learning, programeffectiveness and institutionalaccomplishment of mission." TheCommittee provided guidance to Chancellor'sstaff regarding the content and direction of thatreport. Included in the response to BudgetaryLanguage was a summary of currentassessment practices in the CSU, included inthis report as Attachment A.
3
3. Advice to the California Postsecondaryq, Education Commission
The Committee prepared a report to CPECsummarizing campus responses to spec ificprovisions of AB 2016, and urging heCommission to adopt for California theprinciples recommended by the Committee.The summary of campus input on AB 2016 isincluded as Attachment B. The CSUAdvisory Committee was active in reviewingand commenting on CPEC drafts during theentire period of the CPEC study.
4. Consultation with CSU Constituencies
Committee members recognized that thesophistication about assessment acquired byfaculty and administrators participating inassessment projects and conferences was notwidely shared by CSU faculty. This was seenas a major obstacle to the practical usefulnessof any positive recommendations theCommittee might make. To probe thereceptivity toward outcomes assessment withfaculty and administrators whoseunderstanding and acceptance would bedeterminative in implementingrecommendations to develop assessmentprograms, the Committee assisted in planningand conducting the Second CSU Conferenceon Assessment (November 1988 at LakeArrowhead).
The conference revealed that the number ofCSU faculty and administratorsknowledgeable about assessment andsupportive of many of its purposes was largerthan the Committee had anticipated. It alsoconfirmed strong negative reaction toassessment on the part of some faculty. Theprincipal objections expressed were: demandsto develop and implement outcomesassessment programs would imposeunreasonable additional burdens on faculty;information produced through assessmentprograms would be of little value to faculty;and data could be misused in ways detrimentalto individuals, programs or campuses.
In keeping with the view that theCommittee's recommendations should begrounded in the realities of institutional life onCSU campuses and build upon widely sharedvalues, the Committee sponsored regionalmeetings to enable anyone in the CSU who feltstrongly about assessment or about theCommittee's recommendations to be heard.
4
Members of the Committee met in April 1989,w.qh students, faculty and administrators on theHumboldt, Sacramento and Fullerton campusesto discuss the draft recommendations. Thesediscussions helped to determine the form,content, and recommendations of this report.
CHAPTER TWO
FACTORS CONTRIBUTING TO INTEREST INSTUDENT OUTCOMES ASSESSMENT
A variety of factors have contributed to thegrowth of interest in assessment. The followingstatements by leaders outside and inside theacademy illustrate the major concerns behind thecall for alternative means to evaluate theeffectiveness of educational programs andinstitutions. The Committee interprets thesestatements to be a recognition of changingcircumstances and evolving priorities, rather thancriticisms of the ccntributions or dedication ofindividual faculty members.
Integrity of the Baccalaureate Degree
By failing to articulate the shared objectivesof a liberal education or to discuss how they arerelated to individual courses in the curriculum,faculties can easily lose sight of their commonpurposes. In such an atmosphere, importantaims acknowledged by all, such as the ability tocommunicate precisely or to reason carefully,remain the responsibility of no one.(Derek Bok. Higher Learning, 1986)
The major in most colleges is little more thana gathering of courses taken in one department,lacking structure and depth . . . or emphasizingcontent to the neglect of the essential style ofinquiry on which the content is based. . . .
Another victim of this posture of irresponsibilityis the general education of the American collegeundergraduate, the institutional courserequirements outside the major. They lack arationale and cohesion. . . It is as if no onecared, so long as the store stays open.(Arnold B. Arons, Ernest L. Boyer, et al., Projecton Redefining the Meaning and Purpose ofBaccalaureate Degrees. Integrity in the CollegeCurriculum, 1985)
Many factors have contributed to the tendencyof colleges and universities to place lessemphasis on explicit, common educationalcharacteristics for the baccalaureate degree. Theyinclude the enormous expansion in the number ofinstitutions and students in American highereducation in the postwar period, growingdiversity of the student population, rapid shifts infederal and state funding policies and priorities,cultural changes that impact the academicreadiness of college bound students and the valuessupporting their commitment to higher
education, increasing suspicion of authority,institutional adjustments to declining enrollmentin the seventies, and the evolution of disciplinaryresearch.
These influences have exerted competingpressures on th; loyalties and priorities of collegeand university faculty. The institutionalprocesses that worked to assure integrity of thedegree--curriculum approval, personnel review,evaluation of student work-are less effectivewhere faculty hold diverse understandings of thegoals and values of tne institution and itsprograms and of how to relate them to a verydiverse student population with differing goalsand values.
Improved Means to Evaluate Teaching andin
We must overcome the lazy habit of gradingand scoring "on the curve" as a cheap way ofsetting and upholding standards. Such a practiceis unrelated to any agreed-upon intellectualstandards and can reveal only where students standin relation to one another. It tells us nothingabout where they ought to be. Moreover,students are left with only a letter or number --with nothing to learn from.(Grant Wiggins, Senior Associate, NationalCenter on Education and the Economy. "A TrueTest: Toward More Authentic and EquitableAssessment" in Phi Kappa Deltan, May 1989)
(The development and implementation ofstudent outcomes assessment] has made facultymembers teaching in the Interdisciplinary GeneralEducational Program more aware of theirinterdependence. . . . The exercise of publiclydiscussing and putting on paper the outcomesexpected of specific courses and the pedagogicalstrategies that will produce them has nurturedcooperation. . . . [As a result of the assessmentactivities,] the end of the quarter is no longerperfunctory; it is an important time to completeconnections that have been building over theterm. . . . The outcome of autonomous learninghas been reinforced through student out omenassessment. Experience with assessment hasempowered students to do this.(Andrew I. Moss, Professor of English,California State Polytechnic University,
6
Pomona. Third National Conference onAssessment in Higher Education, June 1988)
Research in assessment has led to theemergence of promising new approaches andtechniques for analyzing and measuring studentlearning and other developmental changesassociated with higher education. As a result,there are better and better ways for faculty toevaluate the effectiveness of learning' and teachingand there are better ways for educationalinstitutions and accrediting agencies to analyzethe quality and performance of programs.Thoughtfully developed outcomes assessmentprograms have demonstrated significant potentialfor improving communication between facultyand students, enhancing learning and helpingfaculty and staff reach consensus on changes toimprove the quality and performance ofprograms.
Global Competitiveness
There has never been a time in American life,when legislators have believed as firmly thathigher education is a central vehicle for dealingwith a number of problems critical to the futureof the region, state, nation. They want equityand excellence. This is an unprecedentedchallenge. No other country has attempted to dothis.(Frank Newman, President, EducationCommission of the States. Third NationalConference on Assessment in Higher Education,June 1988)
Everyone thinks we in America are numberone. . . The process of falling behind is onlydiscovered after it's happened, not while it'shappening and we're resting on our laurels.(Assemblyman Tom Hayden, dialogue with JohnAshcroft moderated by Frank Newman. AAHENational Conference, March 1987)
State governments fear profound erosion inthe quality of life as production moves offshore,imports gain larger shares of the domesticmarket, and foreign capital exerts greaterinfluence over the national economy. They arelooking to colleges and universities to equipgraduates with the understandings and skills tomeet the technological and managerial challengesconnected with the transition to a globaleconomy.
Declining Educational Achievement
We are being asked to do a better job withmore students who are less well prepared and whowill be called upon to do more on the job. . . .
We have more students who are more dependenton the performance of the institution.(Bob McCabe, President, Miami-Dade College.Third National Conference on Assessment inHigher Education, Chicago, June 1988)
The perception has become commonplace thatdiplomas and degrees from American schools andcolleges no longer guarantee possession of basicliteracy and numeracy. Critical reports frombusiness, industry and the military echo news ofdeclining scores on several indices of educationalachievement: e.g., college aptitude tests,professional and graduate school admission tests,teacher credential tests, and internationalcomparisons of abilities in math, science andgeography. At the elementary and secondarylevels, this has led to the imposition ofmandatory testing programs. In the absence ofmore appropriate methods, some states haveimposed similarly draconian measures oncolleges and universities.
Demog.aphic Changes in American Society
It is not true that access causes a decline inthe quality of higher education. However, accesswithout quality is a cruel deception, while qualitywithout access is a betrayal of the cherishedAmerican ideal of equal opportunity. . . . Thenation simply cannot afford to sacrifice the nextgeneration of emerging Americans in the name ofquality enhancement. (George Deukmejian,Michael Dukakis, John Ashcroft, et al., TaskForce on College Quality. Time For Results:The Governors' 1991 Report on Education, 1986)
States are worried about the emergence of atwo-tiered society in which a large underclass,defined along racial or ethnic lines, is unpreparedfor constructive participation in a technological,information-based economy. Maintenance of astable economic and political environmentrequires that all sq., .ents of society share in itsbenefits. Formal education is the principalgateway to such participation.
Changing Views of Educational Quality
The chief challenge facing higher education isnot just the need to improve public understandingof the academy. The academy must want toperform better in substantive matters in order to
serve the public interest more effectively. Thiscan best be undertaken by building bridgesbetween our colleges and universities and theenormous problems that face American societyand the world.(Gary H. Quehl, President, Council forAdvancement and Support of Education. "HigherEducation and the Public Interest" in Academe,September-October 1988)
The capacity for higher education to be apositive change agent in American society willdepend upon our ability to transcend ourinstitutional egos, our narcissism, and our self-interest, and to concern ourselves more directlywith the impact we are having on our studentsand communities.(Alexander Astin, Director, Higher EducationResearch Institute, UCLA Graduate School ofEducation. Third National Conference onAssessment in Higher Education, June 1988)
As state governments increasingly recognizethe close links between the university and theeconomi nd social well being of theirstates/regi they look at the performance ofpublic universities in a changing light.Criticism of the "reputational" or "resource"notions of educational excellence, as distinctfrom the "outcomes" or "talent development"model, has found a very positive reception frompublic policy makers. The call to measure thecontributions of an institution by how much andhow well its students are learning and how thecollege experience is affecting their values andattitudes, speaks directly to the concerns of stategovernments.
Rising Costs
We need not just more money for highereducation, we need more education for themoney.(John Ashcroft, Governor of Missouri,Chairman, Task Force on College Quality,National Governors' Association. Time ForResults: The Governors' 1991 Report onEducation, 1986)
In recent years the cost of a college degree hasrisen at a rate higher than the index of inflation.Nationally, this is occurring at a time ofdwindling federal resources. In California, theGann Limit has capped state spending in the faceof increased demand for services. The effects ofthis policy will be exacerbated by theimplementation of Proposition 98. Stategovernment must determine what share of fixed
7
resources shall be allocated to meet diverse socialneeds. Elected officials are forced to makepainful choices and they want clear,comprehensible evidence of institutionaleffectiveness upon which to base their decisions.
Accountability
Each campus should assess whether thereceipt of a baccalaureate degree signifies theacquisition of a core of knowledge, along withthe development of abilities to use thatknowledge effectively. . . . A more rigorousundergraduate program will require high schools,middle schools, and elementary schools to do abetter job of preparing students to performcollege level work.(George Deukmejian, Michael Dukakis, JohnAshcroft, et al., Task Force on College Quality.Time for Results: The Governors' 1991 Reporton Education, 1986)
Without constant attempts to redefine andreassert publicly their nature and purpose,universities become frozen in internalmythology, in a complacent self-perpetuation... .When they are not challenged within themselvesto justify themselves, to themselves as well as tothe society they serve; when they are not . . .
constantly urged to examine theirpresuppositions, their processes and acts, theystiffen up and lose their evolvingcomplementarity to other American institutions.(A. Bartlett Giamatti, former President of YaleUniversity. "A Free and Ordered Space: TheReal World of the University" as excerpted in theOpinion section of The Chronicle of HigherEducation, 9 November 1988.)
What's relevant is that an institution that hasas its primary mission research, scholarship,investigation, and inquiry knows so little aboutitself . . . Higher education is a black box.You go in, and come out the other side. Youdon't know what happened in it. . . . What we'retrying to communicate in California is anurgency. . . . We have a breakdown of the K-12system. . . . We have a crisis in minorityenrollment. . . . Just because we're [talkingabout assessment] doesn't mean that we want toimpose something; we want to invite theleadership of academic institutions to help createthe future leadership of the country.(Assemblyman Tom Hayden, dialogue with JohnAshcroft moderated by Frank Newman. AAHENational Conference, March 1987)
8
We have a responsibility to communicate tothe public how well we're doing, regardless ofwhether we are mandated to do so. . . . It's clearfrom our behavior that we think we know whatstudents should know. Its not credible to declarethat we do not. We cannot face the publicschools and say: "We produce 70% of "ourteachers, but we can't certify their competeire."If we are to hold up our end of the commitmentto improve public education, we must pressforward with student outcomes assessmentregardless of how primitive these efforts may be.We cannot take the position that we're so perfectthat we can't improve our own teaching.(Vice Chancellor Lee R. Kerschner. 1988 CSUSystemwide Conference on Student OutcomesAssessment, November 1988)
Elected representatives and governing boardshave the right and responsibility to receivetimely, accurate and useful information about theperformance and effectiveness of institutionssupported through taxation. Traditional modes ofcommunicating such information do not reportthe outcomes of student enrollment at collegesand universities in terms related to the goals andobjectives of the respective institution'seducational programs. Where this does occur, itis usually unsystematic and anecdotal.
Actions of Governmer.tal andProfessional Agencies
At the Federal Level
Government response to these pressuresidentified above has been a vigorous call forreform of undergraduate education and a muchexpanded role for evaluation. The following tenstates have adopted mandatory outcomesassessment programs at the postsecondary level:Colorado, Florida, Georgia, Kansas, Missouri,New Jersey, South Dakota, Tennessee, Texas,and Virginia. Other states, including California,have asked coordinating boards to studyassessment and advise them about the desirabilityof imposing similar requirements. At least onestate system, the State University of New York,is moving toward implementation of assessmentprograms for instructional improvement in theabsence of a state mandate.
Most states, including California, haveinstituted mandatory testing for prospectiveteachers. Efforts to develop a national teachersexam are underway. The National Governors'Association has made strong recommendationsregarding state government's need to fix
accountability for educational quality. TheEducation Commission of the States andvirtually every national professional organizationin higher education have devoted majorconferences to assessment.
The U.S. Department of Education hasresponded to these pressures by revising the"Secretary's Procedures and Criteria forRecognition of Accrediting Agencies"--guidelinesaffecting the distribution of federal funds topostsecondary institutions--to require studentoutcomes assessment. Section 602.17 containsthe following new guideline for determiningwhether accrediting agencies are making asatisfactory effort to evaluate the educationaleffectiveness of postsecondary institutions orprograms:
Determining that institutions or programsdocument the educational achievements oftheir students . .. in verifiable and consistentways, such as evaluation of senior theses,reviews of student portfolios, generaleducational assessments (e.g., standardized testresults, graduate or professional school testresults, or graduate or professional schoolplacements), job placement rates, licensingexaminations results, employer evaluations,and other recognized measures;
Determining the extent to whichinstitutions or programs broadly andaccurately publicize . . . the [educationalobjectives consistent with its mission], theassessment measures described above, theinformation obtained through those measures;
Determining the extent to which institutionsor programs systematically apply theinformation obtained through the measuresdescribed above . . . toward steps to fosterenhanced student achievement ....
At the Regional Level
In its January 1988 Handbook ofAccreditation, the Western Association ofSchools and Colleges incorporates revisionsrequiring the collection and reporting ofoutcomes assessment information.
Standard 2.0 requires each institution todemonstrate that it has "developed the meansfor evaluating how well, and in what ways, itis accomplishing its purposes as the basis forbroad-based, continuous planning andevaluation." Included among the suggested
procedures and treasures used to evaluateinstructional programs are: changes instudents' academic achievement; peerevaluation of educational programs; structuredinterviews with students and graduates;changes in students' values; pre- and post-testing of students; surveys of students,graduates and employers; student scores onstandardized or locally constructedexaminations; performance of graduates asindicated by measures appropriate to the fieldof the major.
Standard 4.A requires universities to specifyclearly for each field of study the: "subjectmatter to be covered, the intellectual skillsand learning methods to be acquired, and theaffective and creative capabilities to bedeveloped." They must also demonstrate that"efforts are undertaken to develop andimplement ways to measure the educationaleffectiveness" of academic programs.
At the State Level -- The Legislature
The California Legislature has, to date,enacted four pieces of legislation directly relatedto student outcomes assessment:
Assembly Concurrent Resolution 141(Hayden 1986) required the CaliforniaPostsecondary Education Commission(CPEC) to develop, in consultation withrepresentatives of the public postsecondarysystems, recommendations regarding "talentdevelopment, value-added, and performance-based budgeting approaches to measuring andimproving the quality of education inCalifornia."
The Budget Act of 1987 incorporated languageof intent requiring The California StateUniversity to report progress made towardadoption and implementation of"comprehensive assessment mechanisms forevaluating student learning, programeffectiveness and institutionalaccomplishment of mission."
Assembly Bill 2016 (Hayden 1987) directedCPEC to develop and present options for"measuring and implementing talentdevelopment or value added approaches tohigher education and for an incentive fundingapproach designed to develop appropriatemethods of assessing the teaching andlearning process."
9
Senate Bill 148 (Bergeson 1987) directs theSuperintendent of Public Instruction and theCommission on Teacher Credentialing toadopt new standards anti requirements forearning a teaching credential. Central to thenew credentialing process is a shift fromprogram approval to comprehensiveassessments of individual candidates as thebasis for granting credentials.
In its final report, California Faces . . .
California's Future: Education for Citizenshipin a Multicultural Democracy (March 1989), theJoint Committee for Review of the Master Planfor Higher Education recommends the fundingand initiation of three kinds of assessmetprograms:
An intersegmental assessment project,developed under the aegis of CPEC, aimed atincreasing: the numbers of currentlyunderrepresented minority students, theretention rates of all postsecondary students,and the number of women and minorityfaculty in regular appointments.
A comprehensive "Student Tracking System,"under CPEC direction, to collect data from allCalifornia postsecondary students.
Campus-based student outcomes assessmentprograms, developed by the faculty of eachpublic college and university, to understandthe "cognitive and substantive development ofstudents."
The California Postsecondary EducationCommission
The California Postsecondary EducationCommission prepared two assessment-relatedstudies in response to Legislative requests.Funding Excellence in California HigherEducation (March 1987) clarifies terminology,outlines several model programs, and formulatessix guiding principles which should guide statepolicy development in assessment. The secondstudy, Beyond Assessment: Enhancing theLearning and Development of California'sChanging Student Population (approvedDecember 1988), describes current assessmentpractices in California, discusses value-added/talent-development assessment methods,and recommends that the Legislature establish aCalifornia challenge grant program "to supportinitiatives for improving teaching and learning inhigher education, including the development ofinstitutional assessment plans.'
10
CHAPTER THREE
GUIDING PRINCIPLES FOR ASSESSINGSTUDENT OUTCOMES IN THE CSU
This chapter describes the reasons for adoptionof twelve principles that guidtd the Committee inresponding to external agencies and proposingrecommendations for CSU policy on outcomesassessment
Assessment and Diversity: For WhatPurposes Should Assessment Be Done?
Many of the pressures for assessment arise froma concern that the baccalaureate degree has lost anycommon meaning. In attempting to address thisconcern, universities run the risk of creating adifferent problem: they may damage that academicdiversity that makes individual faculty andinstitutions unique and causes knowledge toadvance. The balance between common standardsand beneficial academic diversity is delicate andeasily distorted.
There is indeed evidence to support the concernabout lack of common meaning in the bachelor'sdegree and in all its components: generaleducation, basic skills, the major. Thesecriticisms, often carefully researched, have beenbrought to the attention of the academy byrespected members of its ranks in responsible andconstructive forms. The rigorous 'if- examinationcalled for in such analyses as Involvement inLearning and Integrity in the College Curriculumcannot be ignored. It would be ironic, however, ifthe California State University were to respond tothese legitimate demands by adopting the strategiesof other states which have sought to remedy basicskills deficiencies by imposing large-scaleprograms and examinations that standardized thecontent and sequence of instruction at the cost ofthat academic diversity that other nations have longenvied and sought to emulate. For one thing, TheCalifornia State University has alreadyimplemented reforms in writing, mathematics, andgeneral education. For another, the arguments foracademic diversity are valid.
The values and traditions represented in thecommunity of America's 3,000 postsecondaryinstitutions reflect the pluralism of Americansociety. The faculties of these institutions aregeographically dispersed, free from the centralizinginfluence of a natic. ministry, responsive to themanifold interests and needs of the communities inwhich they are located, stimulated by association
with colleagues in independent disciplinarysocieties, and governed by boards representative of abroad range of constituencies. Thesecircumstances, and a widely shared belief in thevalue of academic freedom, have enabled Americanfaculty to pursue and transmit knowledge as eachindividual sees fit.
Each faculty member brings to the classroom aunique collection of diverse knowledge, views, andinterests. Courses of the same name may resembleeach other only in some respects, depending on theinstructor. Students completing a major in aspecific field will possess some 'enowledge incommon with other students completing thatmajor; there will assuredly be much knowledgethey do not have in common, however, owing totheir experience with a different faculty. Thisintellectual and programmatic diversity is theparticular strength of American higher education.And it is this diversity that faculty believe isthreatened by centrally developed and administratedassessment programs.
Measurement of students' performance againstspecific criteria or norms tends to standardize andhomogenize student learning. If institutions arcrated, punished or rewarded on the basis of specificindicators of student learning, they will makewhatever adjustments are necessary to meet theperformance expectations measured by thoseindicators. This may be all to the good when thegoal is to increase student competency inmathematics or writing, but it is enormouslydestructive of the innovation that c^curs inacademic disciplines when individual faculty striveto question or expand the frontiers of theirdisciplines and push their students to do likewise.
Assessment programs can be designed to provideevidence of students' progress toward meeting theeducational goals of programs and institutionswhile preserving and nurturing academic diversity.The growing literature on assessment providesoutstanding examples of programs incorporatingthoughtful approaches and yielding informationwhich can improve teaching and learning. Suchapproaches are consistent with the ethos of theacademy, where the process of collecting evidenceand reflecting On its meaning is a habit of mind anda principal strategy of academicians.
Because of its concern that inappropriate modelsbe avoided but that the benefits of assessment beavailable to CSU campuses, the AdvisoryCommittee adopted the following principles:
The only legitimate purpose ofassessing student outcomes is toimprove teaching, learning andacademic advising at the individual,course, program, and/or institutionallevel. Data from outcomes assessmentprograms will not be used for cross-campus rankings or comparisons ofindividual faculty.
Unique assessment -models, tied to amultiplicity of goals represented bythe different institutions andincorporating the principles adopted bythe Committee, are appropriate to theCSU.
Assessment and Complexity: WhatShould Be Assessed?
The most important outcomes of highereducation are difficult to assess in reliable andaffordable ways. Just as teaching at its mostinspired is as much art as science, drawing uponand stimulating the creative as well as analyticresources of the mind, so can the universityexperience as a whole be infinitely more than thesum of its separate courses and requirements.Among the most compelling arguments for thevalue of higher education is its potential toinculcate disciplined curiosity, tolerance, ethicalcommitment and self-esteem, qualities that do notlend themselves to affordable assessment.
Valid assessment using standardized instrumentsis possible in some areas. Basic skills in writing,computation and reasoning can be (and are)evaluated appropriately and economically usingstandardized tests.
In the major field, tests of students' knowledgefail to identify strengths and weaknesses in suchimportant dimensions as reactivity, enthusiasm,adaptability, and perseverance, and are inadequate asindicators of how well institutions are preparingstudents to continue study or begin their careers.To obtain reasonably complete and valid measuresof progress toward the goals of the major, a varietyof outcomes have to be assessed through a varietyof modes. Measures of student achievement in theabsence of contextual information are of little usefor improving teaching and learning.
I I
Although concern about students' generalintellectual development has motivated severalstates to require its assessment, there are nogenerally accepted methods to measure iteffectively. Evidence suggests that programs orinstruments designed to measure comprehensiveknowledge or intellectual growth apart from aparticular curriculum in fact produce results moreindicative of students' natural abilities orsocioeconomic backgrounds.
In view of the complex nature of the mostimportant outcomes of higher education and of thestrong influence of the variables that contribute toit, the Advisory Committee adopted the followingprinciples:
Meaningful outcomes assessment mustbe multivariate if it is to provide validinformation to campuses for their usein improving academic programs andmodifying institutional practices andfor evaluating their effectiveness.Standardized tests provide specific, butlimited kinds of information.
Campuses need to consider, as part ofan assessment program, studentcharacteristics and academic programvariables that affect student learning.Where these variables can bemonitored using systemwide databases,applicable data should be provided tothe academic departments.
A full student outcomes assessmentprogram will take into considerationsuch factors as: academic advising,counseling and career planning,laboratories, libraries, housing,financial aid services, extracurricularactivities, health services, campussocial life, and the quality andquantity of student contact withfaculty.
Responsibility for Assessment: WhoShould Assess?
Student outcomes assssment programs arebased on two major premises: 1) Changes thatoccur during students' formal education areattributable in some part to the institution(s) theyattend, Lind in particular to the academic and supportprograms with which they are associated. 2) It ispossible to obtain global measures of importantoutcomes--e.g., cognitive development, skillsacquisition, attitudinal changes, values
12
clarification--and to link them to institutionalfactors. Aggregate measures of what studentsactually know, believe, and do, provide informationfor analyzing the effects of programs and planningfor changes to improve them.
To be useful, assessment programs requirepersonal and institutional responsiveness to theinformation generated. If the evidence producedthrough an assessment program is perceived to beperipheral to the interests and efforts of the faculty,the students or the campus administration, it willnot command their respect or attention. In somemandated assessment programs, employment ofstandardized tests selected and developed by personsoutside the institution has led to changes that wereunintended and contrary to the broader concernsunderlying the programs' adoption. The motives ofoutside agencies were perceived as anti-intellectual.Stimulation of campus dialogue about institutionalexcellence and improvement and the means to attainit were not reported as outcomes of this approach.
To contribute significantly to the quality ofeducational programs in the CSU, outcomesassessment programs must be designed to measurethose educational dimensions identified by thefaculty, the students and the administration as mostimportant. The university is a collection of richcultures, each of which must be served by anoutcomes assessment policy.
Persuaded by examples of both beneficial andinjurious assessment plans, and mindful of the needto place any new tools for improving educationalprograms in the hands of those responsible forthem, the Advisory Committee adopted thefollowing principles:
Programs to assess student outcomesshould be campus-based, faculty-centered, and student-responsive.
Faculty of the individual campuseshave the primary responsibility fordeciding how to assess studentlearning. This extends to the designor selection and administration ofassessment methods, the interpretationof the results, and how the data willbe used to improve programs.
Consistent with the principle ofinstitutional responsibility, theresources appropriated for assessmentshould support the development andoperation of programs at the campuslevel. System and State efforts should
be directed to helping campuses deviseassessment programs. For this reason,the CSU opposes creation of acentrally administered Stateassessment program.
Data collected through institutionalassessment programs should begoverned by recognized codes of ethicstreating research with human subjects.
The Priority of Assessment: At WhatCost Should Assessment Be Done?
Higher education in America serves a multitudeof social and personal purposes. Publicuniversities in particular are seen as the vehicle foraccomplishing an enormous range of servicescritical to the well-being of the community, region,state and nation. These expectations translate intocompeting demands upon postsecondaryinstitutions and upon their primary resource, thefaculty.
CSU faculty are expected to perform a variety oftasks. Chief among these is their obligation toteach effectively. The tasks associated witheffective teaching are multiplying as discoveriesresulting from research on teaching and learning areapplied in the classroom. Examples of thesenecessary but time-consuming activities to improveinstruction include: added time for academicadvising; one-to-one student contacts in recognitionof the benefits of direct interaction in improvingthe achievement and persistence of students; and theintegration of technologies into the discipline as ameans of promoting student learning. Becausethere is evidence of program improvement whenfaculty spend time in evaluating programs withwhich they are associated, the CSU now requiresfaculty to devote time to this end. The use ofassessment to promote learning and assist inevaluating program effectiveness is another, recentresult of research on effective practices.
Each of these important instructional tasksrequires time investments on the part of facultywho are conscientious about being effectiveteachers. It is unrealistic to expect faculty toassume these additional responsibilities - -often aspioneers on behalf of their colleagues--withoutsome relief from other obligations. In other words,engagement in outcomes assessment represents acost in terms of faculty workload.
There are also opportunity costs at theinstitutional level. For example, time andresources spent on assessment might prevent
investigation of other variables associated withinstitutional excellence. Class size representsanother dimension of the cost-benefit equation.Some of the most effective approaches toassessment (oral presentations, portfolios, fieldexperiences, written essays) require small class sizeas a means of improving the quality and quantity ofstudent interaction with faculty. The decision topursue any of these priorities must be based upon acareful evaluation of the costs and the foregoneopportunities to pursue other means of improvinginstitutional effectiveness. To assure considerationof these factors, the Advisory Committee adoptedthe following principles:
- Student outcomes assessment, whenappropriately carried out, is just oneof several institutional practices thatmust exist in order to achieveeducational excellence. Studentoutcomes assessment should be linkedwith the academic program reviewprocess presently Inandated by theCSU Board of Trustees.
While the evaluation of studentlearning is a regular facultyresponsibility, implementation ofcomprehensive assessment programswill add significantly to facultyworkload and will be costly. Thesecosts must not be borne directly bystudents. Supplemental funding isessential to the development andoperation of effective assessmentprograms. In the absence of adequatesupport, program implementation mustbe limited.
Before substantial resources arerequested for, or invested in,comprehensive assessment programs,it must be established that theyprovide effective means to improve thequality of educational programs.Because of their high cost and the needto evaluate their effectiveness,assessment programs should beimplemented experimentally andincrementally within the CSU. Itshould be noted that assessment hasthe potential of identifying problemsin educational programs that requireadditional resources for solution.
The "assessment movement," as it came intobeing in the mid-eighties, is the offspring ofparents from very different cultures: one, native to
13
the academy, concerned primarily with being thebest possible alma mater; the other, from beyondthe walls, worried that the baccalaureates theacademy sends forth are inadequately prepared tomeet the challenges they must surmount for thegood of all. In search of accountability, the latterdiscovered the former and recognized immediatelytheir common interest in quality.
Given their disparate histories, customs andlanguages, it is not surprising that the associates ofeach view the new alliance with suspicion. Therelationship is tense. Mutual commitment toquality has kept them together; disputes over howto define and measure it often divides them. Toachieve their respective goals, each must understandand respect the other's motives and work diligentlyto help the other comprehend the complexity andimplications of actions taken or contemplated.
There are inherent antagonisms between thenotion of simple indices of performance and thegoals of higher education. To attain the one, theother must be sacrificed. That is not to say,however, that educational quality cannot or shouldnot be measured, or that its goals must becompromised in order to communicate them. Thenatural links between assessment and the values ofthe academy need to be reaffirmed in a more publiccontext and internalized within the academy. Thevalues of diversity and complexity need to bereasserted and effectively communicated beyond theacademy.
After lengthy study and about thebenefits and risks of student outcomes assessment,the Committee concluded that: 1) studentoutcomes assessment programs, of the kindrecommended in this report, have significantpotential for improving teaching andlearning; 2) the CSU cannot afford to ignoreeducational practices and strategies, includingstudent outcomes assessment, that show greatpromise for the improvement of teaching andlearning; and 3) where they have been demonstratedto be effective, it is the responsibility of CSUfaculty and administrators to adopt them asappropriate to the classroom, the discipline, and thecampus.
14
CHAPTER FOUR
RECOMMENDATIONS FOR A CALIFORNIA STATE UNIVERSITYASSESSMENT POLICY
The following recommendations seek to balancethe need for competent assessment with the need topreserve diversity, complexity, and facultyresponsibility for the quality of academic programs.The recommendations are organized around fourmajor goals: improving teaching and learning,improving assessment, improving communicationwith students, and obtaining support forassessment. They recommend integrativeassessment practices at the level of the individualstudent or faculty member, program or department,campus and system.
Assessment for Improving Teaching andLearning
At the Level of the Individual Student or FacultyMember:
1. Faculty should design evaluations ofstudent performance in their coursesto include elements that assessstudents' achievement in terms of theacademic goals of departments andprograms.
Tests given in classes typically measure howwell students have met specific course objectives.They are used, often in combination withevaluations of ether dimensions of studentperformance, for purposes of assigning a final gradein a course. The results of tests and otherassignments provide information essential foranalyzing the effectiveness of instruction.' Ifevaluations are devised to include them, results canalso indicate how well students are acquiring themastery of content, skills and attitudes expected ofprospective graduates in the particular field ofstudy. By carefully embedding measurements ofspecific programmatic outcomes in appropriatecourses, faculty can evaluate student's progresstoward attainment of the broader goals of thediscipline or program without burdening students orthemselves with additional assessmentrequirements.
At the Level of the Department or Program:
2. The faculty of each department orprogram should have ways ofevaluating student attainment in the
major that go beyond the evidenceprovided by course grades.
Although the CSU professes knowledge of thediscipline to be important, current practices do notgenerally assess to what extent students acquire it.Students often complain of a bewildering mosaic ofdemands and of the lack of opportunity to discoverthe patterns which lend coherence and meaning tothem. The relationship of courses to majorprograms and to general curricular goals and suchlearned abilities as effective written and oralcommunication and critical thinking remainsunclear and unarticulated. One way of assessingsuch attainment is to require, in individual majors,a curricular component which calls upon studentsto integrate general and specialized learning and todemonstrate comprehensive understanding of thefield of study appropriate to the degree level. Thiscomponent could be designed to help studentsapproach their academic experience from a unifyingand creative perspective and to permit faculty toobserve and monitor the effectiveness ofdepartmental programs.
In assessing comprehensive knowledge of thediscipline, faculty may wish to consider: oralpresentations requiring synthesis and integration;senior projects requiring research, scholarship orcreative activities appropriate to the discipline;portfolio development; integrative studies and fieldexperiences; simulations and case studies requiringattention to the ethical, historical and philosophicalfoundations of the disciplines.
At the Campus Level:
3. The faculty of each CSU campusshould have mechanisms to assesshow well students are meeting thegoals of the General Educationprogram of the university.
The General Education program is central to thequality of all CSU undergraduate degree programs.Responsibility for realizing its educational goals,however, is divided among many different academicconstituencies, including the CaliforniaCommunity Colleges. For these reasons, there is aneed for mechanisms to assess students' progresstoward attaining General Education objectives asdefined in Executive Order 338 and in the General
Education Transfer Curriculum. Informationobtained from these assessments should besystematically utilized in periodic reviews ofcampus General Education programs and inevaluating the preparation of transfer students whohave completed CSU General Educationrequirements in whole or in part at otherinstitutions.
4. Faculty, students and academicadministrators should work togetherto develop a campus plan forcoordinating and supporting outcomesassessment activities which examinethe interaction between academicprograms, student services, and thecampus environment.
In California Faces . . . California's Future, theJoint Committee for Review of the Master Plan forHigher Education calls for "campus-based studentoutcomes assessment intended to understand boththe cognitive and substantive development ofstudents, as well as their opinions concerning theireducational experience" and suggests that suchprograms include "a ;vide range of issues (quality ofinstruction, campus housing, effectiveness ofstudent services)." Accreditation standards of theWestern Association of Schools and Colleges nowrequire each postsecondary institution to "measurethe educational effectiveness of programs" as onemeans of demonstrating that it has developedprocedures for "evaluating how well, and in whatways, it is accomplishing its purposes." Studentoutcomes assessment programs can meet theseexpectations if they are developed around the goalsof each campus and with the participation of theconstituencies whose contributions are to beevaluated.
At the System Level:
5. The CSU should seek to restrict theproliferation of new, standardizedtests, encouraging instead thedevelopment and use of approaches toassessment that address programmaticneeds and curricular goals.
One major consequence of educational reform inCalifornia has been the adoption of an increasingnumber of tests as prerequisites for students tocomplete and progress beyond specific educationallevels. While each new testing requirement wasconceived and implemented to address importantconcerns, the cumulative effect is to burdenstudents with a series of isolated, often duplicativetesting requirements, each of which harbors the
15
potential for driving curricular development andinstructional practices in divergent directions. TheCSU should work toward the adoption ofintegrative assessment programs which reduce thenumber of tests demanded of students while makingthe results of each useful for multiple purposes.Such an approach is particularly desirable in theassessment of entry-level skills for purposes ofacademic advisement and placement, and indetermining the readiness of students to enterpostbaccalaureate professional programs.
Research for Improving Assessment
At the Level of the Individual Student or FacultyMember:
6. Each CSU faculty member shouldreview current student evaluationpractices for possible improvement inlight of evolving research on teachingand learning.
Teaching is the primary mission of TheCalifornia State University, and assessment isintegral to the teaching and learning process.Research on cognition, motivation and performanceassessment is growing and becoming moresophisticated. This literature is important to theCSU in meeting its primary mission of providingquality instruction for an increasingly diversestudent population. CSU faculty would benefit byhaving access to the findings of assessmentresearch, thereby enhancing their understanding ofthe uses and limitations of various modes ofinstruction and evaluation. Better, not necessarilymore, assessment should be the goal.
At the Level of the Department or Program:
7. CSU campuses should encouragesome faculty in each academicdepartment to engage in assessmentresearch related to teaching andlearning.
Most departments do not have, among theirfaculty, experts in evaluation and assessment. Onemeans for departments to acquire such residentexpertise is to encourage faculty to pursue scholarlyactivities related to instruction. Engagement instudent outcomes assessment is consistent withboth the teaching and the research responsibilitiesof CSU faculty. It represents the application of theprinciples of scholarly inquiry to the teaching-learning process and to other environmental factorsthat directly or indirectly contribute to theeffectiveness of educational programs. While the
16
object of such research may represent a newdirection for many faculty, the rationale andmethodology for pursuing such study are groundedin and supportive of the values and traditions of thedisciplines.
At the Campus Level:
8. Academic and research administratorsshould actively assist faculty tosecure resources to engage in thedevelopment of, and research on,assessment related to teaching andlearning.
Progress toward the development and adoptionof assessment approaches depends in part upon theavailability of supplementary resources to fume.professional and curricular development activities.Prior to the receipt of specific funding for thispurpose, departments should consider utilizingresources that may be temporarily targeted towardthe creation of assessment programs. Possiblesources of support for assessment activities includelottery and professional development funds,systemwide grant programs, state and federal grantprograms, and assigned time.
At the System Level:
9. The Office of the Chancellor shouldassist campuses in establishing newoutcomes assessment programs and inimproving current assessmentpractices through the disseminationof information about assessmentmethodology and research.
The process of building consensus on policiesand practices for measuring program effectivene sincludes providing access to information andexpertise in its interpretation. The Office of theChancellor should support campus efforts to acquireand exchange information about assessmentthrough systemwide programs. The Institute forTeaching and Learning is the logical sponsor for avariety of assessment related activities, including:systemwide and/or regional (tele)conferences,workshops, research projects. Other programs thatmay be appropriate include: Academic ProgramImprovement, the Lottery Revenue Program, andthe Teacher-Scholar Summer Institute.
Assessment for Improving CommunicationWith and About Students
At the Level of the Individual Student or FacultyMember
10. Students should receive the results ofassessments of their performance in atimely fashion, with adequateinformation to permit accurateinterpretation.
Students are entitled to know and understand theresults of any measurement of their academicperformance. The results of examinations and testsneed to be interpreted to students to guide theirfuture efforts. The pertinence of particular tests tothe broader goals of courses or programs, and ofperformance on these measurements as an indicatorof progress toward meeting the expectations of theprogram, must also be made clear.
The CSU has a very diverse student population.There is a need for faculty to recognize that the waythey convey evaluative information to students canbe conducive to learning or it can be very harmfulto learning, depending on individual studentcharacteristics. Faculty should discuss test resultsin class and advise individual students regardingtheir progress, with awareness of and sensitivity tothe complex relationships between testing andpersonal variables, and the impact of thisinteraction on individual students.
At the Level of the Department or Program:
11. Each academic department shouldutilize information about how wellstudents are meeting overarchingprogram goals to a) advise students atkey points in the major, and b)analyze and improve the effectivenessof academic programs.
The fragmentary character of students'educational experience is often mirrored by faculty'sfragmentary understanding of students' educationaldevelopment. Departments and programs shouldensure that faculty evaluations of students'performance yield information to assess the extentto which students are making acceptable academicprogress. The availability of such evidence willenable faculty to discharge, more effectively, theirindividual responsibilities as academic advisors andtheir collective responsibility for recommendingstudents for degrees. This information will also beuseful in understanding the effectiveness of program
components and in evaluating changes introduced toimprove them.
At the Campus Level:
12. The administration of each CSUcampus should assist academicdepartments in a) collecting,analyzing and reporting informationabout current and former students'characteristics, development andattainment of degree and programgoals, b) better utilizing datacurrently collected by the campus,and c) incorporating these outcomemeasures in academic program review.
Assessment of student learning in the major andGeneral Education, and of other programs andservices to support students' progress toward degreecompletion, can produce new information foranalyzing and improving institutional performance.Its value is limited, however, if not accompaniedby an understanding of student characteristics andopinions and how these relate to generalinstitutional patterns, particularly over time.Campuses regularly collect demographic andinstitutional data that could help provide such acontext, but often do not make it available informats useful for evaluating the impact and qualityof specific programs. Campus offices involved ingathering and utilizing relevant data, includingalumni associations, should work with academicdepartments and student services to identify andprovide important information currently notavailable for this purpose.
At the System Level:
13. The Office of the Chancellor shouldassist campuses in acquiring data in aformat useful to departments in theirself-analyses.
Survey infor-- cation is essential to understandingthe impact of educational institutions on the livesof students who attend them. The usefulness ofsuch information is enormously increased whendata are scientifically and systematically collectedover time and with attention to demographiccharacteristics. Presently, surveys of current andformer students are undertaken at various levels forvarious purposes and with varying degrees ofsophistication. At the department level, opinionsurveys are typically conducted in connection withself-studies for program review and accreditation.Campus alumni associations regularly seekinformation about former students for the specific
17
purposes of the associations. Two systemwidesurveys are periodically done to provide aggregateinformation about CSU students.
In the interest of greater efficiency and ofmaking high quality data available to all campusesand departments, the Office of the Chancellorshould work with the campuses to develop andadminister surveys to produce data to meet theneeds identified in these recommendations.Custody of data so generated should be theresponsibility of the unit of analysis; e.g., datapertaining to departments are returned to therespective departments, campus data to therespective campus. Use of the data by persons oroffices external to the unit of analysis should be atthe discretion of the unit. Where such use isapproved, anonymity of the data should besafeguarded.
System Support for Assessment
14. Developing and implementingcampus-based student outcomesassessment programs of theintegrative nature proposed in theserecommendations are costly. TheOffice of the Chancellor should workwith CSU campuses, the AcademicSenate, and educational constituenciesoutside the CSU to acquire adequateresources for this purpose above andbeyond the funding for currentprograms.
The successful acquisition of resources forassessment requires a prior consensus thatassessment programs will serve importanteducational goals. The process of buildingconsensus on policies and practices for measuringprogram effectiveness demands committedleadership, access to information and expertise inits interpretation, and the ability to release facultytemporarily from regular workload obligations todevote adequate time and effort to this purpose.Implementation of programs will require additionalfiscal support for faculty development activities,planning and coordination, instrument design andadministration, analysis and publication.
Successful operation of comprehensiveinstitutional assessment programs is ultimatelydependent upon reliable, predictable fund. ,g of thekind provided through an augmentation of thebudget for instruction. A major purpose of the"Challenge Grants," as proposed in the CPECstudy, Beyond Assessment: Enhancing theLearning and Development of California's Diverse
18
Students, is to provide multiple-year funding tocampuses for establishing student outcomesassessment programs. The Office of the Chancellorshould seek legislative adoption of therecommendations of this CPEC study as one sourceof fiscal support for developing and implementingassessment programs on CSU campuses.
Adoption Of Assessment PolicyFramework
15. The Board of Trustees of TheCalifornia State University should:a) adopt the Statement of GuidingPrinciples for CSU Policy on StudentOutcomes Assessment as theframework for developing outcomesassessment programs in the CSU; b)adopt the recommendations containedin this report; c) recognize that theadoption of these recommendationswill require additional resources andsupport the office of the Chancellorin building a level of consensusregarding the priority of assessmentadequate to pursue successfully theacquisition of budgetary resources toimplement these recommendations.
APPENDIX A
CURRENT ASSESSMENT PRACTICES INTHE CALIFORNIA STATE UNIVERSITY
EXCERPT FROM A REPORT TO THE LEGISLATURE ON CALIFORNIA STATEUNIVERSITY PROGRESS IN DEVELOPING AND IMPLEMENTING
COMPREHENSIVE STUDENT OUTCOMES ASSESSMENT IN RESPONSE TOSUPPLEMENTAL LANGUAGE TO THE 1987-88 BUDGET ACT
Appendix APage 20
Report of the Advisory Committee onStudent Outcomes Assessment
CURRENT ASSESSMENT PRACTICES INTHE CALIFORNIA STATE UNIVERSITY
The following inventory of established CSU programs is limited to systemwide assessmentprograms and practices. It does not include a discussion of traditional faculty gradingpractices, the most basic and important means for ;valuating students' educational progressand attainment, or of assessment programs and activizies existing or being piloted on thenineteen campuses.
Evaluation of Student Performance and Development
The English Placement Test (EPT)
In May 1976, the Board of Trustees, recognizing the need for greater attention to theproblem of student writing skills, approved the establishment of a systemwide writingproficiency and diagnostic examination for entering lower-division students. The policyauthorized support for developmental writing programs to remediate the writingcompetency of underprepared students. It also fixed responsibility for oversight andperiodic evaluation and reporting.
Since its first administration in September 1977, the English Placement Test has been takenby approximately 350,000 students. Its effectiveness in meeting policy goals was thesubject of a major evaluation undertaken with the aid of external consultants duringacademic year 1985/86. After reviewing information and data connected with this complexprogram, the EPT Evaluation Committee found that:
The EPT is a valid instrument for the measurement of the writing skills of enteringstudents and for placement in appropriate composition courses;
Study data suggest a positive relationship between success in writing skills andpersistence at the University;
The clean,: identification of student needs and abilities made evident through the testingprogram has, on various campuses, led to curricular revisions, given direction to facultyresearch interests and prompted the establishment of supplemental assessment programs.(Report of the English Placement Test Evaluation Committee, Office of the Chancellor, TheCalifornia State University, December 1986)
The Graduation Writing Assessment Requirement (GWAR)
The Writing Skills policy approved by the Board of Trustees in 1976 included a provisionthat all students demonstrate writing competency as a requirement for graduation and as aprerequisite to classified standing in graduate programs. The Chancellor's AdvisoryCommittee on Student Writing Skills, the task group charged with implementing the newpolicy, recommended against development of a common systemwide examination along thelines of the Placement Test. They were persuaded that the purpose of the requirementwould be best served by an approach enabling campuses to tailor programs to suit localsituations and particular needs of the disciplinary majors.
An evaluation of the implementation of the Graduation Writing Assessment Requirement oneach CSU campus through academic year 1986-87 was completed in October 1988. Whileackowledging and identifying the need to continue efforts to improve it, the evaluator
Report of the Advisory Committee onStudent Outcomes Assessment
Appendix APage 2,
concluded that "the GWAR is substantive, fair, rigorous, and appropriate as an upper-division measure of writing proficiency."
The Entry-Level Mathematics Examination (ELM)
To implement provisions of the new General Education-Breadth Requirements adopted bythe Trustees in 1980, the chairs of the mathematics departments of the CSU campuses,with the support of campus administrations and the Academic Senate, recommended that asystemwide policy on entry-level mathematics skills be adopted and that a uniformexamination be developed and given on all campuses to evaluate student entry-level skills.
The ELM must be taken and passed by all CSU students prior to enrollment in a course thatsatisfies the general education requirement in quantitative reasoning. Data collected sincethe exam was introduced in 1983 indicate that its use has contributed to improving the levelof high school preparation for college math and, as a result of greater readiness and betterplacement, student performance in CSU math classes as well.
Assessment of students' writing and computational skills, as currently exemplified in theEnglish Placement Test and the Elementary-Level Mathematics Examination, has enabledCSU campuses to do a better job of:
determining the readiness of entering students to do college level work andcommunicating information on the effectiveness of preparatory programs to high schools;
placing students in classes appropriate to their individual level of preparation;maintaining the quality of curriculum by establishing minimum criteria for baccalaureate-
level courses.
Comprehensive Examinations for Licensure, Certification, or AdvancedStudy
Graduates of many CSU degree programs enter fields requiring completion ofcomprehensive examinations for licensure, certification or registration. Nursing,architecture, marriage and family counseling, and dietetics are examples of areas ofemployment to which access is controlled by state boards. CSU academic departmentspreparing students for entry into careers requiring licensure are keenly interested in theperformance of their graduates on state examinations. Where resources and policy permit,this information is collected and reported in program review and accreditation studies.
CSU graduates applying for admission to graduate degree programs at other universitiesare often selected on the basis of their performance on comprehensive examinations ofgeneral knowledge and specialized knowledge in their field. Since graduate schoolacceptance rates are an important indicator of how well local programs are preparingstudents, many academic departments attempt to track their graduates' progress and includesummaries of the information in self-studies.
Systemwide Surveys of Current and Former Students
Since 1975 The California State University has conducted a biennial employment survey ofall spring graduates from all CSU campuses. The purpose of the survey is to providefaculty, counselors, students and prospective students with information on the employmentof recent CSU bachelor's and master's degree recipients relevant to career and life
Appendix APage 22
Report of the Advisory Committee onStudent Outcomes Assessment
planning. The data collected in the Survey of Spring Graduates provide campuses withanswers to such questions as:
What do CSU students do after graduation?What are rates of employment for women, minorities and older graduates who seek
employment?Do CSU graduates get the jobs for which their major programs prepare them?
What are starting salaries for CSU graduates?How do CSU graduates find jobs?
Whereas the biennial spring employment survey attempts to document the status of studentsthat have successfully completed academic programs in the CSU, the Student Needs andPriorities Survey (SNAPS) seeks information from currently enrolled students regarding:
Life goals and educational priorities;Levels of satisfaction with various aspects of their academic and social experiences
on campus;Obstacles or problems, whether institutional or personal in nature, which might
hinder progress toward their educational goals.
Data collected through SNAPS allow campuses to compare opinions and characteristics ofstudents from their own campus over time and with those of CSU students in general.
Evaluation of Program Quality
Program Review
The formal requirement for a qualitative review of existing academic programs has been inplace in The California State University since 1971, when the Board of Trustees directedthat each campus review each academic degree program on a periodic basis. In 1985, theGeneral Education Advisory Committee, noting that the program review schedules did notincorporate a review of the effectiveness of the General Education program, askedcampuses to devise separate, additional procedures to judge the quality of this componentof the curriculum.
While the requirement to review all degree and general education programs is systemwidepolicy, criteria and procedures for conducting the review are unique to the respectivecampuses. Generally, program review begins with a departmental self-study (usuallyinvolving surveys of students, faculty and alumni) which is submitted to a School Dean orto a School review committee. An external reviewer or team of reviewers including expertsin the discipline is typically invited to the campus to analyze the self-study, inte iewstudents, faculty and administrators, and to offer observations on program strengths andweaknesses.
Program review examines numerous aspects of the environment that are associated withand necessary to learning, and critical indicators of quality such as departmental gradingpractices, coherence of discipline coverage in the curriculum, and faculty involvement inscholarly and community affairs.
Institutional Ace' alitation
Report of the Advisory Committee onStudent Outcomes Assessment
Appendix APage 23
All campuses of The California State University have been accredited by the AccreditingCommission for Senior Colleges and Universities of the Western Association of Schoolsand Colleges. Each institution undergoes a full review every tenth year and submits fifth-year reports midway in the cycle. In order to qualify for accreditation, campuses must meetnine standards.
The accreditation standards have been developed by the Commission to understand theunique aspects of an institution and its relative success both in achieving individuality andin meeting regional and national expectations. The nine complex and essential dimensionsof an institution of higher education addressed in the standards encompass: institutionalintegrity, purpose, governance and administration, educational programs, faculty and staff,learning resources, student services and activities, physical resources and financialresources.
Program Accreditation
Currently, over two hundred CSU degree programs are accredited by some thirtyspecialized accrediting bodies. Each program accreditation involves site visits andaccreditation standards in addition to the instiosional accreditation reviews described above.
The specialized accrediting associations, all themselves accredited by the Council onPostsecondary Accreditation, are supported by many professional organizations or groupsof such organizations. Included among these are: the American Assembly of CollegiateSchools of Business, the Accreditation Board for Engineering and Technology, theAmerican Medical Association, the National League for Nursing, the American Speech-Language-Hearing Association, the National Council for Accreditation of TeacherEducation.
APPENDIX B
CSU CAMPUS VIEWS REGARDINGSTATE INCENTIVE FUNDING PROGRAMS
A SUMMARY OF CAMPUS RESPONSESTO A REQUEST FOR CONSULTATION ON
ASSEMBLY BILL 2016 (HAYDEN)"HIGHER EDUCATION TALENT DEVELOPMENT"
State of California Trustees of The California State University
Memorandum
To: Members, CPEC Advisory Committee Date: January 22, 1988on State Incentive Funding Approaches
From: Frank W. Young, Associate DeanAcademic Affairs, Plans
Subject: POLICY ISSUES AND PROBLEMS RELATED TO STATE INCENTIVEFUNDING: REPORT OF CONSULTATION WITH ACADEMICLEADERS IN THE CSU
As agreed in October, CSU members of the Advisory Committee haveundertaken to consult broadly with faculty and administrators from thecampuses regarding State Incentive R -ding Approaches for PromotingQuality in California Higher Education. Efforts to engage affectedconstituencies in discussions directly connected with the work of theAdvisory Committee included the following:
Chancellor Reynolds established a systemwide advisory committee onstudent outcomes assessment to examine assessment issues and makepolicy recommendations. Members of this committee include a campuspresident and academic vice president, three representatives of theAcademic Senate, two representatives of the California State StudentAssociation, and two members of the Chancellor's staff.
The Academic Senate of the CSU discussed outcomes assessment at itsannual systemwide conference in November and requested subsequentinput from campus senates.
Vice Chancellor Kerschner requested campus commentary on questionsposed by the State Incentive Funding study. (A copy of Dr. Kerschner'smemorandum is attached.)
The California State Student Association addressed talent developmentand outcomes assessment in meetings of the Board of Directors over thelast six months. In addition, CSSA has sponsored two student forumsfocussing on these issues.
cc: Dr. Lee R. KerschnerMembers, CSU Advisory Committee on Students Outcomes Assessment
Report of the Advisory Committee onStudent Outcomes Assessment
Appendix BPage 27
An information presentation on assessment issues was made to the Board ofTrustees of The California State University at its January meeting.
The CSU Advisory Committee on Student Outcomes Assessment convened onJanuary 13 to review the responses of campuses to the question posed in ViceChancellor Kerschner's memorandum and to provide guidance in the preparation ofthis report.
The campus responses confronted the Committee with an enormous wealth anddiversity of perspectives and views regarding educational quality and how it mightbest be promoted through a State incentive funding plan. The recommendationslisted below represent a homogenization of a substantial range of these views. Lostas a result of this process is a sense for the vitality and persuasiveness of the originalformulations and with this much of the information value as well. A small numberof campuses declined to reply to the request at this time, explaining that consultativeprocesses could not be concluded within the time allowed.
Apart from strong consensus about fundamental purposes and principles, there islittle common agreement regarding most of the key questions raised in theProspectus. The points listed below reflect findings, opinions and recommendationscontained in the campus responses as identified and synthesized by the committee inthe short amount of available time.
It is clear that campus understanding of the question asked of them was not uniform:some respondents interpreted AB 2016 and the proposed State Incentive Programbroadly to include approaches to improving educational quality beyond assessment;others understood the bill and the CPEC focus to be strictly limited to outcomesassessment.
To assure at least minimally representative character, the following summary reportsrecommendations made in one form or another by more than a single campus. Forpurposes of economy of presentation, similar recommendations differing as to levelof implementation (e.g., departmental or university) are presented under only onerubric. The order of listing reflects generally the degree of consensus found in theseparate campus replies.
Appendix BPage 28
Report of the Advisory Committee onStudent Outcomes Assessment
RECOMMENDATIONS FOR INCENTIVE FUNDING BY LEVEL
AT THE COURSE LEVEL
Instructional Improvement
A majority of responding campuses expressed approval of a program that would makeadditional funds available to improve instruction. The recommendations included:assessment centers aimed at helping faculty assess/improve the effectiveness of theirteaching/courses, applied research into factors affecting teaching and learning in specificdisciplines, assessment of student learning in relation to course/program objectives,reduction in student-faculty ratio for certain types of instruction or students (to allowindividualized attention for students with developmental needs).
Student Involvement in Learning
Funding to develop instructional materials and strategies aimed at involving students moreactively in the learning process was recommended. A variety of approaches wassuggested: e.g., student-faculty research, social service projects, faculty developmentactivities aimed at changing modes of instruction and motivating students, interdisciplinary-integrative learning, cooperative learning.
AT THE DEPARTMENT LEVEL
Curricular Improvement
Support was recommended for activities designed to: enable faculty to define programmaticgoals and develop assessment mechanisms appropriate to them; integrate critical thinking,verbal proficiency and international/multicultural perspectives into curricula; developcapstone/stepping stone courses; define articulation in terms of content mastery.
Diagnostic and Placement Testing
Substantial interest was expressed in external support to develop and administer, eitherthrough the department or the campus testing offices, additional diagnostic tests intended toprovide information for accurate placement and referral of students and for collection ofbaseline data for program evaluation.
Academic Advising
Funding to improve the quality of academic advising was encouraged in some responses.The link between effective advising and efforts to retain and assist students fromunderrepresented groups was stressed. Advisement training and innovative approaches toimprove and expand academic advisement were suggested.
Assessment in the Major
A few campuses approved the use of incentive funding to allow departments to assess thestudent learning and program effectiveness in the major provided the responsibility for
Report of the Advisory Committee onStudent Outcomes Assessment
Appendix BPage 29
developing and administering the assessment and for utilizing the data in improvingcurricula and instruction remains with the faculty of the department.
Partnerships with Business. Industry and Government
The many educational benefits of integrating practical applications into the curricula ofacademic majors has long been recognized. Some campuses would like additionalresources to support programs featuring student internships/practicums, community servicecomponents and programs enabling faculty and experts from outside academe to learn andteach in their counterparts' environments.
AT THE CAMPUS LEVEL
Educational Equity
Consensus among respondents was highest regarding the need to make our institutionsmore effective in meeting the educational requirements of students from underrepresentedsegments of the State's population. Clearly, CSU campuses regard this issue as a priorityconcern of all segments of California higher education and one which may be usefullyaddressed through a State incentive funding program. Specific suggestions as to whatkinds of programs might be funded include support to:
identify and address the factors that contribute to attrition of students who have potentialto complete postsecondary educationtrack the progress of individual students coming to, dropping out, and graduating fromour campusesassess the individual/group learning styles of studentsexpand peer support programs and increase tutorial services in basic skills and lowerdivision GE programsimprove/expand orientation and study skills programsdevelop assessment programs for oral communication, computational and criticalthinking at the entry and exit levels
Underprepared Students
The problem of inadequately prepared and poorly motivated students was mentioned inseveral campus reports. Campuses recognize that these students are often educationallydisadvantaged and represent, collectively, a significant element in the State's economic andpolitical mainstream. Assistance through a State incentive funding program would provideadditional resources to address this problem. Possible approaches, in addition thosementioned under one or more of the preceding headings, include:.
development/implementation of intervention strategies for high-risk studentsimprovement of academic advising, personal counseling and other opportunities forindividualized attention to students with remediational and adjustment needspartnership programs with secondary and middle schools to improve the academicpreparation of educationally disadvantaged students
Appendix BPage 30
General Education
Report of the Advisory Committee onStudent Outcomes Assessment
About half of the responding campuses expressed interest in outcomes assessmentapproaches to measuring the effectiveness of their General Education programs. Thedesign and evaluation of such programs should be the primary responsibility of eachinstitution's faculty and might involve collaboration with counterparts from communitycolleges in the respective campus service areas. Locally designed and administeredprograms might include a component common to all campuses of the system that wouldreflect the CSU general education-breadth requirements outlined in Title V.
International/Multicultural Education
The need to assure that CSU graduates acquire a useful understanding of the global contextof major political and economic problems confronting the United States was evident in theresponses of several campuses, as was the obligation to prepare students to live and workin a multiethnic, multicultural society. Supplementary resources would allow campuses toaccelerate and intensify efforts towards these ends.
Integrating Existing Assessment Programs
Most of the campus reports allude to the many kinds of assessment currently employed thatdo, in fact, measure outcomes. Additional funding to integrate and utilize moresystematically and publicly the data already collected was suggested as a valuable, cost-effective addition to current practices.
Faculty/Staff Development
Strong support was evidenced for use of incentive funds to support faculty and staffdevelopment activities targeted toward institutional priorities. Those mentioned included:use of instructional technologies, crosscultural communication, integrating writing andcritical thinking across disciplines, applied research on domain-specific learning, alternativemodes of instruction.
EsaandinzaillcknLaw-Qyav
Measures of student engagement/effort, moral and social development, and changes inbehavior were identified as important outcomes, a better understanding of which wouldhelp institutions identify programmatic and environmental factors in need of improvement.Financial support for surveying students over time should be included in a State incentiveplan to promote higher quality.
AT THE SYSTEM LEVEL
Basic Skillsiissessment and Development
Funding to measure and to enhance the development and retention of skills in oralcommunication, critical thinking and quantitative reasoning, using something like theEnglish composition model already in place, was recommended by about half of therespondents. Most of them supported system-level assessment, though lesser numbersthought it would be more effective to do such eva'iation at the campus level or on a
01 0
Report of the Advisory Committee onStudent Outcomes Assessment
Appendix BPage 31
Statewide basis. (Rationales and programs to assure student retention of basic skills arementioned above in the discussion of underprepared students.)
Assessment Technoloay
Mindful of the limitations of available instruments to measure student learning anddevelopment in any comprehensive way, some campuses saw support for research anddevelopment of evaluation tools as a major function for a State incentive funding programfocussing on outcomes assessment. Resources should also be made available to enable thesystems to monitor the cost-effectiveness of assessment programs.
AT THE STATE LEVEL
Longitudinal Data Co needed.'
CSU campuses signaled very strong support for the funding of comprehensive longitudinaldata collection systems. We know that students attending the CSU come with widelydivergent backgrounds, preparations -ind aspirations. Enrollment, course-taking, degreeprogress and post-graduation employment patterns vary enormously. Systematicidentification of these and other variables, many of which extend beyond the CSUadmissions and enrollment databases and the resources of the system, would provide apowerful tool for more effective advising, counseling and for academic and studentservices planning.
Educational Policy
About a quarter of the responding campuses suggested that much in the way of creatingconditions for improvement of quality could be accomplished through the funding ofsystem- and/or state-level forums for discussion of educational issues with representativesof industry, government, the media as well as all segments of education. Such exchangeswould enable academic leaders to gain broad societal perspectives regarding educationalpriorities and to inform key constituencies of the priorities and constraint.- undergirdingpolicy development in higher education. One particular policy issue related to AB 2016 thatcould be addressed in such a forum would be the cost-benefit question connected withoutcomes assessment: To what extend should funds be expended to obtain more exactmeasures of value added by institutions/programs versus use of these resources to improvedirectly the quality of the programs and the conditions for learning ?
Student-Faculty Ratio
Several campus responses suggested that budgetary formulas resulting in rather rigidproduction "quotas" of targeted student-faculty ratios was a principal constraint affectingthe ability of CSU campuses to address effectively some of the problems noted above.Supplementary resources to permit lower ratios and/or reduced faculty teaching loads wererecommended as an appropriate approach for a State program to promote higher educationalquality.
Financial Aid
A second major constraint impinging upon campuses' ability to resolve problems cited inthe discussion of educational equity and underprepared students is the dependency of many
Appendix BPage 32
Report of the Advisory Committee onStudent Outcomes Assessment
students upon income generated from full- or nearly full-time employment at jobsunconnected with their studies. Resources to allow students to concentrate their energieson learning would, some campuses suggest, go far toward enhancing the educationalexperience and the value of the time spent on campuses for many of these students.
RECOMMENDATIONS REGARDING PRINCIPLES
Agreement on basic principles that should govern creation of a State incentive fundingmechanism was far more evident than on how the funds ought to be used. Mostrespondents explicitly endorsed the six principles contained in the CPEC report, FundingExcellence in California Higher Education, Echoes of these principles recur in all of thecampus responses.
Purposes of Outcomes Assessment
Without exception, campuses supported the view that assessment of student outcomesshould serve the purpose of improving teaching and learning at the individual course,program and/or institution level. It should not be used for purposes of comparing theperformances of any one of these to others or to norms.
Institutional Autonomy and Faculty Responsibility
There was near unanimous agreement regarding the need to assign to the faculty of theindividual campuses the primary responsibility for designing, administering andinterpreting the results of programs to assess student learning, program and institutionalperformance. The diversity of institutional missions and the rapid shifts in thedemographics of the State require flexibility in approaches to measuring quality andeffectiveness.
Multiple Measures
Meaningful outcomes assessment must be multivariate if it is to provide reasonably validinformation for use in improving academic programs and modifying institutional practicesand for evaluating program or institutional performance. Standardized tests providevaluable but very limited kinds of information that do not necessarily relate to careerproductivity.
Adequate, Supplemental Funding
A good deal of skepticism was expressed about the real source of fiscal support foroutcomes assessment programs. All respondents concurred that the dollars for thesepurposes should be in addition to regular institutional budgets; most, however, thought thatthe money would be taken "off the top," resulting in a distribution of the costs to allcampuses.
Fears were also evident that the real costs of implementing comprehensive assessmentprograms in terms of money, time and political stability have not been recognized or takeninto consideration. Several CSU campuses have recent, apparently painful experience withunderfunded efforts to implement subject-matter competency tests for students enteringcredential programs.
1 ,
t s-J
Report of the Advisory Committee onStudent Outcomes Assessment
Cost-Benefit Ratio
Appendix BPage 33
A State incentive program of the kind under study should be structured in such a way as tominimize the resources required for its administration; i.e., the dollars should supportdevelopment and operation of assessment, not the administrative apparatus charged withdistributing supplementary funds to do so. (It was suggested that existing system grantprograms--e.g., the merit-based instructional improvment funds--might be appropriatevehicles for administering funds targeted for encouraging assessment.)
A strong public commitment to careful analysis of the cost-effectivess of the outcomesassessment approach should be made if it is adopted as the goal of a State incentive plan.
Reductive Effects on Academic Programs
The CSU opposes the creation of an incentive funding approach that rewards institutionsfor producing simplistic, quantitative measures of student performance. Funding shouldgo to institutions proposing to address complex educational problems in commensurateways. Performance funding, as aconcept and as operates at the University of Tennessee,Knoxville, is not acceptable.
CAVEATS
Educational and Institutional Priorities
A major danger of targeting additional resources for assessment is that it reduces supportfor other programs to address concerns of high priority to the State, the system and theinstitutions. There was substantial consensus that additional money could be spent moreeffectively on approaches to improving quality not related to outcomes assessment.
Erosion of Diversity
Adoption of standardized testing as a prominent index of institutional quality acrossinstitutions within/between systems will cause a shift away from complex educational goalstoward achieving good ratings as measured by assessment results. "Teaching to the test"and homogenization of curriculum are inevitable consequences of such an approach.
Distortions of Reality
The existence of comprehensive assessment programs does not guarantee improvement ofquality. One cannot rely on summative evaluation approaches to achieve formative ends.Indices of quality derived from limited measures may be far more misleading than students'grade point average or other currently available measures.
Date: October 28, 1987
To: Vice PresidentsAcademic Affairs
From:
Subject:
Lee R. KerscVice ChancellorAcademic
THE CALIFORNIA STATE UNIVERSITYOffice of the Chancellor
400 Golden ShoreLong Beach, California 90802-4275
(213) 590-
State Incentive Funding Approaches
Code: APPS 87-31
Reply Requested byJanuary 11, 1988
Attached please find a copy of Assembly Bill No. 2016, "Higher Education TalentDevelopment," that has been signed by the Governor. In accordance with the bill, theCalifornia Postsecondary Education Commission has established an Advisory Committeeon State Incentive Funding Approaches for Promoting Quality in California HigherEducation. Six members of that committee are from The California State University:
Dr. Bernie Goldstein, Academic Senate, CSU, San FranciscoDr. Diane Halpern, CSU, San BernardinoDr. Glenn Irvin, CPSU, San Luis OblspoDr. Leigh R. Mintz, CSU, HaywardMs. Sherry Skelly, CSSADr. Frank Young, Chancellor's Office
The anticipated outcomesof the CPEC study (for which a proposed prospectus isattached) are a list of quality improvement options or proposals designed to:
1. Stimulate institutional practices to promote quality in higher education;
2. Provide greater accountability for the quality and content of college instruction;
3. Understand better how the budget process can be used to improve the educationalprocess.
As its title implies, Assembly Bill 2016 specifies that the options CPEC is to develop andpresent to the Legislature focus on measuring gains in student learning and on assessingthe effectiveness of the teachinglearning process.
In fulfilling its charge to develop State funding incentives to achieve these ends, AB 2016requires CPEC to consult with "students, faculty, staff, and administrators at the stateand local campus level." This provision is consistent with the guiding principles developedby the ACR 141 Task Force last year and incorporated into the text of the legislation(Section 66912). These principles recognize that "the definition and assessment of studentoutcomes and competency standards at the course, program and departmental level isprimarily a faculty responsibility."
2
The CPEC advisory committee has ascertained that it will begin its work by requestingthat as many faculty and administrators as possible be polled on the following question:
How can incentive funding be used at the course, departmental, university, systemand state levels to effect improvement in educational quality?
I am asking each campus to consider this complex question and to propose optionsamenable to incentive funding. In soliciting recommendations, campuses should keep inmind the legislative stipulation (page 5) that "State funding incentives to promote qualityin California higher education should be funds that are supplementary to the institution'sbase budget . . .." Please return your recommendations to Dr. Frank Young by January 11,1988.
To coordinate the CSU responses to the CPEC advisory committee, and to address relatedissues raised in supplemental budget language (copy attached), the CSU AssessmentAdvisory Committee is being established. Dr. Young will convey campus responses to thisgroup whose members will be asked to compile the system report.
You may be interested to know that this question will be discussed with the campusSenate chairs when they meet at Asilomar on November 13, and will be a topic ofdiscussion at an Academic Senate Retreat workshop on November 14. However, theRetreat discussions are not intended to produce campus or system responses.
Please call Dr. Frank Young at (213)590-5856 or KISS 635-5856 if you have any questions.
LRIC/na(APPSTWO:1161n)
Attachments: 2
Copies to:
PresidentsAssociate Vice Presidents, Academic AffairsAcademic Senate, CSUChancellor's Office StaffCampus Faculty Senate ChairsCPEC Committee RepresentativesCSU Assessment Advisory Committee RepresentativesChair, California State Student Association
APPENDIX C
SELECTED LIST OF SOURCES CONSULTED
Appendix CPage 38
Report of the Advisory Committee onStudent Outcomes Assessment
SELECTED LIST OF SOURCES CONSULTED
Adelman, C. (Ed.). (1988). peformance and Judgement: Essays on Principles and Practice inthe Assessment of College Students. Washington, D.C.: U.S. Department ofEducation.
Adelman, C. (Ed.). (1986). Assessment in American Higher Education; Issues andContexts. Washington, D.C.: United States Department of Education.
Adelman, C. (Ed.). (1986). From Reports to Response. Proceedings of RegionalConferences on the Quality of American Higher Education. Washington, D.C.: UnitedStates Department of Education.
Alexander, J. M. & Stark, J. S. (1986). Focusing on Student Academic Outcomes. AWorking Paper. Ann Arbor, MI: National Center for Research to ImprovePostsecondary Teaching and Learning.
Alexander, L., et al. (1986). Time for Results. The Governor's 1991 Report on Education.Washington, D.C.: National Governors' Association.
Arons, A. B., Ernest I. Boyer, et al. (1984, October). "Integrity in the College Curriculum:Report of the Project on Redefining the Meaning and Purpose of BaccalaureateDegrees." The Chronicle of Higher Education. pp. 35 48.
Ashcroft, J. & Hayden, T. (1987). "The Assessment Debate. What Do Students Learn byGoing to College and How Do We Know?" Washington, D.C.: AAHE.
Astin, A. W. (1985). Achieving Educational Excellence. A Critical Assessment of Prioritiesand Practices in Higher Education. San Francisco, CA: Jossey-Bass.
Astin, A. W., Blake, J.H., et. al. (1984). Involvement in Learning: Realizing the Potential ofAmerican Higher Education. Final Report of Study Group on the Conditions ofExcellence in American Higher Education. Washington, D.C.: U.S. Department ofEducation.
Banta, T.W. (1987). Bibliography of Assessment Instruments. Knoxville, TN: Univeristy ofTennessee, Assessment Resource Center.
Bergen, S. S., Jr. (1987). Report to the New Jersey Board of Higher Education. From theAsi22asslumiktettQlheColkgeatitcomesayaluatio2LwesdiProject. New Jersey Board ofHigher Education.
Bok, D. (1986). Higher Learning. Cambridge, London: Harvard University Press.
Boyer, C. M., Ewell, P. T., et al. (1987, March). "Assessment and Outcomes MeasurementA View from the States. Highlights of a New ECS Survey and Individual State
Profiles." AAHE Bulletin. pp. 8 12.
California Postsecondary Education Commission (1987). Background Papers of the ACR141 Task Force on Funding Excellence in Higher Education; Correspondence from
Report of the Advisory Committee onStudent Outcomes Assessment
Appendix CPage 39
Task Force Members Preparatory to the Commissions' 1987 Report. Sacramento, CA:California Postsecondary Education Commission.
California Postsecondary Education Commission (1987). Beyond Assessment: EnhancingLearning and Development of California's Changing Student Population. Sacramento,CA: California Postsecondary Education Commission.
California Postsecondary Education Commision (1986). Funding Excellence in CaliforniaHigher Education. Sacramento, CA: California Postsecondary Education Commission.
Cross, K. P. (1988). Feedback in the Classroom: Making Assessment Matter. Washington,D.C.: AAHE Assessment Forum.
Darling-Hammond, L. (1988, June). "Assessment and Incentives: The Medium is theMessage." Three Presentations: From the Third National Conference on Assessment inHigher Education. Washington, D.C.: AAHE Assessment Forum.
El-Khawas, E. (1986). Campus Trends: Higher Education Panel Report Number 73.Washington, D.C.: American Council on Education Higher Education Panel.
Ewell, P. T. (1985). Assessing Educational Outcomes. San Francisco, CA: Jossey-Bass.
Ewell, P. T. The Self-Regarding Institution: Information for Excellence. Boulder, CO:National Center for Higher Education Management Systems.
Farmer, D. W. (1988). Enhancing Student Learning. Emphasizing Essential Competencies in
Academic Programs. Wilkes-Barre, PA: King's College.
Fond, B. (1987). The External Examiner Approach to Assessment. Washington, D.C.:American Association of Higher Education.
Giamatti, A. B. (1988, November). "A Free and Ordered Space: The Real World of theUniversity." The Chronicle of Higher Education. 35, p. B If.
Halpern, D. F. (1987). Student Outcomes Assessment: What Institutions Stand to Gain. SanFrancisco, CA: Jossey-Bass.
Harris, J. (1985). "Assessing Outcomes in Higher Education: Practical Sugggestions forGetting Started." Columbia, S.C.: Paper presented at the National Conference onAssessment in Higher Education. ERIC ED 260277.
Hartle, Terry W. "The Growing Interest in Measuring and Educational Achievement ofCollege Students" in Clifford Adelman, ed. Assessment in American HigherEducation: Issues and Contexts. Washington, D.C.: U.S. Department of Education,Fall 1985.
Hayden, T. (1986). Beyond the Master Plan, A New Vision for Higher Education inCalifornia. Sacramento, CA: California State Assembly.
Heffernan, J. M., Hutchings, P. et al., (1988). Standardized Tests and the Purposes ofAssessment. Washington, D.C.: AAHE Assessment Forum.
Appendix CPage 40
Report of the Advisory Committee onStudent Outcomes Assessment
Kean, T. H., et al. (1986). Transforming the State Role in Undergraduate Education.Denver, CO: Education Commission of the States.
Lamb, M. B., (1986). Assessing the Outcomes of Higher Education. Proceedings of the1986 ETS Invitational Conference. Princeton, NJ: Educational Testing Service.
Members of Challenge Grant Evaluation and Implementation Committee. (1989). Excellenceund Equity. The Kean College Challenge Grant Final Report. Union, NJ: KeanCollege.
Mentkowski, M. & Doherty, A. (1984). Careering After College. Milwaukee, WI: AlvernoCollege.
Neill, D. M. & Noe J. M. (1989, May). "Standardized Testing: Harmful to EducationalHealth." Phi Delta Kappan. pp.688 696.
Nichols, J. 0. (1989). Institutional Effectiveness and Outcomes Assessment Implementation.A Practitioner's Handbook. New York, NY: Agathon Press.
Northeast Missouri State University. (1984). In Pursuit of Degrees with Integrity: A Value-Added Approach to Undergraduate Assessment. Washington, D.C.: AmericanAssociation of State Colleges and Universities.
Novak, J. D. & Ridley, D. R. (1988). Assessing Student Learning in Light of How StudentsImo. Washington, D.C.: AAHE Assessment Forum.
Paskow, J. editor. (1988). Assessment Programs and Projects: A Directory. Washington,D.C.: AAHE Assessment Forum.
Quehl, G. H. (1988, September). "Higher Education and the Public Interest." Academe. pp.7 - 11.
Rossmann, J. E. & El-Khawas, E. (1987). Thinking About Assessment: Perspectives forPresidents and Chief Academic Officers. Washington, D.C.: American Council onEducation & AAHE.
Vasconcellos, J. (Chair), Allen D., et. al. (1989, March). California Faces...California'sFuture: Education for Citizenship in a Multicultural Democracy. Final Report of theJoint Committee for Review of the Master Plan for Higher Education. Sacramento, CA:Government Printing Office.
Western Association of Schools and Colleges: Accrediting Commission for Senior Collegesand Universities. (1988). Handbook of Accreditation.
White, E. M. (1985). Teaching and Assessing Writing. Recent Advances in Understanding,EnbalinganCLIMPIQYing211C1faltEttallina110. San Francisco, CA: Jossey-Bass.
Wiggins, G. (1989, May). A True Test: Toward More Authentic and Equitable Assessment.Phi Delta Kappan. pp. 703 713.
Report of the Advisory Committee onStudent Outcomes Assessment
Appendix CPage 41
SPECIALIZED SOURCES OF CURRENT INFORMATION
"Assess Net:" Comprehensive Research Universities Assessment Network. BITNETAddress: ASSESS@ COLORADO.
Banta, T.W. (Ed.). Assessment Update. (quarterly newsletter). San Francisco, CA: Jossey-Bass.
Weiner, J.R. (periodically updated). National Directir,zc Assessment Programs and Projects.Washington, D.C.: American Association for Higher Education.
CALIFORNIA STATE UNIVERSITY PUBLICATIONS
Barnes, S.F., Coleman, J. et. al. (1969, May). Report of the Educational Support ServicesMaster Plan Task Force. Long Beach, CA: The California State University, Office ofthe Chancellor.
Boxer, M. J. & Richmond, J. E. (1988). Resource Guide: Subject Matter Assessment ofProspective Elementary School Teachers. Report of CSU Workgroup on Assessmentof Prospective Elementary School Tgachers. Long Beach, CA: The California StateUniversity, Office of the Chancellor.
Christensen, B. (Ed.) (1989) Resource Guide: Subject Matter Assessment of ProspectiveForeign Language Teachers. Report of CSU Workgroup on Assessment of ProspectiveForeign Language Teachers. Available from Long Beach, CA: The California StateUniversity, Office of the Chancellor.
Harris, 0. (Ed.). (1989). From Projects to Policy: Seize the Agenda." ConferenceProceedings: The 1988 California State University. Systemwide Conference onOutcomes Assessment. Long Beach, CA: The California State University, Office of theChancellor.
Holder, C. & Arena, N. (1989). Academic Program Improvement: The Campus GrantsProgram 1988-89. Long Beach, CA: The California State University, Office of theChancellor.
Kantz, T. and Baad, M. (Eds.) (1989) Resource Guide: Subject Matter Assessment ofProspective L ife Science Teachers:. Report of CSU Workgroup on Assessment ofProspective Life Science Teachers. Available from Long Beach, CA: The CaliforniaState University, Office of the Chancellor.
Lindeman, C. (Ed.) (1989) Resource Guide: Subject Matter Assessment of ProspectiveMusic Teachers. Report of CSU Workgroup on Assessment of Prospective MusicTeachers. Available from Long Beach, CA: The California State University, Office ofthe Chancellor.
Reeser, R. (Ed.) (1989). Resource Guide: Subject Matter Assessment of Prospective ArtTeachers. Report of CSU Workgroup on Assessment of Prospective Art Teachers.
Appendix CPage 42
Report of the Advisory Committee onStudent Outcomes Assessment
Available from Long Beach, CA: The California State University, Office of theChancellor.
White, J. 0. & Denham, C. H. (Eds.) (1987). Subject Matter Assessment of ProspectiveEnglish Teachers. Report of CSU Workgroup on Assessment of Prospective EnglishTeachers. Long Beach, CA: The California State University, Office of the Chancellor.
