The Texas experience with accountability and student learning assessment

The University of Texas experience with a public account-ability reporting system foretells the prospects of relatedefforts like the Voluntary System of Accountability.

The Texas Experience withAccountability and Student Learning Assessment

Pedro Reyes, Roberta Rincon

The national focus on the quality of higher education centers directly onstudent learning outcomes. The Spellings Commission on the Future ofHigher Education recommended that higher education institutions measurestudent learning and make learning assessment results available to the pub-lic (U.S. Department of Education, 2006). The recent rollout of the Volun-tary System of Accountability (VSA) aims to do just that. It contains acomponent that requires participating institutions to assess their studentsusing one of three standardized learning assessment instruments.

In Texas, the accountability movement was initiated by Governor RickPerry, who issued an executive order in 2004 requiring all public institu-tions of higher education to work with the Texas Higher Education Coordi-nating Board to create a comprehensive system of accountability. Its purposewas one of public accountability: to the citizens of Texas, the governor, andthe legislature. By the end of the year, the board had rolled out its onlineaccountability system.

The University of Texas (UT) System began looking into the need tobecome more transparent and accountable to its stakeholders earlier thanother state and national organizations. In 2000, UT System regent CharlesMiller led the move toward the development of a systemwide accountabil-ity system focused on three institutional missions: learning and teaching,

49

4

NEW DIRECTIONS FOR INSTITUTIONAL RESEARCH, Assessment Supplement 2007, Fall 2008 © Wiley Periodicals, Inc.Published online in Wiley InterScience (www.interscience.wiley.com) • DOI: 10.1002/ir.261

50 ASSESSMENT SUPPLEMENT 2007

NEW DIRECTIONS FOR INSTITUTIONAL RESEARCH • DOI: 10.1002/ir

research, and service. The accountability system was the first in the nationto require the use of standardized tests and public dissemination of theresults (Traub, 2007). The system helped determine, measure, and reporthow well UT institutions achieved their missions.

Chancellor Mark Yudof and the board of regents identified one of theirhighest priorities as accountability––taking responsibility for measuring andreporting effectiveness and applying the information to continuouslyimprove performance. To make its accountability more transparent, consis-tent, and useful, the UT System began in 2004 to publish a comprehensiveannual report, Accountability and Performance Report. The system’saccountability framework was designed to facilitate planning, make strate-gic resource allocations, offer incentives for exemplary performance, pro-mote greater efficiencies and higher productivity, establish expectationswhere improvements are needed, and be outcome oriented. It serves as avehicle for communication with policymakers and the public.

The UT System’s approach to accountability addresses more than juststudent learning outcomes. It includes affordability, the impact of researchon economic development, and efficiency. But the issue of learning out-comes has received significant attention and investment.

The system began addressing learning outcomes in 2002, developingmultiple approaches to student learning assessment. First, a systemwide fac-ulty committee was created to develop performance standards in collegewriting and mathematics. Second, a pilot learning assessment test programwas launched using a standardized test developed by the Educational Test-ing Service (ETS) and the College Board in fall 2003 to see the effectivenessof using the results to improve institutional teaching. A year later, the UTSystem began using the Collegiate Learning Assessment (CLA) to assess stu-dents’ skills in problem solving, analytical and critical thinking, and writ-ten communication. The results obtained would be used to determine thevalue added to student learning at each institution.

Initially faculty members throughout the academic institutions wereupset with the notion of introducing standardized testing within the system.The faculty at different campuses reacted negatively when members of theboard of regents talked about accountability and assessment. The negativereactions were in response to the idea of an accountability and assessmentprogram modeled after the one installed for public education.

Yet the members of the Faculty Advisory Council, a systemwide orga-nization, embraced the challenge. In fact, they were partners with the Officeof Academic Affairs in developing the first generation of assessment metrics.The faculty advisory group, along with academic officials, devised a plan toinvolve faculty in the development of assessment strategies in two discipli-nary areas: one in writing communication and the other in mathematics.

Faculty committees across institutions came together to work on theassessment tools. However, it became clear that due to many logistical prob-lems, it was a difficult process to accomplish as a group. Faculty members

51THE TEXAS EXPERIENCE


therefore decided to develop assessment tools within their own campusesrather than design one for all institutions. The argument was that each insti-tution had some unique aspects that needed consideration in developing thestudent learning assessment program. Thus, faculty leaders formed theirown campus-based committees to deal with assessment.

After two years of work, all institutions had developed rubrics to assesswritten communication, had created grading groups, and had reported tothe system their assessment of written communication. At the same time,faculty realized the enormous time commitment and expense required toassess 100 or 150 students at random on a yearly basis. The mathematicsgroups did not have the time to achieve consensus on the assessment toolfor mathematics or the method to collect the data.

Given this time commitment and expense, system officials offered toreview or pilot-test the Measure of Academic Proficiency and Progress(MAPP) developed by ETS, which the system faculty committee reviewed.The MAPP test was pilot-tested, and the results were found to be inade-quate. Once the faculty reviewed the results, they decided that the test didnot address key areas such as calculus in the mathematics section. There-fore, it was eliminated after its first cycle.

System officials then explored the idea of using the CLA as one of thetools to account for one area of student learning and skill development. CLAoffered a much more economical and less time-consuming way to collectthe data about general cognitive skills such as problem solving, criticalthinking, and written communication––skills typically associated with thegeneral curriculum of any college or university. The consensus was that fac-ulty, along with national accrediting bodies, should develop learning assess-ment tools within their own disciplines to assess student learning. The otherreason for adopting CLA as a tool was that it permitted analysis at the insti-tutional level.

Student Outcomes Model

The UT System uses a set of performance indicators for student outcomesto assess institutional success. This model has five key elements: persistenceand graduation rates, license exam pass rates in critical fields, postgraduateexperience, student experience, and student learning assessments.

Student persistence and graduation rates are two of the indicators usedto assess institutional performance. The UT System board of regents adopteda resolution for each academic institution to set specific targets and initia-tives to improve persistence and graduation rates. These five- and ten-yeartargets allow institutions to benchmark against national trends. Initiativesundertaken to meet these targets included modified tuition structures, on-campus employment opportunities, and combined academic and financialaid advising to help students understand the effects of continuous enroll-ment on their financial status.



Pass rates on licensing exams and postgraduation employment are alsoused to evaluate the quality of the academic programs. These pass-rateexams, although program or discipline specific, provide a way to determinethe preparation levels of graduates in fields such as engineering and educa-tion. In addition, a postgraduation employment and graduate programenrollment indicator is used to assess the quality of student preparation. Thedata for this measure are somewhat limited because of the inability to trackstudents who leave Texas after graduation. But the information provides avaluable measure because it could potentially help address questions of howprepared students are to join the workforce or pursue further study.

Understanding students’ educational experiences is another element inthe student outcomes framework. The UT System institutions participatein the National Survey of Student Engagement, which has been administeredto a random sample of freshmen and seniors within the system for the pastseven years. The survey assesses the extent to which the students engage ineducational practices associated with high levels of learning and develop-ment. Institutions obtain the results of these surveys for benchmarking pur-poses and for sharing with the board of regents, the legislature, and thepublic.

Student learning assessment is perhaps one of the most critical ele-ments of the student outcomes model. The purpose of learning assessmentsat the system level is to promote quality, comparability, and information thatsupport policy development. At the institution level, the purpose of assess-ment is to give faculty and administrators the information they need toimprove student learning. Effective academic assessment can help institu-tions determine where and how to improve teaching and learning (De León,2007). Assessments can also help determine whether academic programsare accomplishing what they intend to accomplish and help to identify cur-ricular areas in need of adjustment.

All of the performance indicators for student outcomes allow UT Systemand individual institutions to gauge their success against other institutions inthe state and the nation, but more important, they give the institutions a wayto judge the impact of improvement made to programs and practices. Insti-tutions track their own performance on a yearly basis, comparing themselvesagainst their own past performance and setting targets for improvement.

The Collegiate Learning Assessment: Outcomes,Shortcomings, and Possibilities

Student learning assessment is an intricate part of the accountability frame-work within the UT System. Two interrelated programs address account-ability concerns. One level of accountability addresses the disciplinarycontent as required by national accrediting bodies. The other addressesexternal accountability groups, such as taxpayers, policymakers, parents,business and industry, and other employers. For example, to satisfy disci-



plinary concerns, every UT institution has student outcomes expectationsset at the program level. These expectations are discipline specific and areguided by external accrediting organizations such as the ABET for engineer-ing programs and the Association to Advance Collegiate Schools of Busi-ness. At the institution level, requirements are set by the SouthernAssociation of Colleges and Schools (SACS) and must be met for accredita-tion purposes. The association recently required all institutions to includea system of assessing student learning into their institutional self-reviews.Student learning contracts and outcomes are displayed to the public by eachacademic program.

At the external level of accountability, institutions use a standardizedlearning assessment tool that has a national framework to benchmark against.The UT System began using the CLA in the 2004–2005 academic year. At thattime, over 120 other colleges and universities across the country were partner-ing with the Council for Aid to Education and the Rand Corporation to admin-ister the CLA. By 2007–2008, the number of institutions that participated inthis project increased to 370. The system chose the CLA because it wasdesigned to assess general problem solving and critical and analytical writingabilities of freshmen and seniors. The level of participation by other highereducation institutions across the country makes it possible to benchmark per-formance against other institutions with similar student bodies. Testing offreshmen and seniors allows a comparison of performance, resulting in a value-added measure that had not been available before the use of this assessment.

The nine academic institutions within the UT System have been test-ing samples of freshmen and seniors for four years. The results of these testshave been presented to the UT System board of regents since that time, andthe regents have shown their support of these assessments through theircontinued funding of the program. The system has been very open with theresults, publishing them in the annual Accountability and PerformanceReport. The CLA results provide system institutions with a measure of per-formance that is easy for external stakeholders, such as parents, employers,and legislators, to understand.

The CLA estimates the value added by the institutions through a com-parison of freshman to senior results. The way the CLA computes this mea-sure accounts for differences in student body characteristics. It provides away to evaluate institutional effectiveness through a comparison of expectedresults based on students’ scores on entrance examinations against theactual results obtained. It thus avoids institutional comparisons within the system. Institutions are instead compared against peer institutions thatshare similar characteristics, particularly similar student academic abilities.Evaluating against national benchmarks in this manner gives stakeholdersa better understanding of the value that each institution offers.

The system institutions have used the CLA results to help them deter-mine what interventions may play a role in their effectiveness in improvingstudent success. For example, one institution has begun using the CLA



results in its quality enhancement plan for its SACS reaffirmation. Anotherhas used the results to fine-tune its writing program, identifying holes in thecurriculum by analyzing areas of weakness that the results reveal. One couldargue that this may be interpreted as teaching to the test. However, that isnot possible given the test architecture. Unlike many other assessment tools,the CLA is not a multiple-choice test. It is a Web-based test requiring open-ended responses. The holistic manner in which the test is presented assessesmore complex higher-level skills rather than knowledge acquisition.

Some institutions that have seen positive results relating to the value-added component have begun using the CLA results as a recruiting tool. Ingeneral, all the UT institutions have used their results to benchmark againstthe national norms or against a set of peers that institution officials haveselected for comparison.

Limitations and Possibilities

Although the system has seen success in the use of student assessments,some limitations still need to be addressed. Until the 2008–2009 academicyear, UT institutions sampled one hundred freshmen and one hundredseniors. For large institutions, the small sample size does not allow resultsto be used to inform improvements at the college level, particularly when arandom sample is required. Typically those institutions oversample to com-pensate for the potential lack of representation from different colleges. Thereason is that although the sample is random, student participation is vol-untary, which makes it difficult to accomplish the stratification and inclu-sion of all colleges in the sample. To help with this limitation, the UTSystem board of regents provided funds to help the institutions provideadditional incentives to students, particularly to help motivate seniors to sitfor the exam and provide their best effort.

One positive aspect of the CLA report is the unit of analysis. The CLAis focused on the institutional results, allowing comparisons across institu-tions and comparisons over time (Benjamin, 2008). The ability to compareinstitutional results is what puts the value-added piece into perspective.However, it is important to compare an institution against its peers––inother words, those with similar average SAT scores (Benjamin, 2008). Insti-tutions make comparisons based on the national sample where there is arange of values across institutions. At the same time, UT institutions areable to show where their students begin academically. Some of the institu-tions that have students entering with low SAT scores are able to see wherethey begin and make comparisons that way. They use national benchmarksfor those comparisons.

At the same time, it is impossible to avoid inter-institution comparisonswithin the system. In fact, that was the initial approach some system offi-cials used. However, UT System staff working on the project convincedthese officials that it was the wrong approach. Similarly, the presentation to



the board of regents strongly emphasized the point to not make internalcomparisons among institutions.

Another benefit of using the CLA is that participating students haveaccess to their individual scores (released only to students), which help themidentify areas of weakness that need more focus. However, the way the CLAinstitutional scores are reported presents a limitation. The CLA report usestwo scores: one for analytical writing and the other for what is called “per-formance task.” The performance task incorporates critical thinking, prob-lem solving, and analytical reasoning. The lack of separate scores for thatassessment component makes it difficult for an institution to determineareas of weakness and allow focused improvement in curriculum andinstructional practices.

University of Texas institutions have experienced some challenges withdetermining exactly how to use the CLA results to improve the curriculumand programs. The Council for Aid to Education recognized the need toaddress this and developed a new training program, CLA in the Classroom.Faculty members have taken advantage of this program, learning how to usethe CLA results for diagnostic purposes. They come away from the trainingwith a better understanding of how to create their own performance tasks,modeled after those used in the CLA, to help students apply critical think-ing and problem-solving skills in their course work. Although this is a newprogram, it should help the institutions integrate assessment tools into theircurriculum and instruction frameworks.

The board of regents has already shown its continued support of theassessments by increasing the funding to double the sample size and pro-vide more incentives to students. Institutions have been creative in their useof incentives to increase student participation. Some are paying a smallmonetary stipend for participating, and others have offered graduation-related items such as diploma frames and graduation gowns. Some haveplaced participant names in drawings for more expensive items such as MP3players and gift certificates. One institution has already made the exam acourse requirement for freshmen and seniors.

One change being considered is a move to a longitudinal design toallow more control over individual differences and the ability to assess thevalue added more carefully. The only limitation is the need for a largeenough sample to account for dropouts, transfers, and other students wholeave the institution. The system experimented with a different approach inits most recent student assessment cycle. Rather than using the differencescores between freshmen and seniors (in other words, the value-added scorefor seniors), their expected and current scores on CLA were used to ascer-tain the value-added score for the individual and then for the institution asa whole. This approach eliminates the problem of using the difference scoresfrom two different samples.

Another possibility is to embed the exam into classroom expectations,such as in a capstone course. Some institutions have begun doing so. The



possibility of course credit can act as a strong motivator, particularly forseniors. Campuses that have experimented with incorporating the CLA intotheir courses have seen increases in student participation (Council for Aidto Education, 2005).

As for evidence that improvements have been achieved as a result ofadopting this system, it is too soon to tell––the system has not been in placelong enough to yield definitive results. Moreover, the system had been usedas a pilot with small samples and incentives to motivate students to take thetest seriously. However, the mere notion that academic institutions and fac-ulty are willing to try new methods of student learning assessment has gen-erated goodwill and trust among policymakers. Once the VSA is in place,parents, students, and others will be able to read the student outcomes in asimplified format. Such portraits are expected to satisfy the informationneeds of these constituents.

The other key issue in this system is the value-added component. Thiscomponent, given the cross-sectional nature of the system’s applicationamong the UT institutions, is not fully useful to system administration atthis time. Different approaches are being attempted to calculate value added,as already noted. Therefore, the index has not been correlated with anyactivities at the campus level. The closest activity related to the CLA is oneacademic institution’s use of the CLA as an intricate element of the qualityenhancement plan required by SACS.

Finally, there was some question as to the utility of and resistance to thisapproach among UT’s two highly selective institutions. The two institutionsembraced the system with some trepidation at first. However, over the years,they have fully adopted the system because it provides them with national databenchmarks for comparison purposes. Indeed, those two institutions foundthat they did much better on freshman and senior scores than the nation as awhole when comparing absolute scores with the national benchmark.

Moreover, since disciplinary student learning assessment is left up to thefaculty, these institutions do not see the CLA results as a threat to faculty gov-ernance in this area. In fact, these institutions use these data to draw atten-tion to the high quality and high levels of achievement of their students. Theseinstitutions, as well as their students, are able to point out that their studentsare in the ninetieth percentile or higher on that student learning assessment.

Discussion

With the pressure of accountability coming from both the state and federallevels, the push for student learning assessments in higher education con-tinues to grow. The UT System board of regents was the first in the nationto require its institutions as a whole (fifteen academic and health sciencecenters together) to share their results with the public. Student learningassessments are a critical element of the system’s student outcomes model.As the system’s assessment instrument of choice, the CLA continues to be a



valuable tool in helping to meet the expectations of students, parents, fac-ulty, legislators, and the public.

Having begun using the CLA four years ago, the UT institutions havehad the opportunity to identify the positive and challenging aspects of itsuse. On the positive side, the CLA has provided a way to present the valueadded in attending a UT institution. The determination of value added isone that focuses on the institution rather than the student, and with overtwo hundred higher education institutions across the country using theCLA, there is now the ability to compare against institutions outside the sys-tem. The comparison therefore is more balanced because it compares insti-tutions with student populations that have similar levels of academicpreparation.

Each of the nine academic campuses in the system has analyzed itsresults to determine what interventions can be made that may have animpact on its effectiveness in improving student success. Some have tied theCLA into their quality improvement plans, while others have added selectcourses to assist students in improving writing skills. However, not everycampus has found it easy to determine exactly how to use the results tomake curriculum and program improvements. The limited availability ofscores for specific components such as problem solving makes it more dif-ficult to determine what needs to be addressed.

In general, the use of the CLA has been positive, but the system contin-ues to modify its use and the analysis and application of assessment results.For example, the UT institutions have struggled with trying to increase smallsample sizes. Students have not always been easy to recruit, and the incentivesthat the institutions offer are not always enough to draw in the necessary par-ticipation levels to fully participate in the CLA. With the continued supportof the UT System board of regents, more funding is being applied to increasesample sizes and provide more incentives to freshmen and seniors to encour-age participation. Institutions are also considering alternative ways to increaseparticipation, including incorporating the assessments into the curriculum.

At the system level, the use of cross-sectional analysis has been anissue. A longitudinal analysis would allow more control over individual dif-ferences, but obtaining a large enough sample size to account for dropoutsand transfers would be costly. Continued use of the CLA may lead to thismore robust analysis, but for now, the CLA is providing a valuable tool inthe system’s quest toward transparency, accountability, and overall improve-ment in student success.

References

Benjamin, R. “The Contribution of the Collegiate Learning Assessment to Teaching andLearning.” Council for Aid to Education, 2008. Retrieved July 23, 2008, fromhttp://www.cae.org/content/pdf/The_Contribution_of_the_%20Collegiate_Learning_Assessment_to_Teaching_and_Learning.pdf.



Council for Aid to Education. Collegiate Learning Assessment: Guide to Recruiting Stu-dents. New York, 2005. Retrieved July 23, 2008, from http://www.cae.org/content/pdf/CLA.GuideForRecruitingStudentsSpring05.pdf.

De León, A. G. “The Collegiate Learning Assessment: A Tool for Measuring the ValueAdded of a Liberal Arts Education.” In E. Lerman (ed.), Carnegie Results. New York:Carnegie Corporation of New York, 2007. Retrieved July 23, 2008, from http://www.carnegie.org/results/19/index.html.

Traub, J. “No Graduate Left Behind.” New York Times Magazine, Sept. 30, 2007, pp.106–109.

University of Texas System Board of Regents. Accountability and Performance Report2003-04. Office of Strategic Management, Austin, Tex., 2004. Retrieved July 23, 2008,from http://www.utsystem.edu/osm/accountability/2003/completereport.pdf

U.S. Department of Education. A Test of Leadership: Charting the Future of U.S. HigherEducation. Report of the Commission Appointed by Secretary of Education MargaretSpellings. Washington, D.C.: Education Publications Center, 2006.

PEDRO REYES is associate vice chancellor for academic planning and assessmentfor the University of Texas System.

ROBERTA RINCON is research and policy analyst in the Office of Academic Affairsfor the University of Texas System.

Documents

The Texas experience with accountability and student learning assessment