Upload
azura
View
43
Download
0
Embed Size (px)
DESCRIPTION
Voluntary System of accountability and learning outcomes: an update. Teri Hinds Voluntary System of Accountability Natasha Jankowski National Institute for Learning Outcomes Assessment. October 30, 2012 Assessment Institute. Overview NILOA’s work with Transparency - PowerPoint PPT Presentation
Citation preview
VOLUNTARY SYSTEM OF ACCOUNTABILITY AND LEARNING
OUTCOMES: AN UPDATE
Teri Hinds Voluntary System of AccountabilityNatasha JankowskiNational Institute for Learning Outcomes Assessment
October 30, 2012 Assessment Institute
Overview NILOA’s work with Transparency Internal and External Communication Voluntary System of Accountability Evaluation of VSA College Portrait Pilot Future of Student Learning Outcomes on
College Portrait VSA as a communication tool
The Heat is On Unprecedented public challenges for higher
education to be more accountable for results. Accompanying external demands for
information about student and institutional performance are growing calls for institutions and accreditors to become more transparent about what they do and the results they achieve.(Ewell, 2010)
Overview of NILOANILOA’s mission is to document student learning outcomes assessment work, identify and disseminate best practices, and support institutions in their assessment efforts.SURVEYS ● WEB SCANS ● CASE STUDIES ● FOCUS GROUPS ● OCCASIONAL PAPERS ● WEBSITE ● RESOURCES ● NEWSLETTER ● LISTSERV ● PRESENTATIONS ● TRANSPARENCY FRAMEWORK ● FEATURED WEBSITES ● ACCREDITATION RESOURCES ● ASSESSMENT EVENT CALENDAR ● ASSESSMENT NEWS ● MEASURING QUALITY INVENTORY ● POLICY ANALYSIS ● ENVIRONMENTAL SCAN ● Degree Qualifications Profile
www.learningoutcomesassessment.org
www.learningoutcomeassessment.org
Reports on Transparency
Question What information does your institution
share publically on its website? How are results of student learning
shared internally?
Transparency Framework
http://www.learningoutcomesassessment.org/TransparencyFramework.htm
Transparent Communication
Internal communication External communication Telling an institutional story about
student learning
Questions What types of stories does your
institution tell about its students? What types of stories does your
institution tell about how it assures quality?
Background VSA launched in 2007 Partnership between APLU and AASCU 320 participating institutions Three original objectives
Demonstrate accountability and transparency
College search toolSupport innovation in the measurement of
student learning outcomes
Student Learning Outcomes Pilot
Student learning outcomes page includes link to institution specific learning outcomes data;
Selection of the CLA, CAAP or ETS Proficiency Profile to measure value-added learning gains
Report results no more than four years after joining
Link to information on test administration, sample and response rate
Pilot period ends 12/31/12 What was learned from the pilot?
Evaluate the Effectiveness of SLO Pilot
Focus groups Interviews Surveys Google Analytics College Portrait statistics
Findings related to Transparency Both VSA-participating and nonparticipating institutions
agree that VSA in 2007 was a wise, timely, useful, and necessary response to the accountability and transparency demands of the time
Yet, nearly half of the participants have not yet met expectations of College Portrait student learning outcomes pilot
45% of participating institutions have yet to post student learning outcomes information
Information that was provided on student learning outcomes lacked explanatory or contextual material
0 20 40 60
Student Experiences
Student Learning Outcomes
Majors, Graduation, & Next
Classes & Campus Life
Student Success and Progress
Admissions
Students
College Cost and Financial Aid
College Cost Estimator
Percentage of Total Views
82% of participating institutions had traffic
on their student learning pages but almost 1 in 5 did not have a single visitor to their student learning outcomes page since 2009.
Information posted on the College Portrait may not reflect the needs of prospective students and families or provide the information they actually seek to make decisions about where to attend college.
The authorized standardized test measures of student learning outcomes lack broad credibility and acceptance in the higher education community – it is thought the tests do not accurately reflect student performance within the institution.
Concerns about displaying results include: Inability to understand what scores
represent Questions regarding what tests measure Issues of reliability and validity Difficulty in using scores to improve
student learning Motivation of students & difficulty to obtain
a sample Questions regarding quality of results
81% of survey respondents stated that the pilot requirements did not align with their assessment practices and that the results were not usable for campus improvement efforts
Participating institutions agree -
expanding the number & nature of student learning measures will:
produce more accurate portrayals of student attainment,
provide more useful information for campuses, make information more meaningful for all
audiences.
Recommendations Focus on specific audiences and
communicating meaningful information to them
Contextualize information Expand range of assessment tools and
approaches Increase traffic to site Include examples of use of evidence to
improve learning Consider creating template which could be
used by all postsecondary institutions
Working Group Deliberations Technical work group considered
alternative measures of learning outcomes to recommend to VSA Board
Confirmed importance of student learning outcomes reporting within the VSA
No perfect of measure of students learning exists for all audiences External accountability Institutional improvement College selection
Working Group Deliberations Make value-add measurement using
existing measures (CAP, CLA, ETS Proficiency Profiler) optional
Introduce option to use VALUE Rubrics based on AAC&U essential learning outcomes (written communication and critical thinking)
Introduce option to use aggregate scores from professional and graduate admissions exams such as the GRE, GMAT, LSAT, and MCAT
Work Group Ratings of Recommended VSA Instruments in Key Areas
Instrument1= lowest rating 10=highest rating
CLA CAAP ETS PP(short form)
ETS PP (long form)
VALUE Rubrics GRE
Allow comparisons, benchmarking across institutions? 6 8 9.5 9.5 6 7
Offer a representative sample? 7 7 7 7 7 2 Usefulness to faculty/students for learning improvement?
inst-2class-7
5 2 2 9 3
Transparency of method, results 6 8 8 8 9 8
External accountability 8 7 7 7 5 5.5
Ease of administration on campus 5 6 7 6 4 9.9
Reasonable costs (time, resources) 5 7 7 6 startup -3 marginal- 8
9
Motivation for students to do well? 3 3 3 3 9 9.9 Likely interest to consumers? 2 2 2 2 5 6
Valid learning outcomes instrument for average student? 5.5 6 6 6 portfolios -9
assign -7 3
VSA Board Decisions College Portrait re-focus from “college selection tool for
prospective students” to “consumer information tool for institutions to demonstrate effectiveness of educational programs.” Primary audiences: state officials, policy makers,
consumers Instrument options:
CAAP, CLA, ETS Proficiency Profile AAC&U VALUE rubrics – critical thinking, written
communication GRE General Test
Reporting options: value-added, benchmarking
Implementation/Next Steps All VSA institutions report results of pilot by
end of 2012 to fulfill reporting requirement 3 part report with tabbed display
Did you participate (or try to)? If so, how’d it go? If not, why not?
Give an example of another institutional-level evaluation
Which of the new options are you considering for the future?
Reporting Format Preview: http://www.collegeportraits.org/slo_preview.html
Implementation/Next Steps Communications Task Force working with College
Portrait Support Team on display of new options Emphasis on more graphical displays Additional guidance on what the scores and results
mean Click-through links for additional details, e.g., subscores
New options released for 2012-13 data cycle (Jan 2013)
Develop outreach strategy to recruit new participants and connect with key audiences – state officials, accreditors
MORE INFORMATIONWebsites www.voluntarysystem.orgwww.collegeportraits.orgwww.learningoutcomesassessment.org
Teri Hinds: [email protected] Jankowski: [email protected]
Let’s Discuss.