Upload
wcet
View
844
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
LEARNING ASSESSMENT SYSTEMS THAT WORK!A FOLLOW-UP REPORT
23nd Annual Conference,Denver, COOctober 28, 2011
Jeff GrannGeorge LucasThayer Reed
Jennifer StephensModerator: Patricia Case
© 2008 Capella University - Confidential - Do not distribute 2
• Founded in 1993• Charter AQIP participant in 2000• Courses 100% online• For-profit university• Enrollment: 39,447 (average age 39)
• Doctoral: 31%• Master’s: 47%• Baccalaureate: 21%• Certificate: 1%
• Enrollment status: 94% part-time• Faculty: 1,398 (82% doctoral degree)
Capella University
© 2008 Capella University - Confidential - Do not distribute 3
Value Creation Model
© 2008 Capella University - Confidential - Do not distribute 4
Academic Program Definition
© 2008 Capella University - Confidential - Do not distribute 5
© 2008 Capella University - Confidential - Do not distribute 6
© 2008 Capella University - Confidential - Do not distribute 7
Online Course Design and Development
© 2008 Capella University - Confidential - Do not distribute 8
CRITERION
LINK
© 2008 Capella University - Confidential - Do not distribute 11
Fully-Embedded Assessment Model
• Extends outcome assessment throughout program to develop actionable leading indicators.
• Documents alignment of all scoring guide criteria to course competencies and program learning outcomes.
• Projected university completion by Q4 2012.
© 2008 Capella University - Confidential - Do not distribute 12
FEAM Worked Example
Principles of Organizational Communication
Course Competencies
1. Analyze business communication situations.
2. Analyze the interrelationships of communication within organizational systems.
3. Communicate effectively.
Pre FEAM1. Provides Henry advice.2. Identifies Henry's educational/training needs in regard to communication
competency.3. Compares the relationship between communication and motivation.4. Applies the theories of motivation.5. Applies the 10 related evaluations.
Post-FEAM1. Tailors coaching based on analysis of a direct report’s motivational needs. (3)2. Validates approach relative to the theories of motivation. (1)3. Predicts stakeholder consequences of communication. (2)4. Identifies educational/training needs based on limited work sample. (1)5. Analyzes peer evaluations relative to communication competencies. (2)
u03a1 Scoring Guide Criteria
© 2008 Capella University - Confidential - Do not distribute 13
© 2008 Capella University - Confidential - Do not distribute 14
Make learning matter broadly
• Implementing a fully-embedded assessment model (FEAM) has proven effective in:• Promoting a coherent learning experience.• Faculty collaboration.
• Students have a shared basis to view their learning experience:• As a long-term effort to demonstrate professionally-relevant
learning outcomes, not just a series of assignments and courses.• Faculty have a shared basis to establish a community of practice
around their program’s learning outcomes:• By sharing professional experiences, teaching practices,
instructional strategies, performance expectations, technical innovations, and illustrative case studies.
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
WHO WE ARE
American Public University System has been dedicated to educating those who serve since our doors first opened in 1991. The American Public University System (APUS) is a fully online system, encompassing both American Military University and American Public University. We are regionally accredited by the Higher Learning Commission (HLC) of the North Central Association.
“Educating Those Who Serve”
• Founded as American Military University (AMU) in
1991 by James P. Etter, a Marine Corps Officer
• In 2002, AMU expanded into the American Public University System (APUS), adding American Public University (APU)
• Nationally Accredited (DETC) in 1995
• Regionally Accredited (NCA) in 2006
• Serving over 96,000 students worldwide
Mission: To provide quality higher education with emphasis on educating the nation’s military and public service communities by offering respected, relevant, accessible and affordable, student-focused online programs, which prepare them for service and leadership in a diverse, global society.
STRATEGY:
• Focus on quality and affordability
• Graduate successful alumni who make important contributions to society and their professions
• Enhance America’s competitiveness by meaningfully increasing the population withcollege degrees
• Collaborate with other universities to create solutions to U.S. higher education challenges
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
2011: Celebrating 20 Years of Service and Leadership
Data Driven Culture
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
“Quality begins on the inside . . . and then works its way out.” -Bob Moawad
• Internal and external benchmarking
• Data tools for stakeholders - dashboards, program and course level fact books, data warehouse, student learning assessment reports.
NATIONALLY BENCHMARKED INSTRUMENTS
18
National Survey of Student
Engagement
Proficiency Profile Tests
Major Field Tests
Community of Inquiry
Survey
APUS utilizes a number of nationally benchmarked tests and validated instruments to measure student engagement, student satisfaction, and the achievement of student learning outcomes.
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
Direct and Indirect Measures of Assessment
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
“Experts often possess more data than judgment.” – Colin Powell
Regular Dissemination of Data to Stakeholders: Dashboards
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
“In the spirit of science, there really is no such thing as a "failed experiment." Any test that yields valid data is a valid test.” - Adam Savage
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
Regular Dissemination of Data to Stakeholders: Course and Program Level Fact Books
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
Student Learning Assessment Report: Alignment of Student Learning Outcomes
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
Student Learning Assessment Report: Culminating Experience
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
Student Learning Assessment Report: Key Course Embedded Assessments
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
Student Learning Assessment Report: Assessment Data
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
Student Learning Assessment Report: Checklist
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
Student Learning Assessment Report: Changes Based on Data
PROGRAM REVIEW PROCESS
External Review
Analysis
Program Review
Meeting
Three Year Plan
Follow-Up
Data Collection
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
“Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many”William A. Foster
“One of the great mistakes is to judge policies and programs by their intentions rather than their results.” Milton Friedman
Data Aggregation & Analysis
• Course books
• Electronic resources
• Learning strategies
Library and Learning Resources
• Student learning outcomes
• Instructional strategies
• Evaluation procedures
• Academic rigor
Curriculum Assessment
• Expert reviewer report
• Industry Advisory Council report
External Reviewer Feedback
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
• Analysis of faculty credentials and expertise to ensure breadth and diversity
Faculty
• Student demographic information
• Enrollment History
• Growth trends
Students
• Curricular Mapping
• Assessment measures
• Fact books
• Student learning assessment reports
• reports
Learning Outcomes Assessment
Data Aggregation & Analysis
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
• Program benchmarking with similar programs and institutions
Program Benchmarking
• Evaluation of findings
• Program recommendations
• Three year proposed strategic plan
Program Directory Summary
• Dean’s observations
• Meeting minutes
Review and Findings
Data Aggregation & Analysis
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
CONTACT INFORMATION
AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE
Jennifer Stephens Helm, [email protected]
Vice-President, Institutional Research and Assessment
Kaplan University
• Offering online programs since 2001• 58,000+ students online; 7,000+ students at campuses• For-profit• Certificates, Associates, Bachelors & Masters• Incoming students have average of four NCES-identified
risk factors• 1,000+ courses• 4 tracks; 14+ term starts per year; overlapping terms• Outcomes-based institution
So We do All this Assessment …
Measuring activities is usefulExample: How many curriculum changes did we make this year?
Assessing what is really going on is essentialExample: How many of those changes improved retention?
Requirement:Sustainable infrastructure to support assessment approach
Result:Meaningful analysis of data to generate actionable information
• Data collection on outcomes• Real-time assessment data
reporting to faculty• Consistency of outcome and rubric• Assignments mapped to outcomes
and rubrics
• Curriculum development• Structured content
development and management
• Centralized framework for consistency across courses
• Content available in a variety of formats
Assessment-Related Infrastructure
Outside VendorUniversity
Course Authoring
Reporting and Analytics Database
Outcome Repository
Outcomes Package
• Single source of truth for course outcomes and rubrics
• Course to program mapping of outcomes
• Course to course mapping• Versions of all outcomes
and rubrics
Course Delivery
• All course content
How do we know if we are
improving learning?
Sustainable Systems
Information at the Course Level
Information at the Institutional Level
Example: Improvement as defined by improvement in outcomes achievement levels and student retention
Source:
Thayer E. Reed, Jason Levin & Geri H. Malandra (2011): Closing the Assessment Loop by Design, Change: The Magazine of Higher Learning, 43:5, 44-52
Assessment and Academic Quality
at the Rockies
George LucasDirector of Academic Quality
University of the Rockies
Private, independent graduate school of the social and behavioral sciences.
Master's and Doctorate degree programs both on campus and online
Two schools including the School of Professional Psychology, and the School of Organizational Leadership
Clinical Specialization at the campus
Academic Quality and Assessmentat the Rockies are grounded in
Learning Outcomes.
Institutional
Programmatic
Course
Using established Learning Outcomes as a Foundation, University of the Rockies has:• Acquired application tools and resources,• Incorporated and leveraged them in a coordinated manner,• Aligned our Learning Outcomes, and (this quarter)• Developed and Launched a Comprehensive Assessment Plan
Institutional
Programmatic
Course
Our Mission
Two Schools w/Master and Doctoral Programs
Our Curriculum and Syllabi
What are the steps we’ve taken to reach this point?
What will we do in the coming months and in 2012?
2009
2010
2011
2012Gates Model
Doctor of Psychology “Gates” Model for Assessment
Admission Process
Courses
(early, mid, late)
Comprehensive
Internship
Dissertation Process
Post Doc and Post
Graduation Licensure
Practicum 1 Thru 5
Multiple Assessment Methods align with the student experience and learning context including but not limited to Holistic Assessment, Evaluations, Grades, Annual Review, Licensure Exams, etc.
2009
2010
2011
2012Gates Model
Curriculum Mapping
Curriculum Mapping and IRMA tables in
Learning Outcomes are identified in all Curriculum with our Development Template
A successful and comprehensive Curriculum Development Process
2009
2010
2011
2012Gates Model
“Convergence”Curriculum Mapping
Usage Statistics
Integrated Rubrics, Holistic and Analytic Assessment
Faculty Development/Accreditation
Mapping, Outcomes Alignment
Quality Assurance, Curriculum Support
Student Lifecycle
2009
2010
2011
2012Gates Model
“Convergence”
Continuous Improvement
Curriculum Mapping
Cycle of Continuous Improvement
www.ala.org
Assessment that is “done right” never ends!
2009
2010
2011
2012Gates Model
“Convergence”
Continuous Improvement
Curriculum Mapping
Fitting it all together…
Our ongoing goal is to assess and continually improve Institutional Quality
and improve the student experience.
Quality and Assessment belong together.
To continually improve, you must continually:
MeasureAssess
InnovateEvolve
Baseline Concepts of our Assessment Strategy:
University of the Rockies Assessment Plan:
Three dimensions:• Academic Achievement• Student Engagement• Faculty Engagement
Seven (w/potentially more) categoriesserving as data sources:
• Surveys• Campus Vue• WayPoint Outcomes• eCollege• Library Writing Center• Curriculum Development• Quality Matters ™
Assessment and
Quality “Coding Table”
Usage Statistics
Integrated Rubrics, Holistic and Analytic Assessment
Faculty Development/Accreditation
Mapping, Outcomes Alignment
Quality Assurance, Curriculum Support
Student Lifecycle
LEARNING ASSESSMENT SYSTEMS THAT WORK!A FOLLOW-UP REPORT
23nd Annual Conference,Denver, COOctober 28, 2011
Jeff GrannGeorge LucasThayer Reed
Jennifer StephensModerator: Patricia Case