Policy Audits to Advance Quality

Collaboratives Goals

A QC Webinar July 7, 2014

Debra Humphreys, VP for Policy and Public Engagement, AAC&U

Judith Ramaley, Senior Advisor, AAC&U

Peter T. Ewell, Vice President, NCHEMS

Karen Paulson, Senior Associate, NCHEMS

Ken Sauer, Senior Associate Commissioner and Chief Academic Officer,

IN Commission on Higher Education

Quality Collaboratives Goals and Strands Goal: Use the LEAP/DQP framework of proficiencies to design more effective ways to assess student learning and assure students’ demonstrated achievement of key learning outcomes, including the context of transfer. • Assessment—develop new tools and strategies that can be used to track, demonstrate

and report on students’ achievement of DQP proficiencies across levels of learning

• Policy—expand frameworks for documenting and tracking student success that include more than just persistence and graduation, but also indicators of demonstrated achievement of DQP proficiencies or other system-wide consensus learning outcomes

• Faculty Leadership and Development—advance campus practices and policies that foster faculty leadership and capacity to use the DQP to map curricular pathways and to foster, align, and assess expected learning outcomes across both general and field-specific areas of learning.

Learning in an Increasingly Connected Age: Impact on the Policy Environment • In today’s environment, people, resources, experiences, diverse content and

opinions, and communities are being connected together in diverse ways.

• The QC project has unfolded as this environment is beginning to take shape.

• New connections are leading to new pathways where learners can see their

progress and can be clear about what comes next.

• The emphasis will be on “pathways not on gateways or gatekeeping.”

• New forms of connectivity will drive changes in both policy and practice.

Source: Diane Oblinger (2013) Higher Education in the Connected Age. EDUCAUSE Review, September/October

2013 p. 4-6

Core Principle

Good educational and assessment practices and shared learning goals should drive development of wise policies rather than policies shaping (and, sometimes, inhibiting) the implementation of good educational practice.

QC Final Publications Final Project Report—focused primarily on new assessment practices and approaches to cross-institutional faculty capacity building and leadership for mapping curricula, developing assignments, and assessing proficiencies in the context of transfer.

QC Final Publications

Shorter Policy Report— “Toward a New Quality Assurance Framework”

The QC policy report will include recommendations for collection and reporting

of data on: learning outcomes; mapping of outcomes to programs; participation

in high-impact practices, faculty and student engagement, and assessment data

focused on students’ demonstrated achievement in authentic curricular-based

work.

The QC policy report also will pose critical questions that should be addressed

in policy and practice and provide examples of how institutions can design or

revise campus policies to ensure that educational practices and student

experiences are aligned with learning goals and to advance and support the

equitable achievement of educational outcomes for all students, including in the

context of transfer.

QC Short Policy Report

Expectations and Outcomes Student Experience Faculty and Staff

Capacity/Structures

System/State/Institutional

Policy

Policy-makers’ and employers’

expectations of a graduate’s

knowledge/skills

The student transfer

experience

Classroom assignments and

assessments

Accreditation processes and

policies

Students’ expectations of their own

knowledge/skills at graduation

Student pathways,

preparedness

Co-curricular offerings System-level or legislative

committees, mandates, etc.

Faculty/administrators’ expectations

of what a student should know and

be able to do upon completion

Actual student learning,

progression of

knowledge/skills

Faculty/administrators’

assessment capacity

Data requested by policymakers

for accountability mechanisms or

incentives

21st century outcomes aligned with

economic and societal needs

Problem-based learning

and other HIPs

Assessments and data

necessary to improve

educational practice

Assessment policy and

mechanisms for data collection

Campus transfer policy

For example, are the following points in alignment?

The Need for a Policy Audit • Many current policies and practices directly or indirectly affect

transfer and student learning outcomes.

• Some are positive, some are negative, and the effects of others are unknown

• There is a need to systematically identify these and document their impact for action

• Policies operate at different levels of analysis.

• State/System

• Campus (2-year colleges and 4-year colleges)

• Curriculum or Program • Some “policies” are not written, but are cultural or perceptual.

A Policy Audit is: a systematic review of existing policies connected to the areas focused on by QC, such as:

• Transfer policies at the two-year institutions, four-year institutions, system-level, and state,

• Policies in support of faculty development,

• Policies governing assessment or placement,

• Institutional, system, and state policies that support needed changes in transfer and faculty culture and practice,

• Other policies that may indirectly affect transfer and learning assessment (e.g. faculty workload, proficiencies needed for particular programs, etc.)

Basic Principles and Questions A policy audit is a “force field” analysis: Does the policy facilitate successful transfer?

Does the policy inhibit successful transfer?

How, specifically, does it exert these influences? How can the policy be modified or addressed? Who “owns” the policy and what will it take to get them to change?

Who shares an interest in changing the policy? What additional effects might the desired change have on other parts of the system? Funding issues

Accreditation issues

Marketability/public perceptions

How much do you already know? (You may have already gathered much of the information needed without really knowing that you have)

Steps to a Policy Audit Collect initial data and information. What story do these data tell? Conduct “desk research” • Search for and review existing policies at all levels that

may positively or negatively influence individual and/or institutional behaviors.

• Interview individuals • Focus on individual and institutional behaviors that result

from existing policies • Try out “what if” scenarios with interviewees

Search for and Review Existing Policies Transfer policies at the two-year institutions, four-year institutions, system-level, and state-level Policies in support of faculty development Policies on assessment and placement Institutional, system, and state policies that support needed changes in transfer and faculty culture and practice

Other policies that may indirectly affect transfer and learning assessment (e.g. faculty workload, proficiencies needed for particular programs, etc.)

Potential Interviewees

• Faculty • Academic administrators • Transfer coordinators • Academic advisors • Transfer students • Curriculum coordinators • (Other student services, financial aid, finance that might

impinge upon or influence student transfer)

Interviews – 2 About an hour long Use data and information and results of “desk research” to frame the situation as you understand it • Does your description ring true? • Are there other or better data that would shed additional

light on the issues?

• Who has that data?

Interviews – 3

What policies or procedures influence their personal behavior or higher institutions’ behavior in the light of the description, “the story”? What policies or procedures assist them in/are barriers to achieving desired behaviors (or to improving performance in relation to the state or institution’s priorities)?

Interviews – 4

During interviews listen and probe for whether there are real policies (if so, ask for copies or where they are found) or perceptions of policies in place. For barriers, should some policies or procedures be eliminated completely or is a specific modification or clarification needed?

“What If” Scenarios

During interviews ask “what if” questions to test out the likely viability of a variety of policy options. This technique allows you to see where the points of leverage are and what is non-negotiable in this particular environment.

Steps to a Policy Audit (Continuing)

More “desk research” to follow up on information obtained from interviews Follow up on any policies or procedures that had not been included previously but were raised by interviewees. Track down additional policies or procedures involved and cited.

Steps to a Policy Audit (Conclusion)

Summarize and write up findings Determine follow up

• Suggest new policies

• Adapt existing policies/procedures

• “Undo” outdated formal statutes or policies

• Delete unneeded policies

• Expose myths about policies; reinforce reality

Scope of a Policy Audit Can be focused at the institutional, system, state level, or multiple levels – depends on your needs Can be conducted by people internal to project, external to project but within unit, or contracted out externally

• External people can be a useful “reality check” and give ideas about how other institutions have approached various policies

Can be completed with modest resources or more fully investigated using more and/or external interviewers

An Example of Policy Change from QC • Indiana Commission on Higher Education Partners with

Legislators on New Transfer Core Legislation;

• Legislation crafted to focus on outcomes rather than courses;

• Legislation delegates to faculty teams task of defining outcomes

and competencies; leaves to individual institutions to craft

specific curricular requirements (with credit limit);

• Sets context for QC discussion about transfer, focus on learning

outcomes, new approaches to assessment and assignment design

Contributing to QC Final Policy Report “Toward a New Quality Assurance Framework” As you examine program, institutional, system, and state-level policies relating to reporting of data (e.g. completion rates, transfer rates, post-graduation employment, etc.), are there recommendations we should make for: • New sources of data on quality of learning to accompany

completion data

• Policies that should change to allow for more use of better quality learning data in accountability reports

Questions and Comments Additional questions?: Peter@nchems.org Karen@nchems.org