Academic Auditor Guide 2012-2013 · of Regents (TBR) system. The 2012-2013 academic year marks the ninth cycle of the Academic Audit initiative in the TBR. Since its inception during

A Guide for the Academic Audit Auditor

2012-2013 Cohort

Tennessee Board of Regents

A Guide for the Academic Auditor 2012-13...

Page 2

CONTENTS

INTRODUCTION …...................................................................................................................................3

OVERVIEW OF THE ACADEMIC AUDITOR PROCESS …..……………………………....................3

PART 1: FORMATION OF THE ACADEMIC AUDITOR TEAM …………………………...................4

PART 2: PREPARATIONS PRIOR TO THE SITE VISIT .........................................................................5

PART 3: THE SITE VISIT EXPERIENCE ……………………….……….………...................................6

PART 4: POST SITE VISIT RESPONSIBILITIES ....................................................................................7

PART 5: SAMPLES AND SCENARIOS ………….…………………………….…….........................….8

APPENDIX A: SUGGESTED ACADEMIC AUDIT SCHEDULE …..…..................................................9

APPENDIX B: ACADEMIC AUDIT ON-SITE EVALUATION CHECKLIST……...........................…10

APPENDIX C: ACADEMIC AUDIT VISITING TEAM REPORT/RECORD OF

COMMENDATIONS, AFFIRMATIONS, AND RECOMMENDATIONS ................................11

APPENDIX D: ACADEMIC AUDITORS REPORT GUIDELINES……………….…............................12

APPENDIX E: ACADEMIC AUDITOR TEAM LEADER RESPONSIBILITIES ..........................……14

APPENDIX F: ACADEMIC AUDIT SUMMARY SHEET FOR PERFORMANCE

FUNDING: UNDERGRADUATE PROGRAMS ..........................................................................17

APPENDIX G: ACADEMIC AUDIT SUMMARY SHEET FOR PERFORMANCE

FUNDING: GRADUATE PROGRAMS ........................................................................................20

APPENDIX H: FOLLOW-UP OF PREVIOUS ACADEMIC AUDIT .......................................................25

REFERENCES …..………………………………….…………………………………...........................…27


Page 3

INTRODUCTION

Which way to quality assurance in higher education? (Through) a new, mutually reinforcing system of institution-based quality assessments of teaching and learning and a coordinated regional system of external academic audits (Dill, 1996).

The commitment to quality assurance in higher education through the continuous improvement of teaching and learning processes lies at the heart of the Academic Audit initiative in the Tennessee Board of Regents (TBR) system. The 2012-2013 academic year marks the ninth cycle of the Academic Audit initiative in the TBR. Since its inception during the pilot year of 2004 – 2005, over 300 academic programs in TBR institutions have undergone the Academic Audit process. In support of the Academic Audit process, more than 400 faculty members and academic leaders from all 19 TBR campuses and other institutions have served as members of Academic Auditor Teams.

The importance of the auditor in the Academic Audit process cannot be overstated. Reasons cited by the director of the Academic Audit Unit in New Zealand for the acceptance of the academic audit process throughout its university system include:

Thoroughness of the auditor training;

Commitment and professionalism of the auditors leading to a rigorous but fair audit process;

The tone of the reports, with commendations as well as recommendations (Meade).

Auditor training has been a foundational element of the Academic Audit process from its inception, and the TBR is committed to conducting a thorough course of preparation for academic auditor team members and team leaders.

The Academic Audit, like more traditional program review process, is an academic auditor process including a self-study and a site visit by peers from outside the institution. The Academic Audit process emphasizes self-reflection and self-improvement rather than compliance with predetermined standards. The purpose of an academic audit is to encourage departments or programs to evaluate their “education quality processes” – the key faculty activities required to produce, assure, and regularly improve the quality of teaching and learning. An academic audit asks how faculty approach educational decision-making and how they organize their work, using the resources available to them and working collegially to provide a quality education in the best interests of the discipline and student learning.

OVERVIEW OF THE ACADEMIC AUDITOR TEAM PROCESS

The TBR Academic Audit program issues a handbook, updated annually, that provides the following description of the academic auditor team review process:

Academic auditors are volunteers (primarily faculty) who receive training on education quality

processes and academic audit methodology. Academic Auditor Teams (2-4 members) most likely but not exclusively come from other TBR

institutions. All departments are given the opportunity to nominate peers for service on their Academic

auditor Teams. Because the academic auditors will be focusing on quality processes, they do not have to come

from the academic discipline of the department being audited although TBR strives to have faculty from the discipline or a closely related discipline on the team.


Page 4

Academic auditor team visits are typically one day per department. Academic auditors meet with campus and departmental leadership, faculty, and students. Academic auditors ask questions similar to the self-study questions. Academic Auditors write a report:

o highlighting examples of exemplary practice, o noting areas for improvement, and o evaluating a department’s approach to educational quality practices.

The purpose of this training document is to further explore and explain these processes and functions.

PART 1: FORMATION OF THE ACADEMIC AUDITOR TEAM

Academic Auditor Team selection process

Each Academic Auditor Team is assembled by the TBR Academic Audit coordinator. The Team consists of a team leader and usually two team members for a total of three members. For some small programs, a two person team is appointed; for a few very large programs or a discipline that issues degrees at more than one level, a four person team may be formed. The TBR Academic Audit Coordinator will assign the team leader (chair) for each academic auditor team. Up to two team members may be nominated by the program undergoing review. These nominees should be from other institutions and not necessarily from the same discipline as the program undergoing review. Also, the nominees need NOT have previously served as members of an academic audit site review team. Training is provided to all members of Academic auditor Teams. If nominations are not made or if a nominee is unable to serve, the TBR Academic Audit Coordinator will identify team members from the pool of auditors.

Discipline-specific or not?

Every effort is made to identify one auditor from the discipline or from a closely related discipline for each program undergoing the academic auditor team review process. As the focus of the academic audit is on the processes - both in place and planned - to continuously improve the quality of teaching and learning, interdisciplinary participation is of great value. Educators share the common challenge of reaching, teaching and assessing students as they aspire to attain their educational goals. A precept of the academic audit process is that such a challenge is not discipline specific and that we can all learn from our colleagues who demonstrate the very best practices of teaching and learning in higher education: “…auditors need not be expert in any particular academic discipline. Disciplinary knowledge is not a prerequisite for conversations about education quality processes. Indeed, too much disciplinary knowledge tempts auditors to advocate their own preconceived views about how things should be done.” (Massy, 2003)

Characteristics of academic auditor team members

Academic auditor team members should be recognized as dedicated and talented faculty members and academic leaders. They may be subject matter experts whose discernment is highly valued, or they may be learning leaders who are prized for their insights into quality teaching and learning processes. Regional/distance considerations should not be the most important criterion for selecting team members. Partnerships and previous associations are acceptable as long as objectivity is not compromised. Also desirable are the abilities to:

Work cooperatively with team members,


Page 5

Communicate effectively in speaking and writing,

Apply a broad perspective in an open-minded fashion, and

Be willing and able to make a commitment of time and effort to this important process.

Commitments required of academic auditor team members

Service as an Academic auditor team member involves these commitments:

Thorough familiarization with the Academic Audit concept and process,

Participation in collaborative work sessions with other academic auditor team members before, during, and after the site visit,

Preparation of questions prior to the site visit, responses on the day of the site visit, and segments of a final written report after the site visit, and

Careful discernment of the strengths and opportunities for improvement in quality academic processes by the program undergoing the academic audit.

Conflicts of interest

Every attempt is made to avoid a conflict of interest between the academic auditor and the program/ institution undergoing the academic audit process. As there are extensive interactions between all professionals within the higher education system and especially within the TBR system, it is not expected that no prior interactions exist between an academic auditor and the program or institution. However, if there is any concern that excessive bias (either favorable or unfavorable) exists by the academic auditor towards the program or institution or by the program /institution towards the academic auditor, that academic auditor would be excused from service on that particular academic auditor team and replaced.

PART 2: PREPARATIONS PRIOR TO THE SITE VISIT

Success of overall academic auditor team experience will be determined in large part to both the level of preparation by the individual academic auditor team members as well as the level of collegiality between the team members. The following steps are encouraged to foster high performance by the academic auditor team:

1. All team members should become familiar or reacquainted with the academic audit process. They should review the Academic Audit Handbook to gain an understanding of the purpose of the Academic Audit process, the nature of the focal areas, the emphasis on “processes”, and the role of the academic auditor team. Those who have served on an academic auditor team previously and/or been a part of a self-study process should reflect judiciously on those experiences.

2. Once the academic auditor team is formed, team members should become acquainted through email exchange, teleconference, or, if possible, a face-to-face meeting. An auxiliary benefit to the academic audit process is the opportunity to build new relationships with colleagues and gain from one another’s experiences and insights.

3. When the self-study document written by the program to be reviewed is received (in early February), each team member should read the document from beginning to end without analysis or notation. The goal is to acquire an overview of the scope and nature of the


Page 6

program, its self-study procedure, and the processes it identified that are either in place or are planned to assure continuous quality improvement in teaching and learning.

4. Team members should also review the appendices for further perspective on the program. The self-study is limited by design to 20 pages for undergraduate programs. Additional information may be included in an appendix as well as via web page links.

5. All team members should clearly understand if the program being reviewed is using the academic audit process as its means to meet the program review requirement by THEC for Performance Funding credit. If so, team members should review the appropriate Performance Funding Summary Sheet (undergraduate or graduate) to prepare for discussions on those criteria with fellow team members.

6. During a second reading of the program’s self-study, each team member should make notes and draft questions that could be used at the site visit.

7. Communication between team members is essential. Organization of these communications will come from the team leader and may be in the form of teleconferences, email exchanges, web postings, etc. Each team member will likely be assigned specific focal areas on which to concentrate.

8. The academic auditor team should develop a proposed site visit schedule which the team leader will finalize with the academic audit campus coordinator at the program’s campus.

9. The team will collectively prepare questions for each set of conversations that will be held on the day of the site visit. These questions should address areas that may have been unclear, not fully explained or not discussed at all in the self-study document.

10. Team members must each make arrangements with their home institution for travel and any missed teaching assignments or other responsibilities for the time period spent on the site visit experience.

PART 3: THE SITE VISIT EXPERIENCE

“The site visit by the audit team sets in motion the academic auditor process…The role of the audit team is to set the tone for a collegial, supportive dialogue” (Massy, 2007). The site visit experience is a very full immersion in the academic auditor process beginning with the gathering of the academic auditor team at an area hotel the evening prior to the site visit date and ending the following afternoon with a report to the program. The success of the site visit is predicated upon preparation both by the academic auditor team and the program itself. Both parties agree upon a schedule well in advance. A sample schedule is included in Appendix A, but the actual schedule will be customized according to the specific nature of the program undergoing the site visit. Key elements of a successful site visit include the following actions by the academic auditor team and its leader:

1. On the evening before and/or the morning of the site visit day, the team should meet together off campus in order to cement collegiality; clarify each individual’s responsibilities; and share any lingering concerns regarding the self-study, appendices, or auxiliary materials referenced in the self-study (for example, postings on a program’s website). Team members should review the site visit schedule [see Appendix A for a sample schedule].

2. From arrival on campus on the morning of the site visit date and throughout the day, the team should make every effort to adhere to the schedule.


Page 7

3. During all conversations (meetings) with faculty, students, and/or other stakeholders, each team member should maintain the role of auditor meaning “listener.” Questions should be succinct without lengthy framing or anecdotal embellishment.

4. An effective way to validate responses is by repeating a question to other participants and/or at another session. For example, if faculty members are asked this question: “How do you determine if your web-based assessments measure student learning?”, then during the session with students, a few students could be asked, “How effectively do the web-based assessments measure what you are supposed to have learned?”

5. Each team member should take good notes. These will be invaluable at two pivotal junctures: a) that afternoon when the team prepares the lists of Commendations, Affirmations, and Recommendations; and b) when each team member writes a part of the academic auditor team report.

6. The host program may provide additional information on site that the team members should also review.

7. After all meetings are over, the team needs to complete the Academic Audit Onsite Evaluation Checklist [see Appendix B] and come to consensus regarding Commendations, Affirmations, and Recommendations [see Appendix C]. If there is a need to ask for information or to hold an additional conversation, this is permissible.

8. When the Performance Funding Summary Sheet is required for the program, the team must make decisions regarding the criteria and complete the Summary Sheet. The Summary Sheet asks for the name, title, institution, and signature of each team member, which should be issued on the form prior to departure [see Appendix D].

9. At the end of the day, the team collectively makes a presentation of about 15 minutes in length to the program faculty, administrators, and guests at which the Commendations, Affirmations, and Recommendations are given.

10. The team leader provides the original, signed copy of the Academic Audit Onsite Evaluation Checklist and, when necessary, the Performance Funding Summary Sheet to the program’s representative. [Note: the team leader and members keep copies one of which is subsequently included in the team’s report sent to the TBR.]

11. Before departing, team members should complete the travel forms and provide receipts for costs incurred related to the site visit to the host institution as team members will be reimbursed by the host institution.

12. Team members should depart with a plan and timetable for completing the written response by the academic auditor team. In several cases, academic auditor teams are able to draft a significant part of the final written report during the early afternoon work session prior to the oral presentation to the program.

PART 4: POST-SITE VISIT RESPONSIBILITIES

While the campus site visit marks the zenith of activity for the academic auditor team, several very important tasks remain. These require the cooperation of all team members.

1. First and foremost, the final written report of the Academic auditor Team needs to be completed in a timely manner but no later than May 17, 2013. While the team leader usually


Page 8

drafts the final document, team members will be expected to submit write-ups of assigned sections.

2. It may also be necessary for team members to participate in a follow-up conference call to address particular points that need clarification and consensus. The team leader may ask team members to review the final draft of the report for accuracy and completeness.

3. When the report is finalized, the team leader sends it to the Academic Audit Coordinator at the Tennessee Board of Regents along with a copy of the Summary Sheet when relevant. The TBR Coordinator will forward the final report to the host campus.

PART 5: ACADEMIC AUDITOR TEAM MEMBER TRAINING: SAMPLES AND SCENARIOS

This overview is intended to provide academic auditor team members (auditors) with a conceptual and pragmatic foundation from which they may engage the academic auditor process as individuals and as team members. The auditor training program will also include an interactive training session. At this session, auditors will have the opportunity to review segments of sample self-study reports; collaborate with peers to evaluate elements of a sample self-study; and formulate possible questions for a site visit conversation with faculty, students, and other stakeholders. A role-playing exercise will also be included in this part of the auditor training experience.

That is what learning is. You suddenly understand something you've understood all your life, but in a new way.

Doris Lessing


Page 9

Appendix A: Suggested Academic Audit Schedule Academic Program, Department and Institution

Date of Visit

Session Time/Attendees Location Breakfast and Team Meeting at Hotel 7:30am – 8:15am Hotel Audit Team Opening Session & Introductions 8:30am – 9:00am ABC Bldg. Room 1 Academic auditor Team – overview Faculty and Administrators Small Group Meeting #1 9:00am – 10:00am ABC Bldg. Room 2 Academic Auditor Team conversation with Faculty Members Small Group Meeting #2 10:15am – 11:00am ABC Bldg. Room 2 Academic Auditor Team conversation with Students Small Group Meeting #3 11:15am – 12:15pm ABC Bldg. Room 2 Academic Auditor Team conversation with

Faculty Members/other Stakeholders

Working Lunch 12:15pm – 1:30pm ABC Bldg, Room 2 Academic Auditor Team only Flexible Meeting and Work Time 1:30pm – 3:30pm ABC Bldg. Room 2 Academic Auditor Team -Possible time for materials review, tours, or additional meetings with faculty or students -Time for group to discuss observations and work on reports Exit Session 3:30pm – 4:00pm ABC Bldg, Room 1 Academic Auditor Team, Faculty and Administrators -Brief report of initial commendations, affirmations, and recommendations -NOTE: If the Performance Funding Summary Sheet is required, a copy of that sheet must be left

with the campus contact prior to departure.

Logistics Information Campus Contact for Academic Audit– Name, Position, Phone (office, cell), E-mail Hotel Accommodations – Hotel Name, Address, Phone


Page 10

Appendix B: Academic Audit Onsite Evaluation Checklist

Institution: _______________________________________________________________

Program: ________________________________________________________________

CIP Code: ________________________________________________________________

Degree Level: Certificate Associate Baccalaureate Master’s Doctoral

Instructions for Audit Chairs and Teams Part I: Academic Audit Visiting Team Report -- Record of Commendations, Affirmations, and

Recommendations

This form must be completed by each audit review team prior to concluding the visit. The original will be forwarded to TBR but a copy must be left with the department prior to departure. All observations included on this form should be represented as commendations, affirmations, or recommendations. Please be concise in your descriptions as you will have opportunity to expand upon your justification for each item in your written report due to TBR by May 17, 2013.

Part II: Academic Audit Summary Sheet (only for use if program is being reviewed for Performance Funding purposes)

This form is only to be completed if the program review is serving as the performance funding review. Using the Academic Audit Summary Sheet complete the 25 elements on the evaluation results checklist (marking “met” or “not met’). This exercise must be completed and signed by the team prior to the Exit Session. The original will be forwarded to TBR but a copy must be left with the department prior to departure.

Part III: Narrative Evaluation and Written Report

The Audit Chair and Team will use their evaluations indicated on the Audit Visiting Team Report and Academic Audit Summary Sheet (if used for Performance Funding purposes) as the basis of a written report. Summarized findings from the self-study report and on-site visit will represent a narrative report of the team’s conclusions and the final responsibility of the visiting team. The template for completing this report (limited to 10 pages) is attached. This report is due to TBR on May 17, 2013.

The Audit Evaluation will become part of the record of the academic program review and will be shared with the academic department/unit, the college, and the central administration, as well as the Tennessee Higher Education Commission. Each department/campus will be provided opportunity to respond and comment on the written report.

Audit Chair’s name, title, and institution: ______________________________________

Audit Chair’s signature: ___________________________ Date_____________________

Names, titles, institutions, and signatures of other Audit Team members:

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________

_________________________________________________________________________


Page 11

Appendix C: Academic Audit Visiting Team Report Record of Commendations, Affirmations, and Recommendations

This form must be completed by each audit review team prior to concluding the visit. All observations included on this form should be represented as commendations, affirmations, or recommendations. Please be concise in your descriptions as you will have opportunity to expand upon your justification for each item in your written report due to TBR by May 17, 2013.

This document should serve as the outline of information to be disclosed during the exit session with the department. The original signed copy is to be forwarded to TBR with one copy left with the campus audit contact or department chairperson prior to leaving campus.

******************************************************************

Total Number of Commendations ___

Commendation #1 –

Commendation #2 –

Commendation #3 –

Commendation #4 –

Total Number of Affirmations ___

Affirmation #1 –

Affirmation #2 –

Affirmation #3 –

Affirmation #4 –

******************************************************************

Total Number of Recommendations ___

Recommendation #1 –





Page 12

Appendix D: Audit Report Guidelines Updated 6/06

Template for Academic Audit Team’s Narrative Report

General Report Outline [NOTE: The report must be limited to ten (10) pages.]

I. Introduction -- Briefly describe the unit/program evaluated, the date of the Audit, the

protocol observed by the auditors, and other relevant information.

II. Overall Performance - What is the team’s overall summary conclusion about the state of the unit/program?

III. Performance in the Focal Areas – How does the unit/program’s work in each focal area measure up against the quality and evidentiary principles?

A. Learning Objectives B. Curriculum and Co-Curriculum C. Teaching and Learning Methods D. Student Learning Assessment E. Quality Assurance

IV. Conclusions. Briefly indicate the team’s conclusions regarding the following by providing each in bulleted or numbered sequence:

A. Commendations – What processes, practices, initiatives, and commitments are particularly commendable and merit recognition?

B. Affirmations – What processes, practices, or plans warrant the team’s affirmation and encouragement? The key idea of Affirmations is that the unit being audited identified the need for improvement before the Audit Team.

C. Recommendations – What are some areas for improvement identified by the team on the basis of the unit/program’s self-study and site visit?


Page 13

Key Themes and Tips for Writing Audit Reports

It is important for respondents to observe the following suggestions when developing the Audit Report.1

1. The purpose of the report is to summarize findings from the self-study report and the visit.

2. Keep the tone of the report developmental – focus on how the department can improve on its

own. Address what it will work on as a result of the review.

3. The report is the synthesis of the teaching and research focal areas pulled together by the

team leader.

4. Provide feedback on all five themes in focal areas in summary form and give overview

perceptions.

5. Reports are written by the audit team and assignments are made at the visit, and members

provide bullet-type comments.

6. Use the debriefing time to gather ideas from members and create an overview of the report.

7. At the visit, agree on a report format and how long each section should be. Find some

consensus from the group and develop a strategy for the report (e.g. bullet comments only).

8. At the visit there is an oral debriefing with the department/unit before the team arrives. The

team decides on overall points. Use these themes to provide the context for the report and

the executive summary.

9. Ask for evidence as you gather impressions. Don’t build summary points around comments

from one faculty member or one small group of students – look for confirming evidence.

10. Keep it simple and tell them what they did well as well as what processes they may want to

address to facilitate continuous improvement.

NOTE: Write the report so that it is helpful for the department, but also write it as if others will read it (i.e., be sensitive to language and tone of comments).

1 Key themes taken from materials provided by Dr. Bill Massy during work with Tennessee Board of Regents in the 2004-05 Pilot Phase of Academic Audit.


Page 14

Appendix E: Academic Auditor Team Leader Responsibilities

1) Pre-Site Visit

Status a) With the Campus Coordinator of the host site…

i) Decide upon hotel selection and related details (restaurants nearby, directions to campus, parking on campus, etc.)

ii) Share contact information including multiple contact numbers

iii) Set up the specific schedule for the site visit day

(1) Establish parameters for each session – with whom, how many, how long, etc.

(2) Make sure needs are addressed including computer access, printer access, visits to a particular program-related facility, coffee, etc.

b) With the Team Members… i) Make initial contact with Academic auditor Team

members via email or telephone

(1) Make sure everyone has the program’s Self Study and Appendices (if provided by the program)

(2) Confirm contact information including multiple phone numbers including cell number

(3) Set up a time for a teleconference to discuss each team member’s primary responsibilities. [NOTE: A typical arrangement is for one team member to focus on Learning Objectives and Curriculum/Co-curriculum; a second takes Teaching & Learning and Student Learning Assessment; and the team leader takes Quality Assurance and also responsibilities for the Introduction and Overall Performance sections.] Each team member takes the lead in her or his assigned areas for the following:

(a) Close reading of those sections of the Self Study

(b) Formulating questions for the onsite visits from those sections

(c) Writing the parts of the final Auditor Report for those sections

ii) Collect suggested questions for the site visit (via email) iii) Share a composite of these questions with team iv) Conduct one or more teleconferences to discuss

questions, concerns, strengths, etc. of the program with


Page 15

team members to finalize the common set of questions to be asked at the site visit

v) Prepare draft of Onsite Evaluation Checklist and Summary Sheet (if required for Performance Funding) to bring on laptop and/or travel-drive

vi) Share information with team members about practical preparations for the site visit [NOTE: Each team member makes her/his own travel request, hotel reservation, and transportation arrangements]

2) Site Visit a) Try to meet as a team either the evening before the site

visit – perhaps over dinner – or at breakfast – often at the host hotel – to get to know one another a little better, to review the site visit schedule, and to confirm roles.

b) Once on campus, guide the conversation sessions especially in terms of managing time and moving through the questions

c) After the morning sessions, lead the process for formulating Commendations, Affirmations, and Recommendations [NOTE: It is ideal if a team member volunteers to serve as scribe, perhaps bringing a laptop. Access to computer and printer should be provided at the site.]

d) Set up a schedule for each team member submitting her/his section of the final auditors report to the Team Leader

e) Be sure that the required forms are completed by the team: i) Checklist ii) Academic Audit Summary Sheet (for Performance

Funding programs only)

f) Give the final report at the end of the site visit day – all team members can participate in this if the team so chooses.

g) Give copies of the report to the program representatives, but retain the original, signed copies for submission to the TBR.

h) Remind team to submit mileage, hotel receipt, and any other expense directly related to the Site Visit to the host institution’s Campus Coordinator.


Page 16

.

For additional information contact:

Randy Schulte, Assistant Vice Chancellor for Academic Affairs Tennessee Board of Regents

1415 Murfreesboro Road – Suite 350 Nashville, TN 37217 – 2833

[email protected] Office: 615-365-1505 Cell: 423-424-6400

TBR Academic Affairs: 615-366-4482

3) Post Site Visit Status a) Writing the Auditors Report

i) Collect the sections written by other team members ii) Assemble these parts into the final report following the

report guidelines contained in the Guide

iii) Add the Commendations, Affirmations, and Recommendations to the final report

b) Submitting the Auditors Report i) Send the report itself as a PDF or Word file to

[email protected]

ii) Mail these original, signed documents to the address below on or before May 17, 2013

(1) Checklist (including Commendations, Affirmations and Recommendations)

(2) Summary Sheet for Performance Funding (only if applicable)


Page 17

Appendix F: Academic Audit Summary Sheet

2010-15 Performance Funding Cycle Appendix H: Academic Audit

Undergraduate Programs

Institution: _____________________________________________________________________________ Program Title(s): ________________________________________________________________________ CIP Code(s): ____________________________________________________________________________ Academic Audit Status: ______ First Academic Audit _____ Second Academic Audit

Instructions for Academic Audit Team:

In accordance with the 2010-15 Performance Funding guidelines of the Tennessee Higher Education Commission (THEC), each non-accreditable undergraduate program undergoes either an academic audit or external academic auditor according to a pre-approved review cycle. If the program under review contains embedded Technical Certificates, the names of each certificate should be included on the “Program Title” line above. The review of embedded certificates must be included as part of the review of the program in which they are embedded. Embedded certificates do not require a separate Summary Sheet.

The criteria used to evaluate a program appear in the following "Academic Audit Summary Sheet.” The Summary Sheet lists 26 criteria grouped into eight categories. THEC will use the criteria in categories 1-6 to assess Performance Funding Standard 1C when the Academic Audit process is used for programs undergoing the Academic Audit process for the first time. For programs undergoing the Academic Audit for the second time, criteria 7 (follow-up) will also be used to assess Standard 1C. The criteria in the eighth category, Support, may be used by the institution and submitted as part of the Performance Funding report. If the Academic Audit process did not include information about criteria 8.1 - 8.3, they should be marked N/A. These criteria will not be included in the THEC Performance Funding point calculation.

These criteria have been selected based on the Academic Audit Focal Areas to be consistent with the spirit and process of the Academic Audit. The program faculty has provided a self-study document that includes information for each criterion within the Focal Areas. Supporting documents will be available as specified in the self study. As the Academic Audit Team Leader, you should assess this and other evidence observed during the site visit to determine whether the process has met each criterion within a category. A checkmark should be placed in the appropriate box to indicate whether you believe that a program has “met” or “not met” each criterion in the table.

The Academic Audit Summary Sheet will be sent to the appropriate campus official for inclusion in the Annual Performance Funding Report. When combined with the self study and the written report prepared by the visiting team, the Summary Sheet will facilitate institutional development of a program action plan to ensure continuous quality improvement.

Your judgment of the criteria will be used in allocating state funds for the community college or university's budget.

Name, Title, and Institutional Affiliation of Academic Audit Team Leader (s):

______________________________________ ____________________________________ Name Name

_____________________________________ ____________________________________

Title Title

______________________________________ ____________________________________ Institution Institution

______________________________________ ____________________________________ Signature Date Signature Date


Page 18

Academic Audit Summary Sheet

Institution: ____________________________________________________________________

Program Title: _______________________________________________________________

Degree Level: __________________________________________________________________

CIP Code:______________________________________________________________________

Academic Audit Status: ______ First Academic Audit _____ Second Academic Audit

Evaluation Results

1. LEARNING OBJECTIVES Met Not Met

1.1 The faculty completed a thorough and candid analysis of their process for developing learning objectives for the program, considering measurability, clarity and what students need to know.

1.2

The faculty documented or proposed a process for developing learning objectives that are based on realistic and appropriate evidence.

1.3

The faculty documented or proposed specific plans to take best practices and appropriate benchmarks into account in the analysis of learning objectives.

2. CURRICULUM AND CO-CURRICULUM Met Not Met

2.1 The faculty completed a thorough and candid analysis of the extent to which they collaborate effectively on the design of curriculum and planned improvements.

2.2 The faculty documented or proposed a plan for analyzing the content and sequencing of courses in terms of achieving program learning objectives.

2.3 The faculty documented or proposed a plan for the ongoing review of curriculum and co-curriculum based on appropriate evidence including comparison with best practices where appropriate.

3. TEACHING AND LEARNING PROCESSES Met Not Met

3.1 The faculty completed a thorough and candid analysis of their process for guiding and improving teaching and learning throughout the program.

3.2

The faculty documented or proposed a plan that promotes the effective use of instructional methods and materials for achieving student mastery of learning objectives.

3.3 The faculty analyzed the extent to which there is true, ongoing collaboration in the design and delivery of the teaching and learning processes of the program.

4. STUDENT LEARNING ASSESSMENT Met Not Met

4.1 The faculty documented or proposed indicators of student learning success that are keyed to the learning objectives of the program.

4.2

The faculty documented or proposed assessments of student learning that are grounded in best practices and appropriate comparisons.

4.3

The faculty documented or proposed a plan for using student learning assessments that leads to continuous improvements in the program.

4.4

The faculty documented or proposed a continuous improvement plan that incorporates multiple measures to assess student learning and program effectiveness.


Page 19

5. QUALITY ASSURANCE Met Not Met

5.1 There is an evident commitment to making continuous quality improvements in the program a top priority.

5.2 The faculty documented or proposed a continuous improvement plan that incorporates multiple measures to assess student learning and program effectiveness.

6. OVERALL ASSESSMENT

Met Not Met

6.1 The Academic Audit process was faculty driven.

6.2 The Academic Audit process (self-study and visit) included descriptions of the program’s quality processes including all five focal areas.

6.3 The process resulted in a candid description of weaknesses in program processes and suggestions for improvements.

6.4 Overall, the visiting team affirms the openness and thoroughness of the program faculty in completing the academic audit of this program.

6.5 The Academic Audit process included involvement of and inputs from stakeholder groups identified by the program’s faculty.

7. FOLLOW-UP OF PREVIOUS AUDIT *

Met Not Met

7.1 An action plan was developed as a result of the previous Academic Audit.

7.2 There is documented evidence that recommendations made by the Academic Audit Team have been considered and, when feasible and appropriate, implemented and tracked.

7.3 There is documented evidence that the program has implemented and tracked the progress of and use of results from improvement initiatives cited by the faculty its self study.

8. SUPPORT (Note: The Support category is NOT included in the Performance Funding calculation. If the Academic Audit process did not address these criteria, they should be marked “NA.”)

Met Not Met

8.1 The program regularly evaluates its library, equipment and facilities, encouraging necessary improvements within the context of overall college resources.

8.2 The program's operating budget is consistent with the needs of the program.

8.3 The program has a history of enrollment and graduation rates sufficient to sustain high quality and cost-effectiveness.

* Criterion only included in the performance funding calculation for programs undergoing the Academic Audit during the 2010-2015 cycle that also used the Academic Audit in the 2005-10 cycle. Note: please be sure that the “Second Academic Audit” is checked on page 1.


Page 20

Appendix G: Academic Audit Summary Sheet: Graduate Programs

2010-15 Performance Funding Cycle Appendix I: Academic Audit

Graduate Programs

Institution: _____________________________________________________________________________ Program Title(s): ________________________________________________________________________ Degree Level(s): _________________________________________________________________________ CIP Code(s): ____________________________________________________________________________ Academic Audit Status: ______ First Academic Audit _____ Second Academic Audit

Instructions for Academic Audit Team:

In accordance with the 2010-15 Performance Funding guidelines of the Tennessee Higher Education Commission (THEC), each non-accreditable graduate program undergoes either an academic audit or external academic auditor according to a pre-approved review cycle.

The criteria used to evaluate a program appear in the following "Academic Audit Summary Sheet.” The Summary Sheet lists 55 items grouped into 14 categories. THEC will use the criteria in categories 1 – 13 to assess Performance Funding Standard 1C when the Academic Audit process is used for programs undergoing the Academic Audit process for the first time. For programs undergoing the Academic Audit for the second time, criteria in category 14 will also be used to assess Standard 1C.

These criteria have been selected based on the Academic Audit Focal Areas to be consistent with the spirit and process of the Academic Audit. The program faculty has provided a self-study document that includes information for each criterion within the Focal Areas. Supporting documents will be available as specified in the self study. As the Academic Audit Team Leader, you should assess this and other evidence observed during the site visit to determine whether the process has met each criterion within a category. A checkmark should be placed in the appropriate box to indicate whether you believe that a program has “met” or “not met” each criterion in the table. If a particular criterion is inappropriate or not applicable to the program, the criterion should be marked “NA”.

The Academic Audit Summary Sheet will be sent to the appropriate campus official for inclusion in the Annual Performance Funding Report. When combined with the self-study and the written report prepared by the visiting team, the Summary Sheet will facilitate institutional development of a program action plan to ensure continuous quality improvement.

Name, Title, and Institutional Affiliation of Academic Audit Team Leader(s):

_________________________________ ____________________________________ Name Name

_________________________________ ____________________________________

Title Title

___________________________________ ____________________________________ Institution Institution

___________________________________ ____________________________________ Signature Date Signature Date


Page 21

Academic Audit Summary Sheet

Institution: _____________________________________________________________________ Program Title(s): _______________________________________________________________ Degree Level(s):_________________________________________________________________ CIP Code(s):____________________________________________________________________

Academic Audit Status: ______ First Academic Audit _____ Second Academic Audit

Evaluation Results

1. LEARNING OBJECTIVES Met Not Met

1.1 The faculty completed a thorough and candid analysis of their process for developing learning objectives for the program, considering measurability, clarity and what students need to know.

1.2

The faculty documented or proposed a process for developing learning objectives that are based on realistic and appropriate evidence.

1.3 The faculty documented or proposed specific plans to take best practices and appropriate benchmarks into account in the analysis of learning objectives.

1.4

The faculty clearly communicates program objectives to current and potential students, employers or other stakeholders.

2. CURRICULUM AND CO-CURRICULUM Met Not Met

2.1 The faculty completed a thorough and candid analysis of the extent to which they collaborate effectively on the design of curriculum and planned improvements which will reflect attained competencies in the outcome data.

2.2

The faculty documented or proposed a plan for analyzing the content, format and sequencing of courses in terms of achieving program learning objectives with appropriate breadth and depth for the degree offered which allows for attainment of the degree in a timely manner.

2.3

The faculty documented or proposed a plan for determining the soundness of and rationale for curriculum and co-curriculum based on appropriate evidence, including comparison with best practices where appropriate, and communicate these views to the student body.

3. TEACHING AND LEARNING PROCESSES Met Not Met

3.1 The faculty examined the extent to which there is focus on and periodic, systematic review of the actual process of teaching and learning throughout the program.

3.2

The faculty documented or proposed a plan that promotes the effective use of instructional methods and materials for achieving student mastery of learning objectives.

3.3

The faculty analyzed the extent to which there is true, ongoing collaboration in the design and delivery of the teaching and learning processes of the program with reliance on best practices and resources beyond the confines of the program or department.

3.4 There is a critical mass of faculty and students to promote a scholarly community and assure an appropriate group of peers.

3.5 Faculty/graduate student ratio, average course load, average thesis/dissertation load per faculty and distribution across department, and teaching evaluations evidence support of graduate teaching and learning processes.


Page 22

3.6 The faculty documented or proposed a plan to inform students of course offerings and the provision of professional development activities and relevant courses to supplement departmental offerings in a timely fashion.

3.7 The faculty documented or proposed a plan to ensure that all students are adequately oriented, advised, mentored and socialized within the discipline and the larger graduate community.

4. STUDENT LEARNING ASSESSMENT Met Not Met

4.1 The faculty documented or proposed indicators of student learning success that are keyed to the learning objectives of the program.

4.2

The faculty documented or proposed assessments of student learning that are grounded in best practices and appropriate comparisons.

4.3

The faculty documented or proposed a periodically and systematically reviewed plan for using student learning assessments that leads to continuous improvements in the program.

4.4

The faculty documented or proposed a continuous improvement plan that incorporates multiple measures to assess student learning and program effectiveness.

5. QUALITY ASSURANCE Met Not Met

5.1

There is an evident commitment to making continuous quality improvements in the program (e.g., student advisement and mentoring, use of best practices, recognition of faculty performance, regular and systematic evaluation of student performance) a top priority.

5.2 The faculty documented or proposed ways to ensure that quality assurance will be a systematic and regular process for program improvement.

5.3 The coursework offers sufficient breadth and depth appropriate for the degree offered.

5.4 The faculty are documented to hold terminal degrees in the discipline in which they teach and have experience sufficient to serve as effective mentors for graduate students.

5.5 Data on current students and follow-up data on graduating students including placement data are regularly and systematically collected.

6. RESEARCH OUTCOMES

Met Not Met

6.1 The faculty documented or proposed a plan to ensure that there is a commitment to matching or exceeding comparable institutions in research activities.

6.2 The faculty documented or proposed a plan to assure sufficient depth and breadth in faculty research expertise to enable competitiveness in the external funding arena while allowing for collaboration when desired.

7. RESEARCH ENVIRONMENT

Met Not Met

7.1 The faculty documented or proposed a plan to ensure a commitment to communicate the program’s research environment, research values and priorities.

7.2 The faculty candidly and thoroughly examined the extent to which the department describes itself accurately and completely to current and prospective students and other “publics”.


Page 23

7.3 The department takes active steps to support both junior and senior faculty in remaining vital in their respective research areas.

7.4 The faculty documented or proposed a plan to ensure that departmental processes, policies and procedures positively influence faculty research activities and program competitiveness.

7.5 The faculty documented or proposed a plan that engages graduate students in inquiry and research in collaboration with faculty.

8. SYNERGY WITH EDUCATION

Met Not Met

8.1

The faculty documented or proposed a plan that honestly evaluates departmental resource demands in light of departmental research and scholarship’s contribution to its educational programs and the mission of the department, college and university.

8.2 There is a commitment to activities designed to keep the faculty and students informed on contemporary issues related to research (e.g., lecture series, responsible conduct of research workshops, professional development activities).

8.3 The program demonstrates best practices in integrating the science with the practice of the discipline.

8.4 The faculty candidly and thoroughly evaluated the extent to which they incorporate research into the educational programs in support of best practices.

8.5 The program demonstrates best practices in addressing workload demands of theses and dissertations supervision.

9. SPONSORED PROGRAMS Met Not Met

9.1 The faculty documented and proposed a plan to strive for sponsored research funding at comparable levels with other comparable departments within the institution and across peer institutions.

9.2 The faculty documented or proposed a plan to assure that faculty are consistently informed of external funding opportunities as well as the availability of assistance in areas such as proposal writing and project management.

10. QUALITY AND PRODUCTIVITY INDICATORS Met Not Met

10.1 There are appropriate indices endorsed by the program faculty as means to gauge faculty quality and productivity.

10.2 The faculty documented or proposed ways to ensure that quality research and productivity will be systematically and regularly examined across the faculty lifespan.

10.3

The culminating experience required by the program both in terms of comprehensives examination and/or research allows the student to demonstrate the breadth, depth and integration of the disciplinary coursework and experiences with the demonstration of communication skills and the ability to apply knowledge independently.

11. CONTRIBUTIONS TO PROGRAM, DEPARTMENTAL AND UNIVERSITY GOALS Met Not Met

11.1

There is a process in place which is communicated to other levels of the institution that evaluates the sufficiency of resources in place to meet the teaching responsibilities while actively engaging in research with graduate students and undergraduates.

11.2 The faculty documented and proposed a plan to encourage and support research


Page 24

outcomes congruent with the department’s purpose and the university mission.

11.3 The faculty clearly state and embrace appropriate admission standards, completion standards and graduation rates which are readily available to prospective and current students.

12. OVERALL ASSESSMENT Met Not Met

12.1 The Academic Audit process was faculty driven.

12.2 The Academic Audit process (self-study and visit) included descriptions of the program’s quality processes.

12.3 The process resulted in a candid description of weaknesses in program processes and suggestions for improvements.

12.4 Overall, the visiting team affirms the openness and thoroughness of the program faculty in completing the academic audit of this program.

13. SUPPORT Met Not Met

13.1 The program regularly evaluates its library, equipment and facilities, encouraging necessary improvements within the context of overall college resources.

13.2 The program's operating budget is consistent with the needs of the program.

13.3 The operating budget is sufficient to attract quality students and provides adequate support without substantially delaying progress toward the degree.

13.4 The program has a history of enrollment and graduation rates sufficient to sustain high quality and cost-effectiveness.

13.5

The operating budget is sufficient to allow faculty regular opportunities for professional development including travel and presentation of research findings, participation in professional organizations, workshops and other learning activities.

14. FOLLOW-UP OF PREVIOUS ACADEMIC AUDIT Met Not Met

14.1 An action plan was developed as a result of the previous Academic Audit.

14.2 There is documented evidence that Recommendations made by the Academic Auditor Team have been considered and, when feasible and appropriate, implemented and tracked.

14.3 There is documented evidence that the program has implemented and tracked the progress of and use of results from Improvement Initiatives cited by the faculty in its self study.


Page 25

Appendix H: Follow-up of Previous Academic Audit

During the 2010 – 2015 Strategic Planning and Performance Funding cycles, many academic

programs will undergo the Academic Audit process for the second time. The expectation of a program

undergoing the Academic Audit process for the second time is that it has acted upon the Initiatives for

Improvements that it set for itself after its first Academic Audit Self Study. Also, that it has responded to

the Recommendations made by the Academic Auditor Team that reviewed the program in its first

Academic Audit experience. Following is the text from the Academic Audit Handbook that provides

guidance to programs undergoing the Academic Audit process a second time regarding reporting its

follow-up processes and activities:

6. Follow-up of Previous Academic Audit [Only for programs undergoing the academic audit process for the second and subsequent times. Not to exceed 5 pages.]

This section of the self-study reflects upon two outcomes of the previous academic audit process: 1) Initiatives for Improvement that were generated by the program itself; and 2) Recommendations by the Academic Auditor Team. The expectation is that the program will have acted in a number of ways to advance improvement in these areas. Thus, a brief discussion of the process that the program followed to develop and enact its action plan is appropriate here.

It is expected that the Initiatives for Improvement that were generated by the program itself have been implemented and that progress on that initiative has been tracked. Further, it is expected that the results of those implementations have been considered and used to further improvement in the program. If the program has a detailed summary report for each initiative or a series of annual reports for each initiative, these do not need to be reiterated in the self-study report. Rather, those reports may be included as appendices. However, a brief narrative describing the process followed by the program to initiate and follow through with its initiatives would be appropriate in the self-study as would be a summary of the effectiveness of the initiatives.

Regarding Auditor Team Recommendations, the self-study report should address how each recommendation was considered by the faculty. If a recommendation was not pursued, then a brief explanation why it was not pursued should be included. If it was pursued as written or pursued in a modified manner, a similar narrative about its inception, follow-through, and effectiveness would be appropriate in the self-study. As with Initiatives for Improvement, if the program maintained detailed annual, periodic or summary reports regarding the implementation of a Recommendation, then these reports could be provided in the Appendix.


Page 26

The Academic Auditor Team is responsible for reviewing and assessing the processes and activities

of the program regarding its Initiatives for Improvement as well as its responses to the initial Academic

Auditor Team’s Recommendations. As noted above, “It is expected that the Initiatives for Improvement

that were generated by the program itself have been implemented and that progress on those initiatives

have been tracked. Further, it is expected that the results of those implementations have been considered

and used to further improvement in the program.” Auditor Team members should carefully review the

self study report and appendices for evidence that these expectations Initiatives for Improvement have

been addressed.

For Recommendations made by the initial Academic Auditor Team, the program may have acted

upon these. If the program has implemented a Recommendation, that implementation should be

documented in a fashion similar to the documentation of an Initiative for Improvement. If a

Recommendation has not been acted upon, it is expected that, “a brief explanation why it was not

pursued should be included.”

For programs undergoing the Academic Audit for the second time and are using the Academic

Audit to meet Performance Funding requirements, the Academic Auditor Team must complete Section 7

of the Undergraduate Summary Sheet for undergraduate programs or Section 14 of the Graduate

Summary Sheet for graduate programs. The three criteria in these sections provide guidance for the

overall review of the program’s follow-up of its previous Academic Audit.


Page 27

REFERENCES

Dill, D. (2000). Designing academic audit: lessons learned in Europe and Asia. Quality in Higher Education, 6, 187.

Dill, D., Massey, W., Williams, P. & Cook, C. (1996, September-October). Accreditation and academic quality assurance: can we get there from here? Change, September-October, 16.

Jennings, J. (2002). Academic audit manual for use by the New Zealand universities academic audit unit. Wellington, New Zealand: Wet Bock Education House.

Massy, W. (2003, June 20). Auditing higher education to improve quality, The Chronicle of Higher Education, B16.

Massy, W., Graham, S. & Short, P. (2007). Academic Quality Work: A Handbook for Improvement. Bolton, MA: Ankar Publishing Company.

Meade, P. & Woodhouse, D. (2000), Evaluating the effectiveness of the New Zealand academic audit unit: Review and outcomes, Quality in Higher Education, 6 (1).

Tennessee Board of Regents. (2007). Educational Quality Improvement: A Handbook for the Academic Audit. Nashville, TN: Author.

Documents

Academic Auditor Guide 2012-2013 · of Regents (TBR) system. The 2012-2013 academic year marks the ninth cycle of the Academic Audit initiative in the TBR. Since its inception during