Embed Size (px)
Citation preview
Program Quality AssuranceAlgonquin College of Applied Arts and Technology
Program Quality Review Manual
Ontario College Credentials
2020-2021
PREFACEThis Program Quality Review (PQR) manual is written to aid Program Quality Reviews for Ontario College Certificate, Diploma, Advanced Diploma and Graduate Certificate programs offered at Algonquin College.
It serves as the master reference document for Program Quality Reviews.
Adaptations are provided separately for Apprenticeship programs, Algonquin College Certificates and Bachelor’s degrees.
Need Assistance?
Visit the Program Quality Review website or contact the Program Quality Assurance Team or Academic Development at any time with questions about Program Quality Review.
Laura Jarrell, Program Quality Assurance Administrator (Acting) [email protected]
Mao Keo, Program Quality Review Specialist Jackson Chafe, Program Quality Review Specialist Maggie Cusson, Dean Academic Development
2
ContentsPREFACE......................................................................................................................................2
PROGRAM QUALITY ASSURANCE – AN OVERVIEW...............................................................5
PROGRAM QUALITY ASSURANCE AT ALGONQUIN COLLEGE..............................................7
Annual Curriculum Review.........................................................................................................7
Program Mix Review..................................................................................................................7
Program Quality Review............................................................................................................7
Cyclical Review of Online Elective General Education courses................................................7
PROGRAM QUALITY REVIEW....................................................................................................8
PQR Goals.................................................................................................................................8
PQR Schedule...........................................................................................................................8
PQR Release Time and Expenses............................................................................................8
PQR Timeline.............................................................................................................................9
PQR Process Overview...........................................................................................................10
PQR REVIEW ACTIVITIES.........................................................................................................11
Document Review....................................................................................................................11
Program Advisory Committee Meeting Minutes.............................................................11
Program Council Meeting Minutes...................................................................................11
Reports from External Accreditation or Licensing Bodies............................................11
Program Monograph..........................................................................................................12
Data Review.............................................................................................................................12
Student Satisfaction Survey..............................................................................................12
Graduate Outcomes and Graduate Satisfaction Survey................................................13
KPI Student Satisfaction – Capstone...............................................................................13
KPI Student Satisfaction – All Questions........................................................................13
KPI Graduate & Employer Satisfaction – Capstone Questions.....................................14
OCAS Program Choice......................................................................................................14
Admissions Summary........................................................................................................14
Registration Milestones by Program................................................................................15
Program Withdrawals........................................................................................................15
Course Failure Rate...........................................................................................................15
Costing Year Detail by Program.......................................................................................15
3
Curriculum Review...................................................................................................................16
Vocational Learning Outcomes........................................................................................16
Essential Employability Skills...........................................................................................16
General Education Requirement.......................................................................................17
Curriculum Mapping..........................................................................................................17
Learning Activities and Assessments Review..........................................................................18
Stakeholder Engagement........................................................................................................18
Student Focus Group.........................................................................................................19
External Stakeholder Focus Group..................................................................................19
Faculty Focus Group.........................................................................................................19
Learner Experience Analysis...................................................................................................19
Human, Physical, Financial Resources Analysis.....................................................................20
PQR Roles...................................................................................................................................21
Program Quality Review Team Lead.......................................................................................21
Chair (also Academic Manager)..............................................................................................22
Program Quality Review Specialist..........................................................................................22
Program Quality Assurance Administrator...............................................................................22
Curriculum Consultant.............................................................................................................23
Dean.........................................................................................................................................23
Program Support Roles...........................................................................................................23
Program Quality Assurance Committee...................................................................................23
THE FINAL REPORT – AN OVERVIEW.....................................................................................25
Body of Report.........................................................................................................................25
Executive Summary...........................................................................................................25
Program Overview..............................................................................................................25
Program Governance.........................................................................................................25
Curriculum Review.............................................................................................................25
Learning Activities and Assessments..............................................................................26
Learner Experience............................................................................................................26
Human, Physical, and Financial Resources....................................................................26
Recommendations.............................................................................................................26
Implementation Plan..........................................................................................................26
The Appendices.......................................................................................................................26
4
PROGRAM QUALITY ASSURANCE – AN OVERVIEWOne of the prerequisites for granting greater autonomy to the Colleges of Applied Arts and Technology in Ontario to govern their own programs was the implementation of a process to assure program quality. The Ontario Colleges of Applied Arts and Technology Act, 2002 outlined two elements of this expectation:
quality assurance at the individual college level, and a self-regulating process at the system level
as noted within the Minister’s Binding Policy Directive, Framework for Programs of Instruction. The Program Quality Review process forms a major part of adherence to providing quality assurance at the College level, and is therefore essential to College operations.
At the system level, that is, at the level that encompasses all Colleges of Applied Arts and Technology (CAATs) in Ontario, a separate quality assurance process takes place; the College Quality Assurance Audit Process (CQAAP), formerly referred to as the Program Quality Assurance Process Audit (PQAPA). This audit is conducted through the Ontario College Quality Assurance Service, OCQAS1. OCQAS was established to provide effective and efficient mechanisms that ensure specific program quality and consistency standards are met by the CAATs in Ontario.
The following six quality standards describe the expectations of the OCQAS regarding quality assurance; the PQR evaluation criteria adhere to and meet these standards, as well as encompassing College specific requirements.
Quality Standard 1 (Program Quality Management System)Effective quality assurance mechanisms ensure the existence of a quality assurance system.
Quality Standard 2 (Vocational Learning Outcomes As The Centre OfProgram Development) Effective quality assurance mechanisms ensure vocational learning outcomes are at the centre of the program’s lifecycle activities.
Quality Standard 3 (Conformity with Government Requirements) Effective quality assurance mechanisms ensure the conformity of programs of study with relevant government requirements.
Quality Standard 4 (Program Delivery and Student Assessment) Effective quality assurance mechanisms ensure the quality of program delivery and student assessment.
1 http://www.ocqas.org/index-en.html
5
Quality Standard 5 (Existence, Monitoring And Communication Of Policies And Practices That Influence And Impact Academic Quality) Effective quality assurance mechanisms ensure the communication and monitoring of established academic policies and practices related to academic issues.
Quality Standard 6 (Availability and Allocation of College-wide Resources) Effective quality assurance mechanisms ensure the existence, availability and allocation of resources (human, physical, financial) and technological infrastructure to support student achievement of PVLOs.
For more information regarding quality assurance at the provincial level, please visit the OCQAS Quality Assurance website.
6
PROGRAM QUALITY ASSURANCE AT ALGONQUIN COLLEGEProgram Quality Assurance at Algonquin College consists of three processes:
Annual Curriculum Review Annual Program Mix Review Cyclical Program Quality Review
Additionally, there is a cyclical review of online elective General Education courses.
Annual Curriculum Review The Annual Curriculum Review (ACR) is an opportunity to make required changes to Programs of Study (POS) and Narrative Information to ensure the quality, relevancy and currency of programs in your area. Proposed ACR changes are program-driven and informed by outcomes from comprehensive program reviews, largely facilitated through Program Quality Review (PQR) activities. Some major drivers for proposed program changes include key performance indicator data (KPIs), the requirement to implement new program standard releases from the Ministry, and feedback from internal and external stakeholders, such as feedback and recommendations from Program Advisory Committee (PAC) members, students, faculty and program councils.
Program Mix Review Staff in the Academic Operations and Planning department compile data that reflect the Quality Measures for a program. Admissions and registration data, employment rates, capstone questions from Key Performance Indicators (KPI) and program costing data are reviewed. Programs not meeting college benchmarks are noted and strategies are developed by the Chair and Dean to improve the quality of the program and/or the financial contribution.
Program Quality Review Program Quality Review (PQR) ensures that each program goes through a comprehensive review, normally every five years. The PQR process builds on annual review activities by including analysis of three to five years of program data, and soliciting direct feedback from stakeholders and students into the program review. Each PQR results in an implementation plan containing resourced actions to maintain and improve the program over the next five years. This manual guides Team Leads when conducting a Program Quality Review.
Cyclical Review of Online Elective General Education courses Online General Education courses form an integral part of many programs offered at the College. General Education course review is supported by the Program Quality Assurance Committee established Fall 2018.
7
PROGRAM QUALITY REVIEWPQR Goals To assess the program against established quality criteria in order to determine whether the
program meets or exceeds College and Ontario College Quality Assurance Service (OCQAS) standards (for Ontario College Credentials) or Postsecondary Education Quality Assessment Board (PEQAB) standards (for Bachelors’ Degrees).
To establish and implement a quality improvement plan for the program, in response to the findings of the review.
PQR Schedule Programs are reviewed according to a defined schedule, normally every five years. The schedule is published on the Academic Development website and revised annually, in consultation with the Academic Departments. The timing of a PQR may be aligned with other Quality Assurance requirements, for example accreditations or consent renewals for bachelor’s degrees.
PQR Release Time and Expenses Team Leads need sufficient time to accomplish the duties and tasks associated with Program Quality Review.
Funding and SWF Allocation for PQRs
Credential SWF Funding Allocation (Up to)1 year programs: College CertificateOntario College CertificateOntario College Graduate Certificate
6 hours complementary time (1-term PQRs)
$7,000
2-3 year programs: Ontario College DiplomaOntario College Advanced Diploma
OCD: 9 hours over 2 termsOCAD: 12 hours over 2 terms3 hours course release + 3 hours prep in first term PLUS 3 hours complementary time in the second term– cluster PQRs of 3 programs or more may be assigned up to 6 hours complementary time in the second term
$10,500 (in cluster up to $14,000)
4 year programs: Honours Degree
15 hours over 3 terms3 hours course release + 3 hours prep in first term PLUS 6 hours complementary time in the second term (preparation of consent renewal application) PLUS 3 hours complementary time during the time of
$17,500
8
the site visit
Where an internal Team Lead is not available to conduct the PQR, an external Team Lead may be assigned to the role. Before the PQR commences, the Chair, in consultation with the PQAA, identifies a Team Lead and negotiates a contract with remuneration within the PQR budget allocation for the program under review. Note that the funds allotted may be less for smaller programs, for example for short College Approved programs. The PQAA provides details on the funds available for review of each program.
The up-to amounts listed above are also intended to cover costs associated with the following: Support Staff: to provide administrative support for meeting minutes Part-time Faculty: $25 per hour meeting rate for meetings and workshops attended
PQR expenses are charged to the program budget code with the exception of travel: Departments are compensated annually by Academic Development for the specified
release time.
PQR Timeline Individual work plans are maintained by the PQR Team Lead with assistance from the Program Quality Assurance Administrator (PQAA) as required.
Fall Course ReleaseTiming Event(s)May Confirmation of Programs to undergo PQR, Team Leads assigned
August/September Team Lead orientation
September to November Review activities
December Draft report due (submit to Chair and PQAA)January Submit final report to PQAA for review/approval by PQAA/Chair
and Dean.
Winter Course Release Timing Event(s)October Winter Team Leads finalized
January Team Lead orientation
January to March Review activities
March Draft report due (submit to Chair and PQAA)April Submit final report to PQAA for review/approval by PQAA/Chair
and Dean.
9
10
PQR Process Overview Program Quality Review follows a systematic process. This guide provides an overview of the main components of a PQR. Team Leads are provided with Workbooks that break down these components in greater detail and provide instructions to facilitate completion of the review.
After the Team Lead has been identified to conduct the review, information pertinent to the program is collected and analyzed. Team Leads review published and internal documentation, program data, and curriculum maps. Additionally, input is collected from faculty, students, and external stakeholders. The process is organized into four phases: Review, Engage, Analyse, and Present.
The review is complete when all findings and recommendations are captured in a draft report and presented for feedback. An implementation plan based on the recommendations carries forward from year to year, with an update on the actions provided to the Program Quality Assurance Administrator (PQAA) 12-18 months after the final report.
11
PQR REVIEW ACTIVITIES
Document Review Team Leads review program documentation, in conjunction with the Chair and faculty as appropriate, to ensure that the information is complete, accurate, unambiguous, current, and relevant. The following documents are reviewed:
Program Advisory Committee Meeting Minutes Program Council Meeting Minutes Reports from External Accreditation or Licensing bodies Program Monograph/Webpages
Program Advisory Committee Meeting MinutesMinutes of Advisory Committee meetings for the past three to five years are reviewed to ensure that:
Program Advisory Committee meetings occur on a regular basis – at least once a term – and are in compliance with Policy AA01 – Advisory Committees.
Issues raised at Advisory Committee meetings are captured, and have an action plan to address these issues.
The review of the minutes may also help identify agenda items for the upcoming focus groups (student, faculty, external stakeholders). The Advisory Committee meeting minutes are included in the appendices of the final PQR report.
Program Council Meeting MinutesMinutes of Program Council meetings for the past three to five years are reviewed by the Team Lead to ensure that:
Program Council meetings occur on a regular basis – at least once a term – and are in compliance with Policy AA03 – Program Council.
Issues raised at Program Council meetings are captured, and have an action plan to address these issues.
There is a feedback mechanism in place to ensure that students are informed of what action, if any, will take place and the proposed timeline for any resolution.
The review of the minutes may also help identify agenda items for the upcoming focus groups. The Program Council meeting minutes are included in the appendices of the final PQR report.
Reports from External Accreditation or Licensing BodiesIf applicable to the program, information regarding formal relationships with external bodies of any kind should be reviewed. The report should include details of the arrangement and note student and graduate success in obtaining licenses and/or external accreditation.
12
Program MonographThe program monograph is reviewed annually as part of Annual Curriculum Review and is composed of the program narrative and program of study. This information is also published to program webpages and should be reviewed for accuracy and truthfulness. Elements of the program monograph to be reviewed include:
Admission Requirements should ensure that students are adequately prepared to enter the program (i.e. that they have the foundational knowledge needed on which to base their learning) without unnecessarily limiting access to a program.
College Eligibility is Ministry mandated and is usually standard for postsecondary programs - Ontario Secondary School Diploma, or at least 19 years of age.
Program Eligibility is determined by the department and should reflect any necessary knowledge and skills that the students must have prior to entering the program, for example a specific level of mathematics or science. Mature students may be required to meet the program eligibility even though they are 19 or older and do not have a high school diploma. Proven competency in English is a College-wide requirement.
Fees and Expenses should list all relevant program expenses to allow students to plan their financial commitment. Fees and Application Information are supplied by the Registrar’s Office.
The Program of Study is the foundation of the College’s contract with an applicant. It outlines the learning opportunities that student can expect in return for paying tuition.
The full program monograph is included in the appendices of the final PQR report, either copied or linked to a reference document.
Data Review Team Leads review program data, with support of the Chair, to gain an understanding of program trends. Reviewing data before focus groups aids with setting the agenda for the meetings.
Program data is collected by the Program Quality Assurance Team and provisioned to the Team Lead. The following are examples of data to be collected and analyzed as a result of the Program Quality Review Process.
Student Satisfaction Survey This survey is administered by an appointed group of administrators to all students in class during a reference week identified in October or February. All information gathered is handled confidentially. Students are asked to comment on experiences with their program, courses, teachers and the facilities/resources and services available.
13
The four KPI Student Satisfaction Survey capstone questions have remained unchanged for the past 8+ years:
KPI Capstone QuestionsOverall, your program is giving you the knowledge and skills that will be useful in your future careerThe overall quality of the learning experiences in this programThe overall quality of the services in the collegeThe overall quality of the facilities/resources in the college
Graduate Outcomes and Graduate Satisfaction Survey Approximately six months following graduation an independent consulting firm contacts graduates by telephone to conduct this system-wide survey. Graduates are asked to comment on their experiences at the college. They are also asked about employment and salary levels to assess the success of a program. All information gathered is handled confidentially.
The graduate is also asked for permission to speak to their employer (their front line supervisor). If the graduate grants the interviewer permission to contact their employer then the employer will receive a call within a month or less of contacting the graduate. If the graduate declines permission to the interviewer to contact their employer then the employer will not receive a call. Response rates tend to be low for employer satisfaction.
KPI Student Satisfaction – CapstoneThis report generates a line graph, bar graph, and table. The line graph displays the percentage of students who responded Very Satisfied or Satisfied for the overall program/college experience. Program data as well as the College average and provincial average responses are displayed. The bar graph is broken down into the four capstone questions of the Student Satisfaction survey. The table displays the same information numerically.
The four capstone statements: Overall, your program is giving you the knowledge and skills that will be useful in your
future career The overall quality of the learning experiences in this program The overall quality of the facilities/resources in the college The overall quality of the services in the college.
KPI Student Satisfaction – All Questions This report displays details of student satisfaction responses per year. When responses are compared over the past five years, trends and areas for further exploration during the PQR may be observed.
14
KPI Graduate & Employer Satisfaction – Capstone Questions This report tabulates the graduate satisfaction and employer satisfaction rates, and presents key items such as employment rate. Sample sizes are included. Aggregate numbers are displayed as a line graph and numerical data in a table.
Graduate Satisfaction Percentage: Percentage of students who rated Very Satisfied or Satisfied for the usefulness of the college education in achieving goals after graduation.
Total Employment: Percentage of students who are employed six months after graduation from their college education.
Related Employment: Percentage of students who responded Yes or Yes, partially to related employment after graduation. This question is only asked of students who are currently employed.
Employer Satisfaction: Percentage of employers who answered Very Satisfied or Satisfied for the graduate’s overall college preparation for current employment.When considering the information from Employer and Graduate surveys, take into account that the trends may not be as valid as those from the Student Survey - there are lower numbers of respondents and students who are working successfully are more willing to have employers contacted than students who may not be performing well.
For detailed information on graduates, and comparable statistics for programs at other colleges, see the MTCU Employment Profile report at http://www.tcu.gov.on.ca/eng/labourmarket/employmentprofiles/ .
OCAS Program Choice The OCAS program choice report shows the number of registered students by their program choice. It may assist with identifying why students have withdrawn from level 01. For example, a program with a significant number of 4th and 5th choice applicants may have higher withdrawal rates.
Occasionally, values of 0 or 6 appear in the program choice report. Where a 0 is present, this means that the student did not apply via OCAS. Where a 6 is present, this means the student is enrolled in a program that was not on their OCAS submission.
Admissions Summary This report displays information about applications to the program. The most relevant categories to PQR are waitlist, offer, confirmation, registration, and melt rate. The melt rate is equal to the number of confirmed students minus registered, divided by confirmed (= confirmation-registration/confirmation). Trends in demand for the program may be observed.
Registration Milestones by Program This report displays actual registration numbers for Day 1, Day 10, the audit date, and final day by term and level. The number of international students is also displayed. The Intake / Projected
15
Enrolment column is the ratio of actual registration (full-time only) to the projected enrolment (full-time only) on the audit date expressed as a percentage. Projected enrolment is used for planning financial, human and space resources. Ideally, the intake / projected enrolment should run close to 100% - significant deviation either way results in recalculation of the budget and related items during the academic year.
Program Withdrawals This report shows the number of program withdrawals by term and the related reasons for withdrawal. It is important to review the reasons given to see if there are any patterns. Note that students do not always fill out the standard withdrawal form and for those that do, the reason given may not be the real reason for withdrawing.
Course Failure Rate This report generates the number of students (registration count on the audit date) who received an F grade. The failure threshold is set by default to 20% though this may be adjusted depending on the program. This report is only produced when five or more students are registered in the course for the term. In some cases, the report will indicate high failure rates as a result of a course being offered off-cycle for a few students who had previously dropped or failed a course, and did not successfully complete the course in the off-cycle term.
The Failure Rate report may identify issues regarding student progression and success. For example:
Are admission requirements adequate to allow the students to be successful (especially for level 01 courses)?
Are the courses sequenced to ensure the students are taught foundational information on which to build their learning?
Are students allowed to progress in the program after failing a course when the failed course should be considered a prerequisite for the next level?
Are the learning materials, the learning activities, and the evaluation methodologies appropriate to allow students with a variety of learning styles to learn the material and demonstrate the course learning requirements?
Is the content or the assessment more difficult than it need be?
Costing Year Detail by ProgramThis report provides a multi-year program contributions trend chart, along with costing figures for the program for a given fiscal year. The report allows drill-through to further data, including space cost, depreciation and enrolment. Details for other fiscal years is accessed via drop-down menu.
Up to five years of data pertaining to the net financial contribution of the program is provided where available. Except in the initial set-up years, while the program is being established, programs are expected to meet a net contribution after overhead of 25%.
16
Curriculum Review Analyzing the program’s curriculum is an essential part of PQR and provides the opportunity to highlight program strengths and develop evidence-based plans and recommendations for curriculum revisions.
Vocational Learning OutcomesVocational learning outcomes represent culminating demonstrations of learning and achievement. They describe performances that demonstrate significant integrated learning by graduates of the program has been achieved and verified. Learning outcomes are interrelated and cannot be viewed in isolation of one another.
All Ontario College Credentials offered at Algonquin College include the institutional Sustainability and Internationalization VLO: “The graduate has reliably demonstrated the ability to identify and apply discipline-specific practices that contribute to the local and global community through social responsibility, economic commitment and environmental stewardship” within the program. The curriculum analysis will help assess progress against this requirement.
Essential Employability Skills Essential Employability Skills relate to the essential employability skills needed for both career and personal success – communication skills, numeracy skills, thinking and problem-solving skills, information management skills, interpersonal skills, and personal skills. They are expressed as learning outcomes and included on course outlines. There are eleven Essential Employability Skills learning outcomes which apply to each program:
1. communicate clearly, concisely and correctly in the written, spoken, and visual form that fulfills the purpose and meets the needs of the audience.
2. respond to written, spoken, or visual messages in a manner that ensures effective communication.
3. execute mathematical operations accurately.4. apply a systematic approach to solve problems.5. use a variety of thinking skills to anticipate and solve problems.6. locate, select, organize, and document information using appropriate technology and
information systems.7. analyze, evaluate, and apply relevant information from a variety of sources.8. show respect for the diverse opinions, values, belief systems, and contributions of
others.9. interact with others in groups or teams in ways that contribute to effective working
relationships and the achievement of goals.10. manage the use of time and other resources to complete projects.11. take responsibility for one’s own actions, decisions, and consequences.
See the EES web page http://www.algonquincollege.com/ees/ or contact the PQAA for more information.
17
General Education RequirementThe General Education Requirement relates to the requirements for general education courses that provide all learners with choice and breadth of experience beyond the vocational areas. General education courses cover areas of general interest and are divided into five themes:
Theme 1 - Arts in Society Theme 2 - Civic Life Theme 3 - Social and Cultural Understanding Theme 4 - Personal Understanding Theme 5 - Science and Technology
See the General Education web page http://www.algonquincollege.com/gened/ or contact the PQAA for more information.
The Essential Employability Skills and General Education requirements apply only to Ontario College Certificate, Diploma and Advanced Diploma programs. Apprenticeship, Locally Approved College Certificate and Graduate Certificate programs are not required to meet these standards. Degree programs also have different standards to meet with respect to breadth courses. For details on each credential, refer to the Credentials Framework in Appendix B of this guide.
Curriculum MappingProgram outcome mapping is the process of documenting and analyzing the structure of a program. Its purpose is to communicate how courses work together to support learners in developing knowledge, skills and attitudes towards the Vocational Learning Outcomes (VLOs) or Essential Employability Skills (EESs)*. Program outcome maps, such as VLO and EES maps, serve as the link between the courses in a program and the program-level learning outcomes.
Curriculum maps have several purposes: To provide a visual representation of the program from a bird’s-eye-view. To communicate to students, faculty, administrators and external stakeholders how
program-level learning outcomes are achieved within a program. To verify that students have opportunities to learn fundamental knowledge, skills and
abilities before approaching advanced outcomes and then graduate-level outcomes. To serve as a review tool to analyze sufficient coverage of program-level learning
outcomes. To verify individual course contribution towards students’ achievement of program-level
learning outcomes. To foster faculty discussion and coordination of program curriculum through regular
collaborative review.
At Algonquin College, program outcome map information is stored in COMMS, the Course Outline Mapping and Management System. This system facilitates the review of various curriculum maps by generating reports at the program and course level. Currently, available maps/reports include
18
Essential Employability Skills Program Summary Essential Employability Skills by Program, Level and Course General Education map Program Overview Vocational Learning Outcome Program Summary Vocational Learning Outcomes by Program, Level and Course
More information on Program Curriculum Mapping, including how to review and interpret program maps and how to locate the maps in COMMS, can be found in the Program Outcome Mapping Resource on the Academic Development website.
The course-level learning outcomes—called Course Learning Requirements (CLRs) at Algonquin College—are mapped to the program-level learning outcomes (VLOs and EESs). This validates the program-level mapping and verifies that the course is contributing to the program-level learning outcomes in the way the program outcome maps indicate. More information on Course Curriculum Mapping can be found in the Course Outcome Mapping Resource on the Academic Development website.
Learning Activities and Assessments Review Team Leads review program activities related to student evaluation and assessment in support of AA13 – Evaluation of Student Learning. Other topics addressed are learning activities, feedback on student work, academic dishonesty and plagiarism. Prior Learning Assessment and Recognition (PLAR) exemptions are noted, as are pathways that facilitate student movement between programs or other educational institutes. Experiential learn activities are documented, including co-op and work-integrated learning (WIL) opportunities.
Stakeholder Engagement Focus groups are held to gather qualitative feedback from participants and are complementary to the quantitative data review. Three groups are consulted – faculty, external stakeholders, and students. Discussion questions for each group will vary but are generally chosen based on the data review and areas of concern specific to that group.
Focus groups are two hours long and depending on the group may be held in a conference room, classroom, or virtually. Team Leads plan and organize all focus groups. A neutral facilitator is required for the student focus group to protect student anonymity and encourage honest feedback. Team Leads can pair up and facilitate each others’ student focus groups. Alternatively, resources from Academic Development or the Academic Department can be assigned to facilitate focus groups. A note-taker is assigned by the Chair to record minutes of the session. Minutes are provided to the Team Lead to analyze and summarize for the report.
In some cases, a survey can be used as an alternative to conducting a focus group.
19
Student Focus GroupGathering feedback from students directly acts as an important complement to KPI data. The focus group setting allows students to elaborate on issues of concern. This often provides important context for KPI results and student feedback gathered through other channels. It is also an opportunity to collect comments on potential areas of change in the program, such as altering the program of study, adding a work term, etc.
External Stakeholder Focus GroupExternal stakeholders provide important insight into what employers are looking for in co-op students and graduates. External stakeholders may also have feedback about graduate success, the future of industry, and the program of study. Program Advisory Committee (PAC) members can be invited to participate, as can other employers and alumni. Faculty are not usually invited as this can create a conflict of interest.
Faculty Focus GroupThe Team Lead invites as many faculty members who teach in the program as possible, including part-time faculty and professors of service courses. Part-time faculty members will be reimbursed for attending this meeting as per College guidelines.
Ahead of the meeting, faculty are asked to complete a short survey. The survey results, as well as the data collected to-date, will inform the meeting agenda. The main agenda items for the faculty focus group meeting are:
Review program monograph/website and program promotional materials Review mapping of vocational learning outcomes Review of course outlines and program of study, including WIL, PLAR and Pathways Discussion of issues arising from the faculty survey, stakeholder feedback, and data
review Other program-specific topics Identification of recommendations for consideration
Learner Experience Analysis The learner experience section is a student-centred view of program quality. Building on the information collected in the Student Focus Group, the Team Leads re-reviews admission requirements and relevant program data that informs trends in registration, progression, retention, and student satisfaction.
Human, Physical, Financial Resources Analysis Learning environments and facilities such as labs, equipment, and technology are described. A summary of faculty experience and credentials is included, as well as a review by the Chair of the financial landscape of the program.
20
PQR RolesThe PQR Team Lead conducts the PQR, with assistance from the Chair, the PQAA, and other supporting roles in Academic Development and the Academic Department.
The Team Lead, faculty, and Chair are at the heart of the PQR process. The Program Quality Assurance Administrator, Program Quality Review Specialists, and Curriculum Consultants all play an active role in supporting the review. The Dean, Program Quality Assurance Committee, and Office of the SVPA have an oversight role. Program support resources in the academic department play a minor role. These roles are described in greater detail below.
Program Quality Review Team Lead The PQR Team Lead is normally a faculty member who is assigned to conduct Program Quality Review for the program in which they teach.
The responsibilities of the Team Lead include: Managing the PQR review work Attending meetings and orientation/training related to the Team Lead role Reporting to the Chair weekly to monthly on the progress of the Program Quality Review Convening meetings of the program faculty team Convening a Student Focus Group Convening an External Stakeholder / Advisory Committee Focus Group Facilitating Focus Groups for another program, as needed Analyzing data and incorporating the results of the analysis in the final PQR report Submitting the draft report to the Chair, who will provide support prior to presentation at
Program Quality Assurance Committee Providing feedback on the Program Quality Review process
21
Chair (also Academic Manager) The responsibilities of the Chair include:
Identifying Team Leads (it is preferred that a full-time professor or coordinator is assigned)
Providing release time for the Team Lead Leading and coaching the PQR work Meeting with the Team Lead early in the process to review the program data Meeting regularly with the Team Lead and monitoring progress Sending an email to all program faculty inviting them to participate in the PQR process Completing the Chair Workbook Assisting with planning and delivery of focus groups Acting as a facilitator for Focus Groups if needed Arranging note-taking and administrative support as needed Providing the relevant budget code for all expenses Providing feedback regarding the PQR process Collaborating in the development of the final recommendations and implementation plan Including PQR recommendations in their Performance Contract Facilitating the implementation of the final recommendations Tracking progress on the implementation plan, and preparing the submission of the PQR
Annual Follow Up Report until all recommendations have been implemented
Program Quality Review Specialist Program Quality Review Specialists collect information and data, guide program faculty in the review of their programs, and facilitate the review process.
The responsibilities of the Program Quality Review Specialist include: Collecting available data Preparing Workbooks Creating work plans and overseeing active reviews Meeting regularly with PQR Team Leads to monitor progress and provide guidance Administering surveys and supporting focus groups, as needed Assisting with report collation and providing guidance on completion and clarity
Program Quality Assurance Administrator Program Quality Reviews are coordinated by the Program Quality Assurance Administrator (PQAA), who reports to the Dean, Academic Development.
The responsibilities of the Program Quality Assurance Administrator include: Ensuring the College is compliant with Ministry requirements regarding Program Quality
Assurance Consulting with major academic constituencies College-wide regarding Program Quality
Review Planning and organizing orientation/training workshops for Team Leads Supporting the Program Quality Review process
22
Communicating regularly with PQR Team Leads during the review process to monitor progress and provide guidance
Communicating regularly with Chairs during the review process to monitor progress and provide guidance
Facilitating evaluation of the Program Quality Review process. Ensuring that the final report is signed off fully by the PQAA, Chair and Dean Arranging archiving of the final report; forward e-copies to the Team Lead, Chair, Dean,
and providing access to all reports to the SVPA Ensuring follow-up reports are completed
Curriculum Consultant The responsibilities of the Curriculum Consultant include:
Reviewing the curriculum mapping and information collected by the Program Quality Review Specialists
Meeting with the Team Lead and a member of the Program Quality Assurance support team to identify required changes to the curriculum and formulate recommendations for the final report
Dean The responsibilities of the Dean include:
Identifying programs for Program Quality Review Supporting the Program Quality Review process Reviewing and signing-off on the Program Quality Review report Following up regarding results and recommendations included in the report Reporting to the SVPA on the progress of Program Quality Review and regarding the
follow-up on the recommendations
Program Support Roles Other roles from the Academic Department may be required to support specific activities related to the PQR.
The responsibilities of program support roles may include: Focus group note-taking (required) Assembling Program Advisory Committee and Program Council minutes (Dean or
Chair’s Assistant) Scheduling focus groups (Dean or Chair’s Assistant) Facilitating focus groups (Student Success Specialist or Faculty Marketing Officer)
Program Quality Assurance Committee The Program Quality Assurance Committee was established in Fall 2018 to support Quality Assurance and Program Quality Review. The committee reviews and comments on the final PQR reports and makes recommendations related to clarity, content, and the implementation plan. More information about the Program Quality Assurance Committee, its mandate, and its membership can be found on the Academic Development website.
23
24
THE FINAL REPORT – AN OVERVIEWThe final report is an extensive document that captures the findings of all review activities. The key elements of the body of the report are analyses, summaries, recommendations, and an implementation plan. Detailed documentation, such as data sets, curriculum maps, and survey results are held in the appendices.
Team Leads are strongly encouraged to capture as much detail as possible in the workbooks to facilitate the compilation and completion of the final report. For example, in the workbook when asked to describe the health of the Program Advisory Council, respond to the question with specific information and by using complete sentences. This content can then easily be copied and pasted into the final report.
The contents of the report must reflect College quality assurance principles and expectations. As such, the Chair and/or PQAA/PQRS may request substantial revisions if these requirements are not met. Once the PQAA, Chair, and Dean have approved the draft report, the Program Quality Assurance Committee (PQAC) will review the report. Once the report has been finalized, the PQAA, Chair, Dean and PQAC will sign-off on the approval of the report. The master copy of the PQR is stored electronically by Academic Development. A hard-copy can be provided on request.
Body of Report
Executive SummaryThe Executive Summary prefaces the entire report. It is a one-page review of the highlights of the Program Quality Review and includes the purpose of the report, a brief history of the program and delivery information, key findings of the review, and a brief summary of the conclusions and recommendations.
Program OverviewThis section provides basic information about the program, a summary of the recommendations from the previous PQR, and how the program meets the strategic direction of the College and the Algonquin Experience.
Program GovernanceThis section documents the activities of the Program Advisory Committees (PAC) and the Student Program Council. Also included is feedback from the external stakeholder focus group, and where applicable, descriptions of regulatory or licensing requirements maintained or pursued by the program.
Curriculum ReviewThis section documents program Vocational Learning Outcomes (VLOs), the program of study (POS), VLO mapping, Essential Employability Skills (EES) mapping, General Education (Gen Ed) course review, and course learning requirements.
25
Learning Activities and Assessments This section will addresses student evaluation and assessment in support of AA13 – Evaluation of Student Learning. Other topics addressed are learning activities, feedback on student work, academic dishonesty and plagiarism. Prior Learning Assessment and Recognition (PLAR) exemptions are noted, as are pathways that facilitate student movement between programs or other educational institutes. Experiential learn activities are document, including co-op and work-integrated learning (WIL) opportunities.
Learner ExperienceThis section focuses on the learner experience and includes a review of admission requirements, and program data that informs trends in registration, progression, retention, and student satisfaction.
Human, Physical, and Financial ResourcesThis section includes faculty feedback, a description of support services and personnel, program use of space, faculty credentials and experience, and a review of program costing.
Recommendations The recommendations are the improvement plan for the program for the next five years. Careful attention should be paid to developing meaningful and feasible recommendations that are directly connected to the evidence collected as part of the review process; what action needs to be taken and why? Articulate how will these recommendations, independently and as a collective, will contribute to improved program quality.
Recommendations can be organized by report category or by theme.
Implementation PlanFinal recommendations are listed and broken down into action items, typically between one and five in number. Each action item must be sufficiently observable and measurable so that completion may be easily determined. The implementation plan is a living document – it is carried forward year to year, and updated annually until action items are completed or closed.
The Appendices The appendices hold information and documentation collected during the review process. A detailed list of appendices can be found in the report template.
26
