MEMBERS OF THE TEACHING AND LEARNING COMMITTEE€¦ · Teaching Fellowship Scheme $88,000 Improving Student Learning Grants Scheme $30,000 Teaching and Learning Forum – Provision

Teaching and Learning Committee Agenda – March 2012 1

The University of Western Australia

27 February 2012 MEMBERS OF THE TEACHING AND LEARNING COMMITTEE Pro Vice-Chancellor (Education) (Winthrop Professor Jane Long) – Chair Associate Chair of the Academic Board (Winthrop Professor Linc Schmitt) Director, Centre for the Advancement of Teaching and Learning (Winthrop Professor Denise Chalmers) University Librarian and Director (Information Management) (Dr Mary Davies) President, Guild of Undergraduates (Mr Matthew Mckenzie) Coursework Student's Officer, Postgraduate Students’ Association (Ms Aisling Blackburne) Chair/Representative of each Faculty Teaching and Learning Committee:

Faculty of Architecture, Landscape and Visual Arts (Mr Philip Goldswain) Faculty of Arts, Humanities and Social Sciences (Dr Neil O’Sullivan) UWA Business School (Professor Philip Hancock) - Deputy Chair Faculty of Education (Assistant Professor Elaine Sharplin) Faculty of Engineering, Computing and Mathematics (Associate Professor Cara MacNish) Faculty of Law (Winthrop Professor Mark Israel) Faculty of Life and Physical Sciences (Associate Professor Peter Whipp) Faculty of Medicine, Dentistry and Health Sciences (Associate Professor Sandra Carr) Faculty of Natural and Agricultural Sciences (Dr Brenton Knott)

Nominee of the School of Indigenous Studies (Assistant Professor Jeromy Harvey) BY INVITATION (STANDING INVITEES) Mr Brian Greene, Associate Director, Policy and Planning, Information Technology Services (ITS) Dr Sato Juniper, Graduate Research and Scholarships Office Professor Sally Sandover, Co-ordinator of Regional Programs Dr Greg Marie, Director, Institutional Research Unit (IRU) Ms Siri Barrett-Lennard, English Language and Learning Skills Adviser, Student Services Mr Kelly Smith, Director, International Centre Mr Mike Shearer, Head of College (Trinity)

TEACHING AND LEARNING COMMITTEE MEETING – THURSDAY 1 MARCH 2012

This is to confirm that the next meeting of the Teaching and Learning Committee will be held on Thursday, 1 March, 2.00 – 4.00pm in the Senate Room. Parts 1 and 2 of the agenda are to be dealt with en bloc by motion of the Chair. Part 3 is for discussion. A member may request the transfer of an item from Part 1 and/or Part 2 to Part 3. Kaye Macpherson-Smith Executive Officer

AGENDA WELCOME AND APPRECIATION The Chair will welcome members, Ms Aisling Blackburne and Mr Matthew Mckenzie, and Executive Officer, Ms Kaye Macpherson-Smith. Kaye has worked at the University for over seven years and has been the Executive Officer for the New Courses 2012 implementation project since the project commenced.

APOLOGIES The Chair will record any apologies. Members are reminded that apologies should be forwarded to the Executive Officer prior to the meeting. Members representing the faculties are reminded that if unable to attend a meeting, an alternative attendee from the Faculty Teaching and Learning Committee should be nominated prior to the meeting.


Members are also reminded that the meetings are usually scheduled for two hours (2.00-4.00pm), if members are unable to attend for the whole meeting would they please advise the Executive Officer prior as this may have an impact on the Committee proceeding inquorate. DECLARATIONS OF POTENTIAL FOR CONFLICT OR PERCEIVED CONFLICTS OF INTEREST The Chair will invite members to declare potential for conflict or perceived conflicts of interest, if applicable, with regard to items on the agenda. 1. MINUTES – Ref: F12154

The Chair recommends confirmation of the minutes of decisions noted by circular by of the Teaching and Learning Committee on 10 November 2011. Minutes are available from the committee’s web page: http://www.teachingandlearning.uwa.edu.au/staff/committees/tlc/

2. ITEMS/BUSINESS IN PROGRESS FOR NOTING SINCE PREVIOUS MEETING

ITEM/BUSINESS IN PROGRESS

ACTION RESPONSIBLE STATUS

2.1 Proposed framework for decisions regarding adoption of new technologies at UWA

Mr Brian Greene to consult with Faculty IT departments and the eLearning and Learning Spaces Standing Committee; redraft recommendations and draft a policy.

Brian Greene and Denise Chalmers

In progress. Scheduled to report to the Teaching and Learning committee in early 2012.

PART 1 – ITEMS FOR COMMUNICATION TO BE DEALT WITH EN BLOC 3. MEETING DATES FOR 2012 – Ref: F12153 The following principal meeting dates (and corresponding cut-off dates for receipt of material to be placed on the agenda) have been scheduled for 2012. Meetings will commence at 2.00pm, generally finish at 4.00pm. Meeting Date Cut-Off Date for Receipt of

Agenda Material Location

Thursday, 1 March Thursday, 16 February Senate Room Thursday, 5 April Thursday, 22 March Senate Room Thursday, 3 May Thursday, 19 April Senate Room Thursday, 7 June Thursday, 24 May Senate Room Thursday, 2 August Thursday, 19 July Senate Room Thursday, 6 September Thursday, 23 August Senate Room Thursday, 4 October Thursday, 20 September Senate Room Thursday, 8 November Thursday, 25 October Old Senate Room, Irwin St The Chair requests that members note these dates in their diaries for 2011.

http://www.teachingandlearning.uwa.edu.au/staff/committees/tlc/


4. INDUCTION OF NEW COMMITTEE MEMBERS TO THE TEACHING AND LEARNING COMMITTEE– Ref: F12155

An induction for new members will be held on 27 February 2012. Prior to this an “Information Package: A Handbook for Members of the Teaching and Learning Committee” will be provided to each new member. An electronic copy is also available on the Teaching and Learning website at http://www.teachingandlearning.uwa.edu.au/page/74621. 5. PRINCIPLES AND RULES FOR THE OPERATION OF COMITTEES – Ref: F12202

Members will be aware that all committees of the University are expected to operate according to the principles set out in the Principles for the Operation of Committees. In 2005 the Deputy Vice-Chancellor and the Executive Director (Academic Services and Registrar) commissioned a “Working Smarter Through Committees” working party in order to investigate ways of streamlining and improving committee processes without compromising collegiality. In accordance with Recommendation 9 – That the University’s principles and rules for the operation of committees are distributed (as part of the agenda) to all committee members and the Chair explicitly address these at the first meeting of each year – the principles and rules are attached along with the code of conduct. (Attachment A) For noting. 6. TEACHING AND LEARNING COMMITTEE BUDGET – Ref: F12152 2011 Budget As part of its reporting requirements, the Teaching and Learning Committee submits to the Planning and Budget Committee a detailed account of expenditure of its annual budget. For the information of members, attached (Attachment B) is a breakdown of the Committee’s expenditure by cost centre. As can be seen $310,000 was returned directly to faculties in support of teaching and learning or as rewards for excellence. The Chair encourages all faculties and schools to reap the benefits of the Committee’s schemes which include the Distinguished Visiting Teachers Fund, Postgraduate Teaching Internship Scheme, Teaching Fellowship Scheme, a travel grants scheme for members, etc. Members are referred to the Teaching and Learning website for more information on these schemes at http://www.teachingandlearning.uwa.edu.au/page/74625. 2012 Budget The Senate has approved an allocation of $605,000 to the Teaching and Learning Committee for 2012 in accordance with the Committee’s budget submission. The 2012 budget allocation is in support of the following projects: Project/Commitment Budget Reward and Acknowledgement of Excellence in Teaching and Teaching Support $98,750 Distinguished Visitors to UWA – no additional funds requested Nil Teaching and Learning Development Fund in support of OPP activities $150,000 Postgraduate Teaching Internship Scheme $209,520 Teaching Fellowship Scheme $88,000 Improving Student Learning Grants Scheme $30,000 Teaching and Learning Forum – Provision of UWA subsidies – no additional funds requested Nil

Teaching and Learning Month $17,200 Flexible Funding for the Teaching and Learning Committee $4,530

http://www.teachingandlearning.uwa.edu.au/page/74621

http://www.teachingandlearning.uwa.edu.au/page/74625


Introduction to University Teaching – CATL – no additional funds requested Nil New bids for 2012 CATL Sessional Staff Day $3,000 CATL English Language Support for Teaching Staff $4,000 TOTAL Budget Submission for 2012 $605,000

As can be seen a large proportion of the Committee’s funds are tied to existing projects and programs. For members’ information.

PART 2 – ITEMS FOR DECISION TO BE DEALT WITH EN BLOC

There are no Part 2 items.

PART 3 – ITEMS FOR DISCUSSION AND DECISION 7. RECOMMENDATION FOR TEACHING AND LEARNING STANDING COMMITTEES – Ref:

F29274, F27831, F29379, F30116

Members are reminded that the Teaching and Learning Committee operates with four Standing Committees sitting beneath, and reporting to it, to facilitate the transaction of business in key areas of activity. These Standing Committees, established in 2009 are:

• The Assessment and Evaluation Standing Committee • The Awards Standing Committee • The eLearning and Learning Spaces Standing Committee • The Grants and Schemes Standing Committee

Members have before them Attachment C which outlines the current arrangements for the Chairs of the Standing Committees, acting as nominees of the Chair of the Teaching and Learning Committee, to nominate co-optees to the Standing Committees. Information is also provided on the informal review of the Standing Committees to be conducted during 2012 by the Chair and Deputy Chair of this committee. A review report will be provided at the September Committee meeting. In the interim, some Chairs have requested for their capacity to invite people to serve on Standing Committees to be expanded in consultation with the Chair and Deputy Chair of the Teaching and Learning Committee, to assist in progressing business during 2012. In respect of co-optees, and effective for 2012, the Chair recommends:

That the number of co-optees that may be recruited to assist in progressing the Teaching and Learning Standing Committees’ work be increased, from up to two, to up to five, at the discretion of the Standing Committee Chair and with the agreement of the Chair of the Teaching and Learning Committee.

The Chair also seeks in-principle endorsement of a proposal for all Standing Committee members to have terms of two years, renewable by invitation, with the exception of Chairs who are nominees. For discussion and members’ endorsement to assist in progressing business during 2012, and to facilitate the planning work that will accompany the review.


8. TEACHING AND LEARNING FOR 2012 – Ref: F12155

In accordance with the Committee’s practice, the Chair will briefly outline the role of the Committee and the expectations of its members, which are guided by the following University policies and practices:

• The Teaching and Learning Committee’s Constitution • Principles for the Operation of Committees • Rules for the Operation of Committees • University Committee Members’ Code of Conduct • The Effective Committee Member

Item 4 notes the “Information Package: A Handbook for Members of the Teaching and Learning Committee” which has been provided to new members and is available on the web for reference by all members http://www.teachingandlearning.uwa.edu.au/staff/committees/tlc The package is primarily for the information of new members but also serves as a reminder to all Committee members as to the protocols for best practice in committees and details the above policies and practices. Members are welcome to provide any comment on these documents during this item. Members are reminded that the performance of Academic council committees is evaluated in November/December each year via online surveys of members. In addition, for the Teaching and Learning Committee, standing invitees are also surveyed. Attachment D provides a summary report of the Teaching and Learning Committee Survey 2011 comprised by the Institutional Research Unit. The report notes a survey response rate of 44%. The survey addressed the following:

• The role of the Teaching and Learning Committee and its members; • The performance of the Committee; • The clarity, accuracy and timeliness of the Committee’s agendas and minutes; • Key relationships between the Committee and faculty Teaching and Learning

Committees, and between members; • The skill mix and contributions of the Committee members; and, • A copy of the responses to questions regarding the scope for improvement in various

areas.

Also attached (Attachment E) as historical information is a copy of the previous survey summary (2010). Within the context of the Committee’s modus operandi and its self-review, the Chair will invite comment on the outcomes of the Committee’s review, how the Teaching and Learning Committee can contribute to the University’s strategic thinking, how to enhance the student learning experience and how to give operational effect to the University’s strategic directions. The 2011 Survey Report and the Committee’s comments will be forwarded to the Academic Council for its information. For discussion.

http://www.teachingandlearning.uwa.edu.au/staff/committees/tlc


9. REPORT FROM THE GUILD PRESIDENT 2012 – REF: F41149

Matthew Mckenzie (99th Guild President) will report on the Guild’s priorities for 2012, in particular with reference to the New Courses 2012 implementation. 10. REPORT FROM THE POSTGRADUATE STUDENT ASSOCIATION PRESIDENT 2012 –

REF: F41150

Aisling Blackmore (Postgraduate Students Association (PSA) President) will report on the PSA’s priorities for 2012, in particular with reference to the New Courses 2012 implementation and the expected significant growth in postgraduate education. 11. TEACHING DELIVERY AND PEDAGOGY IN THE NEW COURSES: A 2012 PRIORITY –

REF: F41151

The implementation of New Courses has proceeded with a strong emphasis upon curriculum development, and the ratifying of content via the (then) IBOS and BCS. As we move into semester 1 2012, the emphasis shifts to matters of delivery and pedagogy, and the Teaching and Learning portfolio, building upon work already done in the faculties in relation to assessment and feedback, has identified these issues as critical points of focus for 2012. There are an array of important matters for consideration, by those engaged in grassroots classroom work with our students, and other areas of academic delivery. By way of introduction, the Chair invites members of the Teaching and Learning Committee to contribute ideas and issues that are relevant to their own areas of the University in this domain. As well as hearing from student representatives in a separate item, the Chair will invite Associate Deans from each faculty to report on planned activities in the context of New Courses-- changing delivery arrangements; the challenge and differences of pedagogical approaches within Cycle 2; effective online delivery; discipline-based academic induction for new student cohorts-- and will outline for the Committee the array of activities with a focus upon pedagogy and delivery during 2012. For discussion. 12. INFORMAL REPORT FROM THE CHAIR AND DEPUTY CHAIR

This agenda item provides an opportunity for the Chair to advise members on any issues which are of relevance to the education portfolio since the last meeting. The Chair's report will relate mainly to UWA's responses to recent consultation papers (5) from the Department, and the AQFC. In particular, the Department's original consultation paper (Attachment F) and the University's response to it (Attachment G) are attached for information. 13. CHAIR’S WELCOME TO COMMITTEE To conclude the meeting, members are invited to share a ‘welcome’ drink and cake with the Chair. 14. NEXT MEETING

The next meeting of the Teaching and Learning Committee will be held on Thursday, 5 April 2012 at 2.00pm in the Old Senate Room (Irwin Street Building). The cut-off date for submission of items for the Committee’s agenda is Thursday, 22 March 2012. Items should be forwarded to Kaye Macpherson-Smith, at [email protected].

mailto:[email protected]

University Secretariat

Principles

Principles for the Operation of Committees

While the committee structures provide a suitable framework, it is the members of the committees who determine whether good governance and better practice standards are actually achieved and ensure that the committee is adding value.

General:

1. Collegiality: The Committee system is transparent and consultative, and all staff have the opportunity to provide meaningful input into decisions that significantly affect them. 2. Working Smart: Committee time is used in ways which make the most efficient and effective use of staff time to deliberate on significant issues and policies. 3. Good Conduct: Committee members abide by a Code of Conduct that requires them to be appropriately informed and prepared before the meeting so that they can contribute to the decisions of the committee. 4. Review: All committees are regularly reviewed in relation to both the performance of committee business and committee members.

Functional:

5. Policy Decisions: Committees are, wherever possible, engaged in establishing policies and rules within which individual staff can manage and administer University business. 6. Management Decisions: Committees are involved in management and administrative decisions as close to the activity as is compatible with legislative requirements. 7. Communication: Committees are used to provide a contextual framework within which University policies are developed and decisions are made. 8. Membership: Committee membership ensures the broad University community, in all its diversity, is reasonably represented. However, individual committees are not constituted to represent every possible interest group and the number of members should be kept as low as practicable. Revised on 16/02/2006

CRICOS Provider Code: 00126GLast updated 10 Apr 2007 10:41

Location: http://secretariat.uwa.edu.au/page/20929

A1

http://secretariat.uwa.edu.au/page/20929


Rules for the Operation of Committees

Establishment and Review of Committees:

1. Committees are to have a constitution that clearly describes the purpose and operation of the committee including membership, Chair, function, decision-making and communication lines, quorum and conduct of meetings. 2. Mechanisms for reporting the business of each committee should be made clear at the time of establishment of the committee. 3. Committees should establish a review regime addressing the frequency and nature of the review process and the allocation of responsibility for conducting and acting on the review.

Conduct of Meetings:

4. Meetings should only commence with the appropriate quorum. If the meeting is inquorate the meeting can be cancelled and business dealt with by circulation or discussion can take place in committee mode with recommendations ratified at the next meeting or by circular. 5. Meetings should be held in accordance with constitution. However, the committee should not meet simply because a meeting has been scheduled rather only when there is important business to transact. If business is limited, consideration should be given to circulating items with the Chair’s recommendation rather than to deferring items to a later meeting. 6. Items brought to the committee for a decision for noting or for communication should be relevant to the business or role of the committee. Consideration should be given to holding a joint meeting of two committees when there is an item of interest for more than one committee. When it is clear that the committee has insufficient knowledge to add value to a decision, then other means of processing the decision should be found within existing policy and legislative provisions. 7. Where decisions are to be referred to other University committees then meeting dates should be scheduled to facilitate the efficient flow of business to these committees. 8. There should be an opportunity at the start of each meeting for members to declare any potential or perceived conflict of interest in respect of any item and where appropriate to absent themselves from discussion and/or the room while the item is being considered. 9. Documentation for committees should be written precisely, accurately, clearly and succinctly. The level of formality and detail should match the purpose and readership of the document. 10. Agendas of meetings should: • Contain or have attached any background material necessary to enable members to make informed decisions. If items have been discussed previously a chronology of key elements of the discussion should be attached. • Be divided into three parts (Attachment C): Part 1. Items for Communication to be dealt with En Bloc - only items for communication that require no decision or discussion, but are relevant to the business of the committee or its future decision-making, or require dissemination by members to staff within their

A2

areas; Part 2. Items for Decision to be dealt with En Bloc - only items for decision with clear recommendations by the Chair that are likely to require no further discussion; Part 3. Items for Discussion and Decision. • Include an Item/Business in Progress List providing an update of actions since the last meeting, where appropriate • At the discretion of the Chair allow items of ‘Other Business’ provided there is advance notice to the Chair and the item is not a major policy item. • Be distributed to members at least three working days before the meeting and items perceived by the Chair to be key issues should be highlighted. Deadlines for placing formal items on the agenda should be enforced by the Chair and the Executive Officer. Rather than delay the distribution of an agenda because one item is not ready, the use of supplementary agendas should be considered. • Be posted on the Web, with attachments where possible and available to all staff on the University’s intranet, unless issues of confidentiality preclude this. 11. Minutes of meetings should: • Contain a summary or précis of events, in dot form wherever possible, rather than a detailed account of every contribution. In general names of individuals should not be recorded in the discussion. • Contain all resolutions and agreements, whether reached formally through motions or through general consensus, and a clear statement of the action to be taken in relation to each item including the person responsible for the action and a timeline for completion, where appropriate. • Be distributed to committee members within 10 working days of the close of the meeting. • Be posted on the Web and available to all staff on the University’s intranet, unless issues of confidentiality preclude this.

Role of Members:

12. Members are required to conduct themselves in accordance with the University’s Code of Conduct and the University Committee Members’ Code of Conduct. 13. Members are required to participate actively in committee business and provide appropriate contributions to decision making for the betterment of the University as a whole.

14. University members should only nominate for committees for which they feel they are well placed and appropriately informed to contribute to the business of that committee. 15. Members who represent constituents on a committee should make every attempt to canvass the views and opinion of that group to bring back to the discussion of the committee and report committee decisions back to their constituents.

Role of Chair and Executive Officer:

16. Chairs and Executive Officers are responsible for ensuring these rules for the operation of committees are followed including the appropriate recording of decisions and actions. 17. Chairs should conduct meetings with the degree of formality appropriate to the committee. Generally, greater formality is needed with major committees, committees with a larger membership and where it is required by constitutional and statutory provisions. 18. Chairs should ensure meetings are conducted fluently so members understand the

A3

matters at hand and have the opportunity to discuss them, and the voting processes and resolutions are clear. Chairs should try to ensure the active participation by all members of the committee. 19. Chairs and the Executive Officers have a responsibility to ensure the business of the committee is either referred for a decision to the relevant committee or is communicated effectively to relevant areas of the University. 20. Committees should make provision for delegating responsibilities to the Chair and Executive Officer as far as possible, ensuring accountability for these delegations. 21. Chairs and Executive Officers should meet to discuss the draft agenda so both understand the purpose and possible outcomes of the meeting. 22. Executive Officers should familiarise themselves with the working of the University committee system as a whole, and as much as possible work with Executive Officers of other committees to ensure smooth transition of business through the committee system. 23. Executive Officers should work with the Chair to provide all new members with induction briefing material and appropriately induct new members into the committee. Where possible, this should occur well before the member’s first attendance at a meeting of the committee. It is recommended that briefing material provided to new members includes:

The constitution of the committee including information on the position of the committee in the University committee structure (namely maps with pathways for decisions and communication).

The Principles and Rules of the Operation of Committees. The Code of Conduct of committee members. Committee meeting dates. Major items of business of the committee of the previous year. Commonly used acronyms and abbreviations which might be used in the conduct

of the committee’s business. Effective Meetings - A Guide to Good Practice

Any enquiries relating to the rules for the operation of committees may be directed to the Academic Secretariat.

Revi sed 16 February 2006

CRICOS Provider Code: 00126GLast updated 17 Sep 2007 15:24

Location: http://www.secretariat.uwa.edu.au/page/88164

A4

http://www.secretariat.uwa.edu.au/home/policies/commconst/effective_meetings

http://www.secretariat.uwa.edu.au/page/88164


University Committee Members' Code of Conduct

Code of Conduct

Membership of a University Committee is an important role and brings with it key responsibilities and obligations. The specific membership requirements for University Committees have been prepared to promote good practice and give committee members a summary of their obligations and provide guidance on ethical conduct. The University Committee Members’ Code of Conduct is based on the Western Australian Public Sector Code of Ethics and the template for Code of Conduct for Government Boards and Committees (http://www.opssc.wa.gov.au/documents/ethicsintegrity/templatecodeofconductboardsandcommittees.pdf)

1. Personal Conduct: All members of University committees are required to conduct themselves at all times in accordance with the. University’s Code of Ethics and Code of Conduct. A copy is available at: http://www.hr.uwa.edu.au/publications/code_of_ethics

Members of committees are also required to:

Understand the committee’s role and purpose within the University. Stay informed about relevant matters affecting the committee’s business. Attend all committee meetings or where attendance is not possible, submit an

apology. Participate actively and work cooperatively with other committee members and

University staff. Prepare for all committee meetings by reading and considering the agenda items,

papers circulated and other relevant documents. Not improperly influence other committee members. Make new points succinctly without reiterating at length points already made.

2. Accountability. All members have a responsibility to ensure efficient and effective operations of the committee, avoid extravagant and wasteful use of resources and ensure actions are consistent with the role and purpose of the committee.

Members of committees are also required to:

Participate constructively in committee activities in a lawful, ethical and justifiable manner.

Ensure decisions are consistent with any statutory and legal requirements. Ensure resources, funds and staff are used effectively and economically for

committee business.

3. Record Keeping and Use of In formation: All documentation produced by the committee forms part of the University records and should be maintained in accordance with University’s Record Keeping Plan (http://www.archives.uwa.edu.au/rkp).

In conjunction with specific responsibilities of the Executive Officer, members of committees are also required to:

Ensure adequate procedures are followed for documenting decisions and actions of the committee.

A5

http://www.opssc.wa.gov.au/documents/ethicsintegrity/templatecodeofconductboardsandcommittees.pdf

http://www.opssc.wa.gov.au/documents/ethicsintegrity/templatecodeofconductboardsandcommittees.pdf

http://www.hr.uwa.edu.au/publications/code_of_ethics

http://www.archives.uwa.edu.au/rkp

Maintain confidentiality of committee business where necessary, ensuring confidential records are subject to appropriate storage and access procedures.

Respect confidential discussions and not misuse any information obtained through membership of the committee.

Openly declare any matters of private interest and record any issues with the potential for conflict or perceived conflict to ensure they are transparent and capable of review.

Where appropriate, disqualify themselves from committee discussions and decisions where a conflict of interest occurs.

Be aware of the FOl Act 1992 and that access may be sought to all records under this legislation

Where members are unsure of their obligations or responsibilities under the University Committee Members’ Code of Conduct, the member should contact the Chair or Executive Officer of the Committee for assistance.

Revised on 16/02/2006

CRICOS Provider Code: 00126GLast updated 17 Jan 2008 11:27

Location: http://www.secretariat.uwa.edu.au/page/88165

A6

http://www.secretariat.uwa.edu.au/page/88165

Attachment B Teaching and Learning Committee

Itemised Budget Allocations for 2011 by Cost Centre

Cost Centre 2011 2011 Total by

Budget Allocations Faculty/Section Income: Central Budget Allocation $605,000 Funds Held in Reserve/Carried Forward $402,501 Funds Returned During 2011 $4,888 Total Income for 2011 $1,012,389 Expenditure: Indigenous Studies Sub-Total $0

Architecture, Landscape and Visual Arts

2011 Faculty Based Teaching Awards $6,000 Teaching Internship Scheme 2011 $5,000 Reward and Recognition for 2011 ALTC Nomination $3,500 Sub-Total $14,500

Arts, Humanities and Social Sciences 2011 Faculty Based Teaching Awards $6,000 Teaching Internship Scheme 2011 $60,000 Reward and Recognition for 2011 ALTC Nominations $2,500 Sub-Total $68,500

UWA Business School 2011 Faculty Based Teaching Awards $6,000 Teaching Internship Scheme 2011 $5,000 Reward and Recognition for 2011 ALTC Nominations $4,500 Teaching Fellowship Scheme 2011 $22,000 Sub-Total $37,500

Education 2011 Faculty Based Teaching Awards 6,000 Teaching Fellowship Scheme 2011 $22,000 Distingushed Teaching and Learning Award for Schools - winner $45,000

B1

Sub-Total $73,000

Engineering, Computing and Mathematics 2011 Faculty Based Teaching Awards $6,000 Reward and Recognition for 2011 ALTC Nominations $2,000 Teaching Internship Scheme 2011 $25,000 Sub-Total $33,000

Law 2011 Faculty Based Teaching Awards $6,000 Sub-Total $6,000


Budget Allocations Faculty/Section

Life and Physical Sciences 2011 Faculty Based Teaching Awards $6,000 Distinguished Teaching and Learning Award for Schools - high commendation $15,000 Teaching Internship Scheme 2011 $15,000 Sub-Total $36,000

Medicine and Dentistry and Health Sciences 2011 Faculty Based Teaching Awards $6,000 Reward and Recognition for 2011 ALTC Nominations $5,000 Sub-Total $11,000

Natural and Agricultural Sciences 2011 Faculty Based Teaching Awards $6,000 Teaching Fellowship Scheme 2011 $22,000 Distinguished Visiting Teachers Fund Grant $2,500 Sub-Total

$30,500

Total Funds Allocated Directly to Faculties and the School of Indigenous Studies in 2011 $310,000

Vice-Chancellery (Office of the Pro Vice-Chancellor)

Education Strategies Office

ALTC Seeding Grants - Funds held in reserve $50,000 Sub-Total

$50,000

Centre for the Advancement of Teaching and Learning (CATL)

Professional development component of the 2011 Teaching Internship Scheme $71,060 2011 Teaching Month $24,000

B2

ULTRIS Scheme (TLDF) $40,000 Assessment Project(TLDF) $246,000 Improving Student Learning Grants Scheme $30,000 Introduction to University Teaching $25,000 Reward and Recognition for 2011 ALTC Nomination $1,000 Teaching Fellowship Scheme 2011 initial allocation $22,000 Distinguished Visiting Teachers Fund Grant $2,500 Distinguished Visiting Teachers Fund Grant $3,204 Teaching and Learning Travel Grant - Director of CATL - TEQSA Conference $956 Sub-Total

$465,720

Total Funds Allocated Directly to the Vice-Chancellery $515,720


Budget Allocations Faculty/Section

Teaching and Learning Committee

Funds allocated

Catering Costs - Associate Deans' Network and Teaching and Learning Committee $946 Provision of Support for Teachers Retreat (Michael Wood) $1,000 Production costs for ALTC Awards $2,161 Sub-Total

$4,107

Funds carried forward to support ongoing schemes

Teaching Fellowship Scheme $27,628 Distinguished Visiting Teachers Fund $26,487 International Visitors $18,000 Teaching and Learning Development Fund $38,674 Postgraduate Teaching Internship Scheme $21,960 Reward and acknowledgement of Excellence in Teaching $26,250 Administrative, Research and Project Management Support $11,185 Uncommitted Funds $12,378 Sub-Total

$182,562

Total Funds Allocated/Held in Reserve by the Teaching and Learning Committee $186,669

Total $1,012,389 $1,012,389 Compiled: 17 February 2012

B3

1

TEACHING AND LEARNING COMMITTEE

Chair’s recommendation on Standing Committee membership

Members will be aware that the Teaching and Learning Committee operates with four standing committees sitting beneath, and reporting to, it, to facilitate the transaction of business in key areas of activity. These standing committees, established in 2009, are:

• The Assessment and Evaluation Standing Committee • The Awards Standing Committee • The eLearning and Learning Spaces Standing Committee • The Grants and Schemes Standing Committee

The membership and constitutions of these standing committees have been endorsed by the Teaching and Learning Committee. In all cases, the Chairs of standing committees, acting as nominees of the Chair of the Teaching and Learning Committee, may nominate co-optees to the standing committees to ensure appropriate expertise in relation to the work at hand.

To ensure that the standing committees’ functions remain relevant, well-informed, and appropriately constituted, the Chair and Deputy Chair will conduct an informal review of the standing committees during 2012: their work, their membership, and their constitutions in the context of the teaching and learning agenda at UWA-- and will report to the Teaching and Learning Committee with any recommendations at the September meeting of the Committee.

In the interim, some Chairs have requested that the capacity for them to invite people to serve on standing committees be expanded in consultation with the Chair and Deputy Chair of the Teaching and Learning Committee, to assist in progressing business during 2012.

It has further been mooted that, with the exception of Chairs who are nominees, all members of standing committees should have terms of two years, renewable by invitation. Whilst this proposal will be considered more fully in the context of the informal review, the Chair seeks in-principle endorsement of this proposal at this earlier stage, to facilitate the planning work that will accompany the review.

In respect of co-optees, and effective for 2012, the Chair recommends:

That the number of co-optees that may be recruited to assist in progressing the Teaching and Learning standing committees’ work be increased, from up to two, to up to five, at the discretion of the standing committee Chair and with the agreement of the Chair of the Teaching and Learning Committee.

W/Professor Jane Long Chair, Teaching and Learning Committee 20 Feb 2012

C1

Report No. 11/29a – UWA Academic Council Committee Performance Evaluation 2011 - Teaching and Learning Committee Institutional Research Unit, The University of Western Australia, December 2011

1.

UWA Academic Council Committee Performance Evaluation 2011


Introduction The performance of Academic Council committees is evaluated in November/December each year via online surveys of members. For two committees, Admissions and Teaching and Learning, standing invitees are also surveyed. This report summarises the responses to the Teaching and Learning Committee survey in 2011. Ten of the 23 members and standing invitees of the Teaching and Learning Committee participated, a response rate of 44%. Summary of results All respondents (100%) agreed that the role of the Teaching and Learning Committee was clearly defined in its Constitution. All respondents (100%) agreed that they were aware of the University’s Principles for the Operation of Committees and all respondents (100%) agreed that the Committee operated according to those Principles. One respondent was a new committee member and this person agreed that the written and verbal information given during their induction provided a satisfactory introduction to the work of the Committee. Questions 1, 2, 3 and 4 There was considerable variation in how members perceived the performance of the Teaching and Learning Committee during 2011. One half or more of all respondents indicated the Committee had been extensively or fully involved in giving advice and making recommendations to the Academic Council and other University bodies or officers on means of encouraging and rewarding high quality teaching and learning (100%), on means of assessing and improving the quality of teaching and learning (56%), on matters relating to teaching and learning including the University’s Teaching and Learning Management Plan/Strategy and the OPP (50%) and/or on matters arising through liaison with relevant external bodies (50%). Fewer respondents indicated the Committee had been extensively or fully involved in giving advice and making recommendations to the Academic Council and other University bodies or officers on the quality of teaching and learning in the Institution (40%), on the use of technology in teaching and learning (40%); and/or research studies on teaching and learning (33%). Almost all respondents (90%) believed that the Committee had allocated its annual budget to support and promote high quality teaching and learning. Question 6 Over two-thirds of the respondents (70%) reported being able to effectively or very effectively use their skills, abilities and experience to participate in the work of the Committee, and most (90%) agreed that the Committee had been appropriately informed by the relevant University officers or sections on key issues and problems affecting the areas of interest of the Committee. All respondents (100%) were confident that they could obtain any information or advice relevant to the workings or considerations of the Teaching and Learning Committee from the Chair and/or Executive Officer. Questions 7, 8 and 9 Respondents rated the agendas, minutes and meetings of the Teaching and Learning Committee positively. All respondents (100%) indicated that the Teaching and Learning Committee agendas were received in sufficient time to allow members to give due consideration to the business, the agendas provided clear, sufficient and focussed information, agenda items made it clear what the Committee was being asked to do, agendas were well organised and easy to navigate, the minutes of the meetings had been accurate, clear and sufficiently full and minutes were received within a reasonable time after meetings. Question 11 Regarding the conduct of the Teaching and Learning Committee meetings, all respondents (100%) indicated the Chair had usually or always conducted meetings efficiently and effectively, enabling all members to participate in meetings and also that members usually or always behaved professionally at meetings and members of the Teaching and Learning Committee had contributed to the Committee’s outcomes. Only

D1


2.

slightly fewer respondents (90%) indicated that usually or always the Chair had ensured the Teaching and Learning Committee devoted the right amount of time to items on its agendas and/or the atmosphere at meetings in 2011 had been conducive to open and productive discussion of issues. Question 13 All respondents (100%) indicated that communication between the central Teaching and Learning Committee and faculty teaching and learning committees was effective. All respondents (100%) indicated the working relationship between the Committee members was satisfactory, good or excellent and almost all (90%) indicated the working relationship between the Chair and the rest of the Committee was satisfactory, good or excellent. Question 15 All respondents (100%) agreed or strongly agreed that the membership of the Teaching and Learning Committee provided a mix of skills, experience and other attributes that enabled the Committee to add value to the Academic Council’s operations. Two-thirds of the respondents (67%) agreed that most or all members had made useful contributions to the work of the Committee during 2011. Question 17 These results are presented in tabular format on the following pages. Respondents’ further comments have been reported verbatim at the end of the tabular data. A copy of the survey is given at the end of this report.

D2

Report No. 11/29a – UWA Academic Council Committee Performance Evaluation 2011 - Teaching and Learning Committee 3. Institutional Research Unit, The University of Western Australia, December 2011

Role of the Teaching and Learning Committee and its members:

Question n Number Percentage % top two

categories*

Strongly disagree Disagree Agree

Strongly agree


Strongly agree

1 The role of the Teaching and Learning Committee is clearly defined in its Constitution. ................. 10 0 0 5 5 0.0 0.0 50.0 50.0 100.0

Not at all To some

extent Considerably Fully Not at all To some

extent Considerably Fully

2 To what extent are you aware of the University's Principles for the Operation of Committees? ...... 10 0 0 8 2 0.0 0.0 80.0 20.0 100.0


Strongly agree


Strongly agree

3 The Teaching and Learning Committee operates according to the University's Principles for the Operation of Committees. ................................................................................................................. 9 0 0 5 4 0.0 0.0 55.6 44.4 100.0


Strongly agree


Strongly agree

4 Please only answer this question if you were a new member of the Committee in 2011. The written and verbal information given during induction provided a satisfactory introduction to the work of the Teaching and Learning Committee ................................................................................ 1^ 0^ 0^ 1^ 0^ 0.0^ 0.0^ 100.0^ 0.0^ 100.0^

* Percentage of responses in the top two categories (Agree or Strongly Agree for Questions 1, 3 and 4, and Considerably or Fully for Question 2) ^ Results based on a small number of respondents. Interpret with caution

Performance of the Teaching and Learning Committee:

* Percentage of responses in the top two categories (Extensively or Fully)


categories*

To what extent has the Teaching and Learning Committee been valuably involved in the following key facets of its role, as described in its Constitution?

Not

at all To some

extent Extensively Fully

Not

at all To some

extent Extensively Fully

6a Giving advice and making recommendations to the Academic Council and/or other University bodies or officers on:

i) matters relating to teaching and learning including the University's Teaching and Learning Management Plan/Strategy and the OPP ...................................................................................... 10 0 5 4 1 0.0 50.0 40.0 10.0 50.0

ii) the quality of teaching and learning in the Institution ..................................................................... 10 0 6 3 1 0.0 60.0 30.0 10.0 40.0 iii) means of assessing and improving the quality of teaching and learning ...................................... 9 0 4 4 1 0.0 44.4 44.4 11.1 55.6 iv) means of encouraging and rewarding high quality teaching and learning .................................... 9 0 0 6 3 0.0 0.0 66.7 33.3 100.0 v) the use of technology in teaching and learning ............................................................................ 10 0 6 3 1 0.0 60.0 30.0 10.0 40.0 vi) research studies on teaching and learning ................................................................................... 9 1 5 3 0 11.1 55.6 33.3 0.0 33.3 vii) matters arising through liaison with relevant external bodies ...................................................... 10 0 5 4 1 0.0 50.0 40.0 10.0 50.0

6b Allocating its annual budget to support and promote high quality teaching and learning 10 0 1 4 5 0.0 10.0 40.0 50.0 90.0

D3



* Percentage of responses in the top two categories (Effectively or Very Effectively for Question 7, Usually or Always for Question 8, and Confident or Very Confident for Question 9)

Teaching and Learning Committee agendas and minutes:


categories*

With reference to the Teaching and Learning Committee agendas and minutes in 2011, please indicate how often the following statements were true: Rarely Sometimes Usually Always Rarely Sometimes Usually Always

11a Agendas were received in sufficient time to allow members to give due consideration to the business ..... 10 0 0 3 7 0.0 0.0 30.0 70.0 100.0 11b Agendas provided clear, sufficient, and focussed information .................................................................... 9 0 0 3 6 0.0 0.0 33.3 66.7 100.0 11c Agenda items made clear what the Teaching and Learning Committee was being asked to do ................ 10 0 0 5 5 0.0 0.0 50.0 50.0 100.0 11d Agenda papers were well organised and easy to navigate ......................................................................... 10 0 0 4 6 0.0 0.0 40.0 60.0 100.0 11e Minutes were accurate, clear and sufficiently full ........................................................................................ 9 0 0 3 6 0.0 0.0 33.3 66.7 100.0 11f Minutes were received within a reasonable time after meetings ................................................................. 10 0 0 3 7 0.0 0.0 30.0 70.0 100.0

* Percentage of responses in the top two categories (Usually or Always)


categories*

Ineffectively Not very

effectively EffectivelyVery

effectively Ineffectively Not very


effectively

7 As a member of the Teaching and Learning Committee how effectively have you been able to use your skills, abilities and experience to participate in the work of the Committee? ................................................ 10 0 3 6 1 0.0 30.0 60.0 10.0 70.0

Rarely Sometimes Usually Always Rarely Sometimes Usually Always 8 How often has the Teaching and Learning Committee been appropriately informed by the relevant

University officers/sections on key issues and problems affecting the areas of interest of the Committee? ................................................................................................................................................... 10 0 1 7 2 0.0 10.0 70.0 20.0 90.0

Not at all confident

Not very confident Confident

Very confident



Very confident

9 How confident are you that you can obtain any information or advice you need relevant to the workings or considerations of the Teaching and Learning Committee from the Chair and/or Executive Officer? ..... 10 0 0 3 7 0.0 0.0 30.0 70.0 100.0

D4


Conduct of Teaching and Learning Committee meetings:


categories*

With reference to the Teaching and Learning Committee meetings in 2011, please indicate how often the following were true: Rarely Sometimes Usually Always Rarely Sometimes Usually Always

13a The Chair conducted meetings efficiently and effectively .................................................................................... 10 0 0 3 7 0.0 0.0 30.0 70.0 100.0 13b The Chair has enabled all members to participate in meetings ........................................................................... 10 0 0 3 7 0.0 0.0 30.0 70.0 100.0 13c The Chair ensured that the Committee devoted about the right amount of time to items on its agendas. .......... 10 0 1 4 5 0.0 10.0 40.0 50.0 90.0 13d The atmosphere at meetings during 2011 was conducive to open and productive discussion of issues ............ 10 0 1 3 6 0.0 10.0 30.0 60.0 90.0 13e Members of the Committee have behaved professionally at meetings ............................................................... 10 0 0 0 10 0.0 0.0 0.0 100.0 100.0 13f Members of the Committee have contributed to the Committee’s outcomes ...................................................... 10 0 0 8 2 0.0 0.0 80.0 20.0 100.0

* Percentage of responses in the top two categories (Usually or Always) Key relationships:

Question n Number Percentage % top

categories*

With reference to the Teaching and Learning Committee meetings in 2011: Ineffective Not very effective Effective

Very effective Ineffective

Not very effective Effective

Very effective

15a How effective was communication been between the central Teaching and Learning Committee and faculty teaching and learning committees? ............................................................................................... 8 0 0 7 1 0.0 0.0 87.5 12.5 100.0

Less than satisfactory Satisfactory Good Excellent


15b The working relationship between the Chair and the rest of the Committee was: ..................................... 10 1 2 1 6 10.0 20.0 10.0 60.0 90.0 15c The working relationships between members of the Committee were: ..................................................... 10 0 2 4 4 0.0 20.0 40.0 40.0 100.0

* Percentage of responses in the top categories (Effective or Very effective for Question 15a, and Satisfactory, Good or Excellent for Questions 15b and 15c) Current membership:


categories*

With reference to the memberships of the Teaching and Learning Committee during 2011: Strongly disagree Disagree Agree

Strongly agree


Strongly agree

17a The mix of skills, experience and other attributes which has enabled the Committee to add value to the Academic Council’s operations..................................................................................... 10 0 0 8 2 0.0 0.0 80.0 20.0 100.0

Few Some Most All Few Some Most All 17c What proportion of members of the Teaching and Learning Committee have made useful

contributions to the work of the Committee during 2010? ................................................................ 9 0 3 6 0 0.0 33.3 66.7 0.0 66.7 * Percentage of responses in the top two categories (Agree or Strongly Agree for Question 17a and Most or All for Question 17c)

D5


6.



Additional Comments Comments are reported verbatim below. Role of the Teaching and Learning Committee and its members 5. General Comments (n = 0) Performance of the Teaching and Learning Committee and its members 10. General Comments (n = 2)

Pip Rundle as Executive Officer during 2011 has done an outstanding job as Executive Officer. Friendly, knowledgeable and well-informed, she will be sorely missed.

The PVCE was a very effective Chair - would always endeavoured to engage members in discussion and decision-making.

Teaching and Learning Committee agendas and minutes 12. General Comments (n = 0) Conduct of the Teaching and Learning Committee 14. General Comments (n = 1)

The Committee has worked hard in the context of New Courses and its role will be pivotal to the successful running of New Courses in matters of pedagogy, delivery and its contribution in future to specialist learning areas such as the BPhil

Key relationships 16. General Comments (n = 1)

Discussion was open and productive.

Current Membership 17. If there were any skills, experience or other attributes not provided by the current

membership, please name them: (n = 0) Other comments 18. Observations about the performance of the Teaching and Learning Committee in 2011, or suggestions for improvements in 2012 (n = 1)

Do the standing committees of the Teaching and Learning Committee do a performance evaluation? If not, they probably should.

D6


7.

D7


8.

D8


9.

D9


10.

D10

Report No. 10/46a – UWA Academic Council Committee Performance Evaluation 2010 - Teaching and Learning Committee Institutional Research Unit, The University of Western Australia, February 2011

1.



Introduction The performance of the Academic Council of the University is evaluated every year using a survey of the UWA Academic Council Sub-Committees. In November 2010, members of the Teaching and Learning Committee were asked to express their views on the performance of their Committee during 2010. 26 individuals, committee members and standing invitees, were invited to take part in the online survey and 13 did so, giving a response rate of 50%. This report relates to the Teaching and Learning Committee survey responses. Some of the results are based on small numbers of respondents, less than 10 people. In these cases the data are marked with a ^ indicating these results should be interpreted with caution. Summary of results All respondents (100%) agreed that the role of the Teaching and Learning Committee was clearly defined in its Constitution. Most respondents (92%) agreed that they were aware of the University’s Principles for the Operation of Committees and all respondents (100%) agreed that the Committee operated according to those Principles. Seven respondents were new committee members and all seven (100%^) indicated that the written and verbal information they were given during induction provided them with a satisfactory introduction to the work of the Committee. Questions 1, 2, 3 and 4 There was considerable variation in how members perceived the performance of the Teaching and Learning Committee during 2010. All respondents (100%) believed that the Committee had been valuably involved to a considerable or full extent in giving advice and making recommendations to the Academic Council and other University bodies in matters relating to teaching and learning including the University’s Teaching and Learning Management Plan/Strategy and the OPP. Three-quarters or more of all respondents believed that the Committee had been valuably involved (considerably or fully) in: giving advice and making recommendations on the quality of teaching and learning in the Institution (85%); in matters arising through liaison with relevant external bodies (85%); and/or on means of assessing and improving the quality of teaching and learning (77%). Between one-half and three-quarters of all respondents believed that the Committee had been valuably involved, either considerably or fully, in: giving advice and making recommendations on means of encouraging and rewarding high quality teaching and learning (69%); the use of technology in teaching and learning (62%); and/or research studies on teaching and learning (54%). Almost all respondents (92%) believed that the Committee had allocated its annual budget to support and promote high quality teaching and learning. Question 5 Over three-quarters of the respondents (77%) reported being able to effectively or very effectively use their skills, abilities and experience to participate in the work of the Committee and most (92%) agreed that the Committee had been appropriately informed by the relevant University officers or sections on key issues and problems affecting the Committee. All respondents (100%) were confident that they could obtain any information or advice relevant to the workings or considerations of the Teaching and Learning Committee from the Chair and/or Executive Officer. Questions 6, 7 and 8 Respondents rated the agendas, minutes and meetings of the Teaching and Learning Committee positively. All respondents (100%) indicated that the Teaching and Learning Committee agendas were received in sufficient time to allow members to give due consideration to the business, the agendas provided clear, sufficient and focussed information, agenda items made it clear what the Committee was being asked to do, were well organised and easy to navigate, the minutes of the meetings had been accurate, clear and sufficiently full and minutes were received within a reasonable time after meetings. Question 9

E1


2.

Regarding the conduct of the Teaching and Learning Committee meetings, all respondents (100%) indicated the current Chair had usually or always conducted meetings efficiently and effectively and members had behaved professionally. Well over three-quarters of all respondents indicated that usually or always the Chair had enabled all members to participate in meetings (92%), the Chair had ensured the Teaching and Learning Committee devoted the right amount of time to items on its agendas (92%), that members of the Teaching and Learning Committee had contributed to the Committee’s outcomes (92%) and/or that the atmosphere at meetings in 2010 had been conducive to open and productive discussion of issues (85%). Question 10 When queried regarding the effectiveness of communication between the central Teaching and Learning Committee and faculty teaching and learning committees all respondents (100%) indicated the communication was effective. Similarly, all respondents (100%) described the relationship between the current Chair and the rest of the Committee and between members of the Teaching and Learning Committee as being satisfactory, good or excellent. Question 11 All respondents (100%) agreed or strongly agreed that the membership of the Teaching and Learning Committee provided a mix of skills, experience and other attributes that enabled the Committee to add value to the Academic Council’s operations. Most respondents (90%) indicated that all desirable skills, experience or other attributes required were provided by the current membership of the Committee. Over two-thirds of respondents (69%) agreed that most or all members of the Committee had made useful contributions to the work of the Committee during 2010. Questions 12 and 13 These results are presented in tabular format on the following pages. Respondents’ further comments have been reported verbatim at the end of this report.

E2

Report No. 10/46a – UWA Academic Council Committee Performance Evaluation 2010 - Teaching and Learning Committee 3. Institutional Research Unit, The University of Western Australia, February 2011

Role of the Teaching and Learning Committee and its members:


categories*


Strongly agree


Strongly agree

1 The role of the Teaching and Learning Committee is clearly defined in its Constitution.................. 13 0 0 6 7 0.0 0.0 46.2 53.8 100.0

Not at all To some

extent Considerably Fully Not at all To some


2 To what extent are you aware of the University's Principles for the Operation of Committees?...... 13 0 1 4 8 0.0 7.7 30.8 61.5 92.3


Strongly agree


Strongly agree

3 The Teaching and Learning Committee operates according to the University's Principles for the Operation of Committees.................................................................................................................. 13 0 0 6 7 0.0 0.0 46.2 53.8 100.0


Strongly agree


Strongly agree

4 This question is for new members only. The written and verbal information which I was given during induction provided me with a satisfactory introduction to the work of the Teaching and Learning Committee ......................................................................................................................... 7^ 0^ 0^ 4^ 3^ 0.0^ 0.0^ 57.1^ 42.9^ 100.0^

* Percentage of responses in the top two categories (Agree or Strongly Agree for Questions 1, 3 and 4, and Considerably or Fully for Question 2) ^ Results based on a small number of respondents. Interpret with caution


* Percentage of responses in the top two categories (Considerably or Fully)


categories*

To what extent has the Teaching and Learning Committee been valuably involved in the following key facets of its role, as described in its Constitution?

Not

at all To some


Not

at all To some


Giving advice and making recommendations to the Academic Council and/or other University bodies or officers on:

5a * matters relating to teaching and learning including the University's Teaching and Learning Management Plan/Strategy and the OPP ...................................................................................... 12 0 0 9 3 0.0 0.0 75.0 25.0 100.0

5b Allocating its annual budget to support and promote high quality teaching and learning 12 0 1 8 3 0.0 8.3 66.7 25.0 91.7 * the quality of teaching and learning in the Institution ..................................................................... 13 0 2 8 3 0.0 15.4 61.5 23.1 84.6 * means of assessing and improving the quality of teaching and learning ....................................... 13 1 2 6 4 7.7 15.4 46.2 30.8 76.9 * means of encouraging and rewarding high quality teaching and learning ..................................... 13 0 4 4 5 0.0 30.8 30.8 38.5 69.2 * the use of technology in teaching and learning.............................................................................. 13 1 4 6 2 7.7 30.8 46.2 15.4 61.5 * research studies on teaching and learning .................................................................................... 13 1 5 5 2 7.7 38.5 38.5 15.4 53.8 * matters arising through liaison with relevant external bodies......................................................... 13 0 2 8 3 0.0 15.4 61.5 23.1 84.6

E3



* Percentage of responses in the top two categories (Effectively or Very Effectively for Question 6, Usually or Always for Question 7, and Confident or Very Confident for Question 8)

Teaching and Learning Committee agendas and minutes:


categories* Rarely Sometimes Usually Always Rarely Sometimes Usually Always

9a Teaching and Learning Committee agendas are received in sufficient time to allow members to give due consideration to the business ............................................................................................................... 13 0 0 2 11 0.0 0.0 15.4 84.6 100.0

9b Teaching and Learning Committee agendas have provided clear, sufficient, and focussed information.... 13 0 0 4 9 0.0 0.0 30.8 69.2 100.0 9c Agenda items have made clear what the Teaching and Learning Committee is being asked to do ........... 13 0 0 4 9 0.0 0.0 30.8 69.2 100.0 9d Agenda papers have been well organised and easy to navigate ................................................................ 13 0 0 3 10 0.0 0.0 23.1 76.9 100.0 9e Minutes of Teaching and Learning Committee meetings have been accurate, clear and sufficiently full ... 13 0 0 4 9 0.0 0.0 30.8 69.2 100.0 9f Minutes have been received within a reasonable time after meetings ........................................................ 13 0 0 2 11 0.0 0.0 15.4 84.6 100.0

* Percentage of responses in the top two categories (Usually or Always)


categories*

Ineffectively Not very


effectively Ineffectively Not very


effectively

6 As a member of the Teaching and Learning Committee how effectively have you been able to use your skills, abilities and experience to participate in the work of the Committee? ................................................ 13 0 3 7 3 0.0 23.1 53.8 23.1 76.9

Rarely Sometimes Usually Always Rarely Sometimes Usually Always

7 Has the Teaching and Learning Committee been appropriately informed by the relevant University officers/sections on key issues and problems affecting the areas of interest of the Committee?................. 13 0 1 7 5 0.0 7.7 53.8 38.5 92.3



Very confident



Very confident

8 How confident are you that you can obtain any information or advice you need relevant to the workings or considerations of the Teaching and Learning Committee from the Chair and/or Executive Officer? ..... 13 0 0 5 8 0.0 0.0 38.5 61.5 100.0

E4


Conduct of Teaching and Learning Committee meetings:


categories*

Rarely Sometimes Usually Always Rarely Sometimes Usually Always

10a The current Chair has conducted meetings efficiently and effectively................................................................. 13 0 0 2 11 0.0 0.0 15.4 84.6 100.0 10b The current Chair has enabled all members to participate in meetings .............................................................. 13 0 1 3 9 0.0 7.7 23.1 69.2 92.3 10c The current Chair has ensured that the Teaching and Learning Committee has devoted about the right

amount of time to items on its agendas. .............................................................................................................. 13 0 1 4 8 0.0 7.7 30.8 61.5 92.3

10d The atmosphere at meetings this year has been conducive to open and productive discussion of issues......... 13 0 2 2 9 0.0 15.4 15.4 69.2 84.6 10e Members of the Teaching and Learning Committee have behaved professionally at meetings ......................... 13 0 0 1 12 0.0 0.0 7.7 92.3 100.0 10f Members of the Teaching and Learning Committee have contributed to the Committee’s outcomes ................ 13 0 1 7 5 0.0 7.7 53.8 38.5 92.3

* Percentage of responses in the top two categories (Usually or Always) Key relationships:

Question n Number Percentage % top

categories*

Ineffective Not very effective Effective

Very effective Ineffective

Not very effective Effective

Very effective

11a How effective has communication been between the central Teaching and Learning Committee and faculty teaching and learning committees? ............................................................................................... 12 0 0 12 0 0.0 0.0 100.0 0.0 100.0



11b The working relationship between the current -Chair and the rest of the Teaching and Learning Committee has been:................................................................................................................................. 13 0 1 4 8 0.0 7.7 30.8 61.5 100.0

11c The working relationships between members of the Teaching and Learning Committee have been: ...... 13 0 1 6 6 0.0 7.7 46.2 46.2 100.0 * Percentage of responses in the top categories (Effective or Very effective for Question 11a, and Satisfactory, Good or Excellent for Questions 11b and 11c)

Current membership:


categories*


Strongly agree


Strongly agree

12a The membership of the Teaching and Learning Committee during 2010 has provided a mix of skills, experience and other attributes which has enabled the Committee to add value to the Academic Council’s operations. ....................................................................................................... 10 0 0 5 5 0.0 0.0 50.0 50.0 100.0

Yes No Yes No 12b. Are there any desirable skills, experience or other attributes not provided by the current

membership?.................................................................................................................................... 10 1 9 10.0 90.0 Few Some Most All Few Some Most All 13 What proportion of members of the Teaching and Learning Committee have made useful

contributions to the work of the Committee during 2010?................................................................ 13 1 3 8 1 7.7 23.1 61.5 7.7 69.2 * Percentage of responses in the top two categories (Agree or Strongly Agree for Question 12a and Most or All for Question 13)

E5


6.



Additional Comments Comments are reported verbatim below. Key relationships Comments (n = 1)

The UWA communication system is not very efficient and sometimes ineffective. It eiether depends on a mass mail-out or on a cascading system - centre sends to Faculty for further distribution. I am not sure that professors make great posties.I would suggest that the institution develops a portal that allows individual academics to select areas where they would like to receive information and where Faculties can list membership of committees, so that Central Administration can send out information to the appropriate people in one hit and avoid repeat handling.

Other comments Observations about the performance of the Teaching and Learning Committee in 2010, or suggestions for improvements in 2011 (n = 1)

I was concerned to hear that appointments to Associate Dean positions in one Faculty might be based on attempts to equalise workload across the Faculty rather than on ability. If there is any basis for this claim, it does not augur well for leadership in T&L at quite an important time for the University. I think that it might be time to change T&L to L&T across the Uni.

E6

1

Development of Performance Measurement Instruments in Higher

Education

Discussion Paper

December 2011

F1

2

Table of contents 1. Introduction .................................................................................................................................... 3

1.1. Performance Funding .............................................................................................................. 4

1.2. Advancing Quality in Higher Education .................................................................................... 5

1.3. Purpose of this paper .............................................................................................................. 5

2. Principles and the student life cycle framework............................................................................... 6

2.1. Principles ................................................................................................................................. 6

2.2. Student life cycle framework ................................................................................................... 6

3. Existing surveys ..............................................................................................................................10

3.1. National and cross-institution surveys ....................................................................................10

3.2. Links with new instruments ....................................................................................................11

4. New instruments ............................................................................................................................12

4.1. University Experience Survey ..................................................................................................12

4.2. Assessment of Generic Skills : The Collegiate Learning Assessment .........................................13

4.3. Review of the Australian Graduate Survey ..............................................................................14

5. Issues .............................................................................................................................................16

5.1. Administration of new instruments ........................................................................................16

5.2. Student selection ....................................................................................................................17

5.3. Central sampling of students ..................................................................................................18

5.4. Uses of data............................................................................................................................19

5.5. Intersection of existing and new instruments .........................................................................20

6. Next Steps ......................................................................................................................................22

6.1. The AQHE Reference Group ....................................................................................................22

6.2. Discussion papers ...................................................................................................................23

6.3. Roundtable discussions ..........................................................................................................23

6.4. Next steps ..............................................................................................................................23

Appendix 1 - References ........................................................................................................................24

Appendix 2 – How to make a submission ...............................................................................................25

Appendix 3 – Terms of Reference for the AQHE Reference Group ..........................................................26

Appendix 4 – Membership of the AQHE Reference Group ......................................................................27

Appendix 5 – Summary of existing surveys .............................................................................................28

F2

3

1. Introduction

In 2008, the Government launched a major review to examine the future direction of the higher education sector, its fitness for purpose in meeting the needs of the Australian community and economy, and options for reform. The review was conducted by an independent expert panel, led by Emeritus Professor Denise Bradley AC. The panel reported its findings to the Government in the Review of Australian Higher Education (the Review) in December 2008. The Review made 46 recommendations to reshape Australia’s higher education system.

In the 2009-10 Budget, the Government responded to the recommendations of the Review with a ten-year plan to reform Australia’s higher education system, outlined in Transforming Australia’s Higher Education System. The Government’s response was based on the need to extend the reach and enhance the quality and performance of Australia’s higher education system to enable it to prosper into the future.

To extend reach, the Government holds an ambition to increase the educational attainment of the population such that by 2025, 40 percent of all 25-34 year olds will have a qualification at bachelor level or above. The Government also seeks to increase the higher education participation of those people who are currently underrepresented in higher education. In particular, by 2020, the Government expects that 20 per cent of higher education enrolments at undergraduate level will be of people from low socio-economic backgrounds.

Reforms announced to achieve these ambitions included the establishment of the Tertiary Education Quality and Standards Agency, the introduction of the demand driven funding system and significantly improved indexation on grants, Performance Funding and mission based Compacts.

The Government’s response to the Review shares a number of features with reform agendas in other areas of “human capital development” such as health, employment services, and disability services that are being implemented in Australia and internationally. These common features include: opportunities for citizens to exercise greater choice between alternative providers; the introduction of funding that “follows the consumer” and thus gives them more power in the service relationship and strengthens incentives for providers to tailor their offerings to citizens’ requirements; improved regulation to ensure minimum quality standards; and improved information on performance to allow citizens to make better informed choices.

The Government’s efforts to improve performance reporting and transparency are aimed at enhancing the quality of information available to students, to give them greater confidence that the choices they make are the right ones for them. The performance of universities has a number of domains, including but not limited to: research, teaching, financial performance, student experience, the quality of learning outcomes and access and equity. Each of these domains has a specific mechanism or tool (sometimes more than one) designed to capture relevant information about performance in that domain. For example, the Excellence in Research for Australia (ERA) process captures information about research performance; access and equity outcomes are captured through student data collections that include markers for low-SES status; and TEQSA will be implementing teaching standards by which the performance of universities will be measured.

F3

4

Similarly, the three performance indicators that are the subject of this paper are designed to capture information about how universities perform in the domains of student experience and the quality of learning outcomes. There are likely to be synergies and complementarities with other tools, for example, TEQSA’s teaching standards. They therefore should be seen as part of an overarching suite of performance measures and mechanisms that are designed to capture information across the most relevant domains of university performance, necessary for improving the information available to students as they seek to exercise the choices that are now open to them in the demand-driven system. It should be noted that the newly created MyUniversity website will be used for presenting information to students about performance across the various domains.

1.1. Performance Funding

In late 2009, the Department convened an Indicator Development Group comprised of experts in the higher education sector. The group assisted in the development of a draft indicator framework, outlined in the discussion paper, An Indicator Framework for Higher Education Performance Funding, which was released for consultation in December 2009. The paper proposed 11 possible performance indicators in four performance categories. 61 submissions from the sector were received in response to the discussion paper.

The Government considered the feedback received and refined the framework to include seven indicators in three performance categories: participation and social inclusion, student experience and the quality of learning and teaching outcomes.

The Government released draft Performance Funding Guidelines for discussion in October 2010. The draft Guidelines provided details on the proposed implementation of the Performance Funding arrangements. The Government received 44 responses to the draft guidelines.

In the 2011-12 Mid Year Economic and Fiscal Outlook (MYEFO) the Government announced that it would retain Reward Funding for universities that meet participation and social inclusion targets. The Government discontinued Reward Funding for student experience and quality of learning outcomes indicators in the context of the Government’s fiscal strategy and on the basis of feedback from the sector that there was no consensus on the issue of whether it is appropriate to use such indicators for Reward Funding.

Universities have acknowledged the need to develop a suite of enhanced performance measures for providing assurance that universities are delivering quality higher education services at a time of rapid expansion. The Government will focus on the development of student experience and quality of learning outcomes indicators for use in the MyUniversity website and to inform continuous improvement by universities. The Government has agreed that three performance measurement instruments will be developed over the duration of the first Compact period: a new University Experience Survey (UES), an Australian version of the Collegiate Learning Assessment (CLA) and a Review of the Australian Graduate Survey (AGS). The Government has removed the composite Teaching Quality Indicator (TQI) from the performance indicator framework since universities have made a reasonable case that university performance is best measured using output rather than input indicators.

F4

5

Final Facilitation Funding and Reward Funding Guidelines and Administrative and Technical Guidelines will be released on the Department’s website in December 2011. These will provide an outline of the final Performance Indicator Framework and Performance Funding arrangements.

1.2. Advancing Quality in Higher Education

In the 2011-12 Budget, the Government released details of its Advancing Quality in Higher Education (AQHE) initiative designed to assure and strengthen the quality of teaching and learning in higher education. The announcement provided more information on the new performance measurement instruments being developed and the available funding for developing the instruments. The consultation processes for the initiative were also outlined including the establishment of an AQHE Reference Group to advise on the development and cohesiveness of the performance measurement instruments. The AQHE Reference Group will also assist in the development of discussion papers for each of the instruments. Roundtable discussions with universities, business and students will also be held later in 2012.

1.3. Purpose of this paper

This paper discusses key issues in the design of the performance measurement instruments, assesses their fitness for purpose and their ability to operate together in a coherent way to obtain a comprehensive view of the student’s undergraduate university experience and learning outcomes. This paper does not describe in detail how the measurement instruments will be implemented - these issues will be outlined in separate discussion papers on each of the instruments.

The second section of this discussion paper considers some principles to guide the development of new performance measurement instruments. It also proposes that a framework of the student life cycle be used to situate the development of new performance measurement instruments. The third section briefly discusses existing survey instruments. The development of new performance measurement instruments for use in performance reporting and for other purposes is discussed in the fourth section. The fifth section considers key issues that impact on the coherence and balance of performance measures. The final section outlines a proposed implementation strategy for the new performance measures.

F5

6

2. Principles and the student life cycle framework

2.1. Principles

It is desirable that the development of new performance measurement instruments is guided by principles. The Department has previously published principles underlying the publication of institutional performance indicators (DETYA, 1998 and DEST, 2001). While these focused on existing data and indicators, these principles have been adapted to guide the development of performance measurement instruments as follows:

Fit for purpose – information is used to suit the purposes for which it is designed to be used; Consistency- information is consistently collected and applied across uses and time; Auditability – information can be scrutinised; Transparency – information has clear meaning; and Timeliness – information readily enables institutions to enhance their quality of teaching and

learning.

Questions for Discussion

Are there other principles that should guide the development of performance measurement

instruments?

2.2. Student life cycle framework

This section proposes that the development of new performance measurement instruments is most appropriately situated within a student life cycle framework.

In a globally competitive economy, the demand for highly educated workers is increasing. This is why the Government has set itself the ambition that 40 per cent of all 25-34 year-olds will hold a bachelor level qualification or above by 2025. The move to the new demand-driven funding system will support achievement of this goal and enable universities to respond better to meet the needs of the labour market. As the system expands, there needs to be assurance that universities continue to deliver high quality education services. Given the Government’s ambition, the development of three performance measurement instruments will focus on the quality of undergraduate teaching and learning.

From a life cycle perspective, an undergraduate student can be considered to proceed through three key phases: pre-entry, university study and post-study as shown in Figure 1. Pre-entry refers to when the student first considers entry to university, applying and enrolling in university. There are distinct phases within university study itself, with the first year of study often considered critical followed by the middle years and the final year of study. Post (undergraduate) study includes completion and graduation and post-graduation outcomes, typically focusing on employment and further education.

Within each of these phases, there are different aspects of the undergraduate life cycle as suggested in Figure 1. These aspects may be applicable to other phases, and other years within each phase, than

F6

7

those identified in Figure 1, however the figure represents when these aspects would be considered most appropriate to measure. For example, retention is not an issue that only relates to first year students, however, data suggests the attrition rate is higher for first years; therefore it is of greater importance to measure retention in first year, as compared to later years.

Readiness to study and pathways to study represent key aspects of the pre-study phase. Similarly, support, engagement, experience and satisfaction are crucial for retention in the first year of study. The student’s university experience, engagement, satisfaction. The quality of teaching and learning and student learning outcomes are all aspects that prevail throughout university study. Possibly more pertinent in the final year of study are the preparedness for employment, further study and achievement of skills. In the post-study phase, key outcomes to be measured include graduate satisfaction, employment and further education and these can be measured at various points in time following graduation.

There are a number of uses and purposes for performance information. These include: institutional strategic planning; teaching and learning plans/frameworks; TEQSA provider, teaching and learning and information standards; the Australian Qualifications Framework; and, the MyUniversity website. It is likely that stakeholders will attach different priority to each of these uses. The different uses for performance information will also lead to different points of emphasis and information requirements. Requirements will vary for quantitative and qualitative information and national, institution, course or subject level information. Figure 2 briefly describes how existing performance measurement instruments currently measure aspects of the student life cycle, though this is by no means intended as a comprehensive coverage of existing instruments. The new performance measurement instruments are also situated within the student life cycle framework, as shown in Figure 3. The new performance measurement instruments have been presented as it is currently proposed for their use. For example, it is proposed that the UES primarily measure the experience of first year students, and potentially final year students. Similarly, the CLA has been proposed as an instrument which will assess first and final year students. The AGS is displayed as it is currently administered, that is, in the year after completion. The actual implementation of each of the instruments throughout the student lifecycle will be a consideration throughout the development of each of the instruments.

There is scope for measurement of other aspects of the student life cycle, for example, through outreach, orientation, placement/fieldwork and employer satisfaction surveys. However, the future development of performance measurement instruments in this area would need to be guided by appropriate cost-benefit evaluations of such activities.


Is it appropriate to use a student life cycle framework to guide the development of performance

measurement instruments?

Are there other aspects that should be included in the student life cycle framework?

F7

8

Figure 1 : Student life cycle – key aspects to be measured

Pre-entry

University (undergraduate) study

Post- study

Application/admissions and enrolment

First year Middle years Final year

Completion and graduation (1 year out)

Greater than 1 year out

Readiness for university study

Readiness for university study

Graduate satisfaction with study

Pathway to study

1st year experience/ engagement and

satisfaction with study Overall study experience

Level of support received

Graduate employment and further study

outcomes

Continued employment and education outcomes

University experience and engagement

Course satisfaction

Quality of teaching and learning

Student learning outcomes

Preparedness for employment and further

study

Achievement of skills

F8

9

Figure 2: Student life cycle – measurement instruments commonly in use

Pre-entry

University (undergraduate) study

Post- study



Completion and graduation (1 year out)


AUSSE

AUSSE

FYEQ

Retention and progress

Completions

ISB

Institution course evaluations

AGS - CEQ

AGS - GDS

BGS

Figure 3: Student life cycle – new instruments for performance reporting

Pre-entry University (undergraduate) study Post- study



Completion and graduation (1 year out) Greater than 1 year out

UES

UES

CLA CLA

Revised AGS - CEQ

Revised AGS - GDS

F9

10

3. Existing surveys

3.1. National and cross-institution surveys

Australia is already well served by existing national and cross-institution surveys. National surveys, for example the Course Experience Questionnaire, are administered to at least all Table A higher education providers. Cross-institution surveys, for example the First Year Experience Questionnaire (FYEQ), are narrower in scope and include a selection, but not all, universities. Figure 2 above shows the major existing survey instruments measuring the performance of institutions situated in the student life cycle. As noted earlier, this is not intended as a comprehensive coverage of existing instruments.

Pre-entry The paucity of surveys relating to pre-entry is a notable feature of Figure 2. Though not listed in Figure 2, there is limited information available relating to pre-entry. The National Data Collection does provide sector wide information on trends in applications, offers and acceptances each year, though institution level data are not published. The Higher Education Statistics Collection provides information on the basis of admission of commencing students and the Australian Tertiary Admissions Rank (ATAR) of students admitted on the basis of secondary education and this information is published at sector and institution level. The Higher Education Participation and Partnerships Program (HEPPP), as part of accountability and reporting arrangements, collects information on outreach activities undertaken by institutions. The Longitudinal Survey of Australian Youth (LSAY) is probably the best current source of information available on the pathways of students into higher education.

University study The Australasian Survey of Student Engagement (AUSSE) is probably the most extensive survey measuring the experience of current university students. It is administered to first and final year students at over thirty universities. The survey is based on the United States National Survey of Student Engagement and provides international comparisons with United States and Canadian universities. The First Year Experience Questionnaire (FYEQ), as the name suggests, is focused more on the experiences of first year students and has been conducted every five years among a representative group of around ten to twelve universities. Of more recent origin is the International Student Barometer (ISB) administered for the first time in 2010 to over thirty universities. This survey tracks the decision-making, perceptions, expectations and experiences of international students studying outside of their home country. The survey is also administered to over one hundred and sixty universities in Europe, North America, South Africa, Singapore, New Zealand and UK providing internationally comparable results.

There are a plethora of surveys of teaching and learning conducted within institutions. These are conducted across the institution, at course level and even down to the subject level. These are used extensively to manage the quality of teaching and learning.

In addition, universities make extensive use of data from the Higher Education Student Statistics Collection to monitor key student learning outcomes such as retention and progress rates.

Post-study The Graduate Destination Survey (GDS) is the longest running example of a national survey of institutions. Since 1972, it has measured the employment and further study destinations of recent

F10

11

graduates four months after they have completed their course. The Course Experience Questionnaire (CEQ) has been conducted in conjunction with the GDS since 1993. It measures graduate satisfaction with aspects of their courses known to be associated with the quality of student learning such as teaching, goals and standards, assessment, workload and generic skills. As such, measures of graduate satisfaction are used to drive improvement in the quality of student learning. The United Kingdom adopted the Course Experience Questionnaire in 2005 and consequently provides points of international comparison. The Beyond Graduation Survey is of more recent origin and has been conducted annually on three occasions in over twenty universities. The survey tracks graduates three years after completion of their studies with a primary focus on the main activity of the graduate at the time of the survey.

For a more comprehensive description of the administration, scope and measures underlying existing survey instruments, see Appendix 5.

3.2. Links with new instruments

This brief description demonstrates that there is a considerable level of activity currently underway in relation to performance measurement instruments and surveys. There is a substantive issue in the development of new performance measurement instruments about their overlap with existing survey instruments, given the potential costs and resource burden. There are key considerations that will influence the need to balance existing and new performance measurement instruments. First, providing international benchmarks of performance is a major pillar of quality assurance both at national and institutional level. Second, performance measures developed at national level can provide a more robust and comprehensive assessment of performance than measures confined to a particular institution or subset of institutions. Third, assessment and reporting leading to continuous improvement is a dynamic process. Existing surveys and established research and evidence provide a basis for the development of enhanced performance measurement instruments for use in quality assurance. Fourth, the many and varied uses of performance measures, for example, reporting on the MyUniversity website and continuous improvement, will guide the balance between existing instruments and development of new performance measurement instruments whilst ensuring at the same time that these are fit for purpose.

F11

12

4. New instruments

In December 2009 the Department of Education, Employment and Workplace Relations (DEEWR) released the discussion paper An Indicator Framework for Performance Funding. As a result of feedback from the sector in response to the performance indicator framework, the Government announced that its performance framework would include two newly developed indicators – the University Experience Survey (UES), and an Australian version of the Collegiate Learning Assessment (CLA). In addition, the Government announced in the 2011-12 Budget that there would be a review of the existing Australian Graduate Survey (AGS).

These instruments are being developed jointly to provide a coherent suite of indicators supporting both performance reporting and achievement of continuous improvement in learning and teaching. In developing the indicators, the Department has been informed by the student lifecycle model discussed in Section 2. Using the student life cycle model, it is readily apparent that the development of new performance measurement instruments will not provide a comprehensive assessment of the quality of higher education. Taken together, the indicators will nevertheless provide an enhanced assessment of the impact universities are having on students’ learning as they progress through the higher education system. As noted above, the development of performance measures and their use in quality assurance is a dynamic process.

4.1. University Experience Survey

Purpose Australia has a rich history in designing survey instruments for higher education, providing a strong foundation and setting high expectations for the UES. The UES will be designed for use in performance reporting on the MyUniversity website and for use by universities for continuous improvement purposes. It is intended that the UES will build on existing student experience instruments. It will be a highly-focussed instrument that is operationally efficient to implement. The UES could potentially provide a better and more timely measure of student experience.

The development of the UES will have a strong theoretical and empirical underpinning and will consider existing surveys and scales of student experience both within Australia and internationally. Although intended to be initially used with first-year students, the UES has been designed with the potential to also be used with later-year students, and sector feedback to date suggests there is support for this wider application. UES results for first year students will provide an indicator of university experience as students transition into higher education from school or other entry pathways. This initial transition period is known to be crucial in achieving long-term learning outcomes. Administering the UES to later year students would potentially provide universities with a more complete picture of students’ learning experience as they progress through the higher education system.

F12

13

What the UES measures The UES will measure aspects of student’s university experience associated with high level learning outcomes such as teaching and support, student engagement and educational development. To do this, the UES will measure the most salient aspects of student experience known to be associated with high-level learning outcomes. In this way, the UES allows the sector and individual institutions to monitor and support cycles of improvement in the quality of university teaching and learning.

The UES measures facets of the first-year and later-year experience including learning and education that can be generalised across institutions and contexts, and that can be shaped and influenced by institutions. In doing so the UES uses methods that are scalable and at the same time locally relevant. As well as being generalised in this way, the UES will also be informed by international developments.

How the UES will work The Department selected a consortium, led by the Australian Council of Educational Research (ACER) and including the University of Melbourne’s Centre for the Study of Higher Education (CSHE) and the Griffith Institute for Higher Education, to develop the UES. The consortium has extensive expertise and knowledge of the Australian higher education sector and in designing and conducting complex national student surveys.

The consortium released a design consultation paper for the UES in April and a national forum attended by representatives from the higher education sector was held on 3 May 2011. A Project Advisory Group, consisting of higher education stakeholders including universities and students has also been consulted throughout the development of the instrument.

The UES has been designed, developed and piloted during 2011 among first year undergraduate students at approximately 25 universities. The consortium will provide the department with a report on the pilot project in late 2011 which will provide direction for the future implementation of the UES and its future use. The department will consult with stakeholders regarding the full implementation of the UES and how it will measure student experience.

It is intended that the UES will be conducted annually from 2012 on. A performance baseline will be established using 2012 and 2013 results. Subject to its successful development and trial, it is intended that universities’ UES results will also be published on the MyUniversity website from 2013 onwards.

4.2. Assessment of Generic Skills : The Collegiate Learning Assessment

Purpose By assessing students’ generic skills at the beginning and end of their university career, the impact of their engagement with higher education on these skill sets can be evaluated. In turn, this will drive improvement in the quality of teaching and learning and provide assurance for employers and the wider community that students have acquired the knowledge and skills expected of them for attainment of their degree. The CEQ currently provides a measure of the self-reporting of graduates’ generic skills. Assessment of generic skills providing a direct measure of learning outcomes could potentially be undertaken using the CLA or some variant of the CLA adapted for the Australian context. The OECD has selected the CLA as the basis for assessment of generic skills in its Assessment of Higher Education Learning Outcomes (AHELO) project. Subject to the successful trialling of the CLA in the Australian higher education environment, this offers the prospect of international benchmarking.

F13

14

What the CLA measures The CLA can be administered to first and/or final year undergraduate students to assess their critical thinking, analytic reasoning, problem solving and written communication skills obtained during their university degree. The Government will consult with the AQHE Reference Group and the sector on the best way to measure attainment of generic skills for the purposes of performance reporting and for achieving continuous improvement within universities.

How the CLA will work The Government will consult with the sector to develop a culturally appropriate version of the CLA for the Australian higher education sector. In addition, consideration will be given to the cost and value of inclusion of discipline-specific assessments. A broader indicator of generic skills has to the potential to increase its relevance and widen its appeal. Securing participation of students and universities in the assessment of generic skills will be of critical importance.

The Department will establish and work with the AQHE Reference Group, universities and higher education stakeholders, including business, to adapt the CLA for the Australian higher education environment.

The Department will release a consultation paper in 2011 to identify key issues that will need to be addressed in developing an indicator of generic skills and seek stakeholder feedback on these issues and how best to develop and implement this instrument. This paper will build on the earlier paper, An Indicator Framework for Performance Funding.

It is proposed the Department will develop and pilot the CLA in 2012. This will be for the purpose of implementing the CLA or some version of the CLA suitable for the Australian higher education sector from 2013. Subject to its successful development and trial, it is intended universities’ CLA results will also be published on the MyUniversity website from 2013 onwards.

4.3. Review of the Australian Graduate Survey

Purpose The AGS is a national survey of newly qualified higher education graduates. It consists of two distinct survey instruments – the GDS, which examines employment outcomes for graduates, and the CEQ, which asks graduates about their course experience perceptions (see Section 3 on existing surveys for details). The CEQ provides a measure of student satisfaction with their entire university experience. The GDS indicates graduates’ outcomes in the labour market shortly after they have exited the higher education sector, providing an indication of the value of the skills and qualifications graduates have attained at university in relation to their transition from education to employment.

The AGS is conducted annually by Graduate Careers Australia (GCA), a not-for-profit company governed by a board of directors. The AGS surveys new graduates from all Australian universities, and a number of higher education institutes and colleges, approximately four months after they complete the requirements for their awards. Detailed results are available in the middle of the following year. For students graduating in 2011, for example, surveys will be conducted in October 2011 and April 2012, with detailed results available in mid-2013. The survey response rate for domestic graduates typically ranges from 60 to 65 per cent.

F14

15

Issues A strengthened AGS will form part of the suite of performance measurement instruments developed under the auspices of the AQHE initiative. The review will examine the strategic position of the survey in relation to the other performance indicators, particularly the University Experience Survey which will provide an alternative measure of student satisfaction.

Other aspects which the review will consider include:

Timeliness of reporting; Administrative arrangements, including funding arrangements and the relationship between

GCA, universities and third-party service providers; Data quality, including response rates and bias; Survey methodology, including collection modes and standardisation of the survey instruments

across universities; and Whether the AGS should be conducted on a census or sample basis.

The review will also consider how to better capture aspects of student experience for external, Indigenous, international and low socio economic status students.

How the review of the AGS will work The Department is working with GCA and the higher education sector to review the AGS. The Department will be advised by the AQHE Reference Group, which will include representatives from universities and business, as well as experts in survey design and administration. As a first step, the AQHE Reference Group will provide input and guidance on the development of a discussion paper raising issues and options for the future of the AGS. Stakeholders will be able to provide feedback on this discussion paper on the issues relating to the review of the AGS. The AQHE Reference Group will also be responsible for ensuring that adequate stakeholder engagement is conducted and will identify topics for discussion at the AQHE roundtables.

F15

16

5. Issues

This section discusses key issues that arise from consideration of existing and new performance measurement instruments. This includes the administration of performance measurement instruments, student selection, central sampling of students, uses and the intersection of existing and new instruments. Discussion of these key issues facilitates an assessment of the fitness for purpose of performance measurement instruments and their ability to operate together in a coherent way to obtain as comprehensive view as possible about the student’s undergraduate university experience.

5.1. Administration of new instruments

Student surveys tend to be conducted in Australian universities using one of two broad deployment approaches (ACER, 2011, p.14), specifically:

an independent (or centralised) deployment, in which most if not all survey activities are conducted by an independent agency; or

a devolved deployment (or decentralised), in which institutions and a coordinating agency collaborate on survey operations.

An independent deployment approach involves participating universities providing the independent agency with a list of all students in the target sample at their institution, including student’s contact details. After receiving institutions’ population lists, the independent agency would identify the target population, which could be either a census or sample of students, and invite students to participate in the survey. Responses would then be returned directly to the independent agency for analysis.

A devolved approach involves participating universities supplying the independent agency with a de-identified student list that excludes student contact details. A sample of students would be drawn, online survey links would be allocated to student records, and this list would be sent back to universities who would then merge in student contact details. Under a devolved approach, universities manage the deployment of the survey by sending invitations to sampled students and following up with non-respondents. Responses are provided directly to the independent agency for analysis.

Both deployment approaches have benefits and limitations. A devolved approach has benefits in that it can accommodate the needs and circumstances of a diverse array of universities. On the other hand, the fact that universities are primarily responsible for collecting data about their own performance can be seen as a conflict of interest, leading to perceptions that universities may ‘game’ the system. Given the stakes and uses to which the data collected from the new instruments will be used, on balance, an independent approach is favoured since this will promote validity, consistency and efficiency.

A key issue for any independently deployed survey is privacy and the responsibility of universities (and the Department) to preserve the confidentiality of student information they hold. Universities may be required to amend their agreements with students to permit disclosure of personal information to third parties for the purposes of conducting surveys. Providing privacy laws are satisfied in the development of an instrument, this approach has to date received broad support from the higher education sector, as measured through consultation in the development of the UES.

F16

17

While the deployment approach is a significant consideration in terms of administration of a new survey instrument, there are other issues which should be considered. These include, but are not limited to, the administration method (online, telephone, paper-based), context and timing. These issues will be considered in the context of individual instruments.


What concerns arise from an independent deployment method?

What are the obstacles for universities in providing student details (such as email address, first name

and phone numbers) to an independent third party? Would universities agree to change their privacy agreements with their students to permit disclosure

of personal information to third parties for the purposes of undertaking surveys?

What are the other important issues associated with administration of survey instruments?

5.2. Student selection

The selection of students for the new performance measurement surveys is an issue which has raised a number of concerns. The burden of the existing range of survey instruments on both students and university resources is foremost in the minds of universities and therefore a balance needs to be struck between the need for the collection of new data for the purposes of performance reporting and to assure quality and the requirements on students and universities.

The surveys could be run as a census of all students in scope or by administering the survey to a sample of the students in scope. Deciding between a census and a sample is a complex process that necessarily takes into account many technical, practical and contextual factors.

Two major issues are non-response biases and general confidence in the precision of results, particularly at the sub-institutional level. The advantage of a sample survey approach is that a structured sample can be constructed that deliberately focuses on target groups for which it is necessary to generate meaningful results (for example at the course level or for particular demographic groups) and this may assist in overcoming non-response biases. A sample survey approach would require a relatively sophisticated sampling frame to give adequate coverage across fields of education and demographic characteristics. This process would be simplified if the Higher Education Information Management System (HEIMS) database could be used to construct the sample frame, given that it already records detailed information on student characteristics. Standard techniques to measure the precision of sample survey results could be systematically applied across all results, for example, calculating confidence intervals or standard errors.

F17

18

On the other hand, given the small student populations sometimes under consideration (for example, courses where only a small number of students are enrolled at a particular institution), sample sizes needed to provide confidence in survey results would approach the total population. The intention to publish data from the new performance measurement instruments on the MyUniversity website disaggregated to subject level may influence the decision on whether to conduct a census or survey since a sufficiently large number of responses will be required to ensure data are suitably robust and reliable. In this case, it may be preferable to continue on a ‘census’ basis where the whole population is approached to participate.

Other issues for consideration in deciding between a census or survey approach include:

Support by participating institutions; The size and characteristics of the population; Providing students with opportunities for feedback; Relationship with other data collections, in particular other student surveys; Analytical and reporting goals, in particular sub-group breakdowns; Anticipated response rates and data yield; Consistency and transparency across institutions; Cost/efficiency of data collection processes; and The availability of supplementary data for weighting and verification.

The method of student selection may vary between instruments, and regardless of whether a census or sample approach is used, proper statistical procedures will be used to evaluate the quality and level of response in the long term.


What are key considerations in choosing between a sample or census approach to collection of

performance data?

5.3. Central sampling of students

As discussed above, an important issue regarding the introduction of new surveys within the higher education sector is the perceived burden on university resources and the students who are required to participate.

One method which has been proposed to assist in the reduction of this burden is the possibility of using DEEWR HEIMS (Higher Education Information Management System) data to better control student sampling and also to use stored student demographic level data to pre-populate survey questions where appropriate.

Using the HEIMS data in this way could potentially improve random sampling and avoid oversampling of students invited to participate in surveys, while also reducing the number of questions students are required to answer per survey through the ability to pre-fill and skip questions where data is already available. In addition, by having the Department involved at this level in the survey process, this could

F18

19

improve perceptions of the overall integrity of surveys through making clear that samples are independently constructed.

Note there are restrictions regarding the use of HEIMS data in the Higher Education Support Act. The Department, therefore, is investigating options for the use of this data in the context of individual performance measurement instruments and also as a suite of instruments.


What are the advantages and disadvantages of central sampling of students?

5.4. Uses of data

The collection of data from the new performance measurement instruments will be of major benefit to universities for continuous improvement and to assure the quality of teaching and learning in the sector. The Government has also indicated that, subject to their successful trial and implementation, it is intended that data from the new instruments will be published on the MyUniversity website.

While a number of uses are proposed for the new data collections, other uses may be discovered throughout the development of the new instruments. It will be important, therefore, to consider potential future uses of the data while being aware of the potential for misuse of the data.

By utilising the principles referred to in section 2.1 of this paper in the development of the instruments, the data available should be relevant, reliable, auditable, transparent and timely. These principles will also provide guidance as to the appropriate uses of the data. Further, the range of consultations being undertaken in the development of the new instruments will include roundtables where stakeholders will be able to discuss how the results from the new instruments can be used while raising any concerns regarding the future use of the data. In addition, it may be appropriate to establish codes of practice that guide appropriate uses and interpretation of performance information.

Continuous improvement It will be important that the data collected from the performance measurement instruments is useful for universities, so that they can implement processes and policies for continuous improvement and maintain a high level of quality of teaching and learning.

To ensure the performance measurement instruments collect data that is useful and relevant to universities, the instruments are being developed with significant sector consultation. Stakeholders will be invited to provide feedback throughout the development of the new instruments to allow these to take account of their data and measurement needs.

Using the student life cycle model, consideration needs to be given to what information can and should be collected from the new instruments given their implementation at different stages of the life cycle, and how this information will assist to assure the quality of teaching and learning in Australian universities.

F19

20

MyUniversity It is expected that in future releases, the MyUniversity website may include performance data from the new performance measurement instruments. This would allow prospective students additional information with which they can assess the performance and quality of different institutions in the three performance categories. How the data will presented on the MyUniversity website will be a consideration once the instruments have been tested and the level of data analysis is known.

Information regarding the use of performance data on the MyUniversity website will be made available throughout the development process for the instruments and the website itself.

Another issue that arises in consideration of the MyUniversity website is the level of reporting. A major purpose of the website is to inform student choice about courses and subjects. In this environment, more detailed reporting is likely to be desired, for example, at the field of education level. There is likely to be a trade off between collection and reporting of data from the new performance measurement instruments at a finer level of disaggregation and adding to the complexity and burden of reporting.


What are appropriate uses of the data collected from the new performance measurement

instruments?

5.5. Intersection of existing and new instruments

The development of the University Experience Survey (UES) has raised the issue of whether the new instruments should be focused instruments for the purposes of performance reporting, or whether they could potentially be expanded to replace existing surveys and institution/course specific questionnaires. For example, what is the potential overlap between the newly developed University Experience Survey and the Course Experience Questionnaire in measuring student experience?

This will be a consideration in the development of all new instruments to ensure there is balance between the additional burden on both students and universities of the new instruments and ensuring they are able to capture targeted and purposeful information. Further, there needs to be consideration of the data needs of individual universities and how these differ across the sector.

A key issue in considering the overlap between the new instruments and existing survey instruments is the uses to which performance measurement data will be put as discussed above. It is expected students will benefit from the availability of new data via the MyUniversity website. The new performance measurement instruments will potentially be used to enhance continuous improvement processes within universities. There is also the potential for international benchmarking.

A major dilemma is the need for broad level national data and the requirement for more detailed data to suit universities’ diverse needs and missions. Satisfying both of these goals raises the important issue of costs and resource burden. One suggested solution for the UES is that there could be a core set of items which would be asked of students at all universities, and an optional set of non-core items which universities could select to suit their individual requirements.

F20

21

Ultimately, universities will decide in which instruments they participate, and this will hinge on a range of factors not limited to the type of data collected. By considering, however, the range of existing surveys, what data is most useful to universities, and what additional data universities would like to collect from the new performance measurement instruments, the development of the new instruments has the potential to make this decision by universities considerably easier.


Are there other issues that arise when considering the overlap of existing and new instruments?

F21

22

6. Next Steps

The Government will work closely with higher education stakeholders, including universities, business and students during the continued implementation of the Advancing Quality in Higher Education (AQHE) initiative. To ensure that comprehensive feedback is received, the Government will consult through a variety of means including through the release of discussion papers for sector feedback and holding a series of roundtable discussions.

6.1. The AQHE Reference Group

The Government has established the AQHE Reference Group to provide advice on the design, development and implementation of the new measurement instruments.

A key role of the AQHE Reference Group is to share ideas and contribute expertise in discussions about the suite of performance measurement instruments and provide advice about the cohesiveness of the instruments to develop a comprehensive picture of teaching and learning performance in the Australian higher education sector.

The Reference Group will:

advise the Department on the coherence and balance of the measures derived from instruments, consistent with meeting the Government’s stated objectives;

advise on the approach to implementation of the instruments as a suite; guide consultation and synthesise feedback on the Review of the Australian Graduate Survey; provide input and guidance on the development of a discussion paper on a vision for the future

of student surveys in Australia, and lead consultation on the paper with the sector; and advise on the development and implementation of new performance measurement

instruments.

The full Terms of Reference for the AQHE Reference Group are shown in Appendix 3 and are available on the Department’s website.

The Reference Group will also provide advice and guidance around the detailed design and implementation issues for each instrument. This includes providing advice on the methods, timeliness and costs of data collection. The Reference Group will also advise on stakeholder engagement. This includes providing input and guidance on the development of Discussion Papers for each of the measurement instruments, identifying key issues for discussion at the roundtables and advising on the feedback received at the roundtable sessions.

The Reference Group includes representatives from universities, business, unions and students who are highly regarded for their expertise in relevant fields. The Department will provide secretariat support to the Reference Group. The advice of the Reference Group will inform the Department’s advice to the Minister for Tertiary Education, Skills, Jobs and Workplace Relations. The first meeting of the AQHE Reference Group was held in late August 2011. The Group provided direction and guidance for developing this discussion paper and also advised on the upcoming roundtable discussions.

F22

23

6.2. Discussion papers

A number of discussion papers will be released for public consultation in December 2011. These papers will be developed in consultation with the AQHE Reference Group. These papers include:

the Development of Performance Measurement Instruments in Higher Education; the Review of the Australian Graduate Survey; and Assessment of Generic Skills.

A consultation paper on the design of the University Experience Survey was released in April 2011 to higher education stakeholders. A report presenting details of the development, design and trial of the UES and findings will be published in December 2011.

6.3. Roundtable discussions

Roundtable discussions will be held to provide a forum for higher education stakeholders, including universities, business/industry and students, to debate and contribute to the development of strategies designed to further advance quality in Australian higher education. The principle focus of the roundtables will be the development of an integrated suite of performance measurement instruments that will improve transparency in university performance.

One-day roundtables will be held in major capital cities and will be based on the information and issues outlined in the discussion papers. The roundtables may include presentations, an open forum or a combination of the two.

6.4. Next steps

The consultation mechanisms outlined above will be used to gather sector feedback which will inform advice provided to the Minister for Tertiary Education, Skills, Jobs and Workforce Development.

The Government welcomes feedback on the discussion questions outlined in this paper. Instructions for how to lodge submissions can be found in Appendix 2 to this paper.

The instruments will be developed during 2012 and 2013. It is envisaged this will enable university performance baselines to be established by the end of 2013. Subject to successful development and trial, it is intended that universities’ results against each of the indicators will be published on the MyUniversity website from 2013.

F23

24

Appendix 1 - References

Department of Education, Training and Youth Affairs (DETYA), 1998, The Characteristics and Performance of Higher Education Institutions, Occasional Paper 98-A, Higher Education Division, December.

Department of Education, Science and Training (DEST), 2001, Characteristics and Performance Indicators of Australian Higher Education Institutions, Occasional Paper 01-B, Higher Education Division, December.

Australian Council for Educational Research (ACER), 2011, University Experience Survey Design Paper.

F24

25

Appendix 2 – How to make a submission

We would welcome your comments on the questions and issues raised in this discussion paper. Developing and implementing the new performance measurement instruments requires a strong evidence base and we would ask that you provide any evidence you have to support your views. Submissions received through this process will be used to inform deliberations of the Advancing Quality in Higher Education Reference Group and subsequent advice to the Minister for Tertiary Education, Skills, Jobs and Workplace Relations, the Hon Christopher Evans MP.

Submissions should be lodged by close of business 17 February 2012.

By email: [email protected]

By post: Andrew Taylor, Branch Manager

Policy and Analysis Branch

Higher Education Group

Department of Education, Employment and Workplace Relations

PO Box 9880

CANBERRA CITY ACT 2601

Please clearly identify your submission showing

Name of Organisation or Individual If an Organisation, please indicate the name of a contact person Address Email Phone

Please note that all submissions will be published on the DEEWR’s Higher Education website.

DEEWR will not accept submissions from individuals submitted on a wholly confidential basis; however, submissions may include appended material that is marked as ‘confidential’ and severable from the covering submission. DEEWR will accept confidential submissions from individuals where those individuals can argue credibly that publication might compromise their ability to express a particular view.

Please note that any request made under the Freedom of Information Act 1982 for access to any material marked confidential will be determined in accordance with that Act.

F25

26

Appendix 3 – Terms of Reference for the AQHE Reference Group

Context

The Australian Government is strongly committed to ensuring that growth in university enrolments is underpinned by a focus on quality.

The Government’s Advancing Quality in Higher Education initiative will assure and strengthen the quality of teaching and learning in higher education. As part of this initiative, the Government has announced that an integrated suite of performance measurement instruments will be developed during the first Compact period from 2011 to 2013 for use in performance reporting - the University Experience Questionnaire, an Australian version of the Collegiate Learning Assessment, and the Review of the Australian Graduate Survey.

The Government’s performance indicator framework is to include indicators of the quality of student experience and learning outcomes. It is intended that indicators of performance drawn from the new instruments will also be published on the MyUniversity website.

Collaboration and consultation with sector on the development and implementation of the new instruments is a high priority for Government. It wishes to ensure that the knowledge and expertise of the sector is applied to Australia having the best possible measures of teaching and learning excellence.

The Reference Group

The Government will establish a Reference Group to provide advice to the Department on the development and implementation of the performance measurement instruments.

The Reference Group will:

advise the Department on the coherence and balance of the measures derived from instruments, consistent with meeting the Government’s stated objectives;

advise on the approach to implementation of the instruments as a suite; guide consultation and synthesise feedback on the Review of the Australian Graduate Survey; provide input and guidance on the development of a discussion paper on a vision for the future

of student surveys in Australia, and lead consultation on the paper with the sector; and advise on the development and implementation of new performance measurement

instruments.

The Department will provide secretariat support to the Reference Group.

The advice of the Reference Group will inform the Department’s advice to the Minister for Tertiary Education, Skills, Jobs and Workplace Relations.

F26

27

Appendix 4 – Membership of the AQHE Reference Group

Name Position

Professor Ian O’Connor (Chair) Vice-Chancellor – Griffith University

Professor Richard Henry AM Deputy Vice-Chancellor (Academic) – UNSW

Professor Carole Kayrooz Deputy Vice-Chancellor (Academic) – University of Canberra

Professor Judyth Sachs Deputy Vice-Chancellor (Provost) – Macquarie University

Professor Jim Barber Vice-Chancellor –University of New England

Professor Richard James Pro Vice-Chancellor (Participation and Engagement), Director of the Centre for Studies in Higher Education, University of Melbourne

Ms Sue Mikilewicz Director Planning and Institutional Performance, UniSA

TBA Chair, Tertiary Education Quality Standards Agency (TEQSA) Teaching and Learning Standards Panel

Ms Claire Thomas Policy Director, Business Council of Australia (BCA)

Mr Jesse Marshall President, National Union of Students

Ms Jeannie Rea National President, National Tertiary Education Union

F27

28

Appendix 5 – Summary of existing surveys

Survey Who runs it? Who participates? What does it measure? How does it operate?

Course Experience Questionnaire (CEQ)

Graduate Careers Australia

Recent graduates (domestic and international students) from all Australian Universities – four months after completion

The CEQ surveys recent graduates on perceptions of their university experience.

In the CEQ, respondents are asked to rate the extent to which they agree or disagree with core items and a subset of optional items, the latter varying by institution.

The core CEQ items constitute:

Good Teaching Scale (GTS) Generic Skills Scale (GSS) Overall Satisfaction Item (OSI).

The eight optional CEQ scales comprise:

Clear Goals and Standards Scale (CGS) Appropriate Workload Scale (AWS) Appropriate Assessment Scale (AAS) Intellectual Motivation Scale (IMS) Student Support Scale (SSS) Graduate Qualities Scale (GQS) Learning Resources Scale (LRS) Learning Community Scale (LCS).

While the CEQ is designed to measure the most significant aspects of the student learning experience, it is not designed as a measure of all aspects of the student experience. Rather than seeking to measure

Annual survey completed by recent graduates four months after completion of a degree.

GCA manages the AGS nationally, while institutions generally conduct the surveys of their own graduates and return survey forms and/or data files to GCA for processing. This method of management can be characterised as partially decentralised in that while a great deal of the work is managed centrally by GCA, key tasks such as the distribution of survey instruments and collection of responses are managed by the institutions.

F28

29


the full range of factors that combine to form student experience, the development of the CEQ was premised on the association between the quality of student learning and student perceptions of teaching as reflected in formal student evaluation.

Graduate Destinations Survey (GDS)


Recent graduates (domestic and international students) from all Australian Universities – four months after completion

The Graduate Destination Survey essentially collects data regarding the immediate post-study activities of new graduates (including full- and part-time employment and labour market activity, further study, job search methods, and the relationship between employment and higher education qualifications).

Annual survey completed by recent graduates four months after completion of a degree.

GCA manages the AGS nationally, while institutions generally conduct the surveys of their own graduates and return survey forms and/or data files to GCA for processing. This method of management can be characterised as partially decentralised in that while a great deal of the work is managed centrally by GCA, key tasks such as the distribution of survey instruments and collection of responses are managed by the institutions.

Australasian Survey of Student Engagement (AUSSE)

Australian Council for Educational Research

First-year and third-year onshore students from higher education providers who choose to participate

The AUSSE focuses on what students are actually doing rather than a focus on student satisfaction and agreement.

The AUSSE surveys students on around 100 specific learning activities and conditions along with information on individual demographics and educational contexts. The instrument contains items grouped by six student engagement scales:

Academic Challenge – the extent to which

The AUSSE is undertaken annually and survey administration is managed centrally by ACER and key activities are conducted by institutions.

F29

30


expectations and assessments challenge students to learn;

Active Learning – students’ efforts to actively construct knowledge;

Student and Staff Interactions – the level and nature of students’ contact and interaction with teaching staff;

Enriching Educational Experiences – students’ participation in broadening educational activities;

Supportive Learning Environment – students’ feelings of support within the university community; and

Work Integrated Learning – integration of employment-focused work experiences into study.

First Year Experience Questionnaire (FYEQ)

Centre for Study in Higher Education (University of Melbourne)

First year students at selected institutions

The FYEQ provides information regarding the student experience of the transition to university study, and the quality of the educational experience for first-year students.

Conducted in five-year intervals, the studies allow an insight into the student experience of the transition to university study. Reports of the survey devote particular attention to important subgroups such as international, Indigenous and rural students, and students reflecting low socioeconomic background.

The FYEQ is undertaken every five years. Survey administration is managed centrally by CSHE with assistance from institutions.

Institutions had the choice of mailing out the surveys themselves or providing the CSHE with an electronic list of the sample to be mailed out by a Melbourne-based mailing-house.

F30

31


International Student Barometer (ISB)

i-graduate All international students at participating universities

The ISB tracks the decision-making, perceptions, expectations and experiences of students studying outside their home country.

The dimensions of the international student experience which are covered include decision-making, arrival, learning, living and support services. Within these sections, students are asked to rate the importance of and satisfaction with a number of elements, including for example, the quality of teaching; social activities and facilities; the surroundings outside the university; the library facilities; internet access; accommodation quality and cost; and making friends with local people and students as well as students from other countries.

Undertaken in 2010 only in Australia at this stage.

International students were invited to participate in the survey via email alerts sent by their institutions and they were able to complete the online survey questionnaire.

The survey was conducted in English.

Responses were collected and analysed by i-graduate.

Beyond Graduation Survey (BGS)


All Australian tertiary graduates from all award levels approximately three years after the completion of their studies (at participating universities)

The primary focus of the BGS is the main activity of the graduate at the time of the survey—be it work, study or something else entirely—although information concerning the various other activities in which the graduate has been engaged in the years between course completion and the present is also collected. In addition to detailing their activities, graduates are invited to make a retrospective assessment of both their course experience and the contribution that their higher education experience has made to their lives.

Annual survey undertaken three times to date.

Of the 23 universities that participated in the 2009 BGS, nine opted to conduct their own data collection fieldwork while the remaining 14 elected to have GCA conduct the survey on their behalf.

F31

G1

G2

1

The Development of Performance Measurement Instruments in Higher Education: A Discussion Paper Response

The University of Western Australia February, 2012

The University of Western Australia (UWA) welcomes the opportunity to make this submission in response to the recent discussion paper, ‘Development of Performance Measurement Instruments in Higher Education’. As a response to one of three papers released concurrently by DEEWR, it may be read alongside our other responses to those discussion papers, ‘Assessment of Generic Skills’ and “Review of the Australian Graduate Survey’. The University notes that in respect of performance measurement, it supports the following principles:

the maintenance of robust minimum standards, without unnecessary intrusion upon institutional autonomy;

the avoidance of duplication in data made available to the government by the University, in various contexts;

the adoption of clear definition, and performance measurement instruments that are fit for that defined purpose, characterised by valid methodologies and whose results are subject to expert analysis and peer review;

a focus upon outcomes, rather than upon processes. Alongside these principles, we note those elaborated within your current papers (fitness for purpose; consistency; capacity to be audited; transparency; and timeliness). Others we believe should be added to that list include:

validity and reliability

relevance

cost effectiveness There are some proposals within the papers which UWA would wholeheartedly support (for example, a shift to central administration of the Australian Graduate Survey). However, there are other features within the papers which UWA views with some concern. We note, for example, that matters canvassed in our earlier submissions concerning the validity, reliability, cost and utility of the Collegiate Learning Assessment (CLA) have gone unremarked. Similarly, UWA notes that the purpose of some envisaged data provision and the multiple purposes to which it is envisaged to be put, is not clear. We are concerned that such data collection will exceed the intended purpose of performance measurement for quality assurance, and will instead stray into areas for which TEQSA has recently been established. We have also provided responses to specific questions posed within the original paper. We trust that these may prove useful to the Department in addressing the substantial issues raised, and in devising appropriate performance measures for our sector.

G3

2

Development of Performance Measurement Instruments in Higher Education This Discussion Paper provides an overview of the Government’s proposals to develop and introduce a number of performance measures. The accompanying discussion papers further extend the discussion on the ‘Assessment of Generic Skills’ and “Review of the Australian Graduate Survey’. While UWA will respond to each of these separately, our core response is based in our response to this paper in particular. In overview, UWA would register some areas of general concern about this paper, as follows:

1. it is not explicitly receptive to the institutional responses made to earlier discussion papers released in December 2010 and October 2011, which pinpointed the lack of clarity of purpose surrounding some proposals;

2. in particular, there is overlap into areas of oversight for which TEQSA has been established, particularly in the area of learning outcomes and standards;

3. the performance funding that was to accompany the introduction of these measures has been withdrawn which brings into question the cost of proposed activities relative to their benefit. Cost effectiveness is not addressed in the paper and is an important consideration.

4. the ways in which these performance measures are to be implemented is not addressed nor details provided as to how data is intended to be made available and reported. Clear codes of practice, statements of purpose and application need to be developed prior to the generation and publication of any data.

The paper is unclear about the ultimate purpose of the performance measures and the use of the data. The paper indicates that the purpose of the performance measures is for both quality assurance and quality enhancement. UWA supports and recognises the Government’s legitimate interest in quality assurance and monitoring. It does not support proposals which intrude on institutional responsibility for enhancement of quality. The Government’s concern with the quality of the provision of higher education is indisputably appropriate. It already gathers significant data for review to this end, relating to student enrolment, retention and completion, and monitoring of satisfaction and employment outcomes through the AGS. The proposed additional tools and measures mooted in the current discussion paper, to fill gaps the Government discerns, are the University Experience Survey, the revised Australian Graduate Survey and the Graduate Skills Assessment (Collegiate Learning Assessment). These are coherent, although not uncontroversial ways to ‘assure’ the quality of higher education. There is digression from this legitimate focus on assurance, however, where additional data is gathered for ‘enhancement’ purposes. We contend that the role of enhancing the quality of higher education resides primarily with the institution. Further, UWA is concerned that these performance measures appear to be directed at the university sector only. We would argue that the higher education sector as a whole (public and private) must be required to provide and participate in the measurement of performance using the same instruments and processes, to best assure quality.

2. Principles and the student life-cycle framework. Are there other principles that should guide the development of performance measurement instruments? In addition to the principles listed, the following are suggested:

Relevance – the measurement should be of relevance and meaningful to inform the institution and the students, rather than providing figures for reporting purposes alone. Data should also be generated in a manner to inform the institution at the most useful level, not merely to generate comparison tables. Course level data, rather than fields of study may be the most appropriate level. Often, there is more intra-institutional variability than inter-institution variability: data needs to be

G4

3

gathered appropriately, to inform relevant degree programs, not at artificially constructed fields of study.

Universality of application– performance measures and instruments should apply to all approved higher education providers, not just to Self-Accrediting Institution (SAI) university providers.

Comparability - the ability for an institution to compare performance of student cohorts through time and across institutions for quality benchmarking purposes

Is it appropriate to use a student life-cycle framework to guide the development of performance measurement instrument? Are there aspects that should be included in the student life-cycle framework? While a student life-cycle framework is useful, its framing within the paper is problematic. It assumes a linear progression of students from school, to university study and then employment. While this is the experience of a large number of students at this university, it does not reflect the student experience in many other universities where students enter and leave and re-enter in a much more complex life-cycle than portrayed. In addition, a large number of students do not enter through obtaining an ATAR score but through alternate pathways, including direct entry into the second year of a program. With the development of multiple pathways for entry into university, a first year survey such as the UES will not capture the experience of an increasing number of these students. While the discussion paper relates to national performance measures and a national perspective of the student life-cycle, elements of Figures 1, 2 and 3 have strayed into current institutional-level quality assurance and enhancement practices and measures, and are therefore not appropriate for inclusion in the final papers as their use is much more fragmented and varied than is suggested in Figure 2. All figures need to reflect a national approach, with individual institutions’ specific responsibility for quality practices and measures left to consideration by them, and by TEQSA as the regulator. As such, reference to institutional responsibilities and practices should be removed from the figures, as these are not within the intended purview of these government initiated performance instruments. We have therefore presented some revised figures for consideration below, re-framed on the following bases:

The original Figure 1 has been retained, but with the areas that should be removed highlighted in blue: these blue areas relate to institutional quality practices which will be subject to the scrutiny of TEQSA;

We argue that ‘student learning outcomes’ should not be separated into three distinct categories, ‘student learning outcomes’, ‘preparedness for employment and future study’ and ‘achievement of skills’. In fact, the latter two categories are both sub-sections of student learning outcomes;

By contrast, the Government’s intention to carry out ‘Assessment of generic skills’ requires that the category of “achievement of skills” be separated out, despite the evidence that learning of these skills takes place within a discipline and within a program of study.

Our proposed revised Figure 1 below, therefore, includes only the aspects that are the stated intended focus of Government and include national performance indicators only. All other aspects that relate to institutional practices and responsibility have been removed. We argue that Figures 2 and 3 should be collapsed, to include only the data that is gathered comprehensively from universities at the national level and which directly relate to the revised Figure 1. (i.e., while Figure 1 presents the student life-cycle with its elements that enable the quality of Australian higher education to be monitored nationally, Figure 2 presents the tools which will be used to collect the data (existing and proposed). Many of the instruments in the original Figure 2 are used intermittently and differently by institutions as quality tools, and are not national or universal, and hence should not be included here. If the suggested revisions are made, Figure 3 is no longer required. We would further argue that pre-entry data be included in the data reported. This data includes the ATAR scores of entering students, median entry ATAR scores for the relevant discipline, and the number and

G5

4

type of non-ATAR based entries, which are currently readily available and provide valid pre-entry data which relate to the first two elements identified in Figure 1 (‘readiness for university study’ and ‘pathways to study’.). This data would make a valuable addition to the information sought by students and their parents on the My University website

G6

5

Existing Figure 1 : Student life cycle – key aspects to be measured by national performance measures (with all institutional quality practices highlighted in BLUE to be removed)


Application/admissions and enrolment First year Middle years Final year

Completion and graduation (1 year

out)


Readiness for university

study

Readiness for

university study

Graduate satisfaction

with study

Pathway to study

1st year experience/

engagement and

satisfaction with study

Overall study

experience

There is a limit to the extent that universities can be monitored for their student’s outcomes

Remove -Institutional Level of support

provided

Graduate employment

and further study

outcomes

Remove -Institutional only to review

courses Continued

employment and

education outcomes

Remove -Institutional _University experience and engagement

Remove -Institutional Course satisfaction

Remove -Institutional Quality of teaching and learning

Remove -Institutional Student learning outcomes

Preparedness for employment is encompassed in student learning outcomes

Remove -Institutional Preparedness for

employment and further

study


G7

6

REVISED Figure 1B : Student life cycle – key aspects to be measured by national performance measures




out)


Readiness for university

study

Readiness for

university study

Graduate satisfaction

with study

Pathway to study

1st year experience/

engagement and

satisfaction with study

Overall study

experience

Retention and progress data

Graduate employment

and further study

outcomes


G8

7

(NOTATIONS IN BOLD and highlighted in BLUE )

Figure 2: Student life cycle – measurement instruments commonly in use




out)


ATAR data, pathways data should be provided here as the data is available and standardised

Remove- Institutional

AUSSE

Remove- Institutional AUSSE


FYEQ

Retention and progress data Completions

Remove- Institutional ISB

Remove- Institutional Institution course evaluations

AGS - CEQ

AGS - GDS


BGS

G9

8

Alternative Figure 3: National performance indicators for reporting on My University website Pre Entry University (undergraduate) Study Post study

Application/admissions, enrolment First year Middle years Final year (1 year out) ATAR and alternative entry data

UES CLA Revised CEQ and AGS

Progression Progression Completion

(NOTED IN BLUE and CAPITALS WHERE THESE ARE NOT SUPPORTED

Figure 3: Student life cycle – data and instruments for use for national performance reporting





out) Greater than 1 year

out

No comments on these have been made but these are supported including ATAR and non ATAR entry data

UES

REMOVE UES

REMOVE CLA CLA

Retention and

progress data

Revised AGS -

CEQ

Revised AGS -

GDS

G10

9

3. Existing surveys UWA notes that the surveys discussed in this section are almost exclusively institutionally based tools, used to inform and enhance institutional practice. UWA has readily participated in the First Year Experience Survey and the AUSSE to inform its practice and to benchmark its performance and it surveys its alumni and employers to inform curriculum development. UWA will continue using such tools in the future. However, many of the instruments included in Figure 2 are used only intermittently and partially by institutions in the sector and so do not represent a national or comprehensive data set. For this reason we would argue that they be removed from Figure 2. Since the discussion document relates to a national approach, the focus should be on national performance indicators, including only those surveys and data sources that are national and apply to the whole sector. Pre-entry While as a general principle, the University has reservations about using input measures for the purpose of performance monitoring, our view is that the data on the numbers of applications, basis of admission and the ATAR of applicants obtained from National Data Collections (referred to on page 9 of the Discussion Paper) should be used to provide information of relevance to potential applicants, including alternative entry applicants. This information could be readily included in the My University website, and could be used by applicants in preparing their applications for admission. University Study As noted, the surveys and data sources mentioned here (and in Figure 2) are applicable to quality data gathering and enhancement practices within institutions rather than at national level, and therefore not relevant to the current purpose. Post Study The AGS is a national survey of long standing and the companion discussion paper details our responses to proposed changes to it. The Beyond Graduation Survey is neither national nor widely used. Its purpose has institutional enhancement and course improvement as its focus, and is not relevant to this context. Many universities survey their alumni and employers when undertaking curriculum and department reviews as part of their institutional quality enhancement practice. However, we caution against attempting to attribute continuing employment outcomes to an institution following graduation longer than one year as many factors can impact on employment. National Census data collection is a more appropriate means to gather data on employment and other economic benefits of university study. 3.2 Links with new instruments While we acknowledge that performance measures developed at national level have the potential to provide some more useful assessments of performance, they cannot replace measures confined to a particular institution or subset of institutions. National instruments by necessity focus on issues of common interest across the higher education sector, and, as such, there is an ongoing need for the concurrent development and use of local and institutionally selected benchmarking instruments to address institution-specific issues. 4. New instruments UWA acknowledges the general support in the sector for a University Experience Survey in year one of university study, and also supports the review of the Australian Graduate Survey. There has been, and remains, no widespread support for the introduction of the Assessment of Generic Skills in the higher education sector. UWA’s position has not wavered since its earlier submissions, in respect of the shortcomings and inappropriateness of any planned application of a CLA like instrument in the context of national assessment. Each of these items is addressed briefly here, and developed in more detail in UWA’s responses to the specific companion Discussion papers.

G11

10

4.1 UES The University of Western Australia supports the introduction of the UES as a survey of the experience for first year students. Given that the UES combines aspects of the AUSSE and the first year experience survey, UWA is confident that this new survey will provide the Government, the sector and the University with useful data. UWA is pleased to support its development and implementation. We note the concern expressed by institutions which have a significant number of students entry into the second year of university and so will not be included in a survey of first year students. However, we see no benefit in this survey being administered on a second occasion in the final year of the program of study, beyond the context for which it has been conceived, developed and trialled. While the report on the development of the UES (released on 10 February) suggests it has been developed for use in the first year and final year of undergraduate study, this was not the initial brief outlined in the DEEWR briefing papers, nor in its initial development. Final year /completing students currently complete the AGS, following graduation, which provides data on student satisfaction about their course etc. If a final year survey is considered warranted, then UWA would support consideration of the CEQ being administered in the final year of study, in the same manner as the National Survey of Students in the UK. This approach would provide data that could be benchmarked more legitimately, since the survey would be administered nationally at the same point of the students’ study. The opportunity to benchmark internationally would provide a valuable set of information for institutions and courses to internationally benchmark their courses and for informing international students on the quality of courses in Australia comparative to the UK and NZ. Were the UES deemed a more appropriate measure of students’ experience of a course in the final year of study, then the CEQ component of the AGS should be removed, to minimise duplication. However, this would result in a loss in the opportunity to benchmark with an internationally comparative instrument. 4.2 Assessment of Generic Skills The stated purpose for this measurement strays considerably from the focus upon national performance measurement, into institutional responsibility for determining the quality and standard of their students’ learning outcomes and into the role of TEQSA. The rationale and purpose of this indicator discussed within the paper is the source of particular concern. The argument put for the use of the CLA or a similar test of generic skills is not supported by the CLA originators, the sector, or within the higher educational research literature. Moreover, we note with concern that the AHELO project sponsored by the Government, which currently utilises the CLA, is yet to conclude its trial or report its findings. We therefore urge the Government to consider the removal of the CLA proposal as one that is not supportable, unsuited to the intended purpose, and both cost and labour intensive, or at a minimum, delay until the results and reports of the AHELO project have been received and the ramifications from it are considered. However, while this is our clear and preferred position, and recognising the imperatives of Government and the context in which the paper is proposed, UWA is prepared to cautiously support the administration of an Assessment of Generic Skills in the final year of study. Given the significant issues that will need to be addressed in developing this tool, we urge the Government to revise the proposed timeline to allow for sufficient time for development, consultation and piloting before introducing an assessment instrument. Our concerns regarding the Assessment of Generic Skills are detailed briefly below, and in more detail in our response to that specific Discussion paper.

G12

11

The assertion in the paper that the Assessment of Generic Skills is a “direct measure of learning outcomes” (p13) remains highly contested, given that “generic skills” are actually not generic: they are learned in a context that is specific, based within disciplines and their particular discourses, and therefore also most clearly evidenced in these specific contexts. An assessment of generic skills outside the discipline of study therefore remains an indirect and proxy measure of learning outcomes. UWA fully supports the development of clear learning outcomes and evidence of standards attained, and is participating in a number of projects to achieve this end including the Go8 standards verification project and the ALTC project Achievement Matters: External Peer Review of Accounting Learning Standards. We argue that peer review, moderation and external verification is the most valid way to determine learning outcomes and standards and that TEQSA is best placed to oversight these. Claims that ‘value-added’ can be simply measured using tests such as the CLA have been routinely discounted over many decades, by internationally reputable evaluation experts. As such, the intention to administer the CLA “to first and/or final year undergraduate students” (p14) is not supported by UWA. More importantly, the administration of an Assessment of Generic Skills in the first year of study is effectively an input indicator that is more appropriately captured by their students’ ATAR scores. The first year administration of a CLA-like test as an input indicator has been shown to have limited validity and therefore its cost effectiveness and fitness-for-purpose is questionable. It also is contrary to the Government’s stated focus on outcome measures rather than input measures. We note that Australian universities are able to subscribe to the services of the CLA on an individual basis, should they wish to utilise this test as an enhancement tool within their own institution – the purpose for which the test was developed. If the Government’s intention is to be assured that students graduating from university have attained a minimum standard of critical thinking, analytic reasoning, problem-solving and written communication skills (independent of the learning that has taken place in the discipline they have studied for their undergraduate degree) then the administration of a common test, acceptable to the sector and prior to graduation, would be cautiously supported by the University, though peer review and verification of standards is our preferred option. If such a test were administered, UWA would argue that the numerical results for all institutions should be published, in the interests of transparency. 4.3 Review of the AGS A review is supported, in a form that takes account of the UES and other performance measures so that the AGS can be adjusted to become part of a suite of integrated performance measures. The review of administration arrangements should be linked and mirror those of the UES and any other measures, to minimise the administrative and resource load on institutions and to avoid duplication and confusion. (Further detail is contained in UWA’s companion response to the Review of the Australian Graduate Survey discussion paper.)

5 Issues 5.1 Administration of new instruments UWA supports a common administration process for new and existing instruments. 5.2 Student Selection UWA would propose the following basis for selection of students: University Experience Survey – Survey Collegiate Learning Assessment – Survey AGS/Course Experience Questionnaire – Census

G13

12

5.3 Central sampling of students UWA supports the central, entirely online administration of surveys to ensure a consistent process is used across institutions. Identifiable data enables linking to data held local. 5.4 Uses of data Student data must be made available to the University in a manner which enables analysis at levels appropriate to institutional purposes, i.e. to Course/program level, year level, degree-specific level, as required. UWA supports the reporting of actual numbers, with neither extrapolation nor weighting or other forms of statistical modelling to ‘account’ for contentious input factors. The paper suggests that a key driver for collecting the data is to populate the My University website. However, in order to respond sensibly to questions concerning appropriate use of data, we require more detailed information concerning the proposed level of its aggregation; the uses to which the data will be put; and the explanations that would be furnished for its users. It is not clear what data is proposed to be available on the My University website. For example, will all questions from the UES be available for students or just one or two questions, or will scales be reported? We support the development of guidelines of use and reporting practices that is endorsed by the members of Universities Australia, before the release of data. 5.5 Intersection of existing and new instruments All instruments (new and existing) must be considered as part of an integrated framework. Existing instruments may need to be altered or removed to avoid duplication and to ensure data gathered is relevant to its purpose. The selection and use of the instruments also needs to proceed with the recognition that they provide the Government with samples and snapshots of quality at particular points in time. They cannot and should not be interpreted as comprehensive. As is the case with many universities, UWA has comprehensive systems for data gathering, the outcomes of which it uses for both quality assurance and enhancement purposes. UWA looks forward to contributing to the generation of robust national performance measurement instruments for quality assurance purposes. We do not support the generation and proliferation of additional instruments or the collection of data that relates to institutional quality enhancement, which is appropriately the preserve of institutions themselves, and which is subject to oversight by TEQSA. Should further clarification of any aspect of this paper be required, The University of Western Australia’s institutional contact is: Winthrop Professor Jane Long Pro Vice-Chancellor (Education) M 466, The University of Western Australia 35 Stirling Highway Crawley WA 6009 Email [email protected] Telephone: (08) 6488 2077

G14

mailto:[email protected]

Documents

MEMBERS OF THE TEACHING AND LEARNING COMMITTEE€¦ · Teaching Fellowship Scheme $88,000 Improving Student Learning Grants Scheme $30,000 Teaching and Learning Forum – Provision