Effective application of quality assurance & enhancement

procedures to e-learning courses

Workshop at

University of Reading

7 July 2008

Welcome

Project teamInstitute of Education Dr Harvey Mellar

Dr Magdalena Jara

University of Reading Maria-Christiana Papaefthimiou

University of Derby Dr Dave O'Hare

UoL External System Brian Sayer

University of Teesside Eileen Webb

Quality Assurance Agency

Dr David Cairns

Workshop background IOE Pathfinder Pilot – PREEL

Included revision of QA procedures for e-learning courses

IOE research study on QA/QE in e-learningExamined effectiveness of internal quality assurance procedures for online courses

Reading Pathfinder DIRECTDriving Institutional Reform: Exploring Change with Technology

Quality Assurance AgencyIn 2006 the QAA began a new cycle of audits with a more enhancement-focused approach

Purposes of the workshop Identify any issues in quality assurance/enhancement

procedures that arise from the use of technology in teaching and learning

Identify how these issues are reflected in your own current institutional procedures

Identify how the internal QA/QE procedures of other institutions deal with these issues

Identify ways to make your institutional QA/QE procedures more effective in assuring and enhancing the quality of TEL.

Day overview10.00 Introduction

10.15 Part 1: Making terms explicit

10.55 Part 2: Internal QA/QE procedures for TEL

11.35 Coffee

11.45 Part 3: Issues affecting effective implementation of internal QA/QE procedures for TEL

12.45 Lunch

13.30 Part 4: Planning actions to improve the effective application of QA/QE procedures to TEL

14.40 QA/QE in e-learning SIG

15.00 Close

Part 1

Making terms explicit

Technology Technology enhanced learning E-learning On-line teaching

QAA’s framework - IQAPs

Internal quality assurance proceduresEach [HEI] is responsible for the standards and quality of its academic awards and programmes. Each has its own internal procedures for attaining appropriate standards and assuring and enhancing the quality of its provision.In particular, institutions address their responsibilities for standards and quality through:

the assessment of students; their procedures for the design, approval, and the

monitoring and review of programmes.

QAA (2003) A brief guide to quality assurance in UK higher education. www.qaa.ac.uk

Quality enhancement.. the process of taking deliberate steps at institutional level to improve the quality of learning opportunities....[…] … an aspect of institutional quality management that is designed to secure, in the context of the constraints within which individual institutions operate, steady, reliable and demonstrable improvements in the quality of learning opportunities.

QAA (2006) Handbook for institutional audit: England and Northern Ireland

The definition of 'enhancement' QAA has adopted …. leaves room for institutions to follow their own definitions of 'enhancement'. Some institutions may define enhancement as 'continuous improvement', others as 'innovation' and there may be other definitions.

QAA (2007) Audit and enhancement: further guidance for institutions and QAA audit teams

QE and TEL Enhancement

improvement or innovation

TEL Is an innovation, hence enhancement? Need to improve TEL?

Task 1(15 minutes)

In each group…

What do you mean by QE? How can you use the internal quality assurance

procedures to achieve QE?

Report back – one point per group

Swinglehurst, D. (2006) Peer Observation of Teaching in the Online Environment: an action research approach. Final Report. Centre for Distance Education

QA/QE – the differenceQuality assurance and quality enhancement are not the same as the first is concerned with determining that objectives and aims have been achieved, while quality enhancement is concerned with making improvements.

Quality enhancement is part of a wider framework in which quality control, quality assurance, quality enhancement and transformation are stages in the management of quality

Common belief that quality assurance leads naturally to quality enhancement is not correct, as most quality assurance efforts are by and large concentrated in accountability

Middlehurst, R. (1997), 'Enhancing Quality'. In F. Coffield and B. Williamson (eds), Repositioning Higher Education. Buckingham: SRHE and Open University Press.

Retrospective vs prospectiveQuality assurance may be either ‘retrospective’ or ‘prospective’ depending on the type of quality it is aiming to assure.

Retrospective QA looks into the past to make a judgement with a focus on accountability

Prospective QA is concerned with the present and future, focusing on quality as fit for purpose, and encouraging improvement

Defines QE as the internal mechanisms that an institution puts in place to continually review and improve practice

Biggs, J. (2001), 'The reflective institution: assuring and enhancing the quality of teaching and learning'. Higher Education, 41, 221 - 238

Accountability vs improvement… external evaluations have accountability and compliance focuses rather than the encouragement of continuous quality improvement of the student experience. In most institutions where it occurs, improvement of the student experience is a function of internal review and monitoring processes, usually heavily reliant, nowadays, on student feedback, examiners reports, internal improvement audits, periodic revalidation of programmes of study and staff teams critically self-reflecting on their everyday practice.

All internal processes of quality monitoring have a greater effect on the quality of the provision than the external monitoring processes

Harvey, L. (2005), 'A history and critique of quality evaluation in the UK'. Quality Assurance in Education, 13 (4), 263 – 276.

Part 2

Internal QA/QE Procedures for TEL

Internal QA procedures

1. Validation

2. Module evaluation

3. Annual course review

4. Periodic course review

5. Student representation

6. External examiner

7. Peer observation

8. Other…

Task 2(15 minutes)

In each group…

Considering the internal QA procedures in place in your own institution, discuss:

Does TEL require new/changed internal QA procedures?

Does TEL enable new/changed internal QA procedures?

Report back

National overviewHEIs' Audit Reports overview regarding QA of e-learning

Total = 129

Mentions particular QA procedures of e-learning

(80) 62%

Mentions QA of e-learning in general

terms only(10) 8%

Does not mention QA of e-learning

(8) 6%

Reports no e-learning, (31) 24%

(QAA audits 2003-2006)

Percentage of HEIs mentioning each QA procedure for e-learning

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

External Examiners

Student feedback

Student Representative

Staff feedback

Team meetings

Annual Review

Periodic Review

Validation/approval

Peer review

Other QA procedures

HEIs reporting variations in the application of the procedures to e-learning

ProcedureTotal

(n=129)

External Examiners

Student feedback 14 11%

Student representatives

Staff feedback

Team meetings/staff development

Annual Review 11 9%

Periodic Review 10 8%

Validation/Approval 35 27%

Peer Review

Other QA procedures    

VariationsStudent feedback

Specially tailored forms Implementation of online surveys and discussion

forums to collect feedback (to improve low response rates)

AnnualReview

Additional issues to be considered by review team Additional scrutiny Additional section to be discussed

PeriodicReview

External assessor with experience in e-learning To include an evaluation of the development and

effectiveness of teaching, learning and assessment strategy and method

Validation Additional consultations/approval steps: Consultation/approval by specialised units Proposals seen by additional committees

Sustainability: Additional scrutiny of economic viability and availability

of resources (risk assessment, contingency plans)

E-learning mode of delivery: Demonstrate appropriateness of delivery method,

explain how student discussions will be facilitated

Materials: Provide sample materials for review Internal and external peer review

Validation Panel members: At least one member of the panel needs to have

expertise in the area

Academic staff: Extra checks on staff development needs

Coffee break

Part 3

Issues affecting effective implementation of internal QA/QE procedures for TEL

QA/QE for TEL

The literature identifies differences between TEL and campus-based learning:

Disaggregated processes

Distributed teams

Distant location of students

Openness to review

Research study Four case studies

Postgraduate courses in a range of universities Explore how dual-mode HEIs approach the application of their

internal QA procedures to their e-learning courses

For each case study QA documentation

Interviews with stakeholders

Comparative examination of data: Map of issues not captured by the QA procedures Identification of those aspects of the courses which were

impacting on the implementation of the procedures

Research resultsThe application of the QA procedures was affected by:

Organisational context in which courses are located

Disaggregated processes

Distributed teams

Distant location of students

Openness to review

1. Organizational context - detachmentOnline courses were often in a ‘detached’ position in their institutions, which created both a sense of autonomy and isolation

On-line courses off senior management’s agendas Courses without central oversight This led to failure to collect relevant information

2. Disaggregated processesDistributed organisation of teams in terms of roles affected the levels of coordination and communication among team members

Unclear distribution of responsibilities Collaboration with external and specialised units added further complexity

3. Distributed teams - coordination

Online courses were often taught by a mixture of full time tutors, tutors with fee-based contracts and tutors working from home or elsewhere. Course teams were scattered and course leaders were often not aware of the coordination requirements of a distributed team

On campus team with face-to-face meetings, tutors fully integrated Course leaders tend to use the same coordination and feedback mechanisms used for on-campus staff

4. Distant location of studentsDistant location of students also affected the quality assurance mechanisms, as it obstructed the implementation of some of the procedures in their current form (e.g. the use of student representatives)

Compensated for by a strong and trusting relationship between students and tutors Opportunity was only occasionally taken up by teams

Effect on internal procedures

Annual Reviews Module evaluations Team meetings

Annual Reviews Approached as an administrative burden, which had to

be written up just for accountability purposes Senior management is perceived as not prepared/not

able to understand the relevance of TEL issues

Annual review often detached from its enhancement function Perceived as an event rather than a process Teams render two different accounts, one for external

consumption and another to be used internally

Module evaluations Distant location of students…

low response rates teams discarding the results as invalid, regardless

of their content

Distributed organisation of teams… unclear allocation of responsibilities

led to responses being left untouched or only superficially analysed

Team meetings Distributed teams

absence of a structure of formal meetings put at risk the team’s capacity to deal with the

issues identified and the monitoring of their resolution

Evidence: consistency in the information managed by team members mechanisms for coordination and feedback relying on informal

encounters

Adjustments to overcome the limitations usually took the form of increasing the formalisation of the communication and coordination channels either online or face-to-face

Task 3(35 minutes)

Select one internal QA procedure

In each group: Quickly review the procedure’s purpose and how should it work Quickly review how the procedure is intended to aid quality

enhancement Describe any issues impacting on the use of this procedure for

QA/QE arising from the use of TEL Suggest solutions

Produce a one page written report: List issues identified affecting the procedure selected List suggestions for changes/improvement

Lunch break!

Part 4

Planning actions to improve the effective application of QA/QE

procedures to e-learning courses

Techniques - research & practice Validation criteria

Evaluation of e-learning courses

Embedded evaluation

PROPP Peer Observation

Periodic review framework

1. Additional validation criteria (IoE) Staff experience of, or training in, use of on-line teaching

methods Evaluation plan Arrangements for the support of students with disabilities Explicit course management procedures (including the

role of administrators, and a named person responsible for quality assurance)

Arrangements for obtaining feedback from students Explicit arrangements for tutor peer observation

2. Recommendations for evaluation Obtain feedback from all stakeholders: students, tutors,

administrators, technical support

Organise frequent formal staff meetings, face-to-face and/or online, with agendas defined by the staff, and covering all key issues

Carry out evaluation as an integral part of the activities of the course, both during and after the course

Ensure that responsibility for collection and analysis of results is clearly assigned

Take advantage of the technology in use in the course to collect feedback

Evaluation of e-learning courses WLE Occasional Paper 4

3. Embedded evaluation Project studying e-learners’ experiences in a mixed-

mode course

Student learning is enhanced by evaluation which is concurrent with teaching Embedding evaluation tasks as part of the activities of the course

encouraging students to think about their own learning and how the course design, materials and/or activities have supported them (or not) in this pr

Daly, C. Pachler, N., Pickering, J. & Bezemer, J. (2006)

A study of e-learners’ experiences in the mixed-mode professional degree programme, the Master of Teaching.

Examples of evaluation questionsWriting online or in face to face seminar

MASTER OF TEACHINGOnly two of you had prior experience of learning online before joining the MTeach, so this has been a new way of communicating to learn for everybody else in the group. These discussions form a significant alternative to talking about issues at the face-to-face days or in traditional seminars. What has it been like to ‘discuss’ by using writing to communicate with each other like this?

RESEARCH SYNTHESIS FOR POLICY AND PRACTICE

Conducting a systematic review is a collaborative process, involving both face-to-face meetings and online discussions/activities. The course has been designed to mimic this process - carrying out activities both in the workshops and online - how has it been to learn like this? Please offer your thoughts…

4. PROPP Peer Observation

Project studying the implementation of peer observation of teaching for tutors of an online course

Development & piloting of approach to peer observation regular meetings course tutors are held

at each meeting, individual tutors bring an example of a problem, issue or a project (e.g. feedback prepared for a student, transcript of a virtual seminar, marks awarded for student’s work)

this material is used as the basis for discussion with other tutors supporting reflection, and challenging assumptions

Swinglehurst (2008) Peer Observation of Teaching in the Online Environment: an action research approach

Example of a PROPP meeting Topic How do we assess our students?

Evidence: Two student tutor marked assignments with the marks and tutor feedback

Main outputs The instructions in the assignment task were ambiguous, and so needed to

be altered The task needed to be broken down into smaller components Written feedback should include specific examples rather than relying on

general comments which may not be well understood by students Feedback should refer back to previous feedback given to students, it is

worth exploring the possibilities for enabling this more easily within the VLE

5. Periodic review frameworkUniversity of Reading

Pathfinder DIRECT - Driving Institutional Reform: Exploring Change with Technology

Plenary discussion

Summaries of discussions from Task 3

Discussion:

How can your institutional procedures be made more effective in assuring and enhancing the quality of TEL?

What obstacles stand in the way of improving these institutional procedures ?

QA-QE SIG JISCmail list: ‘QA-QE SIG’

to join go to http://jiscmail.ac.uk and sign up

More information: www.lkl.ac.uk/research/qa-elearning

Evaluation of workshop External evaluator (Charlotte Creed)

By email: link to online evaluation (10 min max)