MAPPING THE QAA FRAMEWORK AND THE EFQM EXCELLENCE … · Mapping the QAA Framework and 5 'Consortium for Excellence the Excellence Model in Higher Education Figure 1 The simple model

Mapping the QAA Framework and 2 ©Consortium for Excellence the Excellence Model in Higher Education

MAPPING THE QAA FRAMEWORK AND THE

EFQM EXCELLENCE MODEL

GMP 143/QAA

FINAL PROJECT REPORT July 2001

Project Manager and Co-author: CAROL STEED, Organisational Excellence Manager,

Sheffield Hallam University

Primary Co-author: GWYN ARNOLD, Deputy Academic Registrar,

Sheffield Hallam University

Additional contributors from Sheffield Hallam University: JANE TORY, Academic Registrar

MIKE PUPIUS, Director of Organisational Excellence JANET HARGREAVES, Academic Registry

ANNE TOPLISS, Academic Registry LORNA DALY, School of Sport and Leisure Management

ALLAN NORCLIFFE, School of Computing and Management Sciences SEAN CAVAN, School of Education ANGELA REES, School of Education


Contents 2

List Of Figures 5

List Of Tables 6

Executive Summary 9

1 Context and Overview 9

2 Comparing the QAA Framework and the EFQM Excellence Model® � a summary of the report 9

3 The Codes of Practice and Precepts and the EFQM Excellence Model® � Key Findings 9

4 Institutional Review, Continuation Audit and the EFQM Excellence Model® � Key Findings 10

5 Subject Review and the EFQM Excellence Model® � a Summary 12

6 The EFQM Excellence Model® and the Integrated QAA Method 13

7 Comparing the Models in Practice � an Overview 13

8 Summary Conclusions 17

Introduction 20

1 Context of the Project 20

2 The Changing HE Environment 21

3 Aims and Objectives 21

4 Project Deliverables and Outcomes 21

5 Project Methodology 21

6 Project Authors and Contributors 22

Section 1 Exploring the EFQM Excellence Model 24

1 The Development of the �Quality� Concept 24

2 The Evolution of the EFQM Excellence Model 25

3 The EFQM Excellence Model � The Foundation Stone for Excellence 26

4 The Criteria and Sub-Criterion 27

CONTENTS


5 Using the Model as a Self-Assessment and Planning Tool 28

6 Key Methods Of Assessment 30

7 The Reliance Upon Evidence 35

8 Benchmarking Against Others 36

9 Practical Use Of The EFQM Excellence Model® 37

10 Summary 38

Section 2 The Quality Assurance Agency Framework 40

1 The Quality Assurance Agency For Higher Education 40

2 The QAA Quality Assurance Framework: Overview 41

3 Key Elements of the Framework 42

4 Key Issues and Implications for HEIs 53

Section 3 The EFQM Excellence Model® And The QAA Framework: Mapping Synergies And Gaps 57

1 Introduction 57

2 The Code of Practice and the Excellence Model 58

3 Mapping the Precepts Against the EFQM Excellence Model Criteria 62

4 Taking the results for each EFQM criterion in turn: 66

5 Outcomes 67

6 Conclusions 69

Section 4 Institutional Review, Continuation Audit and the Excellence Model. 71

1 Introduction 71

2 Continuation Audit: The Context 71

3 The Excellence Model and Institutional Review: Comparison of Purpose and Concepts. 72

4 Criteria for measurement: HEQC/ QAA and EFQM 75

5 The Excellence Model and Institutional Review: the Process 83

Section 5 Subject Review and the Excellence Model 86

1 Introduction 86


2 The process as undertaken by QAA 86

3 The Process from an Institutional Perspective 87

4 Synergies and gaps 88

Section 6 The Excellence Model and the Integrated QAA Method 96

1 Process 99

2 Outcomes 99

3 Conclusions 102

Section 7 Comparing The Models In Practice 104

1 Introduction 104

2 Self-Assessment against Subject Review criteria 104

3 Self-Assessment against the EFQM Excellence Model 107

4 Learning from these experiences 110

5 Using the RADAR concept in Preparation for Subject Review 112

6 Conclusions 112

Section 8 Conclusions 116

1 General Issues 116

2 The Excellence Model and the QAA Integrated Framework: Some Key Similarities and Differences 117

3 The Scope for Development of the QAA Framework and a new EFQM HE Excellence Model 119

4 A Rationale for Implementing the EFQM Excellence Model in HEIs? 119

Section 9 Further Work 123

Section 10 Appendices 125

1 Results Orientation 130

2 Customer Focus 130

3 Leadership and Constancy of Purpose 130

4 Management by Processes and Facts 130

5 People Development and Involvement 130


Figure 1 The simple model for improved performance 26

Figure 2 The EFQM Excellence Model ® 27

Figure 3 Simple assessment concept 28

Figure 4 Integration of the EFQM Excellence Model® with Business Planning 28

Figure 5 The Excellence Journey 29

Figure 6 The key steps to self-assessment 29

Figure 7 Assessment options 30

Figure 8 The RADAR Logic 35

Figure 9 The EFQM Excellence Model�� with weightings shown 36

Figure 10 Framework Inter-Relationships 42

Figure 11 The Concept of the HE Progress File 52

LIST OF FIGURES


Table 1 Questionnaire Approach 31

Table 2 Matrix Approach 32

Table 3 Workshops 32

Table 4 Pro-forma Based Approach 33

Table 5 EFQM European Quality Award Entry 34

Table 6 England, Wales and Northern Ireland HE Qualifications Framework 45

Table 7 QAA Subject Benchmarks: Phase 1 47

Table 8 QAA Subject Benchmarks: Phase 2 47

Table 9 QAA Code: Publication Schedule 49

Table 10 Suggested miminum requirements for Programme Specifications 50

Table 11 Variations of Approach 59

Table 12 Process / Specification Matrix � Code of Practice 61

Table 13 Currently Published Concepts 62

Table 14 Precept example 1 62



Table 18: Outcomes Matrix � QAA Code of Practice 68

Table 19 Key similarities and differences of purpose and concepts of approaches: Excellence Model and Institutional Review 72

Table 20 QAA Continuation Audit mapped against the EFQM Excellence Model® 77

LIST OF TABLES


Table 21 QAA Institutional Review mapped against EFQM Excellence Model® 79

Table 22 Mapping QAA guidance for Auditors Annex E with the EFQM Excellence Model® 81

Table 23 Process/ Specification Matrix 83

Table 24 QAA Subject Review and EFQM: Measurement Criteria Matrix 90

Table 25 QAA Subject Review and EFQM: Purpose/Concepts Matrix 91

Table 26 QAA Subject Review and EFQM Process/Specification Matrix 92

Table 27 QAA Subject Review and EFQM Process: Outcomes Matrix 93

Table 28 EFQM Excellence Model® and the integrated QAA Review Framework 97

Table 29 Intergrated QAA Review Framework and EFQM Model: Outcomes Matrix 100

Table 30 EFQM Excellence Model and the QAA Framework: Process Matrix 101

Table 31 Advantages and disadvantages of the QAA subject review process 107

Table 32 Advantages and disadvantages of self-assessment using the EFQM Excellence Model® 110

Table 33 The synergy between processes, and areas for potential development 111


EXECUTIVE SUMMARY


1 Context and Overview

This project set out to compare and contrast the principles and concepts of the EFQM Excellence Model® with the principles and framework of the new QAA Quality Assurance framework, including Institutional Review, Subject Review, and the Codes of Practice. It forms part of the GMP143 programme, a three year programme which is part funded by the Higher Education Funding Council for England (HEFCE) as part of the Developing Good Management Practice Initiative. Sheffield Hallam University, in partnership with the Universities of Cranfield, Durham, Salford, Ulster and Dearne Valley College (FE representative), is leading the programme. Since this project was first agreed (May 2000), significant changes have developed around the QAA frameworks for the HE sector as a whole. This project has therefore taken into account the impact of the latest thinking and discussions that were underway at the time of writing. It is acknowledged that even when published, further national developments may have been made, but it is hoped that this work will help to inform the continual debate about how �quality� and �excellence� may be defined, and subsequently monitored, within the HE sector.

2 Comparing the QAA Framework and the EFQM Excellence Model® � a summary of the report This report makes detailed comparisons with various aspects of the QAA framework, a summary of which follows: • The Codes of Practice and Precepts • Institutional Review and Continuation Audit • Subject Review It also considers the practical application of both frameworks, and draws conclusions from the work, including reference to future work that could be undertaken. Much of the mapping work undertaken compared each framework from a conceptual, processes and outcomes perspective, as well as more detailed mapping where necessary. The findings of each of these areas have been summarised below.

3 The Codes of Practice and Precepts and the EFQM Excellence Model® � Key Findings This mapping was undertaken in two main stages:

EXECUTIVE SUMMARY


• an overall assessment of the relative purposes of the QAA Code of Practice and the EFQM Excellence Model® � as stated by QAA and EFQM respectively

• a detailed consideration of each published Code section, mapping each precept against the nine criteria in the EFQM Excellence Model®.

The conclusions that emerged highlighted that there are clearly some synergies between the EFQM Model and the QAA Code of Practice but also evidence of different approaches or concepts. Both are intended to provide tools to support effective management, monitoring and ultimately quality improvement or enhancement. The QAA Code is predominantly concerned with the management of academic quality and standards as discharged through institutional processes. It is concerned with only certain aspects of institutional management and is one element of an integrated QAA framework which seeks to be effective via the interaction of the different elements of the framework. The Code does not achieve (nor does it seek to achieve) a holistic or complete coverage of all aspects of organisational effectiveness or �excellence� as defined in the EFQM model. Hence there is only scope for partial mapping against the EFQM Model criteria. The Code reflects the range of relationships between the various stakeholders within Higher Education � staff, students, external examiners, professional bodies, employers. There are some inherent challenges in mapping these various complex relationships using the EFMQ model terminology of people, customer, partner; this may be a terminological issue only or may be a more fundamental issue concerning the applicability of the current model to Higher Education. Certainly, there would seem to be a need to consider in more depth the role of students as participants within processes, rather than recipients of services.

4 Institutional Review, Continuation Audit and the EFQM Excellence Model® � Key Findings There are striking similarities emerging between these frameworks, but also stark contrasts. The key ones are summarised below: Similarities • The fundamental concept of each is similar in that organisations must have

policies and strategies that they implement systematically through a procedural framework to ensure the proper management of their functions i.e. they both have a strong emphasis on policy, strategy and processes.

• Both are concerned with the achievement of good performance in the

interests of stakeholders, and with management of effective relationships with partners and collaborators.


• They also both offer the opportunity to demonstrate that they are managing effectively.

• They are also reliant on an evidence base. Differences • The Excellence Model is about organisational excellence; there is no

equivalent institutional comparative term, in the QAA IR Model. • The EFQM Excellence Model® is for most organisations, a self-assessment

tool intended to encourage and promote good management practice. Any type of organisation may choose to use the model. It can be adopted at any time, using any method, or entry point into the organisation, at any level, over a timescale to suit the resource of the organisation. It is therefore flexible to the needs of the organisation. The QAA IR mechanism, however, is mandatory: applies only to HEIs: focuses mainly on the management of academic standards and the quality of educational provision: and involves an element of self-evaluation but also one of external judgements by academic reviewers.

• The EFQM Excellence Model® involves a systematic review of how

comprehensively approaches have been deployed across the organisation, trends over three or more years, and scoring using the RADAR scoring matrix. Whilst HEIs may choose to use a systematic �scoring� mechanism of this type during their self-evaluation, QAA IR judgements are not made against a scoring framework.

• The EFQM Excellence Model® is concerned with leadership and with the

ways in which policy, strategy and processes are deployed at all levels within the organisation. QAA IR is less explicitly concerned with �leadership� but is described as a �capstone� review and tends to focus on the effectiveness of the �senior layer� which may include both individuals and committees or groups � in discharging the institution�s overall responsibility for quality and standards management.

• The EFQM Excellence Model®, in focussing on management practice in

organisations, facilitates benchmarking between a wide range of organisations, not necessarily in the same sectors. QAA IR, on the other hand, enables an element of benchmarking between institutions, if they choose to scrutinise and learn from the published reports about other institutions, or use advisers to help them prepare for external scrutiny.

• The Excellence Model is concerned with an all-embracing, holistic approach

to the management of an organisation�s business including all aspects of resource management. The QAA model, including IR, does not focus enquiries on academic governance and management, which it is considered would be too sensitive, nor does it cover resource allocation and management � which may or may not be appropriate.


• The QAA IR method includes an element of threat or sanction and there are significant risks associated with adverse outcomes. No such threats are attached to self-assessment using the EFQM Excellence Model®: if an organisation achieves a low score when it undertakes a self-assessment it is up to the organisation itself to decide what action to take and over what time scale.

• The EFQM Excellence Model® is more explicitly demanding than the

Continuation Audit foci in areas such as: • leadership • the management of relationships with and services for external partners • People Results (both perception measures and performance indicators) • Society Results (though again this may be assumed to be implicit in the

QAA model). • The EFQM approach is seen as a change in the way of working.

5 Subject Review and the EFQM Excellence Model® � a Summary Many of the similarities and differences that arise from the comparison at Institution wide level, also relate to the mapping at Subject level: mandatory vs optional; public accountability vs institutional concern etc. However, when mapped in further detail there appears to be a great deal of synergy between the two processes. Again the similarities and differences are highlighted below: Similarities • The most common areas of similarity are around the focus of Policy and

Strategy, Processes and also Key Performance Results. The focus on Partnerships and Resources and Customer Results are also strong.

• Arguably the greatest synergy between the Excellence Model and Subject

Review is that they can both use a range of internally and externally produced evidence to produce a self-evaluation document and undertake an assessment exercise.

• They are also both concerned with customer focus. • Each requires an honest and rigorous approach to self-evaluation and

improvement action which leads to improved performance Differences • It is the areas of Leadership, People, People Results and Society that are not

closely matched to Subject Review. Although these may be implicit as part the process, what is starting to emerge is that Subject Review looks in depth at a particular area of the institution, and does not attempt to present a balanced view of quality at managerial (even at business unit/school) level, or organisational level.


• The Excellence Model is also far more explicit at encouraging target-setting in a balanced set of performance results areas than the QAA framework.

6 The EFQM Excellence Model® and the Integrated QAA Method When the individual elements of the QAA framework are aggregated, the overall �fit� with the Excellence Model becomes clearer. These are demonstrated in the tables overleaf. The QAA integrated Framework is concerned with many of the same issues as the Excellence Model criteria, but not all. The extent to which either or both lead to genuine and sustained improvement in performance is crucially dependent upon the honesty and rigour of self-assessment, and implementation of focussed improvement actions.

7 Comparing the Models in Practice � an Overview Acknowledging that theories are all very well, we wanted to evaluate how this related in practice. A short exercise was conducted to compare and contrast a sample of academic areas and their experience of Subject Review, with the experience within the University of undertaking an assessment against the EFQM Excellence Model®, using the approach used in Sheffield Hallam University. The findings of this exercise should, however, be considered in light of the changes being recommended for Subject Review from 2002 onwards. A number of key points emerged, however, that point to significant areas for improvement at School level, and that support findings from the mapping exercises documented in the sections above. • The preparation time for a Subject Review is now more flexible with individual

institutions having an opportunity to assess and review how they prepare. There is potential to use the EFQM Excellence Model® as a preparatory tool on an annual basis, giving consistent and strongly evidenced results in many areas.

• The Subject Review process requires positive management of change as a

result of the assessment through the implementation and integration of the actions. As the Excellence Model evaluates many of the people and leadership aspects, cultural and management change may be more fluid � i.e. it happens along the way as part of the process. There is potential to learn from this over-arching approach.


The Outcomes of the EFQM Excellence Model® and the Integrated QAA Framework

Outcomes required

EFQM Excellence

Model®

self assessment

QAA Subject Review

QAA Institutional

Review

QAA Review of Collaborative

Provision

QAA Codes of Practice

Reports Optional - can be a short management

report, or a full assessment document

Self Evaluation Document produced prior to review period

by subject area. Report produced by

QAA following review period

Self Evaluation Document produced at institutional level to inform review by external reviewers. Review report

produced by external reviewers.

QAA methodology does not require institutions to provide reports specifically about the Code of

practice. However, it is assumed that Academic Review Reports (subject & institutional) will include reference to

adherence to the Code. Also assume that institutional review and

monitoring of practice against codes is likely to lead to internal reports.

Action plans Optional, but desirable. The outcomes of a self-

assessment activity usually lead to actions which are implemented through an action plan

Internal action plan produced by subject area. External action plan only if provision is not approved by

QAA.

Assumes that action planning is incorporated into the self-evaluation. Most/ all institutions will initiate action

on issues identified. Follow-up action/ progress report required by

QAA.

Again, the QAA methodology does not necessarily result in action plans,

but institutional responses to QAA review and internal review may well lead to action plans for improvement

measures

Evidence of learning

Strong - assessment activity itself evidences

learning. Action plan tracks progress and

learning.

Internal action plan The review will reach a judgement on the institution's capacity for self-evaluation and self-improvement

Institutional consideration of practice against the QAA Code would

presumably need to be evidence based and good practice in ongoing

monitoring and review should demonstrate evidence of learning

Overall improvement activity

Often demonstrated over 12 months or

more, but can show immediate

improvements. Activities range across

wide range of management and

people issues.

Ongoing improvement seen as

embedded in the process

Stimulates improvement activity on an institution-wide basis

See action plans above � assumed that these would lead to overall

improvement of activity

Integration into other aspects of planning and management

Desirable to be fully integrated into the

business planning cycle - allowing the self-

assessment process to be preparation for an

evidence based business plan.

Integrates well with other management

tools.

Subject Review inputs into overall quality review of subject/school/

institution

Depends on the approach adopted by the institution: may be integrated or

may be a separate activity

Whilst the Code deals with individual areas of practice in separate

sections, there would be inevitably be �cross cutting� and broader

implications arising from identifying and actioning improvement measures

in relation to each section. Institutions would need to be able to

make the links and ensure appropriate integration between

sections and with other aspects of planning and management.

Improvement in customer experience

Evidence to show marked improvement in

customer experience amongst organisations

who are using the Model

Improvement in student experience

where specific issues have been identified in the self evaluation

and review

May lead to an improved 'customer' experience if issues to be addressed

are identified either by the self-evaluation exercise or by the external

reviewers and action plans are implemented effectively.

As much of the Code is concerned with ensuring effective processes,

clearer articulation of roles and responsibilities and good information

to students and partners, it is assumed that improvements should accrue from adherence to the Code.

Improvement in staff experience


staff experience amongst organisations


Unlikely to impact on improved staff

experience as the review is student

focussed

May lead to an improved staff experience if issues relating to staff

quality and support are identified and addressed.

The focus of clarity of roles and responsibilities and ensuring

appropriate staff development and training should be of benefit to staff.

The indirect benefits of demonstrating adherence to the Code (good Ac

Review outcomes) will be of value. Improvement in balanced set of performance results

Usually leads to an understanding of the

need for, and the development of, a

balanced set of performance measures (customer, staff, society

and financial)

Possible improvement, but more likely that resources are

focussed on subject review, rather than

overall balance

Possible improvement, but focuses more on 'enablers' than results.

The code is a potentially valuable tool to assist institutions to improve

performance results in selected areas of practice?

Mappin

g th

e Q

AA

Fra

mew

ork

and

15

©

Conso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Th

e P

roce

sse

s o

f im

ple

me

ntin

g t

he

EF

QM

Exc

elle

nce

Mo

de

l® a

nd

th

e I

nte

gra

ted

QA

A F

ram

ew

ork

T

ech

nic

al /

Pro

cess

sp

eci

fica

tion

E

FQ

M E

xc

ell

en

ce

M

od

el

se

lf

as

se

ss

me

nt

QA

A S

ub

jec

t R

evi

ew

Q

AA

In

sti

tuti

on

al

Au

dit

Q

AA

Re

vie

w o

f C

oll

ab

ora

tive

P

rovi

sio

n

QA

A C

od

es

of

Pra

cti

ce

Tim

es

ca

le t

o

pre

pa

re f

or

as

se

ss

me

nt

Fro

m 1

hou

r to

6

mon

ths

Fro

m s

ix m

onth

s to

tw

o ye

ars

Typ

ical

ly 1

-2 y

ears

T

ypic

ally

1-2

yea

rs

Enc

ompa

sses

sel

f-as

sess

men

t an

d as

sess

men

t th

roug

h A

cade

mic

Rev

iew

, T

imes

cale

and

m

etho

d ve

ry d

epen

dent

upo

n in

stitu

tiona

l ap

proa

ches

to

revi

ew o

f pr

actic

e ag

ains

t C

ode

and

issu

es s

peci

fic t

o C

ode

sect

ions

. T

ime

sc

ale

fo

r a

ss

es

sm

en

t a

cti

vity

to

be

u

nd

ert

ak

en

Fro

m 3

hou

rs t

o 3

mon

ths

Allo

cate

d nu

mbe

r of

re

view

er d

ays

with

in a

pe

riod

of

revi

ew f

rom

tw

o m

onth

s to

12

mon

ths

(lik

ely

to s

ettle

on

max

thr

ee

mon

ths)

4-5

mon

ths

4-5

mon

ths

See

abo

ve

Fle

xib

ilit

y o

f a

ss

es

sm

en

t m

eth

od

Ver

y fle

xibl

e -

from

si

mpl

e qu

estio

nnai

re t

o aw

ard

subm

issi

on

docu

men

t

SE

D s

ubm

itted

to

QA

A t

o pr

ovid

e fr

amew

ork

for

revi

ew

Lim

ited

� al

way

s in

volv

es s

elf-

eval

uatio

n do

cum

ent,

pr

epar

ator

y m

eetin

gs,

and

visi

ts.

Sig

nific

ant:

may

invo

lve

sepa

rate

su

bmis

sion

and

eve

nt o

r be

en

com

pass

ed w

ithin

mai

n IR

/CA

.

See

abo

ve

Nu

mb

er

of

pe

op

le

typ

ica

lly

invo

lve

d

Var

ies,

min

imum

us

ually

5,

can

be

ever

yone

Var

ies

on b

asis

of

'siz

e' o

f re

view

and

the

ext

ent

to

whi

ch s

ubje

ct a

nd

orga

nisa

tiona

l str

uctu

res

map

By

QA

A T

eam

of

4: la

rge

num

bers

invo

lved

in

inst

itutio

ns,

depe

ndin

g on

siz

e in

bot

h pr

epar

atio

n an

d vi

sit.

By

QA

A T

eam

of

4: la

rge

num

bers

in

volv

ed in

inst

itutio

ns,

depe

ndin

g on

siz

e in

bot

h pr

epar

atio

n an

d vi

sit.

See

abo

ve

Re

lia

nc

e o

n

evi

de

nc

e

Rel

iant

on

evid

ence

but

al

so t

o so

me

exte

nt

perc

eptio

n ba

sed

Rel

iant

on

evid

ence

and

ef

fect

ive

faci

litat

ion

by

Sub

ject

Rev

iew

Fac

ilita

tor

and

Sub

ject

Con

tact

Tot

ally

rel

iant

on

evid

ence

ob

tain

ed f

rom

and

tri

angu

late

d be

twee

n th

e su

bmis

sion

, m

eetin

gs w

ith s

taff

and

bas

e ro

om e

vide

nce

mat

eria

l.

Tot

ally

rel

iant

on

evid

ence

ob

tain

ed f

rom

and

tri

angu

late

d be

twee

n th

e su

bmis

sion

, m

eetin

gs

with

sta

ff a

nd b

aser

oom

evi

denc

e m

ater

ial.

Add

ition

ally

evi

denc

e is

pr

ovid

ed in

dis

cuss

ions

with

pa

rtne

rs,

staf

f an

d st

uden

ts.

Eff

ectiv

e in

tern

al a

nd Q

AA

ass

essm

ent

of

adhe

renc

e to

the

Cod

e w

ould

hav

e to

be

evid

ence

bas

ed

Fit

s a

t e

very

le

vel

of

the

o

rga

nis

ati

on

Fits

at

pers

onal

, te

am,

depa

rtm

ent

and

who

le

orga

nisa

tion

Fits

at

pers

onal

, te

am,

depa

rtm

ent/

div

isio

nal,

scho

ol a

nd w

hole

or

gani

satio

n

Foc

uses

mai

nly

at in

stitu

tiona

l le

vel b

ut a

lso

mee

ts w

ith a

w

ide

rang

e of

oth

er s

taff

.

Foc

uses

mai

nly

at in

stitu

tiona

l le

vel b

ut a

lso

mee

ts w

ith a

wid

e ra

nge

of o

ther

sta

ff.

Fits

at

inst

itutio

nal a

nd d

epar

tmen

t/ s

choo

l lev

el

As

se

ss

me

nt

ag

ain

st

se

lf

pre

pa

red

d

oc

um

en

t

Ass

ess

agai

nst

self

com

plet

ed

ques

tionn

aire

, w

orkb

ook

or a

war

d su

bmis

sion

doc

umen

t

Rev

iew

aga

inst

sel

f-ev

alua

tion,

Sub

ject

Rev

iew

A

ide-

Mem

oir

and

exte

rnal

re

fere

nce

poin

ts

Judg

emen

ts a

re m

ade

on t

he

basi

s of

all

the

evid

ence

av

aila

ble

to t

he R

evie

w T

eam

�

from

sel

f-ev

alua

tion

docu

men

t,

staf

f/st

uden

t m

eetin

gs a

nd

base

room

evi

denc

e m

ater

ial.

Judg

emen

ts a

re m

ade

on t

he

basi

s of

all

the

evid

ence

ava

ilabl

e to

the

Rev

iew

Tea

m �

fro

m s

elf

eval

uatio

n do

cum

ent

staf

f/st

uden

t m

eetin

gs a

nd b

aser

oom

evi

denc

e m

ater

ial.

Inte

rnal

ass

essm

ents

wou

ld b

e ba

sed

on a

ran

ge

of in

puts

. Q

AA

judg

emen

ts a

re m

ade

on t

he

basi

s of

all

the

evid

ence

ava

ilabl

e to

the

Rev

iew

T

eam

� f

rom

sel

f ev

alua

tion

docu

men

t,

staf

f/st

uden

t m

eetin

gs a

nd b

aser

oom

evi

denc

e m

ater

ial.

Mapping the QAA Framework and 16 Consortium for Excellence the Excellence Model in Higher Education

• One of the outcomes of Subject Review may be that staff/students will change the way activities and task are undertaken, or the way relationships/partnerships are formed, but there is no guidance as to the sustainability of this in the longer term, unless an institution fails a review. Self-assessment using the Excellence Model process, once integrated into the business planning cycle, checks whether continuous improvement is being made in all areas on an on-going basis. Both can work together, with the Excellence Model providing the on-going check and balance of all general issues, with Subject Review providing a spot-check in specific areas.

• The scale and subject of assessment activity with Subject Review is

prescribed by QAA. There is no option to evaluate any other subject areas other than those stated, unless it is undertaken internally. Self-assessment using the Excellence Model can be applied freely to any part of the organisation at any time, as decided by the organisation. It can be as large or small as resource, time and strategy will allow.

• The outcome of the Subject Review provides a UK wide recognised external

validation of provision within the specified subject areas. This is an important benchmark across the sector against which provision can be compared. The outcome of a self-assessment using the Excellence Model provides the opportunity to benchmark within and outside the sector. This potentially adds more fully to the richness of learning and the possibility of gaining new ideas on �standard� management practices and processes from outside the sector.

• Subject Review focuses on the student experience as provided by a specific

subject area. This provides a great deal of scrutiny in specific areas over a short space of time, but can often ignore some of the wider implications and cross-cutting issues that impact on this experience. The process of undertaking a self-assessment against the Excellence Model takes a wider and more holistic view, rather than the detail � for which there is little guidance.

• Subject Review usually involves a wide range of people from the School, and

from across the institution and is well co-ordinated to included views from all areas that may have an impact. The number and type of people involved in an Excellence Model assessment can vary and is dependant on the self-assessment method used by the institution. This means that a small number of people could undertake the assessment meaning that the evidence gathered is not reflective of the real situation.

• The importance of a central team/shared resource of knowledgeable people

is critical for both approaches. This encourages shared understanding and the interchange of best-practices within and across institutions.


8 Summary Conclusions It must be recognised that the EFQM Excellence Model® and the QAA Framework derive from different traditions and have different starting points and languages. EFQM should not be viewed as a substitute for QAA Review, at any level. EFQM is operative now in a large number of organisations who have chosen to use it as a management tool or approach: it is a potential integrator, which can help with the management of multiple requirements. The QAA Framework is actually applicable to a much smaller number of institutions but has to operate on a pragmatic basis (that is, one which is acceptable to both Government and the institutions being judged) in a highly political context. A number of conclusions have been drawn from this work, and it is recommended that these are read in their entirety in section 8, with potential further work listed in Section 9. Building on the conclusions from the previous sections however, a number of questions also emerge. These include: • are there two different concepts of what constitutes �excellence� operating in

these two models? • could UK HE (or UK Plc) afford for the QAA Framework�s processes to

publicly identify deficiencies, weaknesses, or improvement areas on the scale which the Excellence Model appears to do?

• is the Excellence Model over-demanding? • is the QAA Review model insufficiently objective and systematic and too

heavily dependent on subjective peer review? To what extent is it able to achieve finely differentiated, objective judgements which are meaningful in the context of a diverse HE sector?

• to what extent can standardisation of approach and encouragement of good practice via non-prescriptive frameworks, guarantee academic standards and quality of provision/ student experience?

• might it be useful for the QAA Framework to be developed and improved in the light of the EFQM model, to become almost a �sector specific� tailored variant of the EFQM Excellence Model®?

• and finally � if UK HEIs are subject to a mandatory external review system can they afford to resource a different quality review Model (EFQM Excellence) internally � or, can they afford not to if it is accepted that there is an ever-increasing need to bring closer together, business and academic planning, management and review?

EFQM as an approach to self-evaluation at any level � Institution, Site, School/Department can provide a mechanism for all stakeholders to understand what the core issues are. It is a route through which internal quality systems can be managed and enhanced, using a wider perspective as the starting point. It provides a way of joining-up thinking between business planning and academic quality with a focus on customer orientated results. This in turn provides a seamless cycle of planning, development, checking/evaluating and continuous improvement on an on-going basis. It is concluded that there may be scope for assessing further how the QAA framework and the EFQM Excellence Model® could work synergistically, even as


far as developing a �HE Sector� or �Education Sector� model. Any such tailoring might usefully incorporate, for example, a sharper focus on standards and address the issue of learning partnerships between students and staff. In the mean-time, there are such strong synergies in some of the foci of the EFQM Excellence Model® that it may be beneficial for HEIs to consider how they might capitalise on the learning and improvement planning derived from use of the Excellence Model, to improve the ways in which they routinely manage for academic quality and standards and potentially, to reduce the burden of preparation for periodic external reviews by QAA.


INTRODUCTION


1 Context of the Project

GMP143/QAA is one project being undertaken by the Consortium for Excellence in Higher Education. This Consortium is led by Sheffield Hallam University and includes the Universities of Cranfield, Durham, Salford, Ulster and Dearne Valley College (FE representative). It is part of the wider GMP143 programme which is being funded under the Developing Good Management Practice HEFCE initiative (99/54). The programme (which contains 18 projects) has a life span of 3 years and aims to evaluate the benefits of introducing and implementing the EFQM Excellence Model® into the Higher Education Sector. Funding for the programme was awarded by HEFCE on 14 April 2000. The programme commenced on 2 May and is expected to conclude in May 2003. The projects are summarised as follows: The projects fall into 4 main areas: 1. Self-assessment projects

6 projects covering each institution (SHU, the Universities of Cranfield, Durham, Salford, Ulster and Dearne Valley College) are testing the implementation of self-assessment activities against the EFQM Excellence Model. Pilots are taking place in range of areas � schools, departments, research institutes, cross college and faculty wide. Each institution is also testing different assessment methodologies, rather than just using one. This is adding great richness to the programme, and supports our learning.

2. Mapping and research projects

There are 5 projects in this area which seek to address the relationship, inter-relationship, synergy and gaps between the EFQM Excellence Model and other management tools, models, concepts and auditing frameworks that are used within an HE and FE environment. These aim to reduce confusion across the sector about where each may fit within an organisation compared to another.

3. Benchmarking projects

The 2 benchmarking projects aim to compare the work that we are undertaking with educational institutions internationally who are exemplars of excellence, and with other private and public sector organisations within the UK who have won quality awards. This will allow us to develop, enhance and evolve our methodologies and approaches to ensure that they are as excellent as they can be when compared to the best.

4. Communication projects The 5 communication projects include conferences for each year of the programme, the development and maintenance of a programme web site, and a final programme report which will pull together the findings of the programme over the three years. This will be published and disseminated to the sector.

INTRODUCTION


In keeping with the management of the whole programme, this project has been managed using a form of PRINCE2 methodology, with a project plan established at the outset, and a project board created which has met to monitor, advise and steer the development of the work undertaken.

2 The Changing HE Environment Since this project was first agreed (May 2000), significant changes have developed around the QAA frameworks for the HE sector as a whole. This project has therefore taken into account the impact of the latest thinking and discussions that were underway at the time of writing. It is acknowledged that even when published, further national developments may have been made, but it is hoped that this work will help to inform the continual debate about how �quality� and �excellence� may be defined, and subsequently monitored, within the HE sector.

3 Aims and Objectives This project set out to compare and contrast the principles and concepts of the EFQM Excellence Model with the principles and framework of the new QAA Quality Assurance framework, including Institutional Review, Subject Review, Review of Overseas and Collaborative Provisions, and the Codes of Practice. Due to the resource constraints, the project did not set out to conduct primary research across a range of HEIs, but to focus at a generic level, using lessons learned at Sheffield Hallam University as primary examples. Its main starting point was to answer the questions it was thought the sector may ask when considering a new management framework such as the EFQM Excellence Model: �If we have a QAA framework then why do we need this EFQM Model? What is the difference, and can it really add any value?�

4 Project Deliverables and Outcomes The project plan identified three key deliverables for this project, as follows: • A final report which highlights the findings of the mapping exercise, identifying

where the best practice lies within each of the models, and where the EFQM Excellence model is positioned within these contexts. This is the final report.

• Paper published on the web. This final report is published on the web as an open access document.

• Findings delivered and discussed at the Excellence in HE conference in June 2001. A workshop was delivered at the Mirror of Truth Conference on 7 June on this topic. The workshop presentation and notes can also be found on the web site http://excellence.shu.ac.uk

5 Project Methodology

The data for this report was based on a wealth of information and knowledge held mainly by the staff of Sheffield Hallam University who have contributed to its creation. Individual meetings and Project Board meetings have been held to consider, debate, scrutinise and clarify the information presented to ensure that it


is as accurate as possible at the current time. External verification and comment has also been sought from the QAA, and from other interested parties who have been included as part of the Project Board. It is recognised that a detailed analysis of the sector has not been undertaken, but it is felt that the conclusions/issues within the report reflect the situation that the sector faces.

6 Project Authors and Contributors The project has been managed by Carol Steed, Organisational Excellence Manager for Sheffield Hallam University. Carol authored sections of the report that related more particularly to the EFQM Excellence Model, and as a trained external EFQM Assessor and Project Manager for the self-assessment activities within SHU, was able to bring experience of using the Excellence Model in a higher education context. Carol is also Programme Manager for the GMP143 programme. The primary author and co-ordinator of materials for the QAA aspects of the report was Gwyn Arnold, Deputy Academic Registrar for Sheffield Hallam University. Also a trained EFQM assessor, Gwyn has been able to offer considerable insights into the current and future shape of quality frameworks across the sector. Her detailed and intricate knowledge of all aspects of the past and present QAA frameworks enabled detailed mapping work to be clearly directed and expertly authored. The project has also benefited from a range of other valuable contributions from within and outside Sheffield Hallam University. Most notably the contributions of members of the Academic Registry within SHU who took on much of the detailed research and mapping work in addition to their already complex day jobs, with no additional resource to support them. Other members of staff from Schools across the institution also contributed their expertise and experience of using both the QAA and EFQM frameworks, which proved invaluable when considering the detail, impact and implications of this mapping work.


SECTION 1

EXPLORING THE EFQM EXCELLENCE MODEL


1 The Development of the �Quality� Concept

The European Foundation for Quality Management (EFQM) was created in 1988 by 14 leading European businesses. They had been inspired by the philosophies of Total Quality Management (TQM) which had transformed parts of the manufacturing sector, and had started to transfer to other sectors across the world. Originating from the ideas of process quality and control, the concept of quality having a much wider dimension was first introduced by W. Edwards Deming in the 1950�s. Deming set out an approach to total quality management by the introduction of his now famous 14 points (Appendix A). In addition to promoting product or service quality and more efficient business processes, it also gave industry a human face. His work in Japan saw the creation in 1965 of the now coveted Deming Prize which is awarded annually to Japanese companies, worker groups and individuals who have distinguished themselves in the area of total quality.

Joseph M Duran built on Deming�s philosophies, defining quality as fitness for use in terms of design, conformance, availability, safety and field use. Unlike Deming, he focused on top-down management and technical methods rather than worker pride and satisfaction. Juran developed his TQM message around 10 key steps. (Appendix B). He continued to build on Deming�s work by communicating that quality control was an integral part of management at all levels, not just the work of a special quality control department. It was from this that �quality circles� were created � the practice were quality improvement meetings were held at all levels of the organisation.

Kaoru Ishikawa is known as the �Father of Quality Circles�. He continued to develop ideas through the 1960�s and created the �fish bone� or Ishikawa diagram as a management problem-solving tool which is used by quality circles and quality improvement teams world-wide. He also introduced seven basic tools which were viewed as indispensable for quality control. These are Pareto analysis, fishbone diagrams, stratification, tally charts, histograms, scatter diagrams and control charts.

Philip B Crosby popularised total quality through his book Quality is Free. He built on the thinking of Deming and Duran, and added his idea that quality is �conformance to requirement�. His notion of a zero defects goal as something practical to aim for was based on the fact that poor quality on average was costing companies about 20% of their revenue. Crosby cited four new essentials of quality management which he calls �the absolutes�, and also introduced his own 14 points for success. (Appendix C). Crosby stressed motivation and planning were key rather than statistical process control.

SECTION 1 EXPLORING THE EFQM EXCELLENCE MODEL


Although these experts have differing opinions in certain areas, a number of key themes emerged: • Inspection is never the answer to quality improvement. • Involvement of and leadership by top management are essential to the

necessary culture of commitment to quality. • A program for quality requires organisation-wide efforts and long term

commitment, accompanied by the necessary investment in training. • Quality is first and schedules are secondary.

John S Oakland has furthered the work of TQM in Britain. He has been able to explain the theory of TQM, and then demonstrate the practical application of his theory in practice. Oakland has further developed the thinking behind TQM and it�s integration into a company�s strategy. He developed a model which he summarised in five points (Appendix D), and defined total quality management as �a comprehensive approach to improving competitiveness, effectiveness and flexibility through planning, organising, and understanding each activity, and involving each individual at each level. As a result of this evolution in quality thinking, TQM became a driving force for quality improvement within many organisations across the world.

2 The Evolution of the EFQM Excellence Model It was this holistic approach that was considered and evolved by those 14 businesses, whose mission and driving force was to develop these concepts into something that would encourage sustainable excellence in Europe. Their objectives were: �To stimulate, and where necessary, to assist management in adopting and applying the principles of Total Quality Management, and to improve the competitiveness of European industry�. Taking into account research from across the world on other quality awards and systems, such as the Baldridge Award in America, and the Deming Prize in Japan, the EFQM launched the European Quality Award in 1991. This was initially seen as a way of to identify role models within Europe, recognising their excellence and encouraging them to benchmark and lead others. Supporting the award, a range of criteria had been developed against which organisations were assessed. This built on the criteria used for the Baldridge Award, but included the additional features of business results and impact on society. The model and award scheme has been developed over the past ten years to reflect the diversity of the economy and organisations that exist. In 1995 the Public Sector Award was launched along with a revised model against which Public Sector organisations could assess themselves. In 1996 the SME Award and model was launched, followed in the same year by the launch of an Award for operational units. A number of local and national Award Schemes have also been launched. In the UK the British Quality Foundation (BQF) support and work in partnership with


local and regional centres who run award programmes. The BQF also run a national award each year - The model was therefore becoming a widely used tool and it soon became apparent that organisations were using the framework as a measurement and diagnostic tool without entering for the award. It was visually simple to follow, with a range of criteria and sub-criterion that were laid out in an easy to understand format. The philosophies, steps, diagrams and points that evolved through the development of TQM were embodied in one model, providing a common language and framework against which tens of thousands of organisations across Europe could measure their performance in a holistic and consistent way.

3 The EFQM Excellence Model � The Foundation Stone for Excellence The EFQM model recognised that process improvement was at the heart of any organisational development, but it also made the connection that it is through processes that the talents of its people can be released, which in turn produces better performance. It also followed that improvement in the performance can be achieved only by improving the processes by involving the people. This simple model is shown in Figure 1. FIGURE 1 THE SIMPLE MODEL FOR IMPROVED PERFORMANCE Source: Total Organisational Excellence, John S Oakland, 1999.

The EFQM Excellence Model itself (which is a registered trademark) was revised in 1999 to take account of current thinking, practices and working environments. It is described by the EFQM as: �a practical tool to help organisations establish an appropriate management system by measuring where they are on the path the Excellence, helping them to understand the gaps, and then stimulating solutions�. The model, shown in Figure 2, is non-prescriptive framework based on nine criteria, with five �enablers� and four �results�. The enabling criteria cover what the organisation does, and the results criteria cover what the organisation achieves. �Enablers� cause �Results�.

People

Performance

P R O C E S S E S


FIGURE 2 THE EFQM EXCELLENCE MODEL

® Source: European Foundation for Quality Management, 1999.

The model is underpinned by what is termed the �eight essentials of excellence�, or the fundamental concepts. The EFQM believe that the achievement of Excellence requires total management commitment and acceptance of these concepts. These concepts are listed below, and outlined in more detail along with their benefits in Appendix E. The eight fundamental concepts are: 1. Results Orientation 2. Customer Focus 3. Leadership and Constancy of Purpose 4. Management by Processes and Facts 5. People Development and Involvement 6. Continuous Learning, Innovation and Improvement 7. Partnership Development 8. Public Responsibility It can be seen from these concepts that the ethos of TQM has been embodied and evolved further.

4 The Criteria and Sub-Criterion The criteria and sub-criterion are the only prescriptive parts to the model. It is recognised that assessment against all nine criteria is both desirable and accepted as good management practise. Organisations who are applying for the European Quality Award need to demonstrate evidence in each of the sub-criterion areas. The bullet points given within each sub-criterion are lists of possible areas to address. They are prefixed by �should or �may� to indicate that these are areas which give guidance and promote further thought about the sub-criterion, and are not intended to be mandatory or exhaustive.

Processes CustomerResults

SocietyResults

Policy &Strategy

People

LeadershipKey

PerformanceResults

PeopleResults

Partnerships& Resources

Enablers Results

Innovation and Learning


The five enablers each contain either four or five sub-criterion each. The results each contain two sub-criterion each. There are weightings attached to each criteria to reflect importance to that area when scoring is undertaken.

5 Using the Model as a Self-Assessment and Planning Tool The model is used by the majority of organisations as a way of finding out where they are now, considering where they want to improve, and then making decisions on how to get there. This can be simply illustrated as shown in Figure 3: FIGURE 3 SIMPLE ASSESSMENT CONCEPT Source: European Foundation for Quality Management, 1999.

As organisations mature, the model becomes part of the business planning framework, where it is used to gather evidence, with the results being fed in to strategic, operational and personal development plans. This is illustrated in Figure 4:

FIGURE 4 INTEGRATION OF THE EFQM EXCELLENCE MODEL

® WITH BUSINESS

PLANNING Source: Lloyds TSB

Where am I now?

Where do I want to go?

How to get there

Vision, missionvalues

Self-assessment

ActionPlanning

Implementationand review

Vision, missionvalues

Strategic Planning

BusinessPlanning

Implementationand review

� Leadership� Maturity� Continuous Improvement� Strategic Integration

IntegratedPlanningProcess


Further use of the model defines it as a management tool rather than a measurement tool, encompassing many other improvement tools, techniques and frameworks which work at more specific levels. Figure 5 illustrates this journey:

FIGURE 5 THE EXCELLENCE JOURNEY Source: European Foundation for Quality Management

The self-assessment process using the Excellence model provides an organisation with information by which it can monitor and evaluate progress towards its own goal and organisational excellence. It leads to the identification of strengths and areas for improvement in a wide range of activities, and prompts actions to be owned and taken forward. There are a number of simple steps which can be taken through the self-assessment process. These are shown in Figure 6. FIGURE 6 THE KEY STEPS TO SELF-ASSESSMENT Source: Total Organisational Excellence, John S Oakland, 1999.

Data collection

Consensus scoring

Action planning

Strengths and areas for

improvement

Assessment

Review and refinement

Excellence

Maturity of organisation

Health check Planning tool Strategic tool

Self-AssessmentBusiness Plans

&Self-Assessment

Organisational DevelopmentProcess ManagementCorporate Scorecard

Goal DeploymentPersonal Development

Customer and Supplier RelationshipManagement

Partnership & CollaborationLearning Organisation

Integration & alignment


A detailed and working knowledge of the model, and a deeper understanding of the relationship and inter-relationship of the criteria and the balance that it provides in terms of looking at all aspects of an organisation, means that it can be used holistically to make real and continued improvement. Before undertaking a self-assessment, many organisations train Assessors within their organisation who are then knowledgeable in the philosophy, use and application of the model. Assessor training involves attending a two day workshop hosted by a licensed trainer, where the model is explained in detail and the evaluation and scoring of evidence is practised. Pre-work evaluating a case study is also required. Once trained and licensed, an Assessor is better placed to lead and assess the self-assessment process. Trained assessors are able to assess and validate evidence from other organisations as requested. European Quality Award Assessors, and Assessors for Regional Quality Awards undergo additional training. This consistent framework for the training of Assessors ensures that there is continuity amongst the evaluation and scoring of evidence, as most Assessors have been trained to a similar standard.

6 Key Methods Of Assessment Assessment against the model is flexible dependant on the size, type and maturity of the organisation. Assessment can be internal (self), external (assessed by people outside the assessment unit), or a mix of both. The main types of assessment methods are shown in Figure 7:

FIGURE 7 ASSESSMENT OPTIONS Source: European Foundation for Quality Management

DATA

PROCESS RIGOUR LOW HIGH

QUESTIONNAIRE

MATRIX

WORKSHOP

Based on Opinion

Supported by Evidence

PROFORMA

AWARD ENTRY

Copyright EFQM


Each assessment method has a �standard� approach, but because of the nature of many of the assessment methods (particularly if being used as part of a self-assessment rather than an external assessment) these are flexible and can be adapted to meet the timescales and resources of individual organisations. In order to give a clearer picture of the approaches, positives and negatives of each method, they are outlined in a table format. Table 1 Questionnaire Approach

Approach / Process Positive Negative • Completion of a

questionnaire which may or may not follow the criterion areas of the model.

• Completed by one or more people at whichever level of the organisation required.

• Completed as part of a meeting/workshop, or individually.

• Documents are completed, analysed and answers collated.

• Trends and gaps identified.

• Actions identified.

• Can be one of the least resource intensive.

• Simple to use - basic awareness training being sufficient to get things started.

• Can readily involve many people within the organisation.

• The questions asked can be customised to suit the organisation.

• Enables the organisation to receive feedback which can be segmented by function and by level.

• Can be used in parallel with the workshop approach to provide a more balanced view of deployment for the management team.

• Can give a good visual reference if results are graphed.

• A list of strengths and areas for improvement is not generated.

• Accuracy depends upon the quality of questions asked.

• Excessive use of questionnaires in an organisation may result in a low return.

• Wide circulation can raise expectations.

• Questionnaires tell you what people think, not why they think it.

• Does not allow for direct comparison with scoring profiles of European Quality Award applicants.

• Scores can be derived, but can only be used realistically as trend data, rather than real scores.


Table 2 Matrix Approach

Approach / Process Positive Negative • Completion of a proforma

which has questions down the first column, and �current state� statements across the top of the other columns.

• Matrix can be completed by one or more people at whichever level of the organisation required.

• Often led by one or two individuals, who may have a short meeting to explain the concept and hand out the matrix documents.

• Documents are completed, analysed and answers collated.

• Trends and gaps identified.

• Actions identified.

• Simple to use with minimal training required.

• Can be used to involve everyone in self-assessment.

• Provides a practical way of understanding the fundamental concepts of the Model.

• Enables teams to assess and display progress, shows gaps and possible next steps.

• Good for facilitating team discussions and team building.

• Involving the management team of an organisation in developing their own matrix can be a powerful process.

• Can give a good visual reference of gaps between current and future state.

• Does not provide a list of strengths and areas for improvement.

• Does not allow for direct comparison against European Quality Award applicants.

• There is not a direct cross-reference between the steps in the matrix chart and the criterion parts of the Model.

• Time consuming if you develop your own Matrix Chart.

• Scoring using the RADAR concept not usually possible.

Table 3 Workshops

Approach / Process Positive Negative • Awareness raising to bring

everyone up to date with model and assessment process.

• Gathering of data/evidence as a group within a workshop environment, using a series of workshops.

• Final workshop to evaluate all evidence gathered, agree priority actions and allocate owners.

• Develop action plans. • Implementation.

• An excellent way to get the management team to understand the Model and gain their commitment to it.

• Discussion and agreement by the management team helps to build a common view.

• Ownership by the management team of the outcomes and subsequent prioritisation and agreement to action plans.

• Team building opportunity for the management team.

• An agreed list of strengths and areas for improvement is produced which will drive improvement actions.

• A less robust and rigorous process than the Award entry approach.

• Can be a high risk approach and needs excellent preparation.

• Evidence of the deployment of various company processes can be difficult to assess.

• There is scope for unrealistic scoring.


Table 4 Pro-forma Based Approach

Approach / Process Positive Negative • Similar to the Award entry

approach except that the output is recorded in a simpler format.

• Typically a pro forma is produced covering, for each criterion part: • Evidence • Strengths • Areas for Improvement • Score

• The document produced records the information in abbreviated form.

• Can be assessed internally, or by representatives external to the assessment unit.

• Provides a list of strengths and areas for improvement for driving improvement actions.

• Output is clearly documented.

• Scoring profiles can be close to the Award entry approach in terms of accuracy.

• Less resource required than the Award entry approach.

• The output provides only a summary of the current position.

• Can be challenging for managers to collect the right data, particularly in areas where no data may exist.

• Some time and resource required, although not as much as for Award entry approach.


Table 5 EFQM European Quality Award Entry

Approach / Process Positive Negative • Team approach to the

collection and evidencing of data against all sub-criteria of the model.

• Evidence gathered and collated over a period of months.

• Lead co-ordinator(s) collate and refine into one document (approx. 75 pages in length).

• Document follows criterion and sub-criterion giving evidence in each area.

• Document submitted to EFQM for assessment / scoring, or just used internally as an assessment document.

• If assessed well, team of assessors appointed for a site visit � 1 week in the company validating evidence in the document.

• Assessors then score and recommend for an award or prize (or not).

• If did not score well on first submission, document returned with detailed comments so that learning can be gained and further improvement made.

• Provides a powerful and concise way of reflecting the culture and performance of the organisation.

• It is based on the collection of evidence and is therefore fairly robust.

• Clear documentation of output.

• Provides international recognition if successful in winning an Award or Prize.

• If unsuccessful, can still receive constructive and useful comments on which to base further improvement.

• Good way of benchmarking against other �best in class� organisations.

• Produces a list of strengths and areas for improvement.

• Subsequent reports become relatively easier to complete.

• Provides an excellent opportunity for involvement and communication during the data gathering process.

• Provides an excellent communication document inside the company and externally.

• The site visit and presentation from the assessor team are two value added steps.

• Provides an easy way for internal organisational comparisons of processes and results and sharing examples of good practice.

• Scoring and score achieved seen as robust.

• Is resource intensive with the need for a dedicated team to pull the evidence together over a period of months (can be between 3 and 6 months).

• Only provides a snapshot of the organisation at that time.

• Potential for creative writing, covering up the real issues.

• Risk of less involvement by the management team and greater delegation to others.

• Approach may be too ambitious as a first attempt at self-assessment for new recruits to the Excellence journey because of the potentially high volume of areas for improvement produced.


Whichever assessment method is used, it gives organisations the opportunity to look holistically at themselves against a range of criterion which address internal and external factors. The assessment methods can be mixed to ensure that the evidence collected is robust and suitable for the needs of the organisation at that time, for example, the workshop and pro-forma based approach could be used in Year 1, complimented by questionnaires. In Year 2 the questionnaires only could be used as a �health check� to monitor progress. The pro-forma could then be re-visited in Year 3. The option for internal and external assessor validation is also an important part of the assessment process. External assessment can be sought at any time to validate current opinion and evidence to ensure that the organisation is being open and honest with itself. However, internal assessments are also valuable on a more regular basis to ensure that on-going improvement is being made.

7 The Reliance Upon Evidence At the heart of the model is the logic known as RADAR. RADAR consists of four elements, which are also shown in Figure 8.

Results

Approach

Deployment

Assessment and

Review FIGURE 8 THE RADAR LOGIC Source: EFQM Excellence Model

It is RADAR that gives weight to the model in terms of being built on a strong evidence base. For the results criterion, RADAR provides the framework to look at what an organisation has actually achieved across the balanced set of indicators. It requires a demonstration of positive trends and/or sustained performance for 3 or more years; assesses whether targets are appropriate and

Determine

Results required

Plan and develop

Approaches

Deploy Approaches

Assess and Review approaches and their

deployment


whether they have been met, how performance compares with others, and whether the scope of the results actually addresses relevant areas. For the enabling criteria, the framework covers what an organisation plans to do and the reasons for it. They are scored against whether the approach used is sound, having a clear rationale, being well defined and has a clear focus on stakeholder need. It must also be integrated, supporting policy and strategy and linked to other approaches where appropriate. It also scores the deployment of each approach � to assess whether the approach is implemented in all relevant areas in a systematic way. The final aspect of RADAR concentrates on whether an organisation does actually assess and review both the approach and the deployment of the approach, with regular measurement demonstrated and improvements made. The individual scores are then weighted. The weightings are attribute to what are viewed as the most significant aspects of the model. These weightings were derived at the time the model was formed and so are based on a broad range of experience and wealth of information from across the world. The weighting of the criterion are shown in Figure 9 below: FIGURE 9 THE EFQM EXCELLENCE MODEL

�� WITH WEIGHTINGS SHOWN Source: European Foundation for Quality Management

From these final calculations an overall score is derived which can allow an organisation to benchmark all, part or specific activities against others within or outside their own sector, locally, nationally and internationally.

8 Benchmarking Against Others The EFQM Excellence Model� provides a common framework and language which can be used to compare performance in all areas, or specific areas against others within or outside the sector who are using the same assessment framework. It allows organisations to focus on specific areas for improvement, and learn from others who have had similar learning experiences in the past. It also enables organisations to assess how well they are actually doing in comparison to others, and whether they are actually the �best in class� in a specific area, and could therefore help others.

Processes14%

CustomerResults

20%

SocietyResults

6%

Policy &Strategy

8%

People9%

Leadership10%

KeyPerformance

Results15%

PeopleResults

9%

Partnerships& Resources

9%

Enablers Results

Innovation and Learning


Benchmarking is viewed as an important part of the journey to Excellence. The EFQM Excellence Model® provides the framework for analysis, with benchmarking allowing internal and external comparison to promote learning and continuous improvement. There is no final score, or ceiling of achievement, but a progression which needs to be checked and measured through the self-assessment and benchmarking processes. Whilst the outcomes and scores from self-assessments can remain confidential, many organisations are able to share the learning, methodology, process improvements and pitfalls with others, without contravening commercial confidence. The publication of reports and scores is not mandatory, but many award winning organisations publish summary documents of their award entries, with changes or omissions of sensitive data. This fosters a culture of learning and sharing within and between sectors, which again is based on the understanding and sharing of a common language and assessment framework.

9 Practical Use Of The EFQM Excellence Model® There is no reliable data that specifies how many organisations use the model as an improvement/assessment/management/planning tool, but it is estimated that well over 10,000 organisations are using the model within the UK, with many more across the European Union. The Government has committed to adopting this model within local and central government departments, and is currently deploying self-assessment practices across the range. The Higher Education Funding Council for England is also engaged as part of this process in self-assessment activity against the model. The Cabinet Office agenda is very much driven by this directive, and has provided funding to support the Developing Good Management Practice Initiative, of which this programme is a part. The Cabinet Office itself is also applying the fundamental principles of the model to its operations and is working closely with other �standards� institutions, such as Investors in People, as well as developing partnerships with QAA, to investigate where alignments can be made to provide a single and common auditing framework. The European Foundation for Quality Management has established a Public Sector Steering Group, which is seeking to address the particular needs of the public sector, raise awareness and promote the use of the model. A specifically identified area within this group is that of Education. Mike Pupius, Director of Organisational Excellence at Sheffield Hallam University is leading this European Forum, which aims to bring together educational institutes from across Europe to share best practise and ideas about the practical use of the model. Experiences and issues from this group will be feed back to EFQM in order to influence and progress the on-going development of the model and any particular support requirements identified by the sector.


10 Summary The EFQM Excellence Model® is a holistic framework that recognises that there are many approaches to achieving sustainable excellence. It is a tool that can be used purely in a measurement capacity, or can be developed and adapted to suit individual organisational need as a management tool. It is not a replacement for other �quality� tools and techniques, but provides an over-arching environment in which these tools can be applied to make specific improvements in areas identified as needing development. It allows organisations to look at all aspects of their operation at any level � micro, macro or even personal, giving everyone a common language to communicate their findings and build on their results together. It is a model which is used widely in the UK and across Europe and which is being promoted by both the Government and the Cabinet Office as a key part of any management portfolio.


SECTION 2

THE QUALITY ASSURANCE AGENCY

FRAMEWORK


1 The Quality Assurance Agency For Higher Education

The Quality Assurance Agency for Higher Education (QAA) was established in 1997, following a recommendation from the JPG1 on the establishment of a single agency responsible for an integrated process of quality assurance � at subject and institutional level - for the UK. The new agency�s remit was thus to combine and develop the activities previously discharged by the Higher Education Quality Council and the quality divisions of the English and Welsh Funding Councils. The Agency is an independent body, funded by subscriptions from universities and colleges of higher education, and through contracts with the main funding bodies The Agency's mission is �to promote public confidence that quality of provision and standards of awards in higher education are being safeguarded and enhanced�. Its core business is to review the quality and standards of higher education in universities and colleges. It does this by auditing institutional arrangements for managing quality and standards, including arrangements for collaboration with overseas partners; and by assessing the quality and standards of teaching and learning at subject level. In the same year as the establishment of the Agency, the NCIHE2 published its report (the Dearing Report). The report contained a range of recommendations addressed to QAA or directly relevant to the QAA which have set the agenda for the Agency�s development work. Recommendations included: • enhancing the role of external examiners; • developing a qualifications framework; • developing programme specifications; • specifying criteria for franchising arrangements and monitoring compliance; • developing benchmark information on standards In November 1997, the QAA Chief Executive wrote that four principles would underpin the Agency�s approach to implementing the Dearing agenda3: • Accountability � Quality assurance mechanisms must produce public information to

satisfy the legitimate expectation that there will be accountability and value for the money spent on higher education, and clarity and transparency of objectives, qualifications and standards.

• Ownership � Quality of provision �. must be designed into programmes from the

start and driven by a strong sense of personal ownership of and responsibility for, the quality of all aspects of higher education provision. Peer review will continue to lie at the heart of our processes. The diversity of purpose that characterises higher education will be respected and fostered.

1 Joint Planning Group for Quality Assurance in Higher Education 2 The National Committee of Inquiry into Higher Education (NCIHE) chaired by Sir Ron Dearing, with the NCIHE�s Scottish Committee chaired by Sir Ron Garrick 3 QAA (1997) Higher Quality Vol 1 No. 2 p.1

SECTION 2 THE QUALITY ASSURANCE AGENCY FRAMEWORK


• Enhancement � We will endeavour to � promulgate to the sector, through publications and seminars, material designed to assist institutions and individuals to enhance the quality of their provision.

• Reducing the perceived burden of external scrutiny � We will endeavour to

streamline external quality assurance and to create a framework within which all of those with a legitimate interest in the quality of provision can gain the information they require, whilst reducing the number of separate external engagements to which institutions and departments are subject

2 The QAA Quality Assurance Framework: Overview Following an intensive period of development and consultation, the QAA is introducing progressively, from October 2000, the new quality assurance method in the whole of the UK. The Agency aspires to achieve an integrated framework in which the different elements of the Framework work together in dynamic and iterative fashion. The Framework is focused on making more explicit the outcomes of learning in Higher Education and on the establishment, maintenance and enhancement of academic standards. At the time of preparation of this Project Report, further consideration is being given to the proposed processes, particularly Subject Review, in view of sector and government pressure for a reduction in �the burden of accountability�. A figure of 40% reduction has been mentioned but as yet it is not clear how this will be achieved. No firm, agreed revisions to approach or methodology have been issued by QAA. This paper is therefore based on material published by QAA with reference, where appropriate, to possible or likely changes which QAA may introduce. The key elements of the framework are:

(a) Academic Review - a revised system of review, scrutinising performance at subject and institutional level, within a 6 year cycle. Academic Review therefore encompasses Subject Review and Institutional Review. Institutions� arrangements and practice relating to collaborative provision will be covered either during Institutional Review or maybe via a separate institutional Collaborative Provision Audit in some cases. Reviews of overseas collaborative provision will still normally be conducted within nominated countries, but reporting will be on an institutional basis and outcomes will also inform and influence Institutional Review.

(b) A developing range of Reference Elements for Higher Education intended to set out requirements or expectations in relation to standards and quality management. The core reference elements are the National Qualifications Framework, Subject Benchmarking, and the Code of Practice for the Assurance of Academic Quality and Standards in HE. Programme Specifications and Progress Files are the other key elements of the framework, providing vehicles for the specification of the intended outcomes for individual programmes (Programme Specifications) and the record of achievement of outcomes (Progress Files).


An Institutional Profile, compiled using the outcomes of previous reviews will be maintained by the Agency. This is intended to provide much of the primary evidence for Institutional Review and will be used to determine the levels of scrutiny and focus of all reviews. The profile will be �dynamic� and will be continually modified in the light of review outcomes. The interrelationships between the different elements of the Framework can be presented as follows:

FIGURE 10 FRAMEWORK INTER-RELATIONSHIPS Source: Sheffield Hallam University

3 Key Elements of the Framework

3.1 Institutional Review

An Institutional Review (IR) by QAA will take place every 6 years preceded by a 3 yearly Interim Appraisal of the institutional profile. This will replace Continuation Audit (CA) and is seen by QAA as the senior layer or �capstone� review. IR addresses the ultimate responsibility for the management of quality and standards that rests with the institution as a whole, being concerned particularly with the way an institution exercises its powers to award degrees. Using the outcomes of reviews at subject level, and additional information from senior levels of management, it reports on the degree of satisfaction that may


reasonably be placed in an institution�s effectiveness in managing quality and standards. Institutional Review will address the robustness and security of the systems supporting an institution�s awarding function. It will be concerned with: • Procedures for approval, monitoring and review of academic programmes • Procedures for acting on the findings of external examiners, subject reviews,

and other external scrutinise • Overall management of assessment processes • Overall management of any credit systems • Management of collaborative arrangements with other institutions • Important points of reference for IR are provided by the Qualifications

Framework and sections of the Code of Practice, particularly: • Programme approval, monitoring and review • Assessment of students • External examining • Collaborative provision QAA will maintain a dynamic Profile of each institution. This will contain the conclusions from the QAA�s most recent review of each subject, since the last Institutional Review, and from any separate reviews of collaborative provision. This Profile will provide much of the primary evidence required for Institutional Review. Originally it was intended that the Profile should be a confidential document for use only by QAA and the institution concerned, but if there is a reduction in Subject Review visiting, for example so that only sample visits are undertaken, it may be appropriate for the Institutional Profile to become a public document in the interests of public accountability. The self-evaluation document prepared by the institution is the starting point for the IR. The document will contain the institution�s analysis of how effectively it manages the quality of its programmes and the standards of its awards and how it meets the expectations of relevant precepts of the Code of Practice. Institutional Review will provide a summation and renewal of the Institutional Profile. The Institutional Review Report will cover the effectiveness of the institution�s systems for managing the quality of its provision, the standards of its awards and the security of its awarding function, and will include a statement of the degree of confidence that the Agency considers may reasonably be placed in the continuing effectiveness of the institution�s quality assurance arrangements. This will be a major contributory factor in determining the intensity of scrutiny that is appropriate for subsequent Subject Reviews. Action points arising from IR will be categorised as essential, desirable, or advisable. For any �essential� action points, QAA will normally seek from the institution an account of action taken to address the matter, after one year.


Interim Appraisal will • Review progress on action arising from IR and consider whether a different

degree of confidence should be placed in institutional systems • Consider if there are any recurring problems which require attention in

future Subject Reviews • Consider the general trends of Subject Reviews, follow up action identified

and their implications for future intensity of scrutiny.

3.2 Review of Collaborative Provision Reviews of collaborative provision may take place as part of the main Institutional Review. Alternatively, they may be conducted separately, if the institution�s collaborative portfolio is considered to be of a scale which would warrant separate scrutiny. The QAA�s next programme of separate reviews of collaborative provision will commence in 2003/04. QAA Audit of specific overseas collaborative links will continue to take place on a country by country basis as at present. Reviews of collaborative provision are expected to make use of the same set of �reference points� as other reviews, with the section of the Code relating to Collaborative Provision being particularly relevant in this case.

3.3 Subject Review The new QAA Subject Review methodology has already been introduced in Scotland and will be introduced in England from January 2002. At the time of writing this report, as indicated above, the debate about Subject Review has been reopened and further consideration is being given to how the �burden of accountability� may be reduced. As no definitive guidance has been issued, this report is based on the Academic Review Handbook previously published by QAA. It will replace the current �snapshot� visit with a flexible review of provision across a period of up to one year. Judgements will be made about the academic standards of the subject, focussing on whether learning outcomes are appropriate and whether the outcomes achieved are consistent with the intentions. The quality of provision will also be reviewed and judgements made about the extent to which the following aspects of provision contribute to the achievement of the intended learning outcomes: • teaching and learning • student progression • effective utilisation of learning resources. There will be variable levels of scrutiny and focus, based upon the agreed Institutional Profile and subject indicators such as PSB reports and previous reviews. A Self-Evaluation Document will be submitted one month before the visit, to which programme specifications will be appended. The QAA Qualifications Framework, Subject Benchmark Statements, and Code of Practice will all be �points of reference� for the review.


Fewer �reviewer-days� will be allocated to visits than previously, and some scrutiny of evidence will take place off-site. There will be no allocated grades and judgements will be notified in the report rather than at the end of a visit, as is current practice. The 2000-2006 cycle of Subject Review is divided into two 3 year periods. However, the �new� Review Cycle 1 in England and Wales does not begin until January 2002 � giving an intensive 18 month first cycle.

3.4 Higher Education Qualifications Framework

Key Features One of the central recommendations of the Dearing Report was the construction of two inter-linked frameworks for higher education qualifications in the UK (one for Scotland and one for England, Wales and Northern Ireland). The Higher Education Qualifications Framework is one of the measures designed �to promote public understanding of, and confidence in, academic standards�. It is intended to introduce clarity, consistency and shared explicit reference points to distinguish the character, level and intended outcomes of HE qualifications. The principles underpinning the Framework deal with • the defining characteristics of individual qualifications - level, generic qualification

descriptors, required credit • qualifications as marking the achievement of positive outcomes,. • the definition of modules in terms of learning outcomes, level and credit volume • the independence of level from the nature of the learning, or progression in time • credit awarded on the basis of achievement, rather than time served, and

irrespective of when or how the learning has been achieved • consistency and accuracy of qualification nomenclature. The Framework consists of five levels; three at undergraduate level and two at postgraduate level. For England, Wales and Northern Ireland the Framework is as follows: Table 6 England, Wales and Northern Ireland HE Qualifications Framework LEVEL TYPICAL QUALIFICATIONS

D PhD/DPhil

other Doctorates MPhil Masters M Postgraduate Diploma Postgraduate Certificate H Bachelors degree with Hons

Graduate Diplomas and Certificates I Bachelors degree (non honours)

Foundation Degrees Diplomas of Higher Education and other higher diplomas

C Certificate of HE


QAA Implementation Timetable The new framework was issued in January 2001. Institutions are expected to demonstrate that all students successfully completing programmes which commence in the Academic Year 2003/04 and subsequently, will achieve qualifications which are in accordance with new framework.

3.5 Subject Benchmarking Subject Benchmark statements are designed �to make explicit the general academic characteristics and standards of honours degrees in the UK�4. There will eventually be one set of statements published by QAA for each of 42 designated subjects. The benchmark statements are concerned with undergraduate programmes only, and state what a �threshold� and �typical� student might be expected to achieve at final year. They will form one of a number of indicators by which judgements will be made about quality and standards during Academic Review. Benchmarks are about conceptual frameworks; intellectual capability and understanding; techniques and skills; intellectual demand and challenge. Reviewers will use benchmarks as a means of determining whether the intended learning outcomes of a programme are appropriate and whether standards are met. Institutions should be able to demonstrate how benchmark statements have been used to inform decisions about programmes, and in calibrating the overall demands of the assessment framework. For some programmes, more than one benchmark statement may be relevant, for others there may not be any. In all cases the institution remains responsible for ensuring that standards reflect appropriate external indicators.

4 http://www.qaa.ac.uk/crntwork/benchmark/benchmarking.htm � Benchmarking Academic Standards.


QAA Implementation Timetable Statements for 22 subject areas are now published: Table 7 QAA Subject Benchmarks: Phase 1 Accountancy Archaeology Architecture Business and Management Classical Languages and Ancient History Chemistry Computer Science Earth and Environmental Studies Economics Education Studies Engineering English Geography History Librarianship and Information Studies Philosophy Politics Social Policy & Admin and Social Work Sociology Theology and Religious Studies Hospitality, Leisure, Recreation, Sports & Tourism

Law

QAA has already begun the second phase of the benchmarking process in the following subject categories: Table 8 QAA Subject Benchmarks: Phase 2 Materials Nursing Area Studies Mathematics, Statistics & OR Art & Design Medicine Biosciences Music Building & Surveying Physics & Astronomy Linguistics Psychology Dance, Drama & Performance Arts Subjects Allied to Medicine Dentistry Town & Country Planning Languages & Related Studies Veterinary Medicine Agriculture, Forestry, Agricultural & Food Sciences

Communication, Media, Film & Television Studies

This second set of benchmarks will be published in advance of the second cycle of Academic Reviews, which begins in September 2003. Impact at Institutional Level • Given the important role and positioning of benchmark statements within the

overall QAA quality and standards framework, institutions will need to ensure that their frameworks and practices at institutional and School/ Faculty/ Departmental level take due account of them.

• Schools/ Faculties/ Departments will need to begin to map their provision against the benchmarks, make decisions about applicability and conformity and act appropriately.

• Validation exercises will need to make reference to subject benchmark statements where applicable.


• External Examiners will play a key role in determining the outcome in terms of student attainment and guidelines for External Examiners will need to be amended to take Subject Benchmarking into account.

3.6 Code of Practice on the Assurance of Quality and Standards in Higher Education

A Code of Practice was one of the Dearing recommendations. The Code will not have the force of regulations, but will provide a set of criteria against which QAA Reviewers, at both Institutional and Subject levels, will assess quality and standards. QAA�s original intentions were for 20 or more Code sections but the list has been trimmed to 11, possibly 12. Each section comprises 'precepts' backed up by supporting guidance. The understanding is that the precepts are 'principles in action', and are basic statements of expected levels of policy or practice. HEIs will be judged only against the precepts, although Academic Reviewers may explore matters covered in the guidance if they are not satisfied about the evidence presented to support implementation of the precepts. Key Features Each section of the Code comprises a set of precepts, typically between 15 and 40 bullet points, accompanied by outline guidance. While each section contains precepts highly specific to the particular area being covered, the following themes have emerged in the sections published to date: • Information to students, staff, applicants and partners should be clear and

accessible; • Internal Procedures should be explicit, transparent and fair; • Monitoring, Evaluation and Review should be regular and effective, and

monitoring and review mechanisms should themselves be scrutinised regularly;

• Staff Development and Training should be promoted both in academic and non-academic areas;

• Recording, Documentation and Publication of information should be accurate and systematic;

• Routes for feedback and complaints should be widely advertised.

QAA Implementation Timetable QAA expects that by May 2001 HEIs will be able to demonstrate their progress towards implementing elements of the Code. For each Section, HEIs are expected to be able to demonstrate adherence from 12 months after its publication.


Dates of publication so far are as follows: Table 9 QAA Code: Publication Schedule Section of the Code Publication date Postgraduate Research January 1999 Collaborative Provision July 1999 Students with Disabilities October 1999 External Examining January 2000 Academic Appeals and Student Complaints March 2000 Assessment of Students May 2000 Programme Approval, Monitoring and Review May 2000 Career Education, Information and Guidance 2001 (currently in draft) Student Placements (draft Dec 2000) Admissions (draft March 2001) APEL/ Credit Rating not yet known

Impact at Institutional Level • HEIs will need to be able to demonstrate that they fulfil the Precepts for

each section of the Code. They should also consider the introductory Sections, and accompanying guidance and consider how these might guide and enhance detailed practice

• Monitoring/audit mechanisms may need to be developed to ensure that adherence to the Precepts (and learning from the guidance) is maintained

• HEIs� internal quality assurance should ensure that adherence to the precepts is demonstrable

• HEIs may wish to review their policies, principles and procedures in light of the Code of Practice.

3.7 Programme Specifications

HEIs will be required to design and publish Programme Specifications for all programmes offered. These specifications will provide �a concise description of the intended outcomes of learning from a higher education programme, and the means by which these outcomes are achieved and demonstrated�.5 They will make explicit student learning outcomes in terms of skills, knowledge, understanding and other attributes. Specifications are intended to provide information to a range of stakeholders, including students, prospective students and employers.

5 QAA Guidelines for Preparing Programme Specifications.


Key features Teaching teams are required to specify: • The intended learning outcomes of the programme in terms of knowledge,

understanding, skills and other attributes • The teaching and learning methods that enable learners to achieve these

outcomes and the assessment methods used to demonstrate their achievement

• The relationship of the programme and its study to the qualifications framework.

Programme specifications will indicate the reference points used by curriculum designers in the development of outcomes such as institutional policies, Subject Benchmark Statements and PSB requirements. No single prescriptive format has been issued by QAA, but minimum requirements are specified and example templates and formats are available. The QAA suggests that the following minimum information be included: Table 10 Suggested minimum requirements for Programme Specifications • Awarding body/institution • aims of the programme • Teaching institution (if different) • Details of PSB accreditation

• relevant subject benchmarks and other external and internal reference points used

• Name of final award • programme outcomes • Programme title • UCAS code

• teaching, learning and assessment strategies to enable outcomes to be achieved and demonstrated

• Programme structures and requirements, levels, modules, credits and awards

• date at which specification was written or revised

Programme Specifications must be appended to Self-Evaluation Documents submitted for Subject Review. QAA Implementation Timetable Although there is no published implementation date, specifications need to be available for Subject Review by October 2001 for those subject areas undergoing review in the first cycle, and by July 2003 for those subject areas undergoing review in the second cycle.


Impact on Institutions Formats adopted will need to meet the needs of a variety of audiences but remain a concise overview of the programme. Further underpinning detailed information should be available in supporting documents. Creation of programme specifications needs to be embedded in curriculum design, validation, review and enhancement processes. Programme specifications play an important role in Subject Review providing reference points for evaluation of curriculum design, LTA strategies and relationship to external frameworks. Specifications have potential for marketing and publicity. They need to relate closely to student progress files/student transcripts. There are resource implications of maintaining and updating specifications.

3.8 Progress Files Key Features The concept of the Progress File emerged from the Dearing Report of the National Committee of Inquiry in Higher Education. The Report recommended that institutions develop a Progress File, consisting of two elements: • A transcript � recording student achievement • A means which students could monitor, build and reflect upon their personal

development The concept has since been developed in an initiative supported by CVCP, SCoP, CoSHEP and QAA, leading to an agreed policy statement in May 20006 on the two elements � Transcripts and Personal Development Planning. The key features of the policy are:

(a) Transcripts Institutions should provide transcripts for all HE provision for which credit is awarded and for all provision leading to a HE award. Each student should be provided with a transcript showing what they have studied and achieved • After they have completed the programme • After they have completed each stage of their programme • At an appropriate time when a student interrupts their programme • When such information is necessary in order to apply for a job. Individual institutions should determine the detailed content and format of their transcripts, but this should be based on a minimum information content used consistently across the UK.

6 CVCP Information for Members 1/00/80, 17 May 2000


(b) Personal Development Planning Personal Development Planning is �a structured and supported process undertaken by an individual to reflect upon their own learning, performance and/or achievement and to plan for their personal, educational and career development�. It is intended to help students; • Become more effective, independent and confident self-directed learners • Understand how they are learning and relate their learning to a wider context • Improve their general skills for study and career management • Articulate their personal goals and evaluate progress towards their

achievement • And encourage a positive attitude to learning throughout life. Individual institutions should develop their own policies on PDP within a national guidance framework. The different elements of the Progress File and the infrastructure that underpins it are shown schematically in the Figure below. FIGURE 11 THE CONCEPT OF THE HE PROGRESS FILE Source:

Implementation Timetable

Opportunities for development and practising PDP and for recording the results of learning

Processes for recording learning and achievement, and guidance systems to support PDP

Personal Development

Planning Activity/ process to

improve own learning

Personal Development Records

Information

owned by the student

Transcript

Information owned by the

institution

Creative use of information and PDP skills by students as they interact with the world outside HE


Further work is now being undertaken to develop a guidance and support framework to help HEIs develop their own policy and practice in line with the national policy. The policy statement indicates that institutions are encouraged to introduce a transcript with a content by 2001/2002 but the use of such a transcript would not be expected until 2002/2003. A longer lead in time, however, is stated for PDP, given that institutions are at different stages of development. A 5 year lead in time (2005/06 start date) is therefore proposed for PDP implementation across the whole system and for all HE awards.

4 Key Issues and Implications for HEIs The QAA Framework represents a significant shift to an outcomes based approach to quality and standards. This permeates the entire framework and requires HEIs to focus more sharply on how learning outcomes are articulated, related to national reference elements, and delivered through learning and assessment processes, resulting in final outcomes at the appropriate standard. Even for those Universities which start from a relatively strong position in understanding and applying outcome based approaches, there will need to be further and significant development to ensure that these approaches are firmly embedded and consistently applied across the University. Schools/ Faculties/ Departments will need to act quickly in reviewing and mapping the relevant Subject Benchmark Statements to their provision and be able to demonstrate, from the 2000/2001 planning cycle, how they will comply with the relevant statements(s). Programme Specifications will be significant documents in setting out intended outcomes and the associated teaching and learning methods and will require close attention by all course teams. Progress Files will ultimately be key in demonstrating achievement of outcomes. At institutional level, it will be necessary to review awards frameworks against the Qualifications Framework which may entail revisions to institutions� awards and level descriptors. Institutional quality management The Code of Practice will require HEIs to review practice � and potentially policies � in a wide range of areas of quality and standards management. HEIs will be expected to adhere to the Precepts within each Code section, and will be judged against these in Review; HEIs will be expected to describe and demonstrate how they meet Code Precepts, and to indicate the actions they intend to take, in relation to precepts which they do not currently meet. Whilst the accompanying notes of guidance are not meant to be prescriptive, there is still some (sector wide) nervousness that they will evolve into a checklist used by reviewers and require a defence of any different approaches than those suggested in the guidance. HEIs will need to review each section of the Code in some detail to develop a clear � and shared � view across the institution on how to implement the Precepts and the reasons for the chosen approaches.


Continuous Engagement There will be virtually continuous engagement between each HEI and QAA review processes. QAA describes the whole process of institutional review as a �continuous and dynamic engagement with the institution and its internal processes over the six years of the review cycle7 - much of the engagement through the sequence of Subject Reviews, but also through the �capstone� Institutional Review. Subject Reviews will no longer be focused on an event taking place within one week, but will take place over an extended period. For Institutional Review there will be a major event every 6 years, and the necessity to provide an account of action taken on any �essential� action points 12 months after the publication of the Report. There will now also be an Interim Appraisal every 3 years which could in itself become a significant event, if Subject Reviews in the last 3 years have disclosed any �pattern of difficulties�. The precise balance and interrelationship between Institutional Review and Subject Review is yet to be determined. However, it is likely that Institutional Review will assume increasing importance, as the number of Subject Review engagements between QAA and institutions decreases. There is clear evidence of a shift away from periodic large scale visits towards more frequent communication with HEIs, focused more heavily on institutional strategic management of quality, standards and the student experience. Review of collaborative provision may take place separately from the main Institutional Review, again on a 6 year cycle, if warranted by the scale of the HEI�s collaborative portfolio. Similarly, HEIs with a number of international partnerships might expect to be involved in an audit of one or more of those individual links, in most years, as part of the QAA programme of country-by-country audits. In addition, many parts of many HEIs will be subject to the review processes of accrediting bodies. This extent of external scrutiny raises significant issues for HEIs on how they deploy resources, the nature of its internal quality and standards arrangements, and the relationship between the internal and external regimes.

Intensity of scrutiny Much emphasis was given in the new Framework as originally published to variable intensity of scrutiny. The scope for flexibility, or variability, is now itself under scrutiny, as indicated above, to establish an acceptable basis on which �light touch� may be realised. Where the Agency has confidence in an institution�s ability to assure quality and standards, a lower intensity of scrutiny (a �lighter touch�) may be expected. A greater intensity may be necessary where there is no convincing evidence of �robust and effective systems�. The Institutional Profile maintained by QAA will provide a key reference point for decisions on intensity of scrutiny. The nature of an HEI�s profile may itself lead to a relatively high level of scrutiny. Links with professional bodies, arising from institutions� vocational and

7 Handbook for Academic Review (98)


professional orientation, also mean that there may be a wide range of external judgements in the public domain which will inform the institutional profile (for example, PSB reports will be considered at Subject Review). This too may have a cumulative effect on the intensity of scrutiny. HEIs with extensive collaborative provision will be subject to higher levels of scrutiny in comparison with Universities with less extensive partnership portfolios. Accountability, ownership and enhancement: HEIs quality and standards systems Three of the underpinning principles of QAA�s approach (see 3.1 above) concern accountability, ownership and enhancement. The system QAA has developed includes two defined levels of review and responsibility. At subject level there is, appropriately, strong emphasis on the role of the subject team and review by peers in the subject area. At institutional level, QAA is clear that the ultimate responsibility for the management of quality and standards rests with the institution as a whole. The two levels interact dynamically and, in review terms, lead to the Institutional Profile. HEIs will need to consider within the development of their internal systems the relationship between institutional frameworks, requirements and guidance and local practice; and how to promote local ownership and accountability and at the same time ensure that the institution can discharge its responsibilities both effectively and efficiently. Responses need to be measured so that the needs of all stakeholders are met and so that appropriate investments are made in the learning environment of students and in quality management and enhancement.


SECTION 3

THE EFQM

EXCELLENCE MODEL® AND THE QAA FRAMEWORK:

MAPPING SYNERGIES AND GAPS


1 Introduction

For the purposes of this project, work has focussed on a limited number of elements of the new QAA integrated framework. This section of the report presents the outcomes of mapping exercises undertaken looking at the Excellence Model and the QAA Code of Practice, Institutional Review (IR), Continuation Audit (CA), Collaborative Provision Review (CPR) and Subject Review (SR). Although the Code of Practice is essentially a referent for IR/ CA/ CPR/ SR it has been included in this project because of its value as a set of criteria, or reference points, against which judgements are made.

Some difficulties have been encountered arising from: • the lack of detail published to date by QAA about the future IR and CPR

processes • the re-opening of the issues of �lightness of touch� and �variability of scrutiny�. It is likely that this will lead to a reduction in the Subject Review requirements � but there may also be consequential implications for the Institutional level requirements. This Project has therefore focused on the processes as detailed publicly by QAA up to July 2001 and for the institutional level, has drawn to a greater extent than originally intended on the Continuation Audit methodology. It acknowledges the evolution of Institutional Review from the beginnings of the audit approach as applied to the pre-1992 Universities in Academic Audit, through Quality Audit as operated for all HEIs by HEQC, to the current programme of Continuation Audit. During this evolutionary process, there has been an increasing focus on the higher levels of institutions and on academic standards.

Consideration may be useful, at some future date, of the value of extending this initial mapping exercise to incorporate the other main elements of the QAA Framework � viz Subject Benchmarking, the Qualification Frameworks, Programme Specifications and Progress Files. The matching exercise undertaken has taken into account that the Excellence Model and the QAA Framework have fundamentally different starting points. The use of the Excellence Model is internally driven, holistic, and has the potential to influence behaviour and improvements right across an organisation. The QAA Framework involves a number of judgmental elements, is mandatory and is externally driven. The QAA Framework is also only partial, in that it relates mainly to standards and the student experience. It has to be seen as �nesting� within a number of quality requirements and pressures upon HE such as those imposed by HEFCE (mission statement, strategic plan, financial audit, the Research Assessment Exercise), professional and statutory bodies, TTA, OFSTED, the NHS etc. The QAA has taken on board some of the Dearing recommendations,

SECTION 3 THE EFQM EXCELLENCE MODEL® AND THE QAA FRAMEWORK: MAPPING SYNERGIES AND GAPS


but others are addressed by other means within a wider HE quality framework which is somewhat fragmented. This particular project has therefore had a very narrow focus and there may be merit in a wider project which maps the totality of external requirements upon HEIs, against the Excellence Model.

2 The Code of Practice and the Excellence Model The Code of Practice on the Assurance of Quality and Standards in Higher Education is a key reference element in the QAA Framework. The Code is intended to provide a comprehensive codification of �system wide expectations� on matters relating to the management of academic quality and standards in HE. The intention is that the Code is an authoritative reference point: • for institutions - providing a systematic basis on which institutional practice

can be actively reviewed and developed • for academic reviewers � the basis on which institutional practice on quality

and standards management is assessed. The precepts within each section of the Code are intended to identify �those key matters which the QAA expects an institution to be able to demonstrate it is addressing effectively through its own quality assurance mechanisms�.8 The guidance notes within each Section are intended to offer a framework which institutions �may wish to use, elaborate and adapt� 9 in accordance with their own needs. The constituent elements of the Code have been produced as separate sections over the last 18 months or so. Although the method of developing the sections has been variable, QAA�s aim has been to seek advice from knowledgeable practitioners to inform their development. Draft sections of the Code are circulated to institutions to give them an opportunity to see, and if they wish, to comment upon the draft before a final version is considered by the QAA Board and then issued to institutions.

2.1 Mapping Exercise This exercise has been undertaken in two main stages: • an overall assessment of the relative purposes of the QAA Code of Practice

and the EFQM Excellence Model® � as stated by QAA and EFQM respectively

• a detailed consideration of each published Code section, mapping each precept against the nine criteria in the EFQM Excellence Model®.

Consideration has also been given, briefly, to process and outcomes.

2.2 Overall Purposes The purpose of the QAA Code of Practice is �to provide an authoritative reference point for institutions as they consciously, actively and systematically

8 Code Foreword 9 Code Foreword


assure the academic standards of their programme awards and qualifications. QAA will �consider the extent to which individual institutions are meeting the expectations of the precepts in the available sections of the Code of Practice�. The EFQM Excellence Model is described by EFQM® as :

�a practical tool to help organisations establish an appropriate management system by measuring where they are on the path to Excellence, helping them to understand the gaps, and then stimulating solutions�.

There are clearly some similarities in terms of overall purpose. Both the Code and Excellence Model are provided as �external references� against which organisations can review their practice, and through which performance improvements can be identified. In this sense, both can be regarded as �improvement tools�. However, there are a number of key differences in approach which are summarised in Table 11 below. These largely relate to the specificity and prescription of the QAA Code of Practice in comparison with the EFQM Excellence Model, and to the fact that the Code is one element of the broader QAA framework � whereas the Excellence Model is intended to be a holistic framework. Table 11 Variations of Approach

EFQM Excellence Model self assessment

QAA Code of Practice

A 'model' which is intended to be applied to any type of organisation and is mainly internally driven

A framework which applies solely to Higher Education and is externally imposed

A holistic approach which encompasses all aspects of an organisation in an integrated way and is complete in itself.

An approach concerned with academic quality and standards management, addressing areas of practice individually, which is part of a wider framework.

A tool which enables systematic self assessment against broadly defined criteria (Enablers and Results)

A tool which enables systematic self assessment against standardised expectations (Precepts)

Broad criteria supported by sub-criteria accompanied by indications of 'areas to be addressed'

Precepts state explicitly expected outcomes in terms of institutional practice

Subscription to the model is voluntary and its application owned by the organisation

Adherence to the model is, effectively, obligatory and scope for variation is limited in practice

No external verification of application of the model unless organisation elects to submit for the European Quality Awards (unless internal peer review process is adopted)

External verification of adherence to the Precepts is obligatory, via Academic Review

Benchmarking promoted as a concept/approach, but no specific benchmarks established

Benchmarks of good practice are set within each Code section , influenced by advice from 'knowledgeable practitioners' across the sector


2.3 Process: Institutional approach to the Code of Practice There is no fixed model of how institutions might approach the Code, in terms of using it as a tool to review practice and identify areas for improvement/action. It is likely that approaches will vary and will be influenced by a range of issues including institutional organisation and culture and approach to quality and standards. Different sections of the code may elicit different responses in terms of their scope, complexity, and how far they relate to current practice within the institution. Thus it is more difficult to specify timescales and resources related to this part of the QAA Framework than other more prescribed process elements of the integrated QAA system (See Process/Specification Matrix Table 12).


Table 12 Process / Specification Matrix � Code of Practice Technical / Process specification EFQM Excellence Model

self assessment QAA Code of Practice

Timescale to prepare for assessment

From 1 hour to 6 months Encompasses self-assessment and assessment through Academic

Review, Timescale and method very dependent upon institutional

approaches to review of practice against Code and issues specific to

Code sections. Timescale for assessment activity to be undertaken

From 3 hours to 3 months See above

Flexibility of assessment method

Very flexible - from simple questionnaire to award submission document

See above

Number of people typically involved

Varies, minimum usually 5, can be everyone

See above

Reliance on evidence Reliant on evidence but also to some extent perception

based

Effective internal and QAA assessment of adherence to the Code would have to be evidence

based

Fits at every level of the organisation

Fits at personal, team, department and whole

organisation

Fits at institutional and department/ school level

Assessment against self prepared document

Assess against self completed questionnaire,

workbook or award submission document

Internal assessments would be based on a range of inputs. QAA

judgements are made on the basis of all the evidence available to the

Review Team � from self evaluation document, staff/student meetings and baseroom evidence material.

However, a typical institutional approach might be to assign responsibility for overall oversight of the Code to a senior University officer with quality assurance responsibilities; to have mechanisms by which responsibility for analysing institutional practice in relation to individual Code Sections is assigned to the relevant specialist staff; to submit analyses and plans for action and monitoring to relevant University committees concerned with quality assurance and enhancement. For many institutions, one of the biggest challenges will be to raise widespread awareness of the Code and to ensure that any implications for day to day practice in Schools/Departments are widely understood, acted upon


and monitored in ways which will be helpful and promote improvements and enhancements. Institutions will need to be able to demonstrate adherence to the Code and to explain their processes for ongoing monitoring and review of adherence.

3 Mapping the Precepts Against the EFQM Excellence Model Criteria This part of the exercise encompassed those sections of the Code which had been finalised and published at that time: Table 13 Currently Published Concepts

Section Title No of Precepts 1 Postgraduate Research Programmes 25* 2 Collaborative Provision 38 3 Students with Disabilities 24 4 External Examining 16 5 Academic Appeals and Student Complaints on

Academic Matters 14

6 Assessment of Students 19* 7 Programme Approval, Monitoring and Review 9 8 Career Education, Information and Guidance 14

* Includes sub-divisions of precepts Each precept (159 in total) was reviewed against the EFQM nine criteria and an assessment made about the most appropriate �match� or �matches�. This process was inevitably subjective in approach, dependent upon an interpretation of both the precept and the EFQM criteria as �top level� statements (the analysis was not taken to the level of Code guidance and EFQM sub-criteria). A number of iterations were undertaken before there was broad satisfaction with the outcome. At a superficial or mechanistic level it is possible to match the vast majority of the Precepts solely against the EFMQ criterion concerned with Processes. This is perhaps not surprising given that the QAA Code is a Code of Practice, and therefore is concerned with the systems, procedures, and arrangements through with institutions discharge their responsibilities for the assurance of academic quality and standards. So, for example, the following Precepts all embody processes: Table 14 Precept example 1 Academic Appeals� Precept 3 The procedures should be ratified by the

governing body � Programme approval�.

Precept 1 Institutions should ensure that their responsibilities� are discharged effectively through their procedures

Assessment of Students

Precept 7 Institutions should publish, and implement consistently, clear criteria for the marking ��

However, in many cases, the QAA precepts are �richer� in meaning than simple statements about process. For example, they might also make explicit reference


to institutional strategy or policy, or it is clearly implied that the process will be designed and carried out within an established policy framework. For example, the following precept where assessment procedures would need to be discharged within an overarching University policy with respect to academic standards. Table 15 Precept example 2 Assessment of Students

Precept 1 As bodies responsible for academic standards, institutions should have effective procedures for designing, approving, supervising and reviewing the assessment strategies for programmes and award.

Similarly, some precepts concerned with Processes will include a strong dimension about People - specifying roles and responsibilities, staff information and support needs; or Partners - again specifying roles and responsibilities. For example: Table 16 Precept example 3 Academic Appeals �

Precept 5 Information on complaints and appeals should be published, accurate��and issued to students and staff.

Collaborative Provision

Precept 30 The respective obligations and responsibilities of the Partner Organisation and the Awarding Institution, and the external examiners should be appropriately and clearly communicated by the Awarding institution

Thus it was not considered appropriate, for the most part, to match the QAA Precept against only one of the EFQM criteria and a high proportion match against 2 or 3 of the nine criteria. There were also some issues about the treatment of students within the QAA Code in relation to the EFQM Model�s approach to �customers�. There is � appropriately - a student focused approach within the QAA Code of Practice which manifests itself most strongly in those precepts concerned with appeals and complaints and the special needs of students with disabilities � with a strong emphasis on effective communications and fairness of procedures. It was sometimes difficult to find the appropriate match within the EFQM model � these precepts not being concerned necessarily with �customer results� but how staff and students interact within the learning environment. This may highlight some problems about simply defining students as �customers� within the EFQM model, in other words as recipients of a service. The relationship between institutions, their staff, and students is more complex � for example they might be alternatively described as partners or stakeholders. In the absence of a �customer enabler� criterion, however, relevant precepts were matched with the EFQM �customer results� criterion


The results of the mapping are shown in the measurement matrix shown in Table 17. This table shows the distribution of matches between Precepts in the Code to date, mapped against the EFQM Model criteria. The distribution has been calculated within each code section on the following basis: The total number of matches has been calculated � this includes �multiple matches� so where a precept has been mapped against three EFQM criteria, this is defined as 3 matches. The number of matches against each EFQM sub criterion has been calculated as a percentage of the total matches for the code section and the results shown in broad bandings . Key for table 17 (on next page) Key 1 - 20% 21 - 40% 41 - 60% 61 - 80% 81 - 100%

Mappin

g th

e Q

AA

Fra

mew

ork

and

65

C

onso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

EF

QM

Exc

elle

nce

Model

Cri

teri

on u

sed a

s basi

s fo

r se

lf-ass

ess

ment act

ivity

Codes

of

Pra

ctic

e

Criterion 1: Leadership How leaders develop and facilitate the achievement of the mission and vision,

develop values required for long term success and implement these via appropriate actions

and behaviours, and are personally involved in ithtthiti't Criterion 2: Policy and Strategy

How the organisation implements its mission and vision via a clear stakeholder focused

strategy, supported by relevant policies, plans, objectives, targets and processes

Criterion 3: People How the organisation manages, develops and releases the knowledge and full potential of its

people at an individual, team-based and organisation -wide level, and plans these activities in order to support its policy and

ttdthfftitifit Criterion 4: Partnerships and Resources How the organisation plans and manages its

external partnerships and internal resources in order to support its policy and strategy and the

effective operation of its processes.

Criterion 5: Processes How the organisation designs, manages and improves its processes in order to support its

policy and strategy and fully satisfy, and generate increasing value for, its customers

and other stakeholders

Criterion 6: Customer Results What the organisation is achieving in relation

to its external customers.

Criterion 7: People Results What the organisation is achieving in relation

to its people.

Criterion 8: Society Results What the organisation is achieving in relation to local, national and international society as

appropriate.

Criterion 9: Key Performance Results What the organisation is achieving in relation

to its planned performance.

Sect

ion 1

P

ost

gra

duate

Rese

arc

h

Pro

gra

mm

es

Sect

ion 2

C

olla

bora

tive

Pro

visi

on

Sect

ion 3

S

tudents

with

Dis

abili

ties

Sect

ion 4

E

xtern

al E

xam

inin

g

Sect

ion 5

A

cadem

ic a

ppeals

and s

tudent c

om

pla

ints

on

aca

dem

ic m

atte

rs

Sect

ion 6

A

ssess

ment of

students

Sect

ion 7

P

rogra

mm

e a

ppro

val,

Monito

ring a

nd r

evi

ew

Sect

ion 8

C

are

er

educa

tion, In

form

atio

n a

nd g

uid

ance

Ta

ble

17

: E

FQ

M E

xcel

len

ce M

od

el a

nd

th

e Q

AA

Co

de

of

Pra

ctic

e -

Mea

sure

men

t M

atr

ix


Thus, for Section 1 of the Code the highest proportion of matches is against Processes (within the band 41 � 60%) and the remainder of matches are distributed between Policy and Strategy, People, Partnerships and Resources, Customer Results, People Results and Society Results (all within the lowest bands of 1-20%)

4 Taking the results for each EFQM criterion in turn:

4.1 Enablers • Leadership There is no explicit reference within the QAA Code to Leadership issues. The single match in relation to the section on Collaborative Provision is in relation to the responsibilities of the Awarding institution for academic standards (precept 1) and is subsidiary to the match with Policy and Strategy. • Policy and Strategy There is limited explicit reference to policy and strategy but some degree of implicit reference in most sections, where processes are procedures are set within the context of relevant policy frameworks (see earlier examples).

• People All sections of the Code include reference to institutions� staff, their roles and responsibilities, their support and development needs. The strongest element is within the section on Programme Approval, Monitoring and Review with its emphases on ensuring that those involved in these activities are well guided and supported and are able to exercise independent judgements. • Partnerships and Resources Partnerships feature in 6 out of the 8 code sections. Not surprisingly, (provided it is accepted that External Examiners are key partners who support institutions� policy, strategy and the effective operation of processes) the strongest �match� occurs in the section of the code on External Examining. There is also � inevitably - significant coverage within the Collaborative Provision section, although the major substance of the section is concerned with institutional Processes. The section on Assessment of students includes a reference to professional or statutory bodies (Precept 15) which are considered to be partners in this context. The relevant match in Careers Education, Information and Guidance is in relation to collaboration between employers and career guidance providers. • Processes Whilst the Code of Practice may be more complex than a set of statements about required processes, the predominant mapping of precepts is inevitably against the processes criterion. As noted above this is not surprising given the nature and purpose of the Code.


4.2 Results There was limited scope for mapping against the Results criteria as the Code of Practice rarely includes explicit reference to outcomes. However, as described earlier, there has been mapping against the Customer Results criterion where there is explicit reference to meeting student needs. Where there is reference to academic standards of awards, (eg Collaborative, Precept 24) this has been mapped as a �Society Result� in accordance with the view that the assurance of academic standards is a matter of public accountability. Similarly, references to careers/employment (eg Career, Education, information and guidance; Precepts 4,5) have been deemed as relevant to society results. Where there is explicit reference to monitoring of processes and analysis of performance data in relation to intentions/policy/strategy this has been mapped against Key Performance Results, although these instances are rare.

5 Outcomes EFQM Excellence Model® self assessment involves a highly focussed approach with a defined outcome, that is, an improvement action plan for the whole or part of an organisation. The usual experience is that this process will set targets for improvements, which are measurable.

Outcomes from working with the QAA Code may in some areas be similarly clear and measurable but in others, they may be far more diffuse. The Code is wide-ranging and affects many aspects of practice, particularly those relating to students. The identification of clear outcomes from working towards Code compliance and benefiting from its good practice guidance is critically contingent upon an institution�s approach to reviewing, implementing and monitoring practice against the Code (see Table 17 above). The range of outcomes will obviously be dependent upon how institutions perceive their practice against each section of the Code and what action they decide is necessary or desirable in consequence. The amount of action may well vary from Section to Section � as indeed it might arising from an EFQM self-assessment. The ease with which improvement actions arising from self-assessment against the Code may be tracked by institutions, will be dependent upon the progress monitoring arrangements which they put in place. If this is approached systematically, however, it may offer a very effective tool to support self-improvement.


Table 18: Outcomes Matrix � QAA Code of Practice Outcomes required


QAA Code of Practice

Reports Optional - can be a short management report, or a full assessment document

QAA methodology does not require institutions to provide reports specifically

about the Code of Practice. However, it is assumed that Academic Review Reports

(subject & institutional) will include reference to adherence to the Code, and that

institutional review and monitoring of practice against codes is likely to lead to internal

reports. Code adherence will also be covered in SEDs.

Action plans Optional, but desirable. The outcomes of a self-assessment activity usually lead to actions which are implemented through an action plan

Again, the QAA methodology does not necessarily result in action plans, but

institutional responses to QAA review and internal review may well lead to action plans

for improvement measures


Strong - assessment activity itself evidences learning. Action plan tracks progress and learning.

Institutional consideration of practice against the QAA Code would presumably need to be

evidenced based and good practice in ongoing monitoring and review should

demonstrate evidence of learning Overall improvement activity

Often demonstrated over 12 months or more, but can show immediate improvements. Activities range across wide range of management and people issues.

See action plans above � assumed that these would lead to overall improvement of

activity


Desirable to be fully integrated into the business planning cycle - allowing the self-assessment process to be preparation for an evidence based business plan. Integrates well with other management tools.

Whilst the Code deals with individual areas of practice in separate sections, there would be inevitably be �cross cutting� and broader

implications arising from identifying and actioning improvement measures in relation to each section. Institutions would need to

be able to make the links and ensure appropriate integration between sections and

with other aspects of planning and management.


Evidence to show marked improvement in customer experience amongst organisations who are using the Model

As much of the Code is concerned with ensuring effective processes, clearer

articulation of roles and responsibilities and good information to students and partners, it

is assumed that improvements should accrue from adherence to the Code.


Evidence to show marked improvement in staff experience amongst organisations who are using the Model

The focus on clarity of roles and responsibilities and ensuring appropriate

staff development and training should be of benefit to staff. The indirect benefits of

demonstrating adherence to the Code will be significant: good Subject and Institutional

Review outcomes will be of value. Improvement in balanced set of performance results

Usually leads to an understanding of the need for, and the development of, a balanced set of performance measures (customer, staff, society and financial)

The Code is a potentially valuable tool to assist institutions to improve performance

results in selected areas of practice.


6 Conclusions There are clearly some synergies between the EFQM Excellence Model® and the QAA Code of Practice but also evidence of different approaches or concepts. Both are intended to provide tools to support effective management, monitoring and ultimately quality improvement or enhancement. The QAA Code is predominantly concerned with the management of academic quality and standards as discharged through institutional processes. It is concerned with only certain aspects of institutional management and is one element of an integrated QAA framework which seeks to be effective via the interaction of the different elements of the framework. The Code does not achieve (nor does it seek to achieve) a holistic or complete coverage of all aspects of organisational effectiveness or �excellence� as defined in the EFQM Excellence Model®. Hence there is only scope for partial mapping against the EFQM Excellence Model® criteria. The Code reflects the range of relationships between the various stakeholders within Higher Education � staff, students, external examiners, professional bodies, employers. There are some inherent challenges in mapping these various complex relationships using the EFMQ Excellence Model® terminology of people, customer, partner; this may be a terminological issue only or may be a more fundamental issue concerning the applicability of the current model to Higher Education. Certainly, there would seem to be a need to consider in more depth the role of students as participants within processes, rather than recipients of services.


SECTION 4

INSTITUTIONAL

REVIEW, CONTINUATION AUDIT AND THE EXCELLENCE

MODEL.


1 Introduction

The QAA approach to Institutional Review is still evolving and the Institutional Review programme will not commence until 2002. Institutional Review has developed from QAA�s Continuation Audit which, in turn, superseded the previous HEQC Academic Quality Audit methodology and the even earlier Academic Audit process. Audit or Review of Collaborative Provision may be undertaken either as part of the main institutional level exercise or separately, depending on the scale of an institution�s collaborative portfolio. Details of the approach to the Review of Collaborative Provision will not be published by QAA before Summer 2001 and separate Reviews of Collaborative Provision will not be undertaken before 2003. For the purposes of this project, attention has therefore been paid to the developing focus of Review at institutional level and Review of Collaborative Provision is considered alongside IR.

2 Continuation Audit: The Context The QAA�s programme of Continuation Audit will operate from 1999 until the end of 2001. The approach used is detailed in �Notes for the Guidance of Audit Teams�10. It is expected that the approach to IR will be very significantly informed by the Continuation Audit methodology. The Notes state that: �The joint effect of academic quality audit and teaching quality assessment in the UK to date has been to raise awareness of the importance of the proper management of academic programmes and awards, and also to establish a consensus of good academic quality assurance policies and procedures across the whole of the higher education sector. This consensus is encapsulated in the former HEQC�s �Guidelines on Quality Assurance 1996�: �What is now required is for institutions to provide continuing reassurance about the rigour of these policies and procedures in practice, and about the validity and reliability of the evidence used by institutions individually for assuring the quality of their educational provision and the standard of awards.� Underlying the Continuation Audit method were three assumptions which will continue to apply, when Institutional Review is introduced: ��responsibility for the quality of educational provision and its assurance can only effectively lie where the powers exist to control or change practices� �the standard of an award, and the means by which it is assured, is ultimately a corporate responsibility of the awarding body involved�

10 Notes for the Guidance of Audit Teams, QAAHE (August 1999)

SECTION 4 INSTITUTIONAL REVIEW, CONTINUATION AUDIT AND THE EXCELLENCE

MODEL.


�all higher education institutions should by now have in place policies and procedures for assuring the standard of their awards and the quality of their educational provision.�11 The language used is different but the fundamental concept is very similar to that of the EFQM Excellence Model®, viz, that organisations must have policies and strategies which they implement systematically through a procedural framework to ensure the proper management of their functions.

3 The Excellence Model and Institutional Review: Comparison of Purpose and Concepts. Table 19 below illustrates a number of key similarities and differences between the primary purposes and concepts of self-assessment using the EFQM Excellence Model and Institutional Review. Table 19 Key similarities and differences of purpose and concepts of approaches: Excellence Model and Institutional Review


QAA Institutional Review

To review and assess all aspects of an organisation, measuring its ability to achieve and sustain outstanding results for all

stakeholders

Addresses the ultimate responsibility of the institution in managing the academic standards of its awards and the

quality of its programmes

Focus on robustness of evidence and demonstration of continual improvement over 3 or more years

Looks at robustness and security of systems supporting institutions awarding functions

Based on 8 fundamental concepts, 9 criteria and 32 sub-criteria which have been developed and tested by world class

organisations and used widely across Europe

Concerned with : - procedures for approval monitoring and review of academic programmes; - procedures for

acting on findings of External Examiners, Subject Reviews and other external scrutinies; - overall

management of assessment processes; - overall management of any credit systems; - management of

collaborative arrangements with other institutions.

Concerned with looking at 'where we are now' across a wide range of balanced issues looking at the overall management of the organisation as a whole, taking into account differences at

lower levels

This 'capstone' review is concerned with the effectiveness of the 'senior layer' of management in

relation to its overall responsibility for quality and standards, and includes focus on communications and

on improvement and development generally.

Assesses and reviews activities using RADAR matrix - providing a common and consistent scoring matrix

A judgement is made by the Review Team about the degree of confidence that the QAA considers may

reasonably be placed in the institution's quality assurance arrangements. However, in order to promote

consistency, a template with report headings is in development, for use by all Institutional Reviewers.

Reviews internal and external relationships and inter-relationships

Addresses the management of collaborative links, and communication with staff and students (prospective,

current and past) and with other external bodies. Provides a common language and framework for comparison within and outside the sector on a range of management and

operational issues

Provides a standard framework and approach for published reports on the effectiveness of an institution's discharge of its awarding body responsibilities, including in particular its responsibilities relating to standards and

quality.

11 Notes for the Guidance of Audit Teams, QAAHE (August 1999)


There are a number of similarities in terms of purpose and concepts. Both models assume that organisations must have policies and strategies, implemented through a procedural framework, to ensure effective management of functions and delivery of services. Both are concerned with the achievement of good performance in the interests of stakeholders, and with the management of effective relationships with partners and collaborators. Both offer the opportunity to demonstrate that they are managing effectively � in the case of CA/ IR, that they are managing their academic standards effectively.

The most immediately obvious differences are: • The EFQM Excellence Model® is about organisational excellence; there is

no equivalent institutional comparative term, in the QAA IR Model. • The EFQM Excellence Model® is for most organisations, a self-assessment

tool intended to encourage and promote good management practice. Any type of organisation may choose to use the model. The QAA IR mechanism, however, is mandatory: applies only to HEIs: focuses mainly on the management of academic standards and the quality of educational provision: and involves an element of self-evaluation but also one of external judgements by academic reviewers.

• The EFQM Excellence Model® involves a systematic review of how

comprehensively approaches have been deployed across the organisation, trends over three or more years, and scoring using the RADAR scoring matrix. Whilst HEIs may choose to use a systematic �scoring� mechanism of this type during their self-evaluation, QAA IR judgements are not made against a scoring framework. The QAA Model deliberately shies away from scoring to avoid fuelling the '�league tables� culture: QAA operates a pragmatic model in a political climate and there would be enormous antagonism to such an approach. However, IR will be concerned with how HEIs are adhering to, or how their practice is equivalent with, the QAA Code and whilst there will always be a subjective element, judgements will be made against the Institutional Profile and the evidence reviewed by the Review Team.

• The EFQM Excellence Model® is concerned with leadership and with the

ways in which policy, strategy and processes are deployed at all levels within the organisation. QAA IR is less explicitly concerned with �leadership� but is described as a �capstone� review and tends to focus on the effectiveness of the �senior layer� which may include both individuals and committees or groups � in discharging the institution�s overall responsibility for quality and standards management. IR therefore focuses on strategic level management of quality and standards, whereas Subject Review is concerned more with these matters at the operational level.

• The EFQM Excellence Model, in focussing on management practice in

organisations, facilitates benchmarking between a wide range of organisations, not necessarily in the same sectors. QAA IR, on the other


hand, enables an element of benchmarking between institutions, if they choose to scrutinise and learn from the published reports about other institutions, or use advisers to help them prepare for external scrutiny. The body of knowledge built up by QAA via audit, or via IR in future, informs development of the QAA Framework generally, including the Code Sections and their good practice guidance.

• The EFQM Excellence Model® is concerned with an all-embracing, holistic

approach to the management of an organisation�s business including all aspects of resource management. The QAA model, including IR, does not focus enquiries on academic governance and management, which it is considered would be too sensitive, nor does it cover resource allocation and management. This may or may not be appropriate: it could be argued that an HEI�s ability to deliver a quality student experience, or maintain standards, is more crucially dependent on the resources available to it and the institution�s ability to direct those resources to ensure delivery of quality and maintenance of standards, than on quality and standards assurance procedures. These matters are covered by enquiries via other routes, particularly by the Funding Councils. However, if the HE sector continues to expand without a commensurate increase in its funding, it might be argued that there needs to be an ever greater emphasis on quality and standards assurance, in order to generate evidence, for public accountability purposes, that standards are being maintained (despite erosion of resources). It might also be suggested that the QAA framework should have a greater interest in resource management and its interrelationship with delivery of quality and standards � but one might speculate that this would be likely to be even less popular with HEIs than the current regime and that it raises issues about the relationship between QAA and the Funding Councils.

• The QAA IR method includes an element of threat or sanction and there are

significant risks associated with adverse outcomes. An adverse report or judgement can damage an institution�s reputation, particularly if it is publicised and discussed by the media. Externally published reports must therefore be highly sensitive to the political context and to the potential damage they may cause. An adverse report may also lead to heavier scrutiny in future reviews, the requirement to rectify issues identified, and in the worst case, to action by the Funding Council. No such threats are attached to self-assessment using the EFQM Excellence Model®: if an organisation achieves a low score when it undertakes a self-assessment it is up to the organisation itself to decide what action to take and over what time scale. The Excellence Model is also crucially dependent upon self-assessments being honest, or else their value would be diminished.


4 Criteria for measurement: HEQC/ QAA and EFQM

As part of this project, consideration has been given to the synergy between the QAA Audit/ Institutional Review and EFQM Excellence Model® criteria. A �Measurement Matrix� has been used, and four sets of principles or frameworks have been mapped against this. Firstly the HEQC Guidelines on Quality Assurance 199412 were mapped against the EFQM Model�s criteria. These Guidelines have been considered as part of this project, in view of their significance for Continuation Audit and to illustrate the evolution of the Code of Practice and CA and IR in particular. The primary purpose of the guidelines was: �to assist institutions in their quest to maintain and enhance the quality of educational provision for students. The guidelines provide a framework for quality assurance and control which institutions are able to use, elaborate and adopt according to their own traditions, cultures and decision-making processes.� The results of the mapping exercise are given in Appendix F. There is a very strong synergy in terms of policy, strategy and processes � EFQM criteria 2 and 5. The very strong student focus of this first version of the Guidelines is evidenced by the reasonably strong synergy with criterion 6, relating to external customers. There is, however, little evidence of a match between the Guidelines and the concerns of EFQM criteria 1, 7 and 8. There is some overlap in relation to criterion 9, Key Performance Results, but generally the Guidelines are focused more closely on �enablers� rather than on results. One interesting feature to note, however, is the extent to which the Guidelines refer to communication much more extensively than the EFQM criteria. In 1996, HEQC updated, revised and extended the Guidelines in the light of developments, feedback and consultations with institutions. The resultant �Guidelines on Quality Assurance 1996�13 had a much stronger focus on academic standards generally and included much more detail on teaching and learning, student communication and representation, collaborative provision and external examining. The document continued to be endorsed by QAA as a reference point for both institutions and auditors, in the �QAA Notes for the Guidance of Audit Teams 1999�. However its currency and significance is diminishing, with the progressive publication of the Code sections. This second version of the Guidelines has also been mapped against the EFQM criteria. Unsurprisingly a very similar pattern emerges (see Appendix G). Once again there is no strong evidence of a match between the guidelines and EFQM criteria 1, 7 and 8 � concerned with Leadership, People Results and Society Results, respectively. The Guidelines do not make much explicit reference to the benefits society at large may expect to derive from successful performance by HEIs � one rather assumes that they are significant but they are implicit rather than explicitly expressed. They are most explicitly referred to, for example, in:

12 HEQC: Guidelines on Quality Assurance 1994 13 HEQC: Guidelines on Quality Assurance 1996


Section 2 where references are made to: • �changing patterns in the applicant market�, �accurate promotional material �

and the needs and backgrounds of prospective students� • �admissions procedures and criteria � should have concern for students�

abilities and potential to gain from higher education and provide for equality of opportunity for all applicants.�

Section 6 which refers to the: • �Need to have in place an effective response to the expectations detailed in

the various national student charters.� Section 8, which states that: • �institutions should determine and make explicit the precise nature of the

external examiner�s role and the duties the examiner will perform in order to assist the institution in calibrating academic standards�.�

The most obvious synergy, as with the 1994 Guidelines, is around policy, strategy and processes and on the student (customer). However there does appear to be more emphasis on matters relating to EFQM criteria 3, 4, 6 and 9 suggesting a more strategic and integrated approach to delivery of a quality educational experience, and more attention to results to inform evaluation of institutional effectiveness. Communication once again is emphasised strongly, in contrast to the relative lack of emphasis on this in the EFQM criteria. Two further mapping exercises have also been undertaken. QAA�s Continuation Audit, operative from 1999 � 2001, focuses on a range of matters organised under four main headings. These four headings and their sub-headings have been mapped against the 9 EFQM criteria and their sub-criteria (see Table 20 overleaf). From this mapping exercise it would appear that although the process focus continues to predominate, a more even pattern of synergy begins to emerge.

Mappin

g th

e Q

AA

Fra

mew

ork

and

77

C

onso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Tab

le 2

0 Q

AA

Co

nti

nu

atio

n A

ud

it m

app

ed a

gai

nst

th

e E

FQ

M E

xcel

len

ce M

od

el®

EF

QM

Exc

elle

nce

M

odel®

Crite

rion

use

d a

s basi

s fo

r se

lf-ass

ess

ment

act

ivity

Q

AA

Note

s fo

r th

e

Guid

ance

of

Audit

Team

s (1

999)

use

d

as

a b

asi

s fo

r C

ontin

uatio

n A

udit

C

rite

rion

1:

Leaders

hip

C

rite

rion

2:

Polic

y and

Str

ate

gy

C

rite

rion

3:

P

eople

C

rite

rion

4:

P

art

ners

hip

s

C

rite

rion

5:

P

roce

sses

C

rite

rion

6:

C

ust

om

er

Resu

lts

C

rite

rion

7:

P

eople

Resu

lts

C

rite

rion

8:

Soci

ety

R

esu

lts

C

rite

rion

9:

K

ey

Perf

orm

ance

R

esu

lts

The in

stitu

tion's

st

rate

gic

appro

ach

to

qualit

y m

anag

em

ent

✔

?

✔

✔

? (

4a)

✔

✔

✔

The a

cadem

ic

standard

s of

aw

ard

s

✔

✔

The L

earn

ing

in

frast

ruct

ure

✔

(1

d)

✔

(3

a,

b,

e)

✔

(4c,

4d,

4e)

✔

✔

✔

?

✔

Inte

rnal a

nd

Ext

ern

al

Com

munic

atio

n

✔

✔

(3

c, d

)

✔

✔

✔

?

✔


Continuation Audit is concerned with the institution�s strategic approach to quality management in a way which matches reasonably well with 6 of the 9 EFQM criteria. However the focus on the academic standard of awards does not match well, other than in relation to EFQM�s concern with Processes and Performance Results. The Learning Infrastructure section matches well, however, in its concerns for the broad range of resources and facilities deployed to support learners and learning. Finally, there is also reasonably good synergy between the EFQM Excellence Model® and the Audit�s �Internal and External Communication� quartile. In terms of gaps, however, the areas in which the EFQM Excellence Model® is more explicitly demanding than the Continuation Audit foci, include: • leadership • the management of relationships with and services for external partners • People Results (both perception measures and performance indicators) • Society Results (though again this may be assumed to be implicit in the QAA

model). Finally, the emergent QAA Institutional Review model has been mapped against the EFQM Excellence Model® criteria. At the time of writing this paper, debate has recently re-opened about the �burden of accountability� and the ways in which the burden upon institutions may be lightened. It is therefore likely that there will be some reduction - or dilution � of the Academic Review methodology originally published by the QAA. However, current indications suggest that it is the Subject Review Method which will be subject to the most significant revisions. One may speculate, therefore, that the Institutional Review method is highly unlikely to be similarly reduced and that as a consequence of reductions elsewhere in what QAA intended to be a fully integrated method, Institutional Review and even Interim Appraisal may eventually assume more significance than originally intended. Institutional Review (IR) has not yet been fully detailed, in terms of either focus or process. Table U overleaf below shows the outcome of the mapping exercise between the EFQM Excellence Model® criteria and the QAA Self-evaluation Document Guidance for Institutional Review. As with the mapping exercises discussed above, both models show a clear and strong emphasis on policy, strategy and process. IR does not concern itself explicitly with the matters referred to in EFQM Criterion 1, Leadership. IR is concerned with some aspects of Criteria 3, namely those relating to the planning, management, improvement of people resources (3a) and the identification, development and sustenance of people�s knowledge and competencies (3b). So far, IR descriptions are silent on 3c-3e: people involvement and empowerment, people/ organisation dialogue and that people are rewarded, recognised and cared for. Nor is it easy to see how these matters will necessarily be picked up in the QAA Code sections currently published or planned. In a similar way, IR appears to be likely to focus on only one aspect of EFQM Criterion 4�s concerns � namely 4a, management of external partnerships. The

Mappin

g th

e Q

AA

Fra

mew

ork

and

79

C

onso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Tab

le 2

1 Q

AA

In

stit

uti

on

al R

evie

w m

app

ed a

gai

nst

EF

QM

Exc

elle

nce

Mo

del

®

EF

QM

Exc

elle

nce

Mo

de

l Cri

teri

on

use

d a

s b

asi

s fo

r se

lf-a

sse

ssm

en

t a

ctiv

ity

S

elf

Eva

lua

tion

Do

cum

en

t G

uid

an

ce u

sed

as

a b

asi

s fo

r In

stitu

tion

al R

evi

ew

(in

clu

de

s C

olla

bo

rativ

e

Pro

visi

on

Re

vie

w)

Cri

teri

on

1:

L

ea

de

rsh

ip

Cri

teri

on

2:

P

oli

cy

an

d

Str

ate

gy

Cri

teri

on

3:

P

eo

ple

Cri

teri

on

4:

P

art

ne

rsh

ips

a

nd

Re

so

urc

es

Cri

teri

on

5:

P

roc

es

se

s

Cri

teri

on

6:

C

us

tom

er

Re

su

lts

Cri

teri

on

7:

P

eo

ple

R

es

ult

s

Cri

teri

on

8:

S

oc

iety

R

es

ult

s

Cri

teri

on

9:

K

ey

Pe

rfo

rma

nc

e

Re

su

lts

Do

es

the

inst

itutio

n d

isch

arg

e e

ffe

ctiv

ely

its

resp

on

sib

ility

fo

r th

e s

tan

da

rd o

f a

ll a

wa

rds

gra

nte

d in

its

na

me

, a

nd

fo

r th

e q

ua

lity

of

the

ed

uca

tion

pro

vid

ed

b

y it

to e

na

ble

stu

de

nts

to

ach

ieve

th

at

sta

nd

ard

.

✔

✔

(4a

)

Do

es

the

inst

itutio

n c

on

firm

an

d d

em

on

stra

te t

he

wa

ys

in w

hic

h it

ass

ure

s a

cad

em

ic s

tan

da

rds

an

d q

ua

lity

refle

ct t

he

exp

ect

atio

ns

con

tain

ed

in t

he

pre

cep

ts o

f th

e A

ge

ncy

's C

od

e o

f p

ract

ice

.

✔

✔

(4a

) ✔

✔

?

✔

✔

?

De

scri

ptio

n a

nd

an

aly

sis

of

de

velo

pm

en

ts s

ince

th

e

last

HE

QC

/QA

A q

ua

lity

au

dit/

inst

itutio

na

l re

vie

w.

? ✔

✔

(4a

) ✔✔

De

scri

ptio

n a

nd

an

aly

sis

of

resp

on

ses

to in

div

idu

al

sub

ject

re

vie

ws,

an

d h

ow

less

on

s le

arn

t h

ave

be

en

ta

ken

into

acc

ou

nt

to e

nh

an

ce in

stitu

tion

al p

ract

ice

.

✔

✔

?

(3

a,

3b

) ✔

?

(4

a)

✔

?

Ke

y fe

atu

res

of

pro

cess

es

for

ass

uri

ng

aca

de

mic

st

an

da

rds

of

aw

ard

s a

nd

th

e q

ua

lity

of

pro

gra

mm

es.

✔

(2d

) ?

(3

b)

✔

✔

An

y p

rece

pts

of

the

Co

de

s o

f p

ract

ice

no

t b

ein

g

ad

he

red

to

an

d a

ltern

ativ

es

ap

pro

ach

es

be

ing

ta

ken

.

✔

✔✔

Vie

w o

f th

e p

erc

eiv

ed

str

en

gth

s a

nd

lim

itatio

ns

of

curr

en

t in

stitu

tion

al q

ua

lity

ass

ura

nce

arr

an

ge

me

nts

.

✔

(2a

, 2

e)

✔

(4

a)

✔

✔

?

✔

✔

Dis

cuss

ion

of

the

inte

nd

ed

str

ate

gy

for

the

ne

xt t

hre

e

yea

rs t

o e

nh

an

ce p

ract

ice

an

d r

em

ed

y a

ny

ide

ntif

ied

sh

ort

com

ing

s.

✔

✔

✔

(4

a)

✔

✔

✔

✔


other sub-criteria, management of finances, buildings, equipment, materials, technology, information and knowledge are not explicit concerns of IR. Taken as a whole, the Self Evaluation Document Guidance suggests a further step in the HEQC/ QAA evolution from, very crudely, focus on procedures and student experience to focus on institutional management, strategy, standards and delivery of quality for customers. There is more emphasis on self-evaluation, the capacity for an institution to be self-critical, improvement based on evidence and learning, and the integration of evidence from a range of sources which are key features of the EFQM Excellence Model®. For an institution, therefore, overlapping and complementary benefits are likely to be derived from preparation for IR and adoption of the EFQM Model. One difficulty arising from the timing of this project has been that the QAA has not yet fully detailed its approach to IR and there is as yet far less published information about the next rounds of IR and CPR, than there has been to inform the effective implementation of Continuation Audit. We know that IR will be more concerned than any previous Review programme, with academic standards. Yet there is as yet very little information about how this might be pursued at the institutional level, for example, there is as yet no IR equivalent of Annex E of QAA�s 1999 �Notes for the guidance of audit teams�, relating to the academic standards of programmes and awards. A comparison of this Annex with the Excellence Model�s criteria is quite interesting. It may be easy to dismiss the Excellence Model as being only concerned with �generic� management issues and as having nothing to offer in relation to academic standards. However, if you define the setting of academic standards as policy/ strategy - and an HEI�s policies must incorporate academic standards - and their communication and implementation as processes, Table 22 overleaf shows how the detail of Annex E maps against the nine EFQM criteria. There is a strong synergy between this Annex and EFQM criteria 2 and 5 and appropriate synergy, when references are made to staff, students and external examiners, with criteria 3,4,6 and 9. The gaps, as with other elements of the new QAA Framework, relate to EFQM criteria 1, 7 and 8. It may be interesting to review how closely any successor statement to Annex E, also relates to the EFQM Excellence Model®.

Mappin

g th

e Q

AA

Fra

mew

ork

and

81

C

onso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Tab

le 2

2 M

app

ing

QA

A g

uid

ance

fo

r A

ud

ito

rs A

nn

ex E

wit

h t

he

EF

QM

Exc

elle

nce

Mo

del

®

EF

QM

Exc

elle

nce

Mo

de

l Cri

teri

on

use

d a

s b

asi

s fo

r se

lf-a

sse

ssm

en

t a

ctiv

ity

An

ne

x E

: T

he

aca

de

mic

sta

nd

ard

s o

f p

rog

ram

me

s a

nd

aw

ard

s:

gu

ida

nce

to

au

dito

rs

Cri

teri

on

1:

L

ea

de

rsh

ip

Cri

teri

on

2:

P

oli

cy

an

d

Str

ate

gy

Cri

teri

on

3:

P

eo

ple

Cri

teri

on

4:

P

art

ne

rsh

ips

a

nd

Re

so

urc

es

Cri

teri

on

5:

P

roc

es

se

s

Cri

teri

on

6:

C

us

tom

er

Re

su

lts

Cri

teri

on

7:

P

eo

ple

R

es

ult

s

Cri

teri

on

8:

S

oc

iety

R

es

ult

s

Cri

teri

on

9:

K

ey

Pe

rfo

rma

nc

e

Re

su

lts

1. E

sta

blis

hin

g t

he

re

qu

ire

d s

tan

da

rds

Eff

ec

tive

ne

ss o

f A

rra

ng

em

en

ts f

or:

• m

aki

ng

aw

ard

sta

nd

ard

s e

xplic

it

✔

✔

• d

efin

ing

re

spo

nsi

bili

ties

for

est

ab

lish

ing

sta

nd

ard

s

✔

✔

• E

sta

blis

hin

g e

ntr

y st

an

da

rds

✔

• a

ssu

rin

g t

ha

t re

qu

ire

me

nts

re

de

finin

g a

nd

est

ab

lish

ing

st

an

da

rds

are

be

ing

me

t

✔

✔

Eff

ec

tive

ne

ss

of

ma

kin

g s

tud

en

ts a

wa

re o

f

• e

ntr

y st

an

da

rds

req

uir

ed

✔

✔

✔

• a

cad

em

ic s

tan

da

rds

req

uir

ed

to

pro

gre

ss

✔

✔

✔

• p

erf

orm

an

ce c

rite

ria

✔

✔

✔

• th

eir

ow

n r

esp

on

sib

ility

in m

ee

ting

aca

de

mic

sta

nd

ard

s

✔

✔

✔

Eff

ec

tive

ne

ss

of

pro

ce

du

res

at

pro

gra

mm

e a

pp

rova

l s

tag

e

for

• m

aki

ng

exp

licit

lea

rnin

g o

utc

om

es

✔

✔

• id

en

tifyi

ng

gra

du

ate

att

rib

ute

s

✔

✔

• ta

kin

g a

cco

un

t o

f a

pp

rop

ria

te e

xte

rna

l vie

ws

✔

• re

cord

ing

stu

de

nt

ach

ieve

me

nt

✔

2.

Th

e S

tan

da

rds

of

Aw

ard

s

E

xte

nt

to w

hic

h a

n i

ns

titu

tio

n h

as

es

tab

lis

he

d

•

cle

ar

aw

ard

an

d c

lass

ifica

tion

de

scri

pto

rs

✔

✔

• g

en

era

l de

gre

e d

ime

nsi

on

s fo

r its

aw

ard

s

✔

✔

• a

n e

ffe

ctiv

e r

eg

ula

tory

an

d g

uid

an

ce f

ram

ew

ork

✔

✔

Eff

ec

tive

ne

ss

of

ex

tern

al

ex

am

ine

rs i

n

•

pro

vid

ing

an

ind

ep

en

de

nt

pe

rsp

ect

ive

on

aca

de

mic

st

an

da

rds

✔

✔

• p

rovi

din

g a

n in

de

pe

nd

en

t p

ers

pe

ctiv

e o

n s

tud

en

t a

tta

inm

en

t st

an

da

rds

✔

✔

• m

on

itori

ng

an

d r

ep

ort

ing

on

qu

alit

y, c

on

sist

en

cy a

nd

eq

uity

o

f a

sse

ssm

en

t p

roce

sse

s

✔

✔

• m

od

era

ting

ass

ess

me

nt

crite

ria

✔

✔

Mappin

g the Q

AA

Fra

mew

ork

and

82

C

onso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Th

e w

ay

aw

ard

sta

nd

ard

s c

an

be

ca

lib

rate

d t

hro

ug

h

•

info

rma

tion

on

re

sults

pa

tte

rns

✔

✔

•

com

pa

riso

ns

acr

oss

th

e s

ect

or

✔

✔

•

com

pa

riso

n w

ith n

atio

na

l sta

tistic

s

✔

✔

• u

sin

g a

rch

ive

info

rma

tion

to

mo

nito

r co

nsi

ste

ncy

in

ass

ess

me

nt

an

d e

xpe

cta

tion

s o

f st

ud

en

t a

chie

vem

en

t

✔

✔

3.

Imp

rove

me

nt

an

d D

eve

lop

me

nt

H

ow

eff

ec

tive

ly a

n i

ns

titu

tio

n

•

mo

nito

rs a

nd

eva

lua

tes

the

aca

de

mic

sta

nd

ard

s o

f a

wa

rds

✔

• id

en

tifie

s a

nd

dis

sem

ina

tes

go

od

pra

ctic

e r

e a

cad

em

ic

sta

nd

ard

s a

nd

co

rre

cts

po

or

pra

ctic

e

✔

• su

pp

ort

s st

aff

to

se

t a

nd

me

asu

re s

tan

da

rds

✔

• su

pp

ort

s st

aff

to

de

sig

n a

sse

ssm

en

ts a

nd

ma

rkin

g

sch

em

es,

to

ma

inta

in s

tan

da

rds

✔


5 The Excellence Model and Institutional Review: the Process

Consideration has been given to the way in which each of the models operates in practice. There are some similarities but also some quite striking differences, as is shown by Table 23 below. Table 23 Process/ Specification Matrix Technical / Process specification


QAA Institutional Review QAA Review of Collaborative Provision


From 1 hour to 6 months Typically 1-2 years Typically 1-2 years

Timescale for assessment activity to be undertaken

From 3 hours to 3 months

4-5 months 4-5 months


Very flexible - from simple questionnaire to

award submission document

Limited - always involves self-evaluation document, preparatory meetings, and

visits.

Significant: may involve separate submission and event or be encompassed within main

IR/ CA.



By QAA Team of 4: large numbers involved in

institutions, depending on size in both preparation and

visit.

By QAA Team of 4: large numbers involved in institutions,

depending on size in both preparation and visit.

Reliance on evidence

Totally reliant on evidence

Totally reliant on evidence obtained from and

triangulated between the submission, meetings with

staff and baseroom evidence material.

Totally reliant on evidence obtained from and triangulated

between the submission, meetings with staff and

baseroom evidence material. Additionally evidence is

provided in discussions with partners, staff and students.



organisation

Focuses mainly at institutional level but also

meets with a wide range of other staff.

Focuses mainly at institutional level but also meets with a wide

range of other staff.


Assess against self completed

questionnaire, workbook or award submission

document

Judgements are made on the basis of all the evidence

available to the Review Team - from self-evaluation

document, staff/student meetings and baseroom

evidence material.

Judgements are made on the basis of all the evidence

available to the Review Team - from self-evaluation document,

staff/student meetings and baseroom evidence material.

The EFQM Model depends very heavily on the self-assessment, which can be undertaken at whole or part organisation level and can be informed by one or more of a wide range of self-assessment approaches. Many organisations find it helpful for those leading self-assessment exercises to have been trained as EFQM assessors, to ensure good understanding of the Model. Most organisations using the Model as a self-improvement tool, however, do not proceed to prepare a full-scale submission for an award so there is generally no process of external assessment by trained assessors. The time involved in undertaking the self-assessment activity can be reasonably well limited to ensure


that it does not become excessive but improvement action plans need to be realistic and prioritised, and need to inform, or be integrated into, the normal business planing cycle. To enable trends and the effectiveness of improvement actions to be identified, however, self-assessment does need to be undertaken on a regular � possibly annual � basis. The overall cost to an organisation over a number of years, unless EFQM self-assessment becomes the way of working rather than an additional exercise, may therefore be significant. QAA Institutional Review/Continuation Audit/Review of Collaborative provision, on the other hand, operates only at the institutional level and only on a six yearly cycle. Many institutions seek to involve their staff in supporting the work of QAA as auditors/ academic reviewers/ audit secretaries as this is one of the best ways in which to gain good knowledge of how the method is intended to operate. Many such staff then become involved in the institution�s preparations for the QAA Review. Preparations generally take much longer than an EFQM self assessment but each institution approaches the exercise in its own way: a self-assessment document and supporting material is required by QAA but the approaches by which they are produced are as various and diverse as the HEIs themselves. Collaborative provision review may be covered in the main CA/ IR or separately, depending on the scale of the collaborative portfolio. The total cost of the CA/ IR exercise to an institution is therefore very difficult to calculate but is likely to be very significant for a large institution. The cost is likely to be dependent, at least in part, on the extent to which the institution�s own internal systems and arrangements for quality and standards management routinely generate comprehensive evaluative output which can be used as the basis for the self-assessment document. Following the self-assessment phase the element of the process undertaken by an audit/ review team of the QAA, becomes highly formalised in order to ensure consistency of application across the sector. There will always be audit/ review team consideration of the documentation submitted: a preliminary site visit, possibly in future by the whole team and involving a small number of meetings with key staff: a team briefing meeting: visit to institution to meet staff and students: private team meetings to agree conclusion: draft report and finally the published report. This stage in the process, when the organisation�s self-evaluation is considered and in a sense �validated�, or otherwise, by an external team of peers/specialists/experts, is not commonly experienced by organisations using the EFQM model solely for self-improvement purposes. HEIs put varying amounts to efforts into preparing for this element of the process but, reflecting the perceived significance of the published outcome, it is clear that HEIs are increasingly devoting considerable effort to preparation � often involving extensive briefing or coaching programmes and dry run exercises. It is less immediately obvious, however, whether there is greater long-term value arising from the self-assessment and improvement activity, the last minute preparations for the visit itself, from the judgements made by the external audit/ review team (which may or may not always be accepted by the HEI), or from the QAA�s published report of the audit/ review, if the outcome is favourable.


SECTION 5

Subject Review

and the Excellence Model


1 Introduction HEIs are accountable to the Funding Councils, and ultimately to Parliament, for the way in which they use public funding. The Funding Councils contract with QAA for the delivery of Subject Review programmes. The mission of the QAA is to �promote public confidence that the quality of provision and standards in HE are being safeguarded and enhanced�14.

The purpose of review is threefold: • To secure value for public investment • To encourage improvement in provision • To provide accessible public reporting regarding the academic standards and

quality of learning opportunity in each subject area Review at subject level is a major aspect of this process which has evolved through several stages over the past 10 years. Regardless of the changes over time, the central focus has remained the assessment of the quality of educational provision at subject level. The process of assessment is organised between the QAA and HEIs.

2 The process as undertaken by QAA The QAA groups all academic provision into subject areas (e.g. Medicine, Engineering, Communication Studies etc.) and publishes a schedule of review, such that a group of subjects are reviewed in a given period, and all subjects are reviewed in a rolling programme. For example England has 11 subject areas due for review between 2001-2003. The QAA negotiates with each institution regarding the timing of the review within the period. They recruit, appoint and train Subject Review Co-ordinators (SRCs) who oversee the process, and Subject Specialist Reviewers (SSRs) who peer review provision within their speciality in each HEI. They also provide briefing for Subject Review Facilitators (SRFs) who are staff within HEIs who offer facilitation for the process. QAA publishes a handbook, which sets out all aspects of the review, and administers the process. The SRCs all need to have extensive experience of HE and in the assurance of standards and quality. They are usually nominated by an institution and seconded to the QAA, or may be independent consultants. They undergo an induction and training process provided by the QAA, and are expected then to co-ordinate up to eight reviews per year. They have a complex role which must enable �the team to develop a robust evidence base on which to make judgements�. This includes organising their own and their team�s time, maintaining a balance of activities and time, communicating with all parties, and

14

Handbook for Academic review QAA 2000

SECTION 5 SUBJECT REVIEW AND THE EXCELLENCE MODEL


after the review, producing the report. They are specifically not allocated to subject areas where they have subject expertise, as their role is to oversee the process. SSRs need to be academics within the specified subject area, and to have completed successfully the training arranged by the QAA. They are normally nominated by an institution and employed for an agreed number of reviews. Their role is to analyse evidence from the subject under review, including written documentation, information from meetings and from observation. From this they collectively make judgements on the academic standards and the quality of learning opportunity, and contribute to the written report. Institutions may (but are not obliged to) nominate members of staff for the SRF role. This person�s role is to liase between the various parties within the University and the QAA to ensure the effective progress of the review process. Extensive academic or administrative experience within HE is essential. Although the role is not mandatory, skilfully performed it makes a significant difference to the process of review and has been very well evaluated by the QAA and HEIs. Following each subject review the QAA publishes a report which sets out its judgements on the academic standards and the quality of learning opportunities within the area reviewed. This report is a definitive statement regarding whether the area of provision is approved or �failing�. Non approval is a serious judgement, requiring an action plan by the HEI and a revaluation by the QAA. Failure to meet the required standard of approval can lead to withdrawal of funding. Once all HEIs providing a particular subject area have been reviewed, the QAA produces an overview report of all provision within the subject. Both individual and overview reports are public documents which are published openly in paper form and on the web.

3 The Process from an Institutional Perspective Institutions negotiate with the QAA the timing of reviews within the specified period to fit in with their own quality management processes. Through the institutions own quality management intelligence (e.g. statistical data, reports, stakeholder feedback), the subject area is scrutinised against the QAA criteria. Then, using the Review Handbook as a guide, a Self-Evaluation Document (SED) for each subject is produced. In the present method this document has only been able to assess the area of provision against its own internal aims and objectives, due to the lack of any externally validated standards.


However the new method from 2002 involves a series of external referents (see Section 3 above), which include: • Subject Benchmark Statements • A Code of Practice • Programme Specifications • HE Qualifications Framework During the review period the institution facilitates access to materials, information, people etc. so that the SRC can manage the SSRs peer review of the subject area. The institution nominates staff to be SRCs and SSRs and provides a Subject Review Facilitator (SRF) � as identified above - who works closely with the subject area and QAA team to ensure the review is smoothly and fairly conducted. SRCs and SSRs are paid by the QAA for their work and the institution agrees release from their academic posts for the necessary time. SRFs are employed only by the institution and there is no reimbursement for their time. Once the review is completed, the institution can use published reports for marketing and development purposes, to evaluate the provision across the institution, and to evaluate their market position across the sector. In addition, the expectation from QAA would be evidence of action planning and improvement of the area, based on the review report. For both the QAA and the institution, the subject review reports form a significant part of the base from which the overall institutional profile is derived.

4 Synergies and gaps The QAA subject review process is complex, thorough and predominately academically, rather than managerially focused. In order to analyse the areas of synergy, and the dissonance and gaps between the Excellence Model and QAA subject review, a closer look at particular aspects of the process may be helpful. Whilst the Excellence Model is designed and developed externally to HEIs it remains an �opt in� process, unlike Subject Review which is mandatory. It also offers internal rather than external reporting. If and when the reputation and use of the Excellence Model grows within the sector the results may be significant, but they do not constitute a nationally recognised judgement, and the institution can make a strategic decision whether and how to use the outcomes in the public arena.

4.1 Matching measurement criteria between the Excellence Model and subject review As can be seen from the Measurement Criteria Matrix (see Table 24) there appears to be a great deal of synergy between the Excellence Model and QAA subject review process. However, in the following analysis it must be noted that


although Subject Review may be seen to focus on many of the same areas, it does so at one time and only from the perspective of a limited area of activity within the institution. The greatest match is in criteria 2, 5 and 9. This is not surprising as Subject Review favours settings where clear strategies and processes are in place, and in which subject areas can clearly identify and analyse their own performance. Criteria 4 and 6 are also strongly matched which reflects the emphasis Subject Review puts on the effective use the organisation makes of all resources, and on the views and experiences of key �customers� which include most importantly students, but also employers and graduates. A weaker match is present with criteria 3, 7 and 8, this is perhaps indicative of the narrow focus of Subject Review on one particular area at a time. Finally criterion1, leadership is not closely matched to Subject Review. This offers an interesting insight into the Subject Review process, in which individuals within the organisation are not highlighted: �people� are only of relevance with regard to their effect on the quality of the provision. It is not that the process does not acknowledge that good leadership is important for the maintenance of quality, but poor leadership, as well as other factors such as resources constraints can be redeemed within the review system if the quality of the student learning experience is maintained. Thus it may be argued that whilst there is at one level a close synergy between the two systems, Subject Review looks in depth at a particular area of the institution, and does not attempt to present a balanced review of quality at a managerial or organisational level. This latter function relates to the integrated QAA method, including Institutional Review.

Mappin

g th

e Q

AA

Fra

mew

ork

and

90

C

onso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Tab

le 2

4: Q

AA

Su

bje

ct R

evie

w a

nd

EF

QM

: M

easu

rem

ent

Cri

teri

a M

atr

ix

EF

QM

Exc

elle

nce

Mo

de

l Cri

teri

on

use

d a

s b

asi

s fo

r se

lf-a

sse

ssm

en

t a

ctiv

ity

Se

lf E

valu

atio

n D

ocu

me

nt

Gu

ida

nce

use

d a

s a

ba

sis

for

Su

bje

ct R

evi

ew

Cri

teri

on

1:

L

ea

de

rsh

ip

Cri

teri

on

2:

P

olic

y a

nd

S

tra

teg

y

Cri

teri

on

3:

Pe

op

le

Cri

teri

on

4:

P

art

ne

rsh

ips

an

d R

eso

urc

es

Cri

teri

on

5:

P

roce

sse

s

Cri

teri

on

6:

C

ust

om

er

Re

sults

Cri

teri

on

7:

Pe

op

le

Re

sults

Cri

teri

on

8:

So

cie

ty

Re

sults

Cri

teri

on

9:

K

ey

Pe

rfo

rma

nce

R

esu

lts

Ove

rall

aim

s o

f th

e su

bje

ct p

rovi

sio

n

T

he

mis

sio

n o

f th

e in

stitu

tion

an

d h

ow

stu

de

nts

ca

pa

city

to

lea

rn is

en

ab

led

. ✓

✓

✓

✓

✓

✓

✓?

✓

Ho

w in

tern

atio

na

l, n

atio

na

l, re

gio

na

l an

d lo

cal n

ee

ds

are

m

et.

✓

✓

✓

✓

✓?

✓?

✓

Ho

w s

tud

en

ts a

re p

rep

are

d f

or

futu

re e

mp

loym

en

t o

r fu

rth

er

stu

dy.

✓

✓

✓

✓

✓

✓

✓

✓

Ho

w a

cce

ss t

o f

urt

he

r e

du

catio

n is

be

ing

wid

en

ed

.

✓

✓

✓

✓

✓

✓

✓

✓

Eva

luat

ion

of

the

sub

ject

pro

visi

on

Eva

lua

tion

of

the

ap

pro

pri

ate

ne

ss o

f le

arn

ing

ou

tco

me

s in

re

latio

n t

o t

he

ove

rall

aim

s o

f th

e p

rovi

sio

n.

✓

Re

vie

w o

f th

e e

ffe

ctiv

en

ess

of

con

ten

t a

nd

de

sig

n o

f th

e

curr

icu

la a

nd

th

e m

ea

sure

me

nt

pro

cess

of

the

a

sse

ssm

en

t o

f o

utc

om

es.

✓

✓

✓

✓

✓

Eff

ect

ive

ne

ss o

f te

ach

ing

an

d le

arn

ing

in r

ela

tion

to

p

rog

ram

me

aim

s a

nd

cu

rric

ulu

m c

on

ten

t, a

nd

ho

w

stu

de

nt

pro

gre

ssio

n is

me

asu

red

.

✓

✓

✓

✓

✓

✓

✓

✓

Eff

ect

ive

ne

ss o

f th

e m

ea

sure

s ta

ken

to

ma

inta

in a

nd

e

nh

an

ce t

he

qu

alit

y a

nd

sta

nd

ard

s o

f p

rovi

sio

n.

✓

✓

✓

✓

✓

✓

Qu

an

tita

tive

da

ta t

o s

up

po

rt a

ss

es

sm

en

t

Qu

alit

ativ

e d

ata

to s

up

po

rt a

sses

smen

t

Pro

gra

mm

e sp

ecif

icat

ion

s


4.2 Matching purposes/concepts, and process/ specifications

As can be seen from the purposes/concepts, and process/specification matrices (Tables 25 and 26 below) both the Excellence Model and Subject Review use a range of internally and externally produced evidence to produce a robust self-evaluation, and it may be in this area that the greatest genuine synergy is evident. Evidence gathered as part of an Excellence Model self assessment exercise, and the subsequent self-evaluation of the area would be a significant source of evidence for Subject Review, and results from Subject Review would feed into the Excellence Model self assessment process. In addition, key sources of evidence, for example surveys of stakeholder views, are likely to be of equal value to both processes.

Table 25 QAA Subject Review and EFQM: Purpose/Concepts Matrix

EFQM Excellence Model® self assessment

QAA Subject Review

To review and assess all aspects of an organisation, measuring its ability to achieve and sustain outstanding

results for all stakeholders

Reviews educational provision at the subject level, assessing the quality of education through an

approach focussed on the establishment, maintenance and enhancement of academic

standards Reviews whether approaches are in place, if they are

sound and whether changes are systematically deployed in a wide range of areas - covering

management and operational issues

Reviews the effectiveness of the approaches adopted at a local level in a wide range of areas, and

seeks to promote improvements at a national level through publication of review reports, subject overview reports and through sharing of best

practice Focus on robustness of evidence and demonstration of

continual improvement over 3 or more years Focus on robustness of self-evaluation and evidence

over 3 or more years

Based on 8 fundamental concepts, 9 criterion and 32 sub-criteria which have been developed and tested by

world class organisations and used widely across Europe

Based on seven areas in the Subject Review Aide-Memoire conflated to a judgement on academic

standards and three aspects of the quality of learning opportunities: teaching and learning;

student progression; effective utilisation of learning resources

Concerned with looking at 'where we are now' across a wide range of balanced issues - at corporate and/or

business unit level

Concerned with looking at where the subject is against a range of external and internal reference

points

Can look at the overall management of the organisation as a whole, taking into account the direct and indirect

influencing factors

Looks at local quality management and enhancement at the subject level

Assesses and reviews activities using RADAR matrix - providing a common and consistent scoring matrix

Reviews on the basis of the Subject Review Aide-Memoire using other elements of the QAA QA framework as reference points, e.g. Subject

Benchmarking, Programme Specifications, Codes of Practice, National Qualifications Framework

Reviews internal and external relationships and inter-relationships

Reviews internal and external relationships and inter-relationships, including those within the

organisation and within the subject area

Provides a common language and framework for comparison within the organisation, and within and outside the sector on a range of management and

operational issues

Provides a common language and framework for comparisons of outcomes within an organisation,

and within the sector, although not completely consistent

Takes into account developmental activities being undertaken, and improvement trends being

demonstrated

Takes into account developmental activities being undertaken at local and organisational level, and

improvement trends being demonstrated


Table 26 QAA Subject Review and EFQM Process/Specification Matrix Technical / Process specification


QAA Subject Review


From 1 hour to 6 months From six months to two years

Timescale for assessment activity to be undertaken

From 3 hours to 3 months Allocated number of reviewer days within a period of review from two months to 12 months

(likely to settle on max three months)


Very flexible - from simple questionnaire to award submission document

SED submitted to QAA to provide framework for review



Varies on basis of 'size' of review and the extent to which subject and organisational structures

map

Reliance on evidence Reliant on evidence but also to some extent perception

based

Reliant on evidence and effective facilitation by Subject Review Facilitator and Subject Contact



organisation

Fits at personal, team, department/ divisional, school and whole organisation


Assess against self completed questionnaire,

workbook or award submission document

Review against self-evaluation, Subject Review Aide-Memoir and external reference points

4.3 Matching outcomes

Over the present cycle of review the QAA has been developing as part of its Quality Assurance framework a series of external referents as itemised above. This is enabling the QAA, from 2002, to benchmark subjects against nationally recognised and agreed standards. Whilst this has always been possible to an extent, particularly within subject areas where there are statutory professional body requirements, it is only in the forthcoming round of Subject Review that this has been regularised for all subjects. Thus definitive statements about standards are to be made as can be seen in Table 27 (but please note discussion in the section on Institutional Review with regard to mapping standards).


Table 27 QAA Subject Review and EFQM Process: Outcomes Matrix Outcomes required EFQM Excellence Model® self

assessment QAA Subject Review

Reports Optional - can be a short management report, or a full assessment document

Self Evaluation Document produced prior to review period by subject area. Report produced by QAA

following review period

Action plans Optional, but desirable. The outcomes of a self-assessment activity usually

lead to actions which are implemented through an action plan


QAA.

Evidence of learning Strong - assessment activity itself evidences learning. Action plan tracks

progress and learning.

Internal action plan

Overall improvement activity Often demonstrated over 12 months or more, but can show immediate

improvements. Activities range across wide range of management and people

issues.

Ongoing improvement seen as embedded in the process


Desirable to be fully integrated into the business planning cycle - allowing the

self-assessment process to be preparation for an evidence based business plan. Integrates well with

other management tools.

Subject Review inputs into overall quality review of

subject/school/institution


Evidence to show marked improvement in customer experience amongst

organisations who are using the Model

Improvement in student experience where specific issues have been

identified in the self evaluation and review


Evidence to show marked improvement in staff experience amongst

organisations who are using the Model

Unlikely to impact on improved staff experience as the review is student

focussed

Improvement in balanced set of performance results

Usually leads to an understanding of the need for, and the development of, a

balanced set of performance measures (customer, staff, society and financial)

Possible improvement, but more likely that resources are focussed on

subject review, rather than overall balance

The Excellence Model, whilst it may encourage groups to seek appropriate benchmarks, is unable to judge academic standards in quite the same way. This, in conjunction with the mandatory and public nature of subject review, is one of the areas in which there is least synergy. The outcomes matrix further highlights the potential lack of balance if Subject Review is seen in isolation, as the improvements, and performance results are not uniform. This perhaps illustrates an interesting area of potential dissonance


between the two models. The Excellence Model takes as its focus the concept that the organisation has a product and that maximising the quality of its product will improve the overall performance/market position/profitability of the organisation. Within HEIs this concept is problematic, as the actual product is subjective and complex, making judgements on its quality difficult.

4.4 Conclusions In conclusion QAA Subject Review is designed to look closely at academic standards, a major consideration within HEIs, whereas the Excellence Model framework focuses predominately on the quality of the management of the organisation. Thus it may be possible for the two systems to be complementary


SECTION 6

The Excellence Model

and the Integrated QAA Method


In a sense, it may be artificial to consider elements of the new QAA Framework separately against the EFQM Excellence Model. The new QAA Framework is intended, in the future, to be an integrated framework which provides for reviews of institutions� policies and practices at a number of levels, and for the outcomes of reviews to inform future reviews. However the QAA �integrated method� is not yet operative in England and external consideration of whether institutions are adhering to the Code, for example, is only just now beginning to occur, in Continuation Audit. It may therefore be some time yet before it is clear whether QAA Subject/ Institutional Review Reports routinely comment on Code adherence, or in any great detail. However, if the various mappings of the foci of the individual elements of the QAA Framework are aggregated, the overall �fit� between the QAA Framework and the EFQM Excellence Model® becomes clearer, as becomes apparent in Table 28 below.

SECTION 6 THE EXCELLENCE MODEL AND THE INTEGRATED QAA METHOD

Mappin

g th

e Q

AA

Fra

mew

ork

and

97

C

onso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Ta

ble

28

E

FQ

M E

xcel

len

ce M

od

el® a

nd

th

e in

teg

rate

d Q

AA

Rev

iew

Fra

mew

ork

E

FQ

M E

xcelle

nce

Model

Cri

teri

on u

sed a

s basi

s fo

r se

lf-a

sse

ssm

en

t act

ivity

E

lem

ent

of

QA

A

Fra

mew

ork

Crite

rion

1:

Leaders

hip

Crite

rion

2:

P

olic

y and

Str

ate

gy

Crite

rion

3:

P

eople

Crite

rion

4:

P

art

ners

hip

s and

Reso

urc

es

Crite

rion

5:

P

roce

sses

Crite

rion

6:

C

ust

om

er

Resu

lts

Crite

rion

7:

P

eople

R

esu

lts

Crite

rion

8:

Soci

ety

R

esu

lts

Crite

rion

9:

K

ey

Perf

orm

ance

R

esu

lts

QA

A C

ode o

f P

ract

ice

✔

✔

?

✔

✔

✔

? ✔

✔

QA

A S

ubje

ct R

evi

ew

✔

✔

✔

✔

✔

✔

✔

✔

QA

A I

R &

CP

R

✔

✔

✔

(3a,

3b

✔

(4a)

✔

✔

✔

✔


The QAA Framework makes little explicit reference to the matters covered by EFQM Criteria 1 (Leadership) and 8 (Society Results), but it should be remembered that IR will focus on the �senior layer� within an institution. There is some synergy in relation to Criteria 6, 7 and 9, (Customer, People and Key Performance Results). However, there is strong and clear synergy between the Framework and the Model in relation to Criteria 2-4 (Policy and Strategy, People, Partnerships and Resources) with strongest synergy between all elements of the Framework and the EFQM model, occurring in relation to Criterion 5, Processes. Interestingly, although the QAA Code of Practice is building up to form an extensive set of precepts and good practice guidance, on the evidence of this mapping exercise its foci and concerns, to date, appear to be more restricted in scope than the wider concerns of Subject Review, Institutional Review and the EFQM Model. Subject Review, on the other and appears to be the area where there is greater synergy with a broader range of the EFQM criteria, than is immediately apparent in the case of IR/ CPR It may be interesting to consider whether the QAA Framework might be improved in terms of promoting and encouraging better overall performance by HEIs on academic quality and standards, if it were to be further developed in the future to incorporate more explicit reference to more of the matters covered by the Excellence Model�s Criteria, 1, 3, and 6-9: namely Leadership, People, and Customer, People, Society and Key Performance Results. However, it must be remembered that the Excellence Model is essentially a practical tool to support effective, holistic management, and can be used in any way an organisation considers appropriate. The QAA Framework, in contrast, is essentially a pragmatic model, entirely dependent upon what Government, HEIs and other key stakeholders will accept, and which is therefore operating in a political context. Table 29 gives a very broad comparison of outcomes from the two models. As to whether the outcomes are similar, the answer is probably that �it depends�. The QAA Code of Practice is included in this table to illustrate its difference in nature from the other three elements of the QAA Framework included. It is clearly a reference point for institutions and reviewers, and not a review or assessment activity per se. Both models require self-evaluation or self-assessment documents to be prepared but only QAA SR/ IR/ CA/ CPR lead to external judgements, in published reports. Both models assume that organisations will incorporate improvement action proposed into their self-assessment documents: in the case of the QAA Reviews, institutions will be likely to seek to undertake as much improvement action as is appropriate and feasible before scrutiny by the external reviewers. QAA IR/ CPR/ CA includes consideration of the extent to which a HEI has the capacity for effective self-evaluation and improvement activity, as evidenced by the self- evaluation document, discussions with staff and students and scrutiny of supporting evidence. Each model assumes that an honest and rigorous approach to self-evaluation and improvement action will lead to improved performance. The extent to which individual organisations decide to implement improvements will obviously vary


enormously, depending, not least, on whether they accept the conclusions of the Review and on the resources available to them and the deficiencies identified. Improvements in customer and staff experience may therefore be prioritised by some institutions but not by others, arising from QAA Review: the systematic focus on staff experience is strong, in the Excellence Model. Both the Excellence Model and the QAA Code are concerned with �customer focus�. Similarly, the Excellence Model may be more likely to promote improvements in a balanced set of performance results. The Model encourages target-setting and evaluation of performance trends, in a way which is far more explicit than the QAA Framework. The QAA Framework on the whole, focuses to a greater extent on �enablers�, using EFQM terminology, than on �results�.

1 Process Table 30 provides information about the processes involved in Excellence Model self-assessment and four elements of the QAA Framework. There are clear similarities, most notably the reliance on evidence and the attention paid to all levels within the organisation. It is to some extent inappropriate to compare the process of self-assessment to processes of external scrutiny: more relevant comparisons might be between submission for an EFQM Excellence Award and SR/IR, or between EFQM self-assessment and internal preparation for SR/IR. Table 30 merely demonstrates that approaches to EFQM self-assessment can be very flexible, adapted to suit the organisation, and that QAA scrutiny is far more precisely defined. What the table masks (explored to some extent in Section 6) is the enormous variety of approaches institutions adopt to Code compliance, or preparation for SR/IR/CPR. One conclusion from this project, is that HEIs may derive benefit from adopting or adapting some approaches from the EFQM Excellence Model, to aid both their routine management of quality and standards, and their preparations for engagements with QAA.

2 Outcomes Table 29 draws together the outcomes from the QAA Code and the �integrated method�, and EFQM Excellence Model self-assessment. Taken overall, the totality of the QAA outcomes and self-assessment show remarkable similarity. Both encourage and promote a cycle of self-assessment, learning, improvement action and monitoring. Institutions may approach their internal activities and preparations in a wide variety of ways and may choose to integrate these activities with business planning and other management tools. The areas where differences are more marked, however, are that working with the QAA Framework possibly does less to encourage emphasis on the staff experience, and similarly, does not encourage such a strong emphasis on a balanced set of performance results, as the Excellence Model.


Table 29 Intergrated QAA Review Framework and EFQM Model:Outcomes Matrix

Outcomes required

EFQM Excellence Model self

assessment

QAA Subject Review

QAA Institutional

Review

QAA Review of Collaborative

Provision

QAA Codes of Practice

Reports Optional - can be a short management

report, or a full assessment document

Self Evaluation Document produced prior to review period

by subject area. Report produced by

QAA following review period

Self Evaluation Document produced at institutional level to inform review by external reviewers. Review report

produced by external reviewers.

QAA methodology does not require institutions to provide reports specifically about the Code of

practice. However, it is assumed that Academic Review Reports (subject & institutional) will include reference to

adherence to the Code. Also assume that institutional review and

monitoring of practice against codes is likely to lead to internal reports.

Action plans Optional, but desirable. The outcomes of a self-

assessment activity usually lead to actions which are implemented through an action plan


QAA.

Assumes that action planning is incorporated into the self-evaluation. Most/ all institutions will initiate action

on issues identified. Follow-up action/ progress report required by

QAA.

Again, the QAA methodology does not necessarily result in action plans,

but institutional responses to QAA review and internal review may well lead to action plans for improvement

measures


Strong - assessment activity itself evidences

learning. Action plan tracks progress and

learning.

Internal action plan The review will reach a judgement on the institution's capacity for self-evaluation and self-improvement

Institutional consideration of practice against the QAA Code would

presumably need to be evidence based and good practice in ongoing

monitoring and review should demonstrate evidence of learning

Overall improvement activity

Often demonstrated over 12 months or

more, but can show immediate

improvements. Activities range across

wide range of management and

people issues.

Ongoing improvement seen as

embedded in the process

Stimulates improvement activity on an institution-wide basis

See action plans above � assumed that these would lead to overall

improvement of activity


Desirable to be fully integrated into the

business planning cycle - allowing the self-

assessment process to be preparation for an

evidence based business plan.

Integrates well with other management

tools.

Subject Review inputs into overall quality review of subject/school/

institution

Depends on the approach adopted by the institution: may be integrated or

may be a separate activity

Whilst the Code deals with individual areas of practice in separate

sections, there would be inevitably be �cross cutting� and broader

implications arising from identifying and actioning improvement measures

in relation to each section. Institutions would need to be able to

make the links and ensure appropriate integration between

sections and with other aspects of planning and management.



customer experience amongst organisations


Improvement in student experience

where specific issues have been identified in the self evaluation

and review

May lead to an improved 'customer' experience if issues to be addressed

are identified either by the self-evaluation exercise or by the external

reviewers and action plans are implemented effectively.

As much of the Code is concerned with ensuring effective processes,

clearer articulation of roles and responsibilities and good information

to students and partners, it is assumed that improvements should accrue from adherence to the Code.



staff experience amongst organisations


Unlikely to impact on improved staff

experience as the review is student

focussed

May lead to an improved staff experience if issues relating to staff

quality and support are identified and addressed.

The focus of clarity of roles and responsibilities and ensuring

appropriate staff development and training should be of benefit to staff.

The indirect benefits of demonstrating adherence to the Code (good Ac

Review outcomes) will be of value. Improvement in balanced set of performance results

Usually leads to an understanding of the

need for, and the development of, a

balanced set of performance measures (customer, staff, society

and financial)

Possible improvement, but more likely that resources are

focussed on subject review, rather than

overall balance

Possible improvement, but focuses more on 'enablers' than results.

The code is a potentially valuable tool to assist institutions to improve

performance results in selected areas of practice?

Mappin

g th

e Q

AA

Fra

mew

ork

and

10

1

Conso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Ta

ble

30

EF

QM

Exc

elle

nce

Mo

del

an

d t

he

QA

A F

ram

ewo

rk:

Pro

cess

Ma

trix

T

ech

nic

al /

Pro

cess

sp

eci

fica

tion

E

FQ

M E

xc

ell

en

ce

M

od

el

se

lf

as

se

ss

me

nt

QA

A S

ub

jec

t R

evi

ew

Q

AA

In

sti

tuti

on

al

Au

dit

Q

AA

Re

vie

w o

f C

oll

ab

ora

tive

P

rovi

sio

n

QA

A C

od

es

of

Pra

cti

ce

Tim

es

ca

le t

o

pre

pa

re f

or

as

se

ss

me

nt

Fro

m 1

hou

r to

6

mon

ths

Fro

m s

ix m

onth

s to

tw

o ye

ars

Typ

ical

ly 1

-2 y

ears

T

ypic

ally

1-2

yea

rs

Enc

ompa

sses

sel

f-as

sess

men

t an

d as

sess

men

t th

roug

h A

cade

mic

Rev

iew

, T

imes

cale

and

met

hod

very

dep

ende

nt u

pon

inst

itutio

nal a

ppro

ache

s to

rev

iew

of

prac

tice

agai

nst

Cod

e an

d is

sues

spe

cific

to

Cod

e se

ctio

ns.

Tim

es

ca

le f

or

as

se

ss

me

nt

ac

tivi

ty t

o b

e

un

de

rta

ke

n

Fro

m 3

hou

rs t

o 3

mon

ths

Allo

cate

d nu

mbe

r of

re

view

er d

ays

with

in a

pe

riod

of

revi

ew f

rom

tw

o m

onth

s to

12

mon

ths

(lik

ely

to s

ettle

on

max

thr

ee

mon

ths)

4-5

mon

ths

4-5

mon

ths

See

abo

ve

Fle

xib

ilit

y o

f a

ss

es

sm

en

t m

eth

od

Ver

y fle

xibl

e -

from

si

mpl

e qu

estio

nnai

re

to a

war

d su

bmis

sion

do

cum

ent

SE

D s

ubm

itted

to

QA

A t

o pr

ovid

e fr

amew

ork

for

revi

ew

Lim

ited

� al

way

s in

volv

es s

elf-

eval

uatio

n do

cum

ent,

pr

epar

ator

y m

eetin

gs,

and

visi

ts.

Sig

nific

ant:

may

invo

lve

sepa

rate

su

bmis

sion

and

eve

nt o

r be

en

com

pass

ed w

ithin

mai

n IR

/CA

.

See

abo

ve

Nu

mb

er

of

pe

op

le

typ

ica

lly

invo

lve

d

Var

ies,

min

imum

us

ually

5,

can

be

ever

yone

Var

ies

on b

asis

of

'siz

e' o

f re

view

and

the

ext

ent

to

whi

ch s

ubje

ct a

nd

orga

nisa

tiona

l str

uctu

res

map

By

QA

A T

eam

of

4: la

rge

num

bers

invo

lved

in

inst

itutio

ns,

depe

ndin

g on

siz

e in

bot

h pr

epar

atio

n an

d vi

sit.

By

QA

A T

eam

of

4: la

rge

num

bers

in

volv

ed in

inst

itutio

ns,

depe

ndin

g on

siz

e in

bot

h pr

epar

atio

n an

d vi

sit.

See

abo

ve

Re

lia

nc

e o

n

evi

de

nc

e

Rel

iant

on

evid

ence

bu

t al

so t

o so

me

exte

nt p

erce

ptio

n ba

sed

Rel

iant

on

evid

ence

and

ef

fect

ive

faci

litat

ion

by

Sub

ject

Rev

iew

Fac

ilita

tor

and

Sub

ject

Con

tact

Tot

ally

rel

iant

on

evid

ence

ob

tain

ed f

rom

and

tri

angu

late

d be

twee

n th

e su

bmis

sion

, m

eetin

gs w

ith s

taff

and

bas

e ro

om e

vide

nce

mat

eria

l.

Tot

ally

rel

iant

on

evid

ence

ob

tain

ed f

rom

and

tri

angu

late

d be

twee

n th

e su

bmis

sion

, m

eetin

gs

with

sta

ff a

nd b

aser

oom

evi

denc

e m

ater

ial.

Add

ition

ally

evi

denc

e is

pr

ovid

ed in

dis

cuss

ions

with

pa

rtne

rs,

staf

f an

d st

uden

ts.

Eff

ectiv

e in

tern

al a

nd Q

AA

ass

essm

ent

of

adhe

renc

e to

the

Cod

e w

ould

hav

e to

be

evid

ence

bas

ed

Fit

s a

t e

very

le

vel

of

the

org

an

isa

tio

n

Fits

at

pers

onal

, te

am,

depa

rtm

ent

and

who

le

orga

nisa

tion

Fits

at

pers

onal

, te

am,

depa

rtm

ent/

div

isio

nal,

scho

ol a

nd w

hole

or

gani

satio

n

Foc

uses

mai

nly

at in

stitu

tiona

l le

vel b

ut a

lso

mee

ts w

ith a

w

ide

rang

e of

oth

er s

taff

.

Foc

uses

mai

nly

at in

stitu

tiona

l le

vel b

ut a

lso

mee

ts w

ith a

wid

e ra

nge

of o

ther

sta

ff.

Fits

at

inst

itutio

nal a

nd d

epar

tmen

t/ s

choo

l lev

el

As

se

ss

me

nt

ag

ain

st

se

lf

pre

pa

red

d

oc

um

en

t

Ass

ess

agai

nst

self

com

plet

ed

ques

tionn

aire

, w

orkb

ook

or a

war

d su

bmis

sion

doc

umen

t

Rev

iew

aga

inst

sel

f-ev

alua

tion,

Sub

ject

Rev

iew

A

ide-

Mem

oir

and

exte

rnal

re

fere

nce

poin

ts

Judg

emen

ts a

re m

ade

on t

he

basi

s of

all

the

evid

ence

av

aila

ble

to t

he R

evie

w T

eam

�

from

sel

f-ev

alua

tion

docu

men

t,

staf

f/st

uden

t m

eetin

gs a

nd

base

room

evi

denc

e m

ater

ial.

Judg

emen

ts a

re m

ade

on t

he

basi

s of

all

the

evid

ence

ava

ilabl

e to

the

Rev

iew

Tea

m �

fro

m s

elf

eval

uatio

n do

cum

ent

staf

f/st

uden

t m

eetin

gs a

nd b

aser

oom

evi

denc

e m

ater

ial.

Inte

rnal

ass

essm

ents

wou

ld b

e ba

sed

on a

ra

nge

of in

puts

. Q

AA

judg

emen

ts a

re m

ade

on

the

basi

s of

all

the

evid

ence

ava

ilabl

e to

the

R

evie

w T

eam

� f

rom

sel

f ev

alua

tion

docu

men

t,

staf

f/st

uden

t m

eetin

gs a

nd b

aser

oom

evi

denc

e m

ater

ial.


3 Conclusions

The QAA integrated Framework is concerned with many of the same issues as the Excellence Model criteria, but not all. The extent to which either or both lead to genuine and sustained improvement in performance is crucially dependent upon the honesty and rigour of self-assessment, and implementation of focussed improvement actions.


SECTION 7

Comparing The Models In Practice


1 Introduction

Having investigated, compared and contrasted the concepts and purposes of the two assessment frameworks, this section aims to look in more detail at the process and experiences of using these two models in practice. It aims to compare the experience of undertaking a self-assessment using the EFQM Excellence Model® with the experience of undertaking a self-assessment using the QAA Subject Review framework, identifying further learning that can be gained. Comparing and contrasting at subject level was chosen rather than institutional level as there is a greater body of knowledge in terms of application of the EFQM Model in academic schools rather than at University wide level. The use of Subject Review in Schools, and knowledge of the forthcoming change in the review process as currently published, is also well documented and understood. It was also felt that the direct impact of Subject Review on staff within HEI�s is significantly greater, so the nature of a more detailed comparison may yield greater benefits at the micro rather than macro level. Whilst it is acknowledged that the QAA Subject Review process has been amended and is changing from 2001, there is little experience to date that we have been able to draw upon which gives us a real flavour for how the new process will feel. In the same way the changes made to the process in transition from Continuation Audit to Institutional Review have not yet been put into practice. In terms of self-assessment using the Excellence Model, no one Higher Education institution has experienced an organisation wide self-assessment, but a number of academic Schools/ Faculties and central departments have used the model within their individual areas. For this reason we have compared the pre-2001 Subject Review self-assessment process against the EFQM self-assessment process within Sheffield Hallam University from real experiences, and reflected on how the new QAA self-evaluation process may also compare, based on what we know now. This section aims to give a flavour of how the two journeys have been for us as an institution. We acknowledge that no two HEI�s are the same, but there are many transferable lessons that emerge from our analysis. The exploration begins with looking at how it has been for us so far, what have we learnt about using the two processes in tandem, and how will this enable us to move forward in the future.

2 Self-Assessment against Subject Review criteria The QAA groups all academic provision into subject areas (e.g. Medicine, Engineering, Communication Studies etc.) and publishes a schedule of review, so that a group of subjects are reviewed in a given period, and all subjects are reviewed in a rolling programme. For example England has 11 subject areas due for review between 2001-2003. The QAA negotiates with each institution to agree the timing of the review within the period. They recruit, appoint and train Subject Review Co-ordinators (SRCs) who oversee the process, and Subject Specialist Reviewers (SSRs) who peer review provision within their speciality in each HEI. They also provide briefing for

SECTION 7 COMPARING THE MODELS IN PRACTICE


Subject Review Facilitators (SRFs) who are staff within HEIs who offer facilitation for the process. QAA publishes a handbook, which sets out all aspects of the review, and administers the process. Institutions may (but are not obliged to) nominate members of staff for the SRF role. This person�s role is to liase between the various parties within the University and the QAA to ensure the effective progress of the review process. Extensive academic or administrative experience within HE is essential. Although the role is not mandatory, skilfully performed it makes a significant difference to the process of review and has been very well evaluated by the QAA and HEIs. In preparation for the review, a leader is often nominated within the School to co-ordinate the activity. Often a steering group is established, and aspect groups and/or leaders are identified. The SRF helps to train and facilitate the work of the groups, giving guidance and support throughout the process. This centrally shared resource has been key to the success of the Subject Review process within Sheffield Hallam, as it has enabled learning to be shared across all areas. It has also assisted with the co-ordination and integration of the University�s own internal quality management systems (which encompasses many other external audit requirements) as part of the preparation for Subject Review, as well as being a core part of the overall drive for quality management and enhancement. In terms of actually undertaking a Subject Review, there are differences from School to School, subject to subject, and from institution to institution. Figure 12 gives a fairly typical indication of the current experience within Sheffield Hallam University � using the pre-2001 assessment framework. This demonstrates the various stages that are currently undertaken and the relative timescales associated with each stage. At the end of the Subject Review process, the QAA publishes a report which sets out its judgements on the academic standards and the quality of learning opportunities within the area reviewed. This report is a definitive statement regarding whether the area of provision is approved or �failing�. Non approval is a serious judgement, requiring an action plan by the HEI and a revaluation by the QAA within 12 months. Failure to meet the required standard of approval can lead to withdrawal of funding. Once all HEIs providing a particular subject area have been reviewed, the QAA produces an overview report of all provision within the subject. Both individual and overview reports are public documents which are published openly in paper form and on the web. The resultant report and/or action plan that emerges from this Subject Review process can be fed into improvement work or business planning with the Institution. Sometimes this level of integration is dictated by whether or not the Subject Review has taken place at a time when feedback and forward planning is optimal � i.e. as the business plan or school improvement plan is being prepared. The QAA report then stands as a document that marks the assessment of provision at a specific point in time. Future assessments of the provision will


start with this report and measure improvements made over time, but this may anything up to 6 years later. Figure 12 A typical subject review process at Sheffield Hallam University

Briefing on QAA approach, guidance and systems required

Year 1 � Summer/Autumn

Briefing by Registry followed by School briefing

Year 1 - Autumn

Resources allocated are variable. (leader / admin. support, space)

Year 1 � Autumn / Winter

Internal analysis of provision undertaken to identify strengths

and issues Year 2 - Spring

Steering group / aspect groups set up

Year 2 - Autumn

The Subject Review support team undertake a diagnostic exercise to

identify strengths and issues Year 2 - Winter

SAD written and agreed Year 3 - Spring

SAD submitted to QAA Year 3 - Spring

Identification and collation of evidence for the base room

Year 3 - Autumn

Students briefed Year 3 � Spring - Summer

Employers / past students identified

Year 3 - Summer

Dry Run Year 3 - Autumn

Final membership of groups agreed

Year 3 - Autumn

Subject review assessment Year 3 - Winter

(1 week � 3 assessors)


The experience of this process has shown that preparation for a Subject Review assessment can be very intense, requiring a great deal of preparation, forward planning, time and dedication by staff. One of the issues emerges from an analysis of this process is that the time invested in the process may not necessarily reflect the actual and tangible benefit that can be demonstrated. Whilst it is acknowledged that a high or maximum score is both beneficial and desirable in terms of promoting subjects and attracting students, there are questions raised about how �real� this snap-shot is in terms of continual improvement. The new QAA self-evaluation process seems to go some way to bridging this gap. To analyse this further, Table 3 identifies some of the key advantages and disadvantages of the self-assessment process. Table 31 Advantages and disadvantages of the QAA subject review process

PRE 2001 SUBJECT REVIEW SELF-ASSESSMENT Advantages Disadvantages

• Provides a specific cycle against which all areas of provision can be assessed � no area is left out.

• Cycle time is defined by QAA with little flexibility or choice on the timing of reviews

• Peer reviewed • Prescriptive • Transparent process • Time consuming / can be expensive • Public report made available for all to view • �Snapshot� � current analysis of specific areas • Student centred • Not easy to sustain improvement • Thorough • Creating / keeping standards over a period of

time

• Has to be done � it is mandatory. Groups who may not wish to engage in a quality process cannot �opt out�

3 Self-Assessment against the EFQM Excellence Model

In stark contrast to the QAA, the European Foundation for Quality Management (EFQM) is not an auditing body. Are not prescribe how the Model is used within any organisation (although supporting guidelines are given), and it does not specify which part of an organisation should be assessed, although there is best practice which can followed. This provides the largest and most significant contrast between the processes � the element of choice. HEsI can choose if and when they wish to undertake a self-assessment, at what level to introduce or undertake the assessment (institutional or school/department level � or both), and how many people need to engage directly and indirectly in the process. These are internal decisions that are made based on where the institution is, where it wishes to get to, and what it decides are the best tools to enable it to get there. It is, at present, completely internally driven, with little or no direct communication with the EFQM. Once it has been decided that an assessment will be undertaken, at a time that is appropriate, an assessment methodology and timescale is chosen. The assessment process can include using just one of the main methodologies outlined in section one, or a mix of methods depending on time, resource and know-how. There is complete flexibility in this choice, and the methodology can be amended even after the assessment process has begun. For example, it may have been planned to distribute questionnaires as well as complete a pro-forma. Once the initial discussions and workshops have been held, it can be decided not


to use the questionnaires at all. An internal decision is made based on an evaluation of the benefit of using or losing the approach. In Sheffield Hallam University, the self-assessment process that has been adopted includes three main methodoglogies � pro-forma, workshops and questionnaires. This approach has been applied within academic schools, research institutes and central departments. In many ways the process is similar to the QAA self assessment process, but with a few key differences:

Figure 13 EFQM Typical self-assessment process within Sheffield Hallam

University

Briefing by Organisational Excellence Team to School representatives

Month 1

Discussion with Organisational Excellence Team to agree appropriate

methodology and timescale Month 1

Workshop with assessment team to introduce concepts and method

Month 2/3

Evidence collated using pro-forma by assessment team

Month 3

Questionnaires distributed to gain wider views and communicate activity being

undertaken Month 3

Mentoring and coaching sessions given to aid completion of pro-forma

Month 4

Pro-forma completed and brought together as one document

Month 5/6

Actions integrated into business plan as a key driver, with pro-forma a key evidence base

Month 7

Workshop to assess strengths and areas for improvement, and agree

actions Month 6

Key actions agreed, and action plan produced

Month 6

Introductory discussions and gaining of commitment. Key contact / lead within

school identified Month 1

Evaluation and review of outcomes and methodology: planning for next

assessment Month 8

Re-assessment using agreed (possibly revised) methodology

Month 1 � Year 2

On-going

annual cycle


At Sheffield Hallam University, assessor training has been used as a pre-cursor to many of the self-assessment activities. A central resource of EFQM expertise (two people within the Organisational Excellence team) have also supported, mentored, coached and trained staff within the individual areas to help define and implement the most appropriate assessment methodology and to share lessons learnt from other areas. This has also helped to link the methodology with other assessment and auditing frameworks and timescales. Initially the assessment process has required additional effort by the teams to understand and appreciate the methodology and philosophy � this is not untypical of any new system or framework. As it is becoming embedded, it is being viewed as a core management tool with which not everyone needs to engage� it is becoming a way of managing the business, as well as reviewing and assessing the business. Completion of the pro-forma document is based on trust � an acknowledgement that the information collected for all sub-criteria is open and honest, and that nothing is included that cannot be evidenced if required. At this stage, and using this process, the accumulation of evidence for inspection is not required. The pro-forma can be completed on an individual basis, although it is recommended that it is completed in pairs or teams having gained an �overall� view, rather than an �individual� view. Once completed, the pro-forma can be validated using trained assessors within the organisation, but outside the School, or it can be assessed by trained assessors external to the institution � from the sector, from outside the sector, from the UK or from across Europe. There is no stipulation to publish the findings of the self-assessment, although sharing internally is proving to be valuable. The methodology allows for a score to be determined against each of the sub-criterion using the RADAR approach. This is optional, but can add to the learning as it gives the opportunity for the evidence collected to be scrutinised in further detail � particularly in terms of soundness of the evidence and the establishment of trends. It has been demonstrated that by using the RADAR logic when gathering evidence for a Subject Review, a more robust approach was demonstrated with the process of defining and gathering the information needed much clearer than previously experienced. At the end of the process there is no obligation for the findings to be published. At Sheffield Hallam, there are several networks established to enable schools, departments and research institutes who are going through a self-assessment exercise to share their experiences and learn from each other. This has provided transparency of core issues and learning across the whole University structure, engaging staff at various levels of the organisation. The collation of evidence using this model will also allow benchmarking to take place with others who are also using the model � both within and outside the sector. If we wish to learn more about how to improve our customer satisfaction, it is possible to access organisations within the UK and across Europe who are deemed to be excellent in that area. For many of our core management processes, learning can transcend sectors and sometimes countries. Again, there a number of advantages and disadvantages to using this approach, which are highlighted in Table 32.


Table 32 Advantages and disadvantages of self-assessment using the

EFQM Excellence Model® Advantages Disadvantages

• Flexible and adaptable • Driven internally • Time and place to suit current need • Engages as many or as few staff as required • Can be resource intensive, or resource light

dependant upon the methodology adopted • Range of methodologies available, or can

generate own version • Relies on open and honest assessment of

current situation • Does not necessarily require the collation of

large data/evidence base • Provides a holistic framework for assessing

the overall management of the school • Looks at a balanced set of results indicators • Has a fundamental focus on customers • Has been developed from many years

academic research and practical experience • Requires positive trends in results over a

three or more year period � consistently • Accommodates other management tools,

techniques and frameworks. • Provides a common language for

benchmarking purposes. • Can integrate into business planning over a

shorter timescale. • Can have immediate results as well as

longer term improvement. • Provides a mechanism for continual

assessment and review.

• Relatively unknown concept within Higher Education • The Model does not reflect the language, challenges

or idiosyncrasies of the sector • There is relatively little guidance (other than from

trained specialists and consultants) as to how to apply the model in practice

• The range and choice of assessment methods can be daunting

• The jargon and language is a barrier at every level of the organisation (other than to those who have been trained)

• The focus is holistic and generalistic, it does not focus on specifics, such as subject provision, research, accreditation etc.

• If not reviewed or assessed by external assessors to the school, valuable external feedback is not gained.

• No means yet to evaluate against, or share learning experiences with the rest of the sector

4 Learning from these experiences

Key Similarities and Gaps These experiences have been drawn from the existing Subject Review method, which is changing post 2001. It has to be acknowledged that some of the issues from these experiences have already been integrated and are reflected by the new self-evaluation framework that will shortly be coming into practice - the �lighter-touch� approach to the Review process probably being the most significant of these changes. It is still important, however, to reflect on the lessons we have learnt, whilst taking into account our early assumptions of how the new methodology will work, and consider if there is still further potential for the models/frameworks to work together synergistically in the future. It is essential that we continue to look forward whilst learning from all knowledge gathered in the past, the EFQM Model having added a new dimension to this thinking. Given this reflection there appear to be a number of common elements emerging, which are indicated in Table 33, which includes potential areas for further convergence of the two in process terms.


Table 33 The synergy between processes, and areas for potential development

Activity/Process

EFQM Excellence Model Self-Assessment

QAA Subject Review (post 2001)

Potential areas for further synergy / shared learning

between the models Commitment to undertake assessment activity

Flexible � completely internally driven

Mandatory � externally driver with dates and subject areas

determined

Potential for further flexibility in the QAA framework, building on the �light-touch� approach, but allowing more freedom to

self-evaluate within each institution, with interim

assessments Preparation for assessment

Flexible � usually short term and can use a range of

assessment tools over an internally defined timescale

Assessment method defined by QAA in Handbook, within a

cycle defined by QAA

EFQM could be more prescriptive in terms of advising how to plan

assessments � the information is available, but from many

different sources. Timescale guidance also not highly

featured. Flexibility of assessment method

Very flexible - can mix and match methods according to

time and resource

Assessment method set by QAA - little flexibility

Need to establish greater flexibility within the QAA

framework, but potentially be a little more prescriptive for the using the EFQM Model in HE.

Undertaking the assessment process

Can be from 3 hours to 6 months � can be a 3 hour

workshop using a questionnaire, or the

completion of an evidence based award submission

document. Can include as many or as few people as

required.

Each subject area undertakes a comprehensive self-

evaluation over at least 12 months. A number of reviewer

days within a specific period are allocated.

Review will be taking place over approximately 3 months.

Opportunities to undertake a self-evaluation/assessment for both QAA and EFQM together

� the development of a common assessment

framework



Engages a wider number of people across subject groups

and central departments.

Reliance on evidence Reliant on evidence, although robustness of evidence

depends on assessment method employed (rigour vs

evidence/opinion)

Reliant on evidence and effective facilitation by Subject Review Facilitator and Subject

Contact � evidence collated and collected in a base room

Potential for QAA framework to adopt the �trust� ethos of the

EFQM assessment methodology, where

supporting information is not necessarily required (other

than for award submissions) Fits at which level of the organisation

Assessment methods can be applied on a personal level, at a team, department or whole organisation level. It has the

potential to be used at subject level, but has not yet

been tested in this way.

Applied only at subject level within schools, not

departments, although consideration given to the role

departments play in the student experience.

Potential for the QAA framework to engage further

with central departments (e.g. Finance) to track impact and experience right through the

core processes


Assess against self completed questionnaire or

pro-forma, which usually follow the criteria, sub-criteria

and possibly the areas to address.

Review against the self-evaluation document, subject

review Aide-memoire and external reference points.

Outcomes generated and how they are used

Outcomes generally in the form of action points, which

are allocated owners, and an action plan developed. Feed directly in business planning

if the timing is right. Can also obtain a score for

benchmarking. Supports an Institutional wide

assessment.

Self-evaluation document produced. Feedback used to

improve current practices, action plan developed and implemented. Approved or not approved, with report

published on the web by the QAA. Feeds into institutional

profile.

Sharing of best practice by published reports on the web is a good way of sharing best

practice and being honest with the customer.


5 Using the RADAR concept in Preparation for Subject Review

RADAR can be a useful diagnostic tool, in preparation for a QAA Subject Review, as part of a systematic process of annual self-assessment limited to continuous improvement in QSME and to business planning activities. It can help to identify the diversity of perceptions and develop consensus for review of the objectives of key QSME processes & policies which underpin the student experience. This can be used to create a self evaluation document and to identify relevant evidence base appropriate to reflect the QSME activities within the area under review. Using the �Approach� definitions according to the RADAR concept can help define clearly the processes /sub-processes / inter-process relationships and related resources which deliver the objectives. Identification of shortfalls in these, duplication of efforts, and issues which �fall between stools�, etc can also emerge. This can then lead to the: • identification of immediate corrective action / fine tuning of systems , i.e.

learning from the review preparation process, to address immediate problem issues to gain immediate improvements in student experience

• identification of changes to be planned and implemented in a short/medium

term time-window that encompasses the review process. These change plans and the degree of implementation to be evidenced during the review process.

The �Deployment� definition according to the RADAR concept can help when gathering evidence concerning consistency of deployment of polices, procedures and systems, e.g. unit files, action planning and associated feedback loops, mentoring and peer review, resources review, and external liaisons. The �Assessment� & Review RADAR definition can give a clear picture of process for assessment and reviews of policies, procedures and systems related to (old) QME and (new) QSME.

5.1 Summary RADAR used in this context provides for • Consistent approach to information and evidence gathering • Information/ evidence interpretation • Information/ evidence presentation • Immediate actions with respect to problems • Immediate actions with respect to review preparation process • Continuity to the post-review phase of continuous review and improvement This then facilitates reflection of a moving picture, rather than a snapshot.

6 Conclusions


As indicated throughout this section, this has been purely an analysis of Subject Review and self-assessment against the Excellence Model, rather than the whole QAA framework. The conclusions emerging from the analysis, therefore, should be considered in the wider context of how the two models/frameworks could work tangentially. There are a number of key points that emerge, however, that point to significant areas for improvement that can still be made between the use and application of the models when considered at school level: • The preparation time for a Subject Review is now more flexible with individual

Institutions having an opportunity to assess and review how they prepare. There is potential to use the EFQM Excellence Model® as a preparatory tool on an annual basis, giving consistent and strongly evidenced results in many areas.

• The Subject Review process requires positive management of change as a

result of the assessment through the implementation and integration of the actions. As the Excellence Model evaluates many of the people and leadership aspects, cultural and management change may be more fluid � i.e. it happens along the way as part of the process. There is potential to learn from this over-arching approach.

• One of the outcomes of Subject Review may be that staff/students will

change the way activities and task are undertaken, or the way relationships/partnerships are formed, but there is no guidance as to the sustainability of this in the longer term, unless an institution fails a review. Self-assessment using the Excellence Model process, once integrated into the business planning cycle, checks whether continuous improvement is being made in all areas on an on-going basis. Both can work together, with the Excellence Model providing the on-going check and balance of all general issues, with Subject Review providing a spot-check in specific areas.

• The scale and subject of assessment activity with Subject Review is

prescribed by QAA. There is no option to evaluate any other subject areas other than those stated, unless it is undertaken internally. Self-assessment using the Excellence Model can be applied freely to any part of the organisation at any time, as decided by the organisation. It can be as large or small as resource, time and strategy will allow.

• The outcome of the Subject Review provides a UK wide recognised external

validation of provision within the specified subject areas. This is an important benchmark across the sector against which provision can be compared. The outcome of a self-assessment using the Excellence Model provides the opportunity to benchmark within and outside the sector. This potentially adds more fully to the richness of learning and the possibility of gaining new ideas on �standard� management practices and processes from outside the sector.

• Subject Review focuses on the student experience as provided by a specific

subject area. This provides a great deal of scrutiny in specific areas over a short space of time, but can often ignore some of the wider implications and


cross-cutting issues that impact on this experience. The process of undertaking a self-assessment against the Excellence Model takes a wider and more holistic view, rather than the detail � for which there is little guidance.

• Subject Review usually involves a wide range of people from the School, and

from across the institution and is well co-ordinated to included views from all areas that may have an impact. The number and type of people involved in an Excellence Model assessment can vary and is dependant on the self-assessment method used by the institution. This means that a small number of people could undertake the assessment meaning that the evidence gathered is not reflective of the real situation.

• The importance of a central team/shared resource of knowledgeable people

is critical for both approaches. This encourages shared understanding and the interchange of best-practices within and across institutions.


SECTION 8

Conclusions


1 General Issues Firstly, it must be recognised that the EFQM Excellence Model® and the QAA Framework derive from different traditions and have different starting points and languages. EFQM should not be viewed as a substitute for QAA Review, at any level. EFQM is operative now in a large number of organisations who have chosen to use it as a management tool or approach: it is a potential integrator, which can help with the management of multiple requirements. The QAA Framework is actually applicable to a much smaller number of institutions but has to operate on a pragmatic basis (that is, one which is acceptable to both Government and the institutions being judged) in a highly political context. The QAA Framework is prescriptive, mandatory and costly in terms of the resources HEIs need to devote to adherence/compliance in a climate where accountability is of ever-increasing importance. Why then might an HEI choose to voluntarily engage with another Model concerned with quality, with all the attendant resource issues that entails? Although there are some similarities between the Excellence Model and the QAA Framework in terms of overall purpose, there are some significant differences in approach which are partly attributable to the differing primary purposes of the QAA and EFQM. The QAA has important public accountability responsibilities in relation to academic standards and quality assurance across the UK HE sector, as represented through its service level agreement with institutions and contracts with main funding bodies. Its mission is �to promote public confidence that quality of provision and standards of awards in higher education are being safeguarded and enhanced�. The EFQM, in contrast, is a membership based foundation which spans the private, public and voluntary sectors and to which subscription is entirely voluntary. HEIs are required to work within the QAA Framework. They are not required to either subscribe to EFQM or to use the EFQM Excellence Model® as a tool to support institutional management. The processes as indicated by the QAA or EFQM do not necessarily lead to the actual integration of the outcomes of the assessment activities directly into business planning. QAA require a �one year after� follow-up report after Continuation Audit/ Institutional Review which requires a description of action and an analysis of its effectiveness, and discussion of what has been prioritised by the institution. The production of an action plan, and the implementation of those actions is required and deemed as good practice, but there is no defined requirement within each framework that states how or when this should be undertaken on a continual basis. This is an area that could be more prescriptive for both. This lends itself to the consideration of an HE wide agreed �management approach� which is non-prescriptive but provides a broader management framework which interlocks intrinsically into the business planning of HEIs.

SECTION 8 CONCLUSIONS


2 The Excellence Model and the QAA Integrated Framework: Some Key Similarities and Differences

(a) Both the EFQM Model and the QAA Code of Practice are intended to provide

tools to support effective management, monitoring and ultimately quality improvement.

(b) The strongest synergy between the Model and the QAA Framework is to be found, possibly predictably, in the area of Criterion 5, Processes. There is least synergy in relation to 7 and 8, People Results and Society Results.

(c) The QAA Framework includes provision for sanctions/ threats, which the EFQM model does not. It has a public accountability element, which EFQM does not. It involves no scores, no awards, only external judgements that subject areas are approved or failing, or that it is possible to have overall, limited or no confidence in the way in which an HEI�s standards and quality responsibilities are discharged. EFQM on the other hand does involve scoring, always involves self-assessment and may involve external assessment if submission is made for an award, but there are no �public� judgements or direct externally-driven consequences, if an organisation is judged not to be performing well against the model�s criteria. If a good outcome were to be achieved by an organisation under either Model, however, they would be likely to wish to make use of that achievement for promotional purposes.

(d) Both models are ostensibly non-prescriptive frameworks and allow for diversity of organisational approach and practice. However the QAA Framework is clearly the more prescriptive of the two, given the development of the reference elements of the Framework with which HEIs are expected to comply (the Qualifications Framework, Subject Benchmarks, Programme Specifications, Progress Files, Code of Practice).

(e) Overall, the Excellence Model provides for greater emphasis on results than the QAA Framework but as the QAA Framework evolves to include an increasing concern with outcomes, this may gradually diminish.

(f) The QAA Framework is very much concerned with standards and the student experience. These matters are covered to some extent by the criteria and sub-criteria of the Excellence Model but it may be felt that their complexity and importance is difficult to represent adequately, against the Model. HE students are not simply �customers� but also partners in learning, and may in some cases themselves be providers, or producers.

(g) Mapping the complex relationships between all the various stakeholders within HE (e.g. staff, students, researchers, examiners, PSBs, employers, funding bodies, collaborative partners) is challenging, against the terminology of the Excellence Model (people, customer, partner).

(h) It is likely that if Subject Review is �downgraded� within the QAA Framework, Institutional Review will correspondingly increase in significance. If this is the case, it is not clear whether the trend towards review of the more senior levels and strategic approaches of the institution, which has been evident from Academic Audit through to Institutional Review will continue, or whether there


may be a demand for greater emphasis on practice at School/ Faculty/ Department/ Subject level, or both. Changes to SR and IR could impact quite significantly, on the way in which these elements of the Framework map against the Excellence Model.

(i) Careful consideration needs to be given to the way in which the Excellence Model is used in an HE context, where reference is frequently made to the notion of �academic freedom�. A single management model may be anathema to the academic mindset, as evidenced, for example, by the reluctance within the sector to embrace IiP wholeheartedly. However, academic staff do also chafe against QAA requirements and it may not be necessary for the majority of academic staff to be familiar with or accept the Excellence Model, for it to be able to be adopted by an institution.

(k) One of the starkest contrasts between the Excellence Model and the QAA Framework, when applied in practice, is the outcomes achieved by organisations. Most organisations which use the Excellence Model as a self-improvement tool initially self-assess and achieve very low scores, using the RADAR matrix method. Scores of below 300/1000 are the norm. Even organisations externally judged as excellent against the Excellence Model criteria, are only achieving scores of around 600-700/1000. Most organisations would define the pursuit of excellence as a �journey� which will take several years. The QAA Framework, however, ostensibly appears to pitch �excellence� at a different level. Most HEIs achieve �excellent� ratings under the current Subject Review methodology. Very, very few HEIs have received such negative outcomes at Continuation Audit that a �no confidence� judgement, to use the future IR judgement terminology, would have been the final outcome. It is in reviews of collaborative provision and overseas collaborative provision that the most negative judgements by QAA have tended to occur. The time scale permitted by QAA for rectification of deficiencies is one year � much shorter than the several years� improvement trajectory expected under the Excellence Model. Why then are the outcomes so different? A series of potentially highly controversial questions may be posed but this report ventures no answers. For example:

• are there two different concepts of what constitutes �excellence�

operating in these two models? • could UK HE (or UK Plc) afford for the QAA Framework�s processes to

publicly identify deficiencies, weaknesses, or improvement areas on the scale which the Excellence Model appears to do?

• is the Excellence Model over-demanding? • is the QAA Review model insufficiently objective and systematic and

too heavily dependent on subjective peer review? To what extent is it able to achieve finely differentiated, objective judgements which are meaningful in the context of a diverse HE sector?

• to what extent can standardisation of approach and encouragement of good practice via non-prescriptive frameworks, guarantee academic standards and quality of provision/ student experience?

• might it be useful for the QAA Framework to be developed and improved in the light of the EFQM model, to become almost a �sector specific� tailored variant of the EFQM Excellence Model®?


• and finally � if UK HEIs are subject to a mandatory external review system can they afford to resource a different quality review Model (EFQM Excellence) internally � or, can they afford not to if it is accepted that there is an ever-increasing need to bring closer together, business and academic planning, management and review?

Although the Excellence Model is �customer focused� a detailed mapping of the QAA Code against the Model suggests that the Excellence Model�s criteria are insufficiently sophisticated to do justice to the richness and complexity of the way in which HEIs support and provide for their students (customers), as defined, for example, in the QAA Code.

3 The Scope for Development of the QAA Framework and a new EFQM HE Excellence Model

(a) Consideration might be given � by both HEIs and QAA � to the benefits which might flow for academic quality and standards, if more attention were to be paid, in HEIs� quality and standards management and enhancement arrangements, to those areas where the EFQM Model�s criteria are more explicitly demanding than the QAA Framework � for example in relation to Leadership, People, and all the Results criteria. More controversially, consideration might also be given to broadening the remit of the QAA Framework to consider a broader spectrum of management issues, including management of (for many HEIs increasingly scarce) resources to deliver standards and quality, unless this were to be considered politically unacceptable or to duplicate other bodies� requirements of HEIs.

(b) The terminology used in the Excellence Model is clearly very different from that used in the QAA Framework. The two have emerged from different traditions and their main foci are different. However, it may be interesting to speculate whether the mapping of the foci and concerns of the EFQM Model and the QAA Academic Review Framework would show even greater synergy than it does already, if an �Education Sector� or even �HE Sector� version of the EFQM Model were to be developed? Any such tailored model might usefully incorporate, for example, a sharper focus on standards and address the issue of learning partnerships between students and staff.

4 A Rationale for Implementing the EFQM Excellence Model in HEIs?

(a) To what extent should the QAA Framework be viewed as just one element in a range of arrangements concerned with public accountability, and to what extent might HEIs be better placed to respond to that whole range, by using the Excellence Model as part of their routine institutional management? Might HEIs benefit if they were to use the Excellence Model as a basis for promoting, within their own internal systems, a more systematic interrelationship between resource planning and academic management? Might an HEI which enthusiastically adopted the Excellence Model as a way of working, be more likely to develop an embedded culture of continuous learning and self-improvement which would inevitably lead to improved performance on delivery of standards and quality? To what extent is continuous learning and improvement already well-embedded in HEIs without the stimulus of the Excellence Model?


(b) There are such strong synergies in some foci of the EFQM Excellence Model® and QAA IR/ CA/ CPR/ SR that it may be beneficial for HEIs to consider how they might capitalise on the learning and improvement planning derived from use of the EFQM model, to improve the ways in which they routinely manage for academic quality and standards and potentially, to reduce the burden of preparation for periodic external reviews by QAA.

(c) Whilst it is not entirely clear where the outcome of the current re-opening of thinking on the future approach to Subject Review will lead, the simple fact of virtually �continuous engagement� with QAA for HEIs in future, seems unlikely to change. It is too disruptive to routine business, to prepare for each engagement with QAA requirements on a project-by project basis. There is therefore, particularly for large HEIs, a powerful argument for a fully-integrated internal system which promotes systematic deployment of policy, strategy and procedures and promotes continuous improvement.

(d) EFQM as an approach to self-evaluation at any level � institution, site, School/Department can provide a mechanism for: • All stakeholders understanding where Quality Standards Management

Enhancement (QSME) fits in, (Its purpose, role, objectives, demands, benefits etc) to everything else that the Institution/Site/School/Depts does. This can play a key role in developing stakeholder�s (Staff, Managers, Students etc) perspective on QSME as being of value and relevance.

• A monitoring/evaluation function/process for QSME at all levels to facilitate fine tuning to �local� QSME needs by providing consideration of those elements that lie outside QSME�s immediate boundary (eg management activities in the School/Department).

• Providing a joined-up way of thinking between annual business planning and QSME so that links, problem-cause-effect relationships and synergistic action plans can be identified.

• Subject Review is focussed on academic output, not on models of management in the Schools, which is a secondary issue. Annual business planning tends to be the opposite. EFQM self-assessment, integrated into the Annual Planning cycle can provide the connection between them.

• The results, approach, deployment, assessment and review (RADAR) method can be effective in diagnosing activities or processes at all levels which are part of, or impact on, QSME issues.

(e) The current Subject Review process produces a cycle of intense activity that may be detrimental in the following ways:

• diversion of resources � staff, rooms, documents from normal activities. • Workload and pressure,, particularly on key staff. • temptation for quick fixes, fakery, �wet paint syndrome� to meet process of

review rather than medium/long term benefits to students. • the above impacting negatively on staff perception of quality and QSME

processes.


(f) EFQM as an integrated element of an institution site, School/Department self-review offers a mechanism for:

• Promoting QSME by illuminating its role etc. in overall activities. • Facilitating continuous improvement in QSME that is informed by all

stakeholders. • Facilitating continuous improvement in QSME in a manner such that all

direct/indirect impacts on other aspects of the institution/site/School/Department are clear.

• Facilitates continuous improvement in other aspects of the Institution/Site/School/Dept in a manner such that all direct/indirect impacts on QSME are clear.

• The above improving all the aspects of activities within the score of QAA Reviews in a way which provides for better review results through genuine, continuous improvement rather than through intense activities.


SECTION 9

Further Work


During the course of this project, the remit for which is specific and limited, some thought has been given to possible follow-on projects, and some ideas for further consideration are given below • Consider extending this initial mapping exercise to cover the other main

elements of the QAA Framework, viz, Subject Benchmarking, the HE Qualifications Framework, Programme Specifications and Progress Files.

• Map the QAA IR and CPR approaches fully against the EFQM Model, when

they are published. • Consider the impact on the perceived synergies/ gaps between the EFQM

Model and QAA SR, of the final decision about the way in which SR is to be conducted.

• School/ Subject Area which has undertaken an EFQM Self-Assessment and

been subject to QAA SR to compare the exercises and experiences and reflect on the value of the former, in preparing for the latter.

• Explore the scope for and feasibility / value of producing an �HE� version of

the EFQM Model. • Encourage discussion with QAA (by whom?) to establish whether the EFQM

Excellence Model might usefully influence further development of the QAA Framework.

• More detailed consideration of the potential benefits to HEIs (as regards their

performance against QAA requirements) of implementing the EFQM Excellence Model® as a tool to support and promote good management of academic quality and standards.

• In recognition of the very specific context within which HEIs operate, including

in particular pressures to maintain academic standards, consideration might be given to a potential follow-on project involving a number of HEIs in reviewing their experience of working with both the QAA Framework and the EFQM model.

• Consider mapping against the Excellence Model the totality of external

requirements upon HEIs (HEFCE, PSBs, QAA etc) and the experience of responding to them.

• Consider exploring the costs of implementing the EFQM Excellence Model®,

the costs of responding to the QAA Framework and the costs of responding to the HE quality agenda in its widest sense, within one or more HEIs.

SECTION 9 FURTHER WORK


SECTION 10

Appendices


Appendix A Deming�s 14 Points

1. Create consistence of purpose with a plan - The objective is constancy of purpose for continuous improvement. An unwavering commitment to quality must be maintained by management. Quality, not short-term profit, should be at the heart of organisation purpose. Profit will follow when quality becomes the objective and purpose.

2. Adopt the new philosophy of quality � The modern era demands ever-increasing quality as a means of survival and global competitiveness. Inferior material, poor workmanship, defective products, and poor service must be rejected. Reduction of defects is replaced by elimination of defects. The new culture of quality must reflect a commitment to quality and must be supported by all employees.

3. Cease dependence on mass inspection - Quality can�t be inspected in; it must be built in from the start. Defects discovered during inspection cannot be avoided � it is too late; efficiency and effectiveness have been lost, as has continuous process improvement. Continuous process improvement reduces costs incurred by correcting errors that should not have been made in the first place.

4. End the practice of choosing suppliers based on price � Least cost is not necessarily the best cost. Buying from a supplier based on low cost rather than a quality/cost basis defeats the need for a long-term relationship. Vendor quality can be evaluated with statistical tools.

5. Identify problems and work continuously to improve the system � Continuous improvement of the system requires seeking out methods for improvement. The search for quality improvement is never-ending and results from studying the process itself, not the defects detected during inspection.

6. Adopt modern methods of training on the job � Training involves teaching employees the best methods of achieving quality in their jobs and the use of tools such as statistical quality control.

7. Change the focus from production numbers (quantity) to quality � The focus on volume of production instead of quality leads to defects and rework that may result in inferior products at higher costs.

8. Drive out fear � Employees need to feel secure in order for quality to be achieved. Fear of asking questions, reporting problems, or making suggestions will prevent the desired climate of openness.

9. Break down barriers between departments � When employees perceive themselves as specialists in one function or department without too much regard for other areas, it tends to promote a climate of parochialism and set up barriers between departments. Quality and productivity can be improved when there is open communication and coordination based on the common organisation goals.

10. Stop requesting improved productivity without providing such methods to achieve it � Continuous improvement as a general goal should replace motivational or inspirational slogans, signs, exhortations and work force targets. The major cause of poor productivity and quality is the management systems, not the work force. Employees are frustrated when exhorted to achieve results that management systems prevent them from achieving.

11. Eliminate work standards that prescribe numerical quotas � Focus on quotas, like a focus on production, may encourage and reward people for numerical targets, frequently at the expense of quality.

SECTION 10 APPENDICES


12. Remove barriers to pride of workmanship � A major barrier to pride of workmanship is a merit or appraisal system based on targets, quotas or some list of personal traits that have little to do with incentives related to quality. Appraisal systems that attempt to coerce performance should be replaced by systems that attempt to overcome obstacles imposed by inadequate materials, equipment or training.

13. Institute vigorous education and retraining � Deming emphasises training, not only in the methods of the specific job but in the tools and techniques of quality control, as well as instruction in teamwork and the philosophy of a quality culture.

14. Create a structure in top management that will emphasise the preceding thirteen points every day � An organisation that wants to establish a culture based on quality needs to emphasise the preceding thirteen points on a daily basis. This usually requires a transformation in management style and structure. The entire organisation must work together to enable a quality culture to succeed. Source: Principles of Total Quality; Second Edition 1998, Jill A Swift, Joel E Ross, Vincent K Omachonu, pp7-8


Appendix B Juran�s 10 Key Steps

1. Create awareness of the need and opportunity for quality improvement. 2. Set goals for continuous improvement. 3. Build an organisation to achieve goals by establishing a quality council,

identifying problems, selecting a project, appointing teams and choosing facilitators.

4. Give everyone training. 5. Carry out projects to solve problems. 6. Report progress. 7. Show recognition. 8. Communicate results. 9. Keep a record of successes. 10. Incorporate annual improvements into the company�s regular systems and

processes and thereby maintain momentum. Source: Total Quality Management; Second Edition 2000, John Bank, pp 93


Appendix C Crosby�s 14 Steps

1. Make clear management�s commitment to quality. 2. Set up quality improvement teams with representatives from each department. 3. To set in place quality measurement to provide a display of current and potential

non-conformance problems. 4. To determine the �cost of quality� and explain how to use it as a management

tool. 5. To raise the level of awareness and personal concern for the company�s quality

reputation for all employees. 6. To take corrective action on the problems raised in the previous steps. 7. To plan a zero defects programme. 8. To train supervisors actively to carry out their part in the total quality improvement

process. 9. To hold a zero defects day to create an event that will let all employees know

through a personal experience that there has been a change. 10. Goal setting and encouraging individuals and groups to set improvement goals.

Source: Total Quality Management; Second Edition 2000, John Bank, pp 99 - 101


Appendix D Oakland�s 5 Points

1. First one must identify customer-supplier relationships. 2. Then set up a system to manage processes. 3. Change the company culture from what it was to a TQM culture. 4. Improve communications company-wide. 5. Demonstrate commitment to quality.

Source: Total Quality Management; Second Edition 2000, John Bank, pp 103


Appendix E Fundamental Concepts of Excellence

1 Results Orientation Excellence is dependant upon balancing and satisfying the needs of all relevant stakeholders (this includes the people employed, customers, suppliers and society in general as well as those with financial interests in the organisation).

2 Customer Focus The customer is the final arbiter of product and service quality and customer loyalty, retention and market share gain are best optimised through a clear focus on the needs of current and potential customers.

3 Leadership and Constancy of Purpose The behaviour of an organisation�s leaders creates a clarity and unity of purpose within the organiastion and an environment in which the organisation and its people can excel.

4 Management by Processes and Facts Organisations perform more effectively when all inter-related activities are understood and systematically managed and decisions concerning current operations and planned improvements are made using reliable information that includes stakeholder perceptions.

5 People Development and Involvement The full potential of an organisation�s people is best released through shared values and a culture of trust and empowerment, which encourages the involvement of everyone.

Mappin

g th

e Q

AA

Fra

mew

ork

and

13

1

Conso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Ap

pe

nd

ix F

M

ap

pin

g o

f th

e H

EQ

C G

uid

elin

es

on

Qu

alit

y A

ssu

ran

ce 1

99

4 w

ith t

he

EF

QM

Exc

elle

nce

Mo

de

l®

EF

QM

Ex

ce

lle

nc

e

Mo

de

l C

rite

rio

n u

se

d

as

ba

sis

fo

r s

elf

-a

ss

es

sm

en

t a

cti

vity

HE

QC

Gu

ide

lin

es

on

Q

A 1

99

4:

Ch

ec

kli

st

for

QA

Sys

tem

s1

5

Cri

teri

on

1:

L

ea

de

rsh

ip

Ho

w l

ea

de

rs d

eve

lop

a

nd

fa

cil

ita

te t

he

a

ch

ieve

me

nt

of

the

m

iss

ion

an

d v

isio

n,

de

velo

p v

alu

es

re

qu

ire

d f

or

lon

g t

erm

s

uc

ce

ss

an

d

imp

lem

en

t th

es

e v

ia

ap

pro

pri

ate

ac

tio

ns

a

nd

be

ha

vio

urs

, a

nd

a

re p

ers

on

all

y in

volv

ed

in

en

su

rin

g t

ha

t th

e

org

an

isa

tio

n's

m

an

ag

em

en

t s

yste

m i

s

de

velo

pe

d a

nd

im

ple

me

nte

d

Cri

teri

on

2:

Po

lic

y a

nd

Str

ate

gy

Ho

w t

he

o

rga

nis

ati

on

im

ple

me

nts

its

m

iss

ion

an

d

vis

ion

via

a c

lea

r s

tak

eh

old

er

foc

us

ed

str

ate

gy,

s

up

po

rte

d b

y re

leva

nt

po

lic

ies

, p

lan

s,

ob

jec

tive

s,

targ

ets

an

d

pro

ce

ss

es

Cri

teri

on

3:

Pe

op

leH

ow

th

e

org

an

isa

tio

n

ma

na

ge

s,

de

velo

ps

a

nd

re

lea

se

s t

he

k

no

wle

dg

e a

nd

fu

ll

po

ten

tia

l o

f it

s

pe

op

le a

t a

n

ind

ivid

ua

l, t

ea

m-

ba

se

d a

nd

o

rga

nis

ati

on

-w

ide

le

vel,

an

d p

lan

s

the

se

ac

tivi

tie

s i

n

ord

er

to s

up

po

rt i

ts

po

lic

y a

nd

str

ate

gy

an

d t

he

eff

ec

tive

o

pe

rati

on

of

its

p

roc

es

se

s.

Cri

teri

on

4:

P

art

ne

rsh

ips

a

nd

Re

so

urc

es

H

ow

th

e

org

an

isa

tio

n

pla

ns

an

d

ma

na

ge

s i

ts

ex

tern

al

pa

rtn

ers

hip

s

an

d i

nte

rna

l re

so

urc

es

in

o

rde

r to

su

pp

ort

it

s p

oli

cy

an

d

str

ate

gy

an

d t

he

e

ffe

cti

ve

op

era

tio

n o

f it

s

pro

ce

ss

es

.

Cri

teri

on

5:

P

roc

es

se

s

Ho

w t

he

o

rga

nis

ati

on

d

es

ign

s,

ma

na

ge

s a

nd

im

pro

ves

its

p

roc

es

se

s i

n

ord

er

to s

up

po

rt

its

po

lic

y a

nd

s

tra

teg

y a

nd

fu

lly

sa

tis

fy,

an

d

ge

ne

rate

in

cre

as

ing

va

lue

fo

r, i

ts

cu

sto

me

rs a

nd

o

ther

s

tak

eh

old

ers

Cri

teri

on

6:

C

us

tom

er

Re

su

lts

W

ha

t th

e

org

an

isa

tio

n

is a

ch

ievi

ng

in

re

lati

on

to

it

s e

xte

rna

l c

us

tom

ers

.

Cri

teri

on

7:

P

eo

ple

R

es

ult

s

Wh

at

the

o

rga

nis

ati

on

is

ac

hie

vin

g

in r

ela

tio

n t

o

its

pe

op

le.

Cri

teri

on

8:

S

oc

iety

R

es

ult

s

Wh

at

the

o

rga

nis

ati

on

is

ac

hie

vin

g

in r

ela

tio

n t

o

loc

al,

n

ati

on

al

an

d

inte

rna

tio

na

l s

oc

iety

as

a

pp

rop

ria

te.

Cri

teri

on

9:

K

ey

Pe

rfo

rma

nc

e

Re

su

lts

W

ha

t th

e

org

an

isa

tio

n

is a

ch

ievi

ng

in

re

lati

on

to

it

s p

lan

ne

d

pe

rfo

rma

nc

e.

Est

ablis

hing

a

fram

ew

ork

for

Qualit

y (1

,2)

✔

✔

✔

Revi

ew

ing Q

A

Sys

tem

s (3

)

✔

Adm

issi

ons

Pol

icie

s (4

)

✔

✔

✔

Adm

issi

ons

requir

em

ents

(5)

✔

✔

Info

for

pro

spect

ive

students

(6, 7)

✔

✔

✔

? ✔

Pre

-entr

y guid

ance

(8)

✔

✔

S

ele

ctio

n P

roce

ss (

9)

✔

✔

✔

Faci

litatin

g s

tudent

entr

y (1

0)

✔

✔

? ✔

QA

and the d

ivers

ity o

f

HE

(11, 12)

✔

✔

✔

✔

15 H

EQ

C: G

uid

elin

es

on Q

ualit

y A

ssura

nce

(1994)

Mappin

g th

e Q

AA

Fra

mew

ork

and

13

2

Conso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Ap

pe

nd

ix F

(C

on

tinu

ed

) Ext

ern

al P

rogra

mm

e

appro

val (

13)

✔

✔

? ✔

Inte

rnal p

rogra

mm

e

appro

val (

14-1

7)

? ✔

✔

Pro

gra

mm

e in

fo f

or

students

(18)

✔

✔

Teach

ing a

nd

Learn

ing (

19)

✔

✔

? ✔

Eva

luatio

n o

f pro

gra

mm

es

of

study

(20, 21)

✔

✔

Eva

luatio

n o

f T

each

ing a

nd

Learn

ing (

22)

✔

✔

✔

✔

Sta

ff A

ppoin

tment (2

3)

✔

Sta

ff d

eve

lopm

ent and

train

ing (

24)

✔

? ✔

Sta

ff A

ppra

isal (

25)

✔

Qualit

y of

colla

bora

tive

arr

angem

ents

(26)

✔

✔

PG

R s

tudents

(27, 28)

✔

✔

✔

Stu

dent S

upport

S

ervi

ces

✔

✔

✔

Stu

dent G

rieva

nce

(3

0)

✔

✔

✔

Stu

dent P

rogre

ss (

31)

✔

✔

✔

Stu

dent A

ssess

ment

(32, 33)

✔

Appeals

(34)

✔

✔

✔

Ext

ern

al E

xam

iners

(3

5)

✔

✔

✔

Mappin

g th

e Q

AA

Fra

mew

ork

and

13

3

Conso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Ap

pe

nd

ix G

HE

QC

: G

uid

elin

es

on

Qu

alit

y A

ssu

ran

ce 1

99

6:

Su

mm

ary

of

Prin

cip

les16

ma

pp

ed

aga

inst

th

e E

FQ

M E

xce

llen

ce M

od

el®

M

easu

rem

ent

Mat

rix

EF

QM

Exc

elle

nce

Mo

del

C

rite

ria

HE

QC

Su

mm

ary

of

pre

cep

ts

Cri

teri

on

1:

Lea

der

ship

H

ow

lea

der

s d

evel

op

an

d

faci

lita

te t

he

ach

ieve

men

t o

f th

e m

issi

on

an

d v

isio

n,

dev

elo

p v

alu

es r

equ

ired

fo

r lo

ng

ter

m s

ucc

ess

and

im

ple

men

t th

ese

via

app

rop

riat

e ac

tio

ns

and

b

ehav

iou

rs,

and

are

p

erso

nal

ly i

nvo

lved

in

en

suri

ng

th

at t

he

org

anis

atio

n's

m

anag

emen

t sy

stem

is

dev

elo

ped

an

d

imp

lem

ente

d

Cri

teri

on

2:

Po

licy

an

d S

trat

egy

Ho

w t

he

org

anis

atio

n

imp

lem

ents

its

m

issi

on

an

d v

isio

n

via

a cl

ear

stak

eho

lder

fo

cuse

d

stra

teg

y, s

up

po

rted

b

y re

leva

nt

po

lici

es,

pla

ns,

ob

ject

ives

, ta

rget

s an

d

pro

cess

es

Cri

teri

on

3:

Pe

op

le

Ho

w t

he

org

anis

atio

n

man

ages

, d

evel

op

s an

d r

elea

ses

the

kno

wle

dg

e an

d f

ull

p

ote

nti

al o

f it

s p

eop

le

at a

n i

nd

ivid

ual

, te

am-

bas

ed a

nd

o

rgan

isat

ion

-w

ide

leve

l, a

nd

pla

ns

thes

e ac

tivi

ties

in

ord

er t

o

sup

po

rt i

ts p

oli

cy a

nd

st

rate

gy

and

th

e ef

fect

ive

op

erat

ion

of

its

pro

cess

es.

Cri

teri

on

4:

P

artn

ersh

ips

and

R

eso

urc

es

Ho

w t

he

org

anis

atio

n p

lan

s an

d m

anag

es i

ts

exte

rnal

p

artn

ersh

ips

and

in

tern

al r

eso

urc

es

in o

rder

to

su

pp

ort

it

s p

oli

cy a

nd

st

rate

gy

and

th

e ef

fect

ive

op

erat

ion

o

f it

s p

roce

sses

.

Cri

teri

on

5:

P

roce

sses

H

ow

th

e o

rgan

isat

ion

d

esig

ns,

man

ages

an

d i

mp

rove

s it

s p

roce

sses

in

ord

er

to s

up

po

rt i

ts

po

licy

and

str

ateg

y an

d f

ull

y sa

tisf

y,

and

gen

erat

e in

crea

sin

g v

alu

e fo

r, i

ts c

ust

om

ers

and

oth

er

stak

eho

lder

s

Cri

teri

on

6:

C

ust

om

er

Res

ult

s W

hat

th

e o

rgan

isat

ion

is

ach

ievi

ng

in

re

lati

on

to

its

ex

tern

al

cust

om

ers.

Cri

teri

on

7:

P

eop

le R

esu

lts

Wh

at t

he

org

anis

atio

n i

s ac

hie

vin

g i

n

rela

tio

n t

o i

ts

peo

ple

.

Cri

teri

on

8:

S

oci

ety

Res

ult

s W

hat

th

e o

rgan

isat

ion

is

ach

ievi

ng

in

re

lati

on

to

lo

cal,

nat

ion

al

and

in

tern

atio

nal

so

ciet

y as

a

pp

rop

ria

te.

Cri

teri

on

9:

K

ey

Per

form

ance

R

esu

lts

Wh

at t

he

org

anis

atio

n i

s ac

hie

vin

g i

n

rela

tio

n t

o i

ts

pla

nn

ed

per

form

ance

.

1. A

fra

mew

ork

for

Qual

ity

✔

✔

✔

✔

✔

✔

2. E

ntr

y to

Hig

her

Educa

tion

✔

✔

✔

✔

✔

✔

3. P

rogra

mm

e D

esi

gn,

appro

val a

nd r

evi

ew

✔

✔

✔

✔

4. T

each

ing a

nd

learn

ing

✔

✔

✔

✔

✔

5. S

tudent

deve

lopm

ent and

support

✔

✔

✔

6. S

tudent

com

munic

atio

n a

nd

repre

senta

tion

✔

✔

✔

✔

✔

✔

7. S

tudent

Ass

ess

ment

✔

✔

✔

✔

✔

✔

8. E

xtern

al E

xam

iners

✔

✔

✔

? ✔

16 H

EQ

C: G

uid

elin

es

on Q

ualit

y A

ssura

nce

(1996)

Mappin

g th

e Q

AA

Fra

mew

ork

and

13

4

Conso

rtiu

m f

or

Exc

elle

nce

th

e E

xcelle

nce

Model

in

Hig

her

Educa

tion

Ap

pe

nd

ix G

(co

ntin

ue

d)

9. S

taff

ing

✔

✔

✔

✔

✔

? ✔

✔

10. R

ese

arc

h D

egre

e

Stu

dents

✔

✔

✔

✔

✔

11. C

olla

bora

tive

Pro

visi

on

✔

✔

✔

✔

✔

Ta

ble

8