ii

CONTENTS

Page

LIST OF MOST COMMONLY USED ABBREVIATIONS iv

FOREWORD vi

1. INTRODUCTION 1

2. INTERNAL ASSESSMENT AS A CORE PROCESS IN QUALITY ASSURANCE 2

3. THE EUROPEAN REFERENCE FRAMEWORK 7

4. INTERNAL QUALITY ASSURANCE SYSTEMS WITHIN THE NATIONAL JURIDICAL FRAMEWORK 12

5. EUROPEAN TRENDS WITHIN THE FIELD OF QUALITY ASSURANCE 17

5.1 The European dimension of quality assurance 17 5.2 The influence of the Bologna Process 19 5.3 The dynamics of external quality assurance 21

5.4 From quality assurance to quality enhancement 26 5.5 Future directions and challenges 30

6. INTERNAL QUALITY ASSURANCE SYSTEMS – THE EUROPEAN PANORAMA 33

6.1 The implantation of internal quality assurance systems 33

6.2 The valorisation of internal quality assurance systems 36

6.3 Auditing internal quality assurance systems 40

7. INSTITUTIONAL AUDIT PROCESSES IN EUROPE 43

7.1 Institutional audit processes – Case studies 43

7.1.1 ‘Academic Infrastructure’ – External standards for quality assurance in the United Kingdom 43

7.1.2 ‘Quality Enhancement Framework’ – A radical approach to quality in Scotland 48

7.1.3 ‘Quality Reform’ – The Norwegian system 51

iii

7.1.4 Quality Network – The participation of the universities in the Swiss model 55

7.1.5 Postponement of audits – A backward step in Denmark 57 7.1.6 Respect for Autonomy – The Finnish model 59 7.1.7 ‘System Accreditation’ – A new model for accreditation in

Germany 62 7.1.8 Conciliating support and audit– The Austrian Quality Assurance

Agency (AQA) 69 7.1.9 Earning trust – The case of the Netherlands and Flanders 73 7.1.10 The AUDIT programme – Spain 76

7.2 Core characteristics of the institutional audit processes – A comparative analysis 84

7.2.1 Scope and motivation behind audit processes 84 7.2.2 Criteria and methodology 87 7.2.3 Consequences of audit processes 90 7.2.4 Follow-up 92 7.2.5 Summary table 92

8. CONCLUSIONS 95

8.1 Specification of internal quality assurance systems 95

8.1.1 Proposal of a frame of reference for internal quality assurance systems 98

8.2 Core elements for the certification of internal quality assurance systems 103

8.3 Suggestions for future work 107

REFERENCES 110

ANNEX – PART 1 OF THE EUROPEAN STANDARDS AND GUIDELINES 117

iv

LIST OF MOST COMMONLY USED ABBREVIATIONS

A3ES Agência de Avaliação e Acreditação do Ensino Superior (Portugal)

ACQUIN Accreditation, Certification and Quality Assurance Institute (Germany)

ACSUG Axencia para a Calidade do Sistema Universitario de Galicia

ADISPOR Associação das Instituições Superiores Politécnicas Portuguesas

AERES Agence d’Evaluation de la Recherche et de l’Enseignement Supérieur (France)

AHELO Assessing Higher Education Learning Outcomes (OCDE)

ANECA Agencia Nacional de Evaluación de la Calidad y Acreditación (Spain)

APESP Associação Portuguesa do Ensino Superior Privado

AQA Austrian Agency for Quality Assurance

AQAS Agency for Quality Assurance by Accreditation of Study Programmes (Germany)

AQU Agência per a la Qualitat del Sistema Universitari de Catalunya

BFUG Bologna Follow-up Group

CNAVES Conselho Nacional de Avaliação do Ensino Superior

CNE Comité National d’Evaluation (France)

CRE Association of European Universities

CRUP Conselho de Reitores das Universidades Portuguesas

EHEA European Higher Education Area

ELIR Enhancement-led Institutional Review (Scotland)

ENQA European Association for Quality Assurance in Higher Education

EQAR European Quality Assurance Register

ESG Standards and Guidelines for Quality Assurance in the European Higher Education Area

ESIB National Unions of Students in Europe

v

ESU European Students’ Union

EUA European University Association

EURASHE European Association of Institutions in Higher Education

EVA Danish Evaluation Institute

FINHEEC Finish Higher Education Evaluation Council

FUP Fundação das Universidades Portuguesas

GAC German Accreditation Council

HEFCE Higher Education Funding Council – England

HEI Higher education institutions

HRK Hochsschulrektorenkonferenz (German Rectors’ Conference)

KMK Kultusminister Konferenz (Conference of Ministers for Education and Culture – Germany)

NOKUT Norwegian Agency for Quality Assurance in Education

NVAO Accreditation Organisation of the Netherlands and Flanders

OAQ Centre of Accreditation and Quality Assurance of the Swiss Universities

QAA Quality Assurance Agency for Higher Education (United Kingdom)

QAHECA Quality Assurance for the Higher Education Change Agenda Project (EUA)

QEF Quality Enhancement Framework (Scotland)

RJIES Regime jurídico das instituições de ensino superior

SFC Scottish Funding Council

SUC Swiss University Conference

TQA Teaching Quality Assessment (United Kingdom)

ZEvA Central Evaluation and Accreditation Agency Hanover (Germany)

vi

FOREWARD

The mission of the Agência de Avaliação e Acreditação do Ensino Superior (A3ES) is to assure the quality of higher education in Portugal, through evaluation and accreditation of its study cycles and through other types of evaluation of a scientific nature, and thus to ensure Portugal’s integration in the European higher education quality assurance system.

The A3ES is conscious that its action must be underpinned by research and a continuous effort towards improvement. In order to fulfill these objectives, a research and analysis group was set up within A3ES, which is responsible for data collection, information processing, analytical studies and other types of research, within the area of quality assurance in higher education.

Within the terms of the legal Framework which instituted the current system of evaluation and accreditation in higher education, the essential task to be undertaken by A3ES is the promotion and dissemination of a culture of quality within higher education institutions. In order to do this, A3ES will publish a series of studies on themes linked to the quality of higher education systems, which will serve as a basis for permanent discussion with Higher Education Institutions and to make available information concerning the methods and procedures adopted by the agency in the course of its activity.

Within the terms of Law 38/2007, it is the responsibility of higher education institutions to adopt quality assurance policies and procedures for their implementation; to develop a culture of quality and of quality assurance, and to promote and implement a strategy for continuous enhancement of quality. The same Law determines that external evaluation procedures “should take into account the efficacy of the internal quality assurance procedures”. Moreover, in the preamble to Decree-Law 369/2007, one of the mainstays of the new system of evaluation and accreditation is defined as “The demand for the development and implementation by higher education institutions of their own quality assurance systems, which may be subject to certification”

The legislation thus reaffirms that the quality of teaching is primarily the responsibility of each higher education institution, and that they must create internal structures and procedures to assure this quality. It will be the responsibility of A3ES to support the institutions in the implementation of their internal quality assurance systems and to undertake audits with a view to certification of the internal quality procedures of the institutions.

vii

The first publication by A3ES is, very appropriately, a study of European Processes for Assessment and Certification of Internal Quality Assurance Systems, by Professor Sérgio Machado dos Santos, whose expertise in the area is extremely well known. The aim of this study is to make a comparative analysis of internal quality assurance systems implemented in European countries of reference, so as to provide institutions with a basis on which to establish of their own internal systems. It will also underpin the work of supporting the institutions that A3ES must undertake in the ambit of its competences.

1. INTRODUCTION

The A3ES, in its objectives and activity plans, foresees the establishment of institutional audit mechanisms leading to the possibility of certification of internal quality assurance systems within national higher education institutions, as an essential tool for a subsequent simplification of procedures in the external assessment and accreditation processes which apply to institutions and programmes. The priority task for the Agency is, however, by legal requirement, to promote the accreditation, in the short term, of all the degree programmes that are already operating in all higher education institutions, together with the ex-ante accreditation of new study cycles. Therefore, preparation of the audit mechanisms will only start in 2010, for implementation in 2011/2012.

This study corresponds to a preliminary phase of the process, the aim of which is to gather data on the subject, which is relatively new to Portugal, taking advantage of the experience of other countries in which some of the audit mechanisms foreseen are already in use or are in the late stages of development.

The specific aims of the study are: - to establish the importance of internal assessment within the framework of

quality assurance, in the light of the European guidelines for this area; - to analyse the national legal context, with regard to internal quality assurance

systems and their certification; - to analyse the main European trends in the area of specification and

certification of internal quality assurance systems, in order to identify and characterise cases of good practice;

- to identify core components for the certification of internal quality assurance systems which may steer national higher education institutions, without losing sight, however, of the flexibility necessary for the desirable development of innovative models which may arise within the ambit of the responsible exercise of institutional autonomy.

In the final chapter, the study includes a concrete proposal for a frame of reference for internal quality assurance systems in Portuguese higher education institutions, along with suggested guidelines for the design and development of the institutional audit process to be adopted by the A3ES.

These proposals were put to public debate throughout the first semester of 2010 through their presentation to, and discussion by, the representative bodies of the different sectors of higher education, and also in a number of higher education institutions, whenever such was requested.

2

2. INTERNAL ASSESSMENT AS A CORE PROCESS IN QUALITY ASSURANCE

Among the many effects of the radical changes that occurred in the context of higher education in the last quarter of the 20th century was the emergence of serious concern with quality assurance, on the part of both the institutions themselves and society in general, placing assessment incontrovertibly on the agenda of higher education.

Among the changes that occurred, some particularly relevant aspects were:

a) The massification of access, relating to the exponential growth in higher education, the diversification of provision of undergraduate and postgraduate programmes, the diversification of institutions as a consequence of the fast development of the polytechnic and private sectors, as well as the new expectations of the public with regard to higher education as a source of training and education, not just for elites but mainly as a means for the exercise of active citizenship in the context of the knowledge society.

b) Internationalization, namely with regard to the need to confer international validity on the certification of qualifications, in order to facilitate the transferability of academic and professional qualifications and consequently the continuation of studies and the access to a profession, in face of an increased mobility of students and graduates. This question is particularly relevant in the context of the European Union, as a fundamental condition for ensuring the right to free circulation and thus contributing to the full exercise of European citizenship.

c) Greater awareness of their rights on the part of citizens, and the subsequent demand for quality.

As a consequence of these environmental changes in higher education, institutions were faced with new challenges and expectations, namely the problem of how to preserve quality in the face of massive, rapid and sometimes uncontrolled growth, which made it necessary to consider quality from a more institutional perspective. Indeed, with great numbers of students, and a rapid increase in the teaching staff, there tends to be some dilution of the traditionally high level university standards and this no longer suffices to implicitly ensure the quality that is inherent to higher education. At the same time, when faced not only with a variety of supply, but also with marketing strategies which are sometimes aggressive, society and applicants for higher education – together with their families and future employers – feel the need for better information

3

about the quality of the programmes on offer, which has led governments to take measures of “consumer protection”.

In part due to such concerns, quality assurance systems began to be implemented, using different approaches according to their geographical location, particularly in terms of the so-called “evaluation processes”, which predominated in Western Europe, and the “accreditation processes” which were developed in Eastern Europe, for example, where they took advantage of the vast experience of the United States of America in this field.

It is important to note that evaluation and accreditation have complementary objectives and that both are of great social interest. Indeed, evaluation, which aims to monitor and constantly improve quality, represents what could be considered as the more noble function and with greater impact on society, which is to promote the quality of teaching, research, cultural action, and interaction with society. Accreditation, on the other hand, aims to guarantee compliance with the minimum requirements necessary for a programme or institution to gain official recognition as such.

The initial duality between the evaluation and accreditation processes has gradually become more blurred and nowadays it is generally agreed that evaluation and accreditation are inseparable and are, indeed, two sides of the same coin. It would certainly make little sense that the process of evaluation – while transmitting to the public a message concerning the quality of an institution or programme –, should not objectively identify cases in which the minimum requirements of quality necessary for accreditation are not fulfilled. On the other hand, accreditation alone provides society with important information, but which is insufficient because a programme may respect the minimum requirements in order to be officially recognized, but may only be reasonable or mediocre in terms of quality.

Currently, therefore, quality assurance systems tend to combine different complementary aims, with more or less focus on each feature depending on whether the national external quality assurance systems are more concerned with a process of control and certification of conformity and verification associated with the dimension of accreditation or more concerned with transparency and improvement, which has more to do with the dimension of evaluation, focusing on the production of validated, objective and intelligible information, and on the systematic and sustained improvement of quality within the institution.

Another dichotomy regarding quality assurance concerns internal and external assessment, two facets of the system which are also complementary. Here, it is important to keep in mind the idea – which is today consensual – that quality and

4

quality assurance are primarily the responsibility of the higher education institutions themselves. Indeed, it is within the institutions that the battle for quality will be won or lost.

This was the position of Portuguese state universities (including UCP and ISCTE) when, in 1993, they took the initiative of launching an assessment programme, in order to trigger an internal culture of quality, which other higher education institutions later adhered to. In the document of guidelines1 which largely shaped the law on assessment passed in 19942, there is emphasis on the level of responsibility of institutions with regard to the community “which implied continual updating of teaching, with sustained improvement and the ability to respond to the great challenges of our time”, and it advocated “the absolute need for each higher education establishment to institutionalize an internal system of self-assessment based on previously defined indicators”. The success of this initiative in terms of the impact it had within the institutions was made very clear in the report on the self-assessment of the Portuguese system of quality assurance, prepared in the ambit of the assessment promoted by ENQA. Among the conclusions of this report one of the strengths of the system was identified as:

“the good acceptance and enthusiasm by the part of many institutions, which have assumed their participation in the evaluation process as an opportunity to reflect on their organisation and activities, and to take up quality culture and quality assurance as strategic elements for their institutional development”,

noting, consequently, “the positive movement towards the institutionalization of internal quality assurance procedures” 3.

This is also the position that is being taken at European level throughout the Bologna Process, expressed namely in the Berlin Communiqué4, in which Ministers responsible for higher education in signatory states to the Bologna Declaration explicitly stated that:

“Consistent with the principle of institutional autonomy, the primary responsibility for quality assurance in higher education lies with each institution itself and this provides the basis for real accountability of the academic system within the national quality framework”.

1 CRUP (2003). Guiding Principles for the Evaluation and Follow-up of Activities within Higher Education Institutions (translated title). 2 Law nº 38/94, 21st November. 3 Santos, S.M. (Rapporteur) et al. (2006). Review of the Quality Assurance and Accreditation Policies and Practices in the Portuguese Higher Education – Self-Evaluation Report, p. 78. 4 Realising the European Higher Education Area (2003). Communiqué of the Conference of Ministers responsible for Higher Education, Berlin.

5

This declaration is reflected in the document Standards and Guidelines for Quality Assurance in Higher Education, adopted in 2005 at the Bergen Ministerial Meeting, which among its underlying basic principles includes5:

- “Providers of higher education have the primary responsibility for the quality of their provision and its assurance”;

- “There should be encouragement of a culture of quality within higher education institutions”,

and, in accordance with these principles, it gives special relevance to mechanisms of internal quality assurance, as will be seen in more detail further on.

It is patently obvious from this line of thought that internal quality assurance structures and mechanisms must be a basic first line of intervention in assessment procedures, with the ultimate aim of promoting the interiorisation of the need for a culture of quality which permeates all activities undertaken within the institution. The following points also emerge as fundamental in explaining the idea underlying internal quality assurance:

a) More than a bureaucratic exercise of control and certification of conformity with external orientations, internal assessment is essentially a permanent process aiming at enhanced quality. As such, it involves monitoring and control processes, but also reflection and subsequent intervention. It presupposes systematic monitoring of various activities, data collection and the construction of indicators. In other words, it includes a dimension of mensurability, which is essential for the credibility of the assessment process, for the establishment of benchmarks and, above all, as a starting point for subsequent reflection on the information gathered during monitoring, from which conclusions should be drawn and consequences should be expressed in proposals of measures for adaptations or corrections to be made. That is, assessment involves a process of retroaction for quality enhancement.

b) The process of internal assessment requires that all internal activities should be assessed (teaching at undergraduate and postgraduate level, research, community outreach, as well as all of the actors involved). This assessment should occur naturally, as part of everyday activities, by means of simple, agile mechanisms, which at the same time are effective and produce outcomes.

c) Internal assessment should actively involve all relevant actors, in a process implying collective responsibility, as something that is meaningful to and

5 ENQA(2005). Standards and Guidelines for Quality Assurance in the European Higher Education Area, p. 13.

6

shared by everyone. In other words, it should induce a sense of belonging, ownership and co-responsibility among the academic community and not merely be seen as an imposition on the part of the academic management or political power.

d) The process of internal assessment should obey an institutional policy for quality assurance and should follow procedures which have been duly institutionalised. It is not, therefore, compatible with ad hoc procedures; in other words, it requires pre-defined measures and appropriate support structures.

These points concerning the main function of internal assessment as part of the quality assurance systems do not diminish the importance of external quality assurance, which is not only necessary but also essential both to validate internal assessment mechanisms – which must be included in the review to be undertaken by the external assessors – but also as a source of information and an independently validated judgement.

It has been repeatedly pointed out in various documents pertaining to the Bologna Process that higher education is a public good and a public responsibility. Public authorities and society in general cannot, therefore, distance themselves from this public good and even less from its quality. External assessment must thus be seen in the scope of the social dimension of higher education, addressing, in a complementary way, both of the above-mentioned dimensions: on one hand, as a critical view on the quality and quality assurance within the institutions, and on the other, as validation of objective information presented in a manner which is easily understood by society.

7

3. THE EUROPEAN REFERENCE FRAMEWORK

The systems of quality assurance are one of the essential mainstays for the implementation of the aims of the Bologna Declaration. Indeed, they make a decisive contribution to the mutual recognition of degrees and study periods, and the resulting transferability of academic and professional qualifications, which, as has already been pointed out, is a crucial condition for the mobility of European citizens

The Bologna Declaration, however, says little about quality assurance, dedicating only one paragraph to the subject, in which it advocates the promotion of European co-operation in quality assurance with a view to developing comparable criteria and methodologies6. The importance of assessment was thus implicitly taken for granted and the development of assessment systems was left up to each country, within the principle of subsidiarity.

However, some protested that assessment was the absent element in the Bologna Declaration and they insisted on more precise objectives regarding quality assurance. Perhaps because of this, but also as a consequence of the enlargement of the EU to Eastern European countries and due to concerns regarding transnational education, the Prague Communiqué7, in 2001, took up the theme. A specific section was dedicated to it, broadening the concept by establishing, as an objective within the domain of quality assurance, the development of the conditions for the mutual recognition of evaluation and accreditation/certification mechanisms. An appeal was also made to higher education institutions, national agencies and to the ENQA to work together in the establishment of a common frame of reference and in the dissemination of information about good practice.

It was, nonetheless, at the Berlin meeting of 2003 that a most proactive position was taken when, with a view to accelerating the Bologna Process, Ministers defined intermediate priorities in three areas of action: quality assurance, the two-cycle system and the recognition of degrees and periods of study. Returning to the need for the development of mutually acceptable criteria and methods for quality assurance, the Berlin Communiqué established some parameters to be observed in national systems of quality assurance8 and delegated to the ENQA – which was still known as the European

6 The European Higher Education Area (1999). Joint Declaration of the European Ministers of Education Convened in Bologna on the 19th of June 1999. 7 Towards the European Higher Education Area (2001). Communiqué of the meeting of European Ministers in charge of Higher Education in Prague on May 19th 2001. 8 In the Berlin Communiqué it is stated that: “[The Ministers] agree that by 2005 national quality assurance systems should include: a definition of the responsibilities of the bodies and institutions involved; evaluation of programmes or institutions, including internal assessment, external review,

8

Network for Quality Assurance in Higher Education, whose membership included representatives of national and regional evaluation agencies and government representatives9 – the responsibility of preparing, through its associates and in cooperation with the European University Association (EUA), the European Association of Institutions in Higher Education (EURASHE) and the student association National Unions of Students in Europe (ESIB)10, a commonly acceptable set of standards, procedures and guidelines on quality assurance and to explore ways of ensuring an adequate peer review system for quality assurance and/or accreditation agencies or bodies. This work was undertaken between 2003 and 2005 by ENQA in close cooperation with the E4 group11, which was made up of the above-mentioned associations, with the participation of the European Commission. It led to the document Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG).

At the 2005 Bergen meeting, the Ministers approved this document and committed themselves to introducing into their respective national systems of quality assurance the guidelines and criteria established within it. The Bergen Communiqué12 also welcomed the idea of the creation of a European register of quality assurance agencies, requesting ENQA to develop the practicalities of implementation of this register in cooperation with EUA, EURASHE and ESIB. The importance of cooperation between national agencies was also underlined, with a view to enhancing mutual recognition of decisions relating to accreditation or assessment procedures, and an appeal was made to higher education institutions to continue their efforts to enhance the quality of their activities through the systematic introduction of internal mechanisms and their direct correlation to external quality assurance.

In 2007, The London Communiqué13 returned to the theme, emphasising the progress made so far: participation of students and the publication of results; a system of accreditation, certification or comparable procedures; international participation, co-operation and networking”. 9 Later, at the General Assembly of 4/11/2004, the ENQA became an association, calling itself the European Association for Quality Assurance in Higher Education, only including as full members the associated agencies. Representatives of national governments attend the meetings of the General Assembly as observers. 10 In May, 2007 the ESIB changed its name to ESU – European Students’ Union. 11The E4 Group, which included ENQA, the ESIB (now, ESU), the EUA and the EURASHE, began to meet regularly in 2001 with a view to contributing to the creation of a European dimension for quality assurance. It played a decisive role in the preparation of the ESG and in the design and implementation of the EQAR. 12 The European Higher Education Area – Achieving the Goals (2005). Communiqué of the Conference of European Ministers Responsible for Higher Education, Bergen. 13 Towards the European Higher Education Area: responding to challenges in a globalised world (2007). London Communiqué.

9

“The Standards and Guidelines for Quality Assurance in the EHEA adopted in Bergen (ESG) have been a powerful driver of change in relation to quality assurance. All countries have started to implement them and some have made substantial progress. External quality assurance in particular is much better developed than before. The extent of student involvement at all levels has increased since 2005, although improvement is still necessary. Since the main responsibility for quality lies with HEIs, they should continue to develop their systems of quality assurance. We acknowledge the progress made with regard to mutual recognition of accreditation and quality assurance decisions, and encourage continued international cooperation amongst quality assurance agencies”.

The Ministers asked the E4 Group to organize an annual European Forum on Quality Assurance14 to facilitate the exchange of information on good practice and to guarantee continued enhancement of higher education in the European space. They agreed to the implementation of the European Quality Assurance Register (EQAR), along the lines of the model proposed by the E4 Group, namely with regard to voluntary adhesion on the part of agencies, to its independent nature, its transparency and the fact that it should be self-financed. It is the intention of EQAR to encourage registration of quality assurance agencies which are operating within the European Higher Education Area in conformity with the ESG. This body was formalized in March, 2008, and included as actors the E4 Group and social partners, including government representatives as observers. In a recent report on the main developments of the Bologna Process, the decision taken in London to create EQAR was considered a historic landmark as it corresponded to the establishment of the first legal body to be set up through the Bologna Process15.

The Louvain Communiqué16, which was already quite focused on the extension of the Bologna Process beyond 2009, did not add anything of substance to the issues discussed above, but it included, among its priorities and challenges for the next decade, the development of “multidimensional transparency tools” which in practice opened up the door to the establishment of rankings among European higher education institutions17. The E4 group was asked to continue their cooperation in further developing the European dimension of quality assurance, and in particular to ensure that the EQAR is evaluated externally, taking into account the views of the stakeholders.

14A first European Forum on Quality Assurance had been held in 2006, in Munich, organized by the EUA in partnership with the other institutions which make up the E4 Group, which proved to be a great opportunity for discussion of the developments underway in various countries. This was the initiative behind the recommendation of the Ministers that this type of meeting should become an annual event. 15 EC(2009). Higher Education in Europe 2009: Developments in the Bologna Process, p. 16. 16 The Bologna Process 2020 – The European Higher Education Area in the new decade (2009). Communiqué of the Conference of European Ministers Responsible for Higher Education, Leuven and Louvain-la-Neuve. 17This reference, made in the Louvain Communiqué, to instruments aimed at more detailed information on institutions, allegedly to make the diversity of the European Higher Education Space more transparent, met with the strong disagreement of students, but strangely did not provoke a visible reaction on the part of organisations representing higher education institutions.

10

The document on standards and guidelines adopted in Bergen became the main European reference for assessment of higher education, and has naturally conditioned the national quality assurance systems. It is organized in three sections regarding, respectively, internal quality assurance in higher education institutions, external assessment of higher education and quality assurance within the agencies themselves. The standards do not aim to be prescriptive, and in general they do not go into detail concerning procedures, in order not to impose unique models and consequently to provide leeway for the exercise of autonomy by the institutions and by the agencies themselves. Indeed, it is explicitly assumed that, as an underlying principle, the document on standards and guidelines endorses the spirit of the EUA Graz Declaration, where the idea is defended that “the purpose of a European dimension to quality assurance is to promote mutual trust and improve transparency while respecting the diversity of national contexts and subject areas”18.

With regard to internal assessment, which is the main focus of this study, seven standards are defined, in which the previously mentioned principles for internal assessment are clearly in evidence, and which can be organized in three different areas of concern:

a) The first norm deals in detail with the question of institutionalization of internal quality assurance, establishing that higher education institutions must define a policy for quality assurance and also the associated procedures it involves, including a clear definition of the standards of its programmes and degrees They should also make a explicit effort to develop a culture which recognizes the importance of quality and of quality assurance for its activities. With these objectives in mind, they should at the same time develop and implement a strategy for the continuous enhancement of quality. The norm adds that the adopted strategy, policy and procedures should have a formal status, be publicly available, and should incorporate an appropriate role for students and other stakeholders.

b) The next four norms, relating to the most pertinent areas for the assessment of teaching, deal with, respectively, the educational offer and operation of the programmes, the assessment of the students, the teaching staff and the resources. They establish that: - higher education institutions must have formal mechanisms for the

approval, periodic review and monitoring of their programmes and degrees;

18 EUA (2003). Graz Declaration 2003 – Forward from Berlin: the Role of the Universities, p. 9.

11

- students should be assessed using published criteria, regulations and procedures which are applied consistently;

- institutions must have means of ascertaining that their teaching staff is appropriately qualified and competent to teach, and these procedures must be made public;

- institutions must guarantee that the resources available for the support of student learning are adequate and appropriate for each programme offered.

c) The last two norms concern data collection and its use and publication, and specify that: - higher education institutions should develop a system for collection,

analysis and utilization of relevant information for the effective management of their programmes of study and other activities;

- institutions should regularly publish updated, impartial, and objective information, both quantitative and qualitative, about the programmes and degrees they are offering.

The importance given to internal quality assurance systems is reinforced by the fact that the first norm in the second part, regarding external assessment, makes it clear that “external quality assurance procedures should take into account the effectiveness of the internal quality assurance processes described in Part 1 of the European Standards and Guidelines”.

In spite of the relative detail with which the standards are formulated, each one is accompanied by a set of guidelines which justify, clarify and look in detail at the norm, even suggesting possible pathways for its implementation. Thus, this document is not only a crucial reference for the specification of internal quality assurance systems, but also a source of invaluable help for their design and implementation.

In a final note on the European framework described above, it is important to mention that the document approved in Bergen explicitly states that it only refers to the dimension of teaching and learning: “The standards and guidelines relate only to the three cycles of higher education described in the Bologna Declaration and are not intended to cover the area of research or general institutional management”19.

19 ENQA(2005). Standards and Guidelines for Quality Assurance in the European Higher Education Area, p. 11.

12

4. INTERNAL QUALITY ASSURANCE SYSTEMS IN THE NATIONAL JURIDICAL FRAMEWORK

The demand for appropriate assessment of the quality of teaching is foreseen in the Constitution of the Portuguese Republic, in Art 76º relating to the autonomy of universities, which after the constitutional revision of 1997 states: “In the terms of the law, the universities enjoy statutory, scientific, pedagogical, administrative, and financial autonomy without detriment to the proper evaluation of the quality of the teaching”. This constitutional principle reinforces the idea underlying the binomial autonomy/responsibility, in which assessment is undoubtedly seen as a fundamental pillar for the full exercise of institutional autonomy.

The RJIES – the Portuguese juridical regime governing higher education institutions – (Law nº 62/2007, 10th September) reiterates this principle by establishing in Art. 11, nº 5, that “the autonomy of Higher Education Institutions does not preclude (…) accreditation and external assessment, in the terms of the law”.

Internal assessment is dealt with in Art.147, nº 1, of the RJIES: “Within the terms of their statutes, higher education institutions should establish mechanisms for regular self-assessment of their performance”. The legal imperative is thus established for institutions to include in their statutes the way in which their respective internal quality assurance systems will be organised. This statute dignifies and confers a higher degree of institutionalization on internal assessment, which was not previously the case, as up until the present juridical regime was established “only rarely did any university or polytechnic statutes, even those approved after the Assessment Law (Law nº 38/94, 21st November), include any structure specifically responsible for overseeing quality, although in the Constitution this was considered to be interconnected with autonomy”20.

The questions of transparency, information and dissemination, which in the terms of the European standards are essential to internal quality assurance systems, are dealt with in detail in Arts. 161 and 162 of the RJIES, clearly geared towards “consumer protection”:

“Art. 161 - Transparency

1. Higher Education Institutions must make available on their Internet site all the information necessary for easy understanding of the cycles of studies offered and the degrees awarded, research undertaken, and services offered by the institution. 2. This information must include reports on the self-assessment and external assessment of the institution and of its organs, as well as on its study cycles .

20 Simão, J.V., Santos, S.M. & Costa, A.A. (2005). Ambition to Excellence – The Bologna Opportunity (translated title), p. 188.

13

Art. 162 - Information and dissemination

1. Higher Education Institutions must make clear in their documents which are aimed at public dissemination, and in all their publicity, the precise contents of the official recognition of public interest, and of the authorisation for the operation of study cycles and the recognition of awards. 2. Accurate and adequate information must be made available on the following points:

a) The mission and aims of the institution; b) Statutes and Regulations; c) Organic units; d) Study cycles offered, degrees awarded and curricular structure; e) Teaching staff and types of contract and of link to the institution; f) Academic assessment regime; g) Accreditation certificates and results of the assessment of the institution and

its study cycles; h) The rights and duties of students, including all fees and other required

payments; i) Social support services; j) Pass, failure and employability rates, relating to study cycles offered; l) Any other data foreseen the law and in the statutes”.

Internal assessment is also dealt with in Law nº38/2007, 16th August, which approves the legal framework for the assessment of higher education currently in force. Articles 17 and 18 are particularly important, as is nº 1 of Article 19, which basically transpose into the national juridical regime the European standards which have already been discussed:

“Article 17 - Internal quality assurance

1 - Higher education institutions should:

a) Adopt a policy of quality assurance of their study cycles, according to their mission, and the appropriate procedures for this undertaking;

b) Commit themselves, through concrete measures, to the development of a culture of quality and of quality assurance in their activities;

c) Develop and implement a strategy for continuing quality enhancement.

2 - The strategy, the policy and the procedures referred to above must:

a) Be formally approved by the statutorily legal and competent body of the institution and made public;

b) Ensure the participation of students and other parties interested in the process.

Article 18 – Self-assessment

Within the framework of self-assessment, higher education institutions should:

a) Define formal procedures for approval, follow-up and periodic assessment of its study cycles, which must include:

i) the participation of the pedagogic councils and the opinion of students, namely through the pedagogic councils and the student associations;

ii) the participation of research centres which collaborate in the organisation and operation of study cycles;

14

iii) the participation of external advisory bodies that collaborate with the institution;

b) Adopt appropriate procedures to ensure that the teaching staff possess qualifications and competences necessary to carry out their duties, which must be made available to those responsible for external assessment and must be the object of appraisal in the assessment reports;

c) Ensure that the teaching resources available are sufficient and appropriate for each of the study cycles offered;

d) Ensure that they gather, analyse and use the information necessary for efficient management of their study cycles and other activities;

e) Regularly publish quantitative and qualitative information which is up-to-date, impartial and objective, concerning:

i) the study cycles they offer and the degrees and diplomas they award; ii) keeping track of their graduates for a reasonable period of time, with

regard to employability.

Article 19 – Basic principles of external assessment

1. The procedures for external quality assessment should take into consideration the internal quality assurance procedures referred to in the previous article.”

Decree-law nº 369/2007, which instituted the A3ES, and which, as said in its preamble, concluded “the legislative process related to quality assurance in higher education”, obviously does not go into detail about ways of organizing internal assessment, but includes a very important note in its preamble on the system of assessment, by identifying as one of the axes around which quality assurance systems are organised “the obligation for institutions of higher education to create their own systems of quality assurance, which can be certified”. This is an important point, because it indicates the intention of the lawmakers to foresee the possibility of certification of internal quality assurance systems, although this concept is not dealt with explicitly in the articles of any of the legal diplomas cited.

The action plan for 2009 of the A3ES gives some indications on the importance given to internal quality assurance systems, namely in the following excerpts21:

“By taking into account its specific role in the national higher education quality assurance, the Agency does not ignore that the first responsibility for quality assurance lies with the institutions themselves. This means that the mission of the Agency only makes sense when in permanent dialogue with the institutions and with their interested collaboration. Therefore, a substantial component of the Agency’s activity will be dedicated to supporting the implementation of internal quality assurance systems.”

“It results from the normative framework that created the present assessment and accreditation system that the Agency must assume as its main task the promotion and the diffusion of a quality culture amongst the higher education institutions. Actually, we need to start from the principle that the responsibility for the quality of education lies first of all, with each higher education institution itself that must

21 A3ES (2009). Activity plan for 2009, p. 3.

15

create the internal structures and procedures adequate to promote and assure the quality of that education. The Agency will be responsible for realising audits aiming at certifying the institutional internal quality procedures.”

Thus, the Agency reinforces the idea that certification of internal quality assurance systems is implicitly foreseen in the national juridical framework, concluding, in this regard, after an analysis of the legal principles mentioned earlier: “We thus feel we can use the term ‘certification’ to denote the activities of quality auditing promoted by the agency in order to guarantee the efficiency of the internal institutional mechanisms of quality assurance”.

The Agency draws on these conclusions, which are crucial for the future simplification of the assessment system, and which are to be implemented after the intense work of accreditation of programmes to be undertaken in the short term: “In a stable system, to enter into regular operation at the beginning of the academic year 2011/2012, attention must be focused on internal systems of quality assurance, promoting a lighter system for institutions”22, pointing right from the beginning a possible method for an approach of this kind.

In brief, higher education institutions face the challenging yet rewarding task of implementing their internal quality assurance systems, which are crucial for the consolidation and full exercise of institutional autonomy, and for them to be able to make use of a simplified system of external assessment, with obvious gains in terms of effort and costs involved in individual periodic accreditation of their programmes. A number of institutions have begun to work towards this end in a variety of ways, which is in itself an added value for the system as a result of all the experience that will be accumulated.

However, there are advantages to the establishment of some common guidelines which, without stifling creativity and innovation, will guarantee the fulfilment of some requirements/prerequisites which will allow the above-mentioned certification23.

Here it is worth noting that some polytechnic institutions have decided to apply for certification under the rules established by ISO 9001:2000 and various other university and polytechnic institutions have been certifying some of their support services. The

22 Ibidem, p. 6. 23 ENQA, in ENQA Position Paper on Quality Assurance in the EHEA, in view of the Leuven and Louvain-la-Neuve meeting of ministers responsible for higher education of 28-29 April 2009, addresses the need for a balance between respect for diversity and the adoption of common principles in the folloing terms “The diversity of the EHEA makes a single monolithic and prescriptive approach to quality, standards and quality assurance in higher education inappropriate and impracticable. (…) However, while respecting diversity, [ENQA] does promote the harmonisation and convergence of quality assurance processes, based on common principles”.

16

norms set out in ISO 9001 refer to systems of quality management in service providers, so that they are appropriate for this end – the certification of some particular services within higher education institutions – but to apply them to an institution as a whole, including the organization and operation of teaching, raises difficulties due to the complexity of the teaching/learning processes, which do not fit into the categories provider/client. Efforts have been made to produce guidelines for the application of the ISO 9001 in education, in particular in the documents produced within the framework of the International Workshop Agreement24, which, based on the norms mentioned, promote the adaptation of the terminology, principles and concepts to the reality of educational organizations. It would be worthwhile undertaking a specific study on the effectiveness and real impact of the ISO 9001 on the institutions which adopted them, particularly with regard to an appreciation of the effort made and the level of bureaucracy implied in their application, as well as to their capacity to monitor in depth the teaching /learning processes and allow improvement thereof.

24 IWA 2 (2007). Quality Management Systems – Guidelines for the application of ISO 9001:2000 in education.

17

5. EUROPEAN TRENDS WITHIN THE FIELD OF QUALITY ASSURANCE

5.1 THE EUROPEAN DIMENSION OF QUALITY ASSURANCE

The European dimension of quality assurance in higher education began to emerge at the beginning of the 90’s, following the evolution of the assessment systems in member states and as part of a process of internationalization of assessment. The first steps taken were supranational experiments in evaluating programmes in specific areas, carried out by international teams of experts, with the voluntary participation of institutions. The main objective was a comparison of study programmes in different countries and the mutual learning arising from the process.

Examples of exercises of this nature were: - The International Programme Review of Electrical Engineering, promoted by

the Association of Cooperating Universities in the Netherlands (VSNU) in 1992, regarding a comparative study of the programmes of Electrotechnical Engineering in the Netherlands, Switzerland, Germany, Belgium and the United Kingdom, but which also included the programme operating at the University of Porto;

- The study Business Administration and Economics Study Programmes promoted by the Swedish National Board of Universities and Colleges in 1992, comparing courses in the United Kingdom, the Netherlands, Finland, Germany, France and Sweden;

- A joint project by the Centre for Higher Education Policy Studies (CHEPS) of the University of Twent with the American Accreditation Board for Engineering and Technology (ABET), for comparison of courses of Civil, Mechanical and Chemical Engineering in the Netherlands, Belgium, Switzerland, France Germany and Portugal, in which the Universities of Aveiro and Porto were involved;

- A variety of other undertakings within the ambit of the Erasmus networks and in thematic networks, as a basis for the establishment of mutual confidence likely to increase mobility within these networks.

Thus, these initiatives were embarked upon by the higher education institutions themselves, with a number of different motivations, both internal and external to higher education, in order to develop the international dimension of evaluation.

At the time, internal motivation included increasing student mobility and the consequent problems of transparency, comparability and recognition of periods of study, as well as the exportability of educational programmes, which implied the extending of assessment beyond official systems of higher education. External pressures, on the other hand, were related to the globalization of professions and the corresponding problems of recognition of qualifications, with agreements concerning

18

international commercialization of services and with the action of international organizations interested in the processes of assessment and recognition.

The most significant of these actions, which was the first Europe-wide exercise of this kind, was undertaken by the European Commission in 1994/5 with the pilot project Quality Assessment in the Field of Higher Education, which covered 46 higher education institutions in 17 countries in the European Union and the European Economic Association. The project focused on areas such as Engineering Sciences, Communication/Information Sciences, and Art and Design, and the Portuguese institutions which took part were the Instituto Superior Técnico with its Mechanical Engineering programme and the Instituto Politécnico do Porto with its music programme.

In September, 1998, as a result of the experience gained in this pilot study, the Council of Europe passed a Recommendation on European cooperation in the area of quality assurance in higher education25, so that, as part of their responsibility for the organization of their higher education system, each Member State should establish a transparent quality assessment and quality assurance system. The main objectives were to:

- Assure the quality of higher education, taking account of the specific economic, social and cultural context of each country, while at the same time taking into consideration the European dimension and the existence of a world in rapid evolution;

- Encourage and support higher education institutions to use evaluation as a basis for improving the quality of teaching and learning, as well as training and research;

- Support European and international cooperation, in view of the benefits to be gained from exchange of experience in the above-mentioned areas.

The Recommendation, which specifies principles on which the assessment systems should be based, emphasises the need for autonomy for the agencies responsible for external evaluation, and sets out some standards to be used by the agencies in the course of external evaluation.

There was also a proposal for the creation of a European network to increase cooperation among national assessment systems. This idea was developed by an ad hoc group during 1999, coordinated by the European Commission, and in March 2000 the European Network for Quality Assurance in Higher Education (ENQA) was set up with

25Council of the European Union (1998). Recommendation 98/561/CE, 24th September, 1998, regarding European cooperation, with a view to assuring quality in higher education.

19

the aim of promoting cooperation at European level among all the actors involved in the field of assessment and quality assurance.

Members of ENQA included national and regional quality assurance agencies, Governments through representatives of ministries responsible for higher education, European organizations representing higher education institutions (the Confederation of European Union Rectors’ Conferences, CRE and EURASHE) and the European Commission.

5.2 THE INFLUENCE OF THE BOLOGNA PROCESS

The Recommendation of the Council of Europe and the ensuing creation of the ENQA were a huge qualitative step towards the European dimension of quality assurance, at a particularly significant moment when, in a parallel process arising from an intergovernmental initiative, the Bologna Declaration was signed. Among its concerns with intelligibility, comparability and compatibility of European higher education systems, it called for the “promotion of European co-operation in quality assurance with a view to developing comparable criteria and methodologies”. The ENQA, which was born of an idea previous to the Bologna Process, was naturally seen as the right operational body to attempt to reach this objective.

As a result, the past decade has seen enormous progress in the adoption of quality assurance procedures in European higher education systems, particularly following the Berlin Conference, where, as has been seen previously, the Ministers explicitly declared quality to be a core factor in the establishment of The European Space for Higher Education, and the adoption of the ESG in Bergen two years later, whose principles and guidelines were successively incorporated into national juridical regimes governing higher education. This evolution is patently clear in the data, published in 2008, collected in the survey undertaken by the ENQA, which shows a common pattern, whereby Agencies responsible for external quality assurance were created by an act of law – almost 96% of the agencies were created by a specific law or are recognized by public authorities.26

According to this ENQA survey, the influence of the Bologna Process has become more marked in quality assurance processes, largely due to the efforts to bring them in line with the reference points expressed in the ESG, but also due to the influence of the development of the national qualifications frameworks, which in turn create important frames of reference for the work of the agencies. In a comparison with the survey 26 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 21.

20

published in 200327, it is made clear that one of the most important changes concerns the notion and use of specific criteria and standards which were seen as “emerging as a common feature”, in the various systems, but which in 2008 were actually embedded and in general use.28

A closer look shows that the more immediate effect of the ESG has happened within the higher education institutions themselves, at different rates according to the level of development of the internal quality assurance systems. Thus, in countries where this type of system already existed, the ESG were essentially seen as a point of reference with which to validate the procedures already in use, with regard to how far they fit the European models, in a process that started to gear itself towards the certification of internal systems, as will be seen in more detail in the following chapter. But in cases which were still in the early stages of systematic attention to internal quality assurance, or which were at a phase of transition regarding the types of approach, the ESG served as a useful and revitalizing reference for the design and installation of internal quality assurance systems. However, the main impact of the ESG is being seen at the second and third levels (the organization of the work of the agencies and their meta-assessment), as will be analyzed later.

The use and the influence of the ESG was the main theme of debate at the Second European Quality Assurance Forum. The conclusions of this meeting identified a number of positive effects of the use of the ESG, namely the fact that they provide an operational framework which is appropriate for the development of approaches which in terms of institutional policies and strategies for quality are top down, while also being predominantly bottom up in terms of implementation. This provided a balance between centralization and decentralization of internal systems of quality assurance29. Overall, the opinion is that the ESG are reasonably easy to implement, and that they are able to support creativity, reinforcing the individual nature of each institution, which is why they are a good starting point for new initiatives in the area of quality processes.

There is also mention of the fact that the ESG can facilitate transparency of quality assurance systems, improve their comparability and constitute a potential basis for international cooperation. However, it was noted that in order for the full potential of the ESG to be realized, it will be necessary to continue to invest in exchange of experience and good practices, in clarification of concepts and identification of

27 ENQA (2003). Quality Procedures in European Higher Education – An ENQA Survey. 28 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 85. 29 Harvey, L. (2008). Using the European Standards and Guidelines: Some Concluding Remarks. In EUA(2008), p. 82.

21

obstacles and also in the involvement of stakeholders in their implementation and interpretation.

Potential negative effects were also identified, namely the fear that an overly prescriptive interpretation of the ESG could lead institutions to focus excessively on conformity with standards, to the detriment of taking advantage of good practices already in operation and to the detriment of creativity in the development of internal quality processes. The fact that the ESG refer exclusively to teaching and do not cover other areas such as the assessment of research or the research-teaching relationship was also seen s a significant limitation.

The implementation of the ESG has met with some problems, largely associated with academic conservatism and institutional inertia, which have made it difficult to involve not only teachers but also managers, students and non-teaching staff, in enhancement processes which require significant change in practices and mentalities. In particular, a latent conflict was identified between the practices of the academic staff regarding their own subject area and the more institutional practices, which shapes the majority of quality assurance processes. In order to attenuate these concerns and thus reduce this apprehension it will be necessary to invest in mobilising the internal actors in order to convince them of the value of quality assurance processes.

5.3 THE DYNAMICS OF EXTERNAL QUALITY ASSURANCE

The progress report presented by the Bologna Follow-Up Group (BFUG) at the Ministerial Conference in Louvain30 states that the implementation of the external quality assurance had made significant progress, and that every one of the Bologna Process signatory States had already introduced external quality assurance processes which include the phases of self-evaluation and external evaluation and almost all of them publish the results of the evaluations and implement follow-up measures.

A close examination of the scorecards for 2009 shows that, in regard to the indicator relating to the stage of development of the external quality assurance systems, 69% of the countries are in the green or the light green score category, representing fully functioning systems, applying to all higher education institutions, following the four-phase model (self-assessment, external review, publication of the results and follow-up procedures) in which the agency or agencies operating within the country had already undergone external assessment foreseen in the third part of the ESG (dark green) or had already set a date to do so (light green). Included in the yellow category are 29% of

30 Rauhvargers, A., Deane, C. & Pawels, W. (2009) . Bologna Process Stocktaking Report 2009.

22

countries, corresponding to cases in which the system does not cover all higher education institutions, or only covers two or three of the phases mentioned, or when no date has been fixed for the meta-evaluation of the Agency. There is only one case in the orange category, representing an ongoing situation at the very early stages of implementation of external quality assurance. In the indicators relating to ‘level of participation of students’ and ‘level of international participation’, 73% and 58 % of countries, respectively, are in the green category.

The ENQA survey illustrates that in spite of the predominance of the four-phase model, external quality assurance agencies have diversified organizational structures and operational methods, as can be seen from the following:

a) In terms of institutions assessed, 9% of the agencies only evaluate universities, 15% evaluate other higher education institutions, 42% evaluate both universities and other higher education institutions, 24% evaluate other types of institution, and 10% operate in a specific area of knowledge;

b) In terms of the geographical area of action, the range of operation of the majority of agencies is national; there are only seven countries within the European Higher Education Area where there is more than one Agency with responsibility for external quality assurance, which share tasks on a regional, procedural or type of target institution basis, as a rule with no overlapping of competencies (with the exception of the German and Dutch systems, and in part the Spanish system, where there is competition among the agencies operating within the country);

c) With regard to the processes used, there is a clear predominance of assessment and accreditation, followed at some distance by audit processes; almost two thirds of the agencies use assessment and/or accreditation processes at programme level and almost 40% at institutional level; the auditing processes are mainly used at an institutional level;

d) As regards the autonomy of the agencies to define the criteria and standards applicable to the processes of external quality assurance, 79% declared that they participate in the final decision concerning specification of these elements and in 62% of cases this decision is the Agency’s alone; in two cases only, this decision is taken by the government alone.

Closer study of the data collected in the survey reveals that almost all national agencies have more than one type of approach to external quality assurance and some use the three processes (assessment, accreditation and auditing). The survey report presents some interesting correlations31:

- “Agencies that conduct programme evaluation also conduct – at a rate of fifty percent – accreditation of programmes and evaluation of institutions.

31 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 24.

23

- Agencies that conduct accreditation of programmes also conduct – at a rate of fifty percent – evaluation of programmes and accreditation of institutions.

- Almost every Agency that accredits institutions also accredits programmes, and half of them also evaluate programmes and institutions, and audit institutions.

- Almost two-thirds of the agencies conducting audits at the institutional level also evaluate programmes, and half of them also accredit programmes, and evaluate and accredit institutions.”

The general impression gained from the survey is consequently one of well developed external systems of quality assurance and on the whole the agencies use different types of procedures, and have begun to add new activities to their portfolio, which reflects the dynamic nature of the evolution that has been seen in quality assurance in higher education. One particular facet of this evolution – which shows how external pressure, namely from political powers and public opinion, can influence QA systems – was the addition of programme accreditation procedures in systems which were previously oriented towards evaluation processes. This understanding is also expressed in the conclusion of the 2nd European Quality Assurance Forum, where it is stated that “with few exceptions such as Scotland and Finland, when quality assurance starts with an improvement focus, political pressure forces an accountability orientation into the system”32. A prime example of this effect was seen in Sweden in 2001, when after two cycles of institutional assessment, cyclical evaluation of programmes was introduced, largely as a result of pressure from student associations, who complained that institutional evaluation provided no information about the quality of the programmes33.

These changes of approach to external quality assurance, which have been translated into changes in the juridical framework in various countries, are among the most notable findings of the survey: 75% of agencies indicate that either they have recently changed their work approach or are about to do so, in the majority of cases in very significant ways. The reasons given for this are not only in order to respect the ESG but also the change to other procedures or, more often, the addition of another procedure. Generally speaking, a comparison with the 2003 report shows a broadening of the scope of action of the agencies. Some of these changes correspond to in-depth restructuring of agencies with a well-established tradition, as happened for example with the Danish Evaluation Institute (EVA), who saw its role changed as a result of the creation of a new accreditation organ, and the Comité National d’Evaluation (CNE), in France, which was 32 Harvey, L. (2008). Using the European Standards and Guidelines: Some Concluding Remarks. In EUA(2008), p. 84. 33 Serrano-Velarde, K. & Hopbach, A. (2007). European politics of quality assurance and the introduction of the Standards and Guidelines for Quality Assurance in the European Higher Education Area. In HRK (2007), p. 39.

24

incorporated in the new Agence d’Evaluation de la Recherche et de l’Enseignement Supérieur (AERES), which now covers not only all higher education institutions but also all research bodies.

In this brief overview of the dynamics of external quality assurance it is important to mention another point which has been under discussion since the Recommendation of the Council of Europe in 1998 and which concerns the standards applied to the agencies themselves, as organizations. Part 3 of the ESG was a response to this question, and had a strong impact on the organization of the agencies, as was seen previously. The importance of the standards established by the ESG as a frame of reference for the organization and operation of the agencies is also demonstrated by the founding of the EQAR, which uses these standards as a condition for registration.

The fact that the ESG were developed for use in the whole of the European Higher Education Area, regardless of structure, function and dimension of the different national systems, has nonetheless given rise to some problems, as is recognized in the report on the ENQA survey:

“Even though there is some convergence in European quality assurance, there are limitations when the requirements of ESG conflict with national regulations and traditions. (…) The analysis and commentaries [related to questions 11 to 24 of the survey, which paraphrase Part 3 of the ESG and thereby the ENQA membership criteria] acknowledge that there can be a strained relationship between national traditions, legislation and the ESG” 34.

These difficulties have led to the introduction of some adjustments concerning the application of the ESG, illustrated for example in the positions taken by the Board of the ENQA in 2006 with regard to the degree of compliance of the agencies with the ESG:

“Full or substantial compliance may be impossible for some agencies, owing to restrictions placed on them by the very nature of their work and/or legislation in place in their country (ies) of operation. When considering such cases, the ENQA Board will take mitigating circumstances such as these into account” 35.

Moreover, regarding the notion of independence of the agencies, which is one of the basic requirements of the ESG, the Board of the ENQA deliberated at the same time that ‘independence’ for the purpose of the ESG, should be understood as “autonomous responsibility for operations, where conclusions in reports cannot be influenced by HEIs, ministries or other stakeholders”36,37. 34 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 20. 35 ENQA (2006). Guidelines for national reviews of ENQA member agencies, p. 6. 36 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 33. 37 The evolution in ENQA’s position is interesting if we consider that not long before, in the external evaluation of the Portuguese evaluation system, one of the criticisms made by the evaluation team was

25

This change of attitude with regard to compliance with the ESG was corroborated in the London Communiqué, where as regards admission into the EQAR it was stated that this should be decided on a basis of substantial compliance with the ESG, which represents a move away from the original position of the ENQA, which foresaw compliance with all the standards as a condition for inclusion on the register38.

Another of the requirements imposed on agencies by the ESG was the compulsory establishment of procedures regarding their own transparency and accountability, including the existence of a duly publicized policy (with support documentation) for the effect, as well as the undertaking of an external assessment of the Agency at least once every five years. According to data collected in the survey, 89% of agencies stated they had already developed some type of internal procedures, at least partially. Among the measures adopted by the agencies, the following four emerge with a level of development between moderately effective and effective:

- The establishment and enforcement of a no-conflict-of-interest mechanism in the work of the agency’s external experts;

- Mechanisms for collection of feedback from experts and reviewed institutions, and for the analysis and reflection on such data, for the purpose of informing and supporting the agency’s improvement and development;

- Processes and results which reflect the agency’s mission and goals of quality assurance;

- An appeals system.

Regarding external assessment, in compliance with the ENQA membership criteria, only 14 agencies (out of 46) stated they had already been externally assessed, while 20 others indicated they were preparing their external assessment39. The 2009 scorecards presented in the Bologna Process Stocktaking Report in Louvain, also reflect the modest progress in relation to this item, insofar as only 16 countries had organized external assessment of their respective agencies, although a further 17 had set dates for it, and 15 had no plans in this respect, so the conclusion was drawn that “while the scheme of external QA has been widely implemented, in some countries it may not yet operate entirely in accordance with the ESG”40. More up-to date data reveal that 34 agencies have already undergone external evaluation, corresponding to only 17 countries. 41

the “apparently limited independence of the system”, when in reality, the concept of independence as expressed above was entirely respected. 38 In the words of Amaral and Rosa (2008, p. 79), “this lack of definition raises problems of clarity and comparability, which are very much in fashion today”. 39 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 80. 40 Rauhvargers, A. et al. (2009). Bologna Process Stocktaking Report 2009, p. 8. 41 Externally Reviewed Agencies, ENQA, http://www.enqa.eu/reviews_agencies.lasso, retrieved 14.05.2010.

26

The question of financing is another important point regarding European trends in the establishment of agencies, which will certainly be put to debate in Portugal.

As was mentioned in Chapter 2, external assessment derives from the social dimension of higher education and is seen by the political powers and by society as a crucial instrument with which to gauge the quality of institutions and study programmes. It is not, therefore, surprising that in the majority of countries the initiative for the establishment of external quality control measures came from governments or was supported by governments and in most cases, as would be expected, this is reflected in the funding of the agencies.

In reality, according to the ENQA survey, 78% of the agencies are mainly funded by the State (41% of these are exclusively funded by the government and in the case of 37%, the level of public funding ranges from 52-98%), and only 15% (6 agencies) do not receive any government funds (though it should be noted that four agencies included in the survey were international). Two agencies state that they are only supported by higher education institutions and a further two are exclusively funded by payment for services rendered, though the nature of these services is not specified. In 2003, for example, the Dutch agency NAO (currently NVAO) received public funding of 3.6 million Euros, and also charged the institutions 2,500 Euros for each process of programme accreditation42. So it may be concluded that the State is by far the main source of funding of agencies in the European Higher Education Area.

5.4 FROM QUALITY ASSURANCE TO QUALITY ENHANCEMENT

Quality assurance processes have four general aims –accountability, conformity, control and enhancement – which are all present in the external quality processes in Europe, although to different degrees and with frequent changes of emphasis in the various countries.

Accountability and continuous enhancement are aims which are included in the majority of national systems, but there are well-known difficulties in conciliating them. The ESG, which appear to be geared towards continuous enhancement, but which also admit other aims, do not resolve the latent incompatibility between the two approaches (accountability and continuous enhancement). One solution which seems to be gaining some popularity, although it is not an easy one to implement, is the trend to focus

42 Dittrich, K. (2004). Accreditation in the Netherlands. In Di Nauta et al. (2004), p. 56.

27

external quality assurance on accountability but to focus the internal processes on continuous quality enhancement43.

The report Trends V identified a number of serious problems in the balancing the need for the development of forward-looking, innovative internal systems of quality assurance and standardized accreditation procedures in use. This is mainly due to the difficulties caused by the formal accreditation procedures regarding innovation and curricular reform, interdisciplinarity and experimentation within the ambit of new courses adapted to the Bologna Process. In particular, in countries where the national accreditation systems function at programme level, there are often problems of uneasiness regarding emerging institutional procedures and strategies for quality enhancement44.

On this subject, in a paper presented at the opening of the 2nd European Forum on Quality Assurance, Sybille Reichert made the point that the development of quality in higher education is much more than “the formal quality assurance processes that policymakers like to focus upon when they speak about quality in higher education”. Quality enhancement brings together a wide range of institutional development methods and, on its positive side, “the Bologna reforms could improve quality in multiple ways: through the opportunities they offer to reflect and review curricula, to reform teaching methods (student-centred learning, continuous assessment, flexible learning paths) and even through strengthening horizontal communication and institutional transparency.”45

Improvement in the quality of teaching is, indeed, central to the Bologna Process and quality assurance processes must respond to the demands of society (at the risk of losing credit in the eyes of the public if they fail to do so), but must not be an obstacle to innovation and development. The dynamic evolution that the agencies have been registering, which has already been mentioned, would appear to imply that there is receptiveness to this concern, and cases have already arisen where there has been a movement away from accreditation of programmes46 or at least where this has been discussed, towards the adoption of approaches at an institutional level which are less intrusive in the detailed organisation of teaching.

43 “externally-addressed accountability through internally-organised enhancement”, in the words of Lee Harvey (2008). 44 Crosier, D. et al. (2007). Trends V: Universities Shaping the European Higher Education Area, p. 60. 45 Reichert, S. (2008). Looking Back – Looking Forward: Quality Assurance and the Bologna Process. In EUA (2008), p. 5. 46 Costes, N. et al. (2008). Quality Procedures in the European Higher Education Area – Second ENQA Survey, p. 85.

28

The changes occurring in the methods used by the agencies do not, however, only affect the nature of the procedures, but also more basic aspects, such as changing the emphasis from the input indicators to others which , in principle, have more to do with the learning outcomes, with the support of a wide range of objective criteria, which are interrelated, namely: (i) with the qualifications frameworks; (ii) with benchmarks related, for example, to Tuning; and (iii) with the ESG. Proper treatment of learning outcomes within quality assurance will continue to pose challenges to the agencies and in particular to higher education institutions. In effect, with increasing numbers of students from more varied social backgrounds, registering on more and more varied higher education programmes, there is obviously a great advantage in having more detailed information on the quality and the relevance of the teaching. The students themselves are more and more involved in the process of quality assurance and demand new sources of information. Employers also require better knowledge of what the graduates have learnt and what they can do. Consequently, emphasis on learning outcomes will tend to increase47.

In the concluding remarks of the ENQA survey, however, a question is raised: But can quality procedures cope with the amount of work required to provide reliable information on these at a detailed level? The conclusion drawn was “probably not if the emphasis on data collection is merely for accountability and reporting; but if quality assurance has quality improvement/enhancement as an agreed and truly integrated focus, then it is more likely to be able to”48.

The concern with finding a balance between quality assurance processes and innovation and creativity in teaching, in order to give more emphasis to quality enhancement, while maintaining the balance between different approaches to quality assurance, was the object of Project QAHECA49. This project has been developed over the last two years with the participation of about three dozen agencies and higher education institutions, with the aim of exploring how internal and external quality assurance systems can provide support for innovative and creative institutions, able to adopt and implement schemes for modernization of their teaching/learning processes.

In the final report on a previous EUA project, on the development of an internal culture of quality within universities (the Quality Culture Project, undertaken between 47 Ibidem, p. 88. 48 The report rounds this off with the following optimistic note: “it is interesting and encouraging to note that the survey shows, in several places and in different ways, such a change in emphasis from assurance to enhancement” (Ibidem). 49 This was the Quality Assurance for the Higher Education Change Agenda (QAHECA) Project, launched in 2007 by the EUA, in partnership with ACQUIN (Germany), the Higher Education Academy (United Kingdom) and the National University of Ireland, and recently completed. .

29

2002 and 2006), the participating institutions highlighted the risk of internal quality assurance processes becoming mere bureaucratic procedures, even when they are undertaken properly50. To study this issue in more depth and to attempt to better understand how creativity could be reinforced in higher education, the EUA launched the Creativity Project, during which it was concluded that:

“Quality processes have the potential to strengthen creativity and innovation if they are geared towards enhancement and focus on the capacity to change as a way to incorporate a future dimension. However, they can also have highly detrimental effects if they stress conformity over risk taking, are oriented towards the past rather than the future and develop into burdensome bureaucracies.” 51

These concerns were the starting point for QAHECA with a view to exploring the best ways to limit the potential negative effects of (internal and external) processes of quality assurance, and to steer these processes towards constituting an added value with regard to creativity in teaching and learning. One of the outcomes of the project was a set of non-prescriptive recommendations, putting forward some key principles to be considered by higher education institutions and by agencies in the planning of their current practices of quality assurance. Given the relevance of this theme in the current context of evolution of quality assurance in the European Higher Education Area, these recommendations are transcribed below:

Recommendations from the QAHECA project52 “1) First and foremost, quality assurance must be context