1

occasional papers

21

TransparencY oF eUropean HiGHer eDUcaTion THroUGH pUBlic QUaliTY

assUrance reporTs(eQarep)

Final report of the project

Tiia BacH, ĐUrĐica DraGoJeViĆ , peTer FinDlaY, sTepHanie HerinG, liia laUri, orla lYncH, ZeYnep olcen, MaiKi UDaM

21

TransparencY oF eUropean HiGHer eDUcaTion THroUGH pUBlic QUaliTY

assUrance reporTs(eQarep)

Final report of the project

Tiia BacH, ĐUrĐica DraGoJeViĆ , peTer FinDlaY, sTepHanie HerinG, liia laUri, orla lYncH, ZeYnep olcen, MaiKi UDaM

ISBN978-952-5539-74-5ISBN978-952-5539-75-2ISSN1458-1051

ThepresentreportcanbedownloadedfromtheENQAwebsiteatwww.enqa.eu/index.php/publications

©EuropeanAssociationforQualityAssuranceinHigherEducationAISBL2014,BrusselsENQAAISBL,AvenuedeTervuren36-38bte4,1040Brussels,BelgiumQuotationallowedonlywithsourcereference.

Coverdesignandpagelayout:double-id.comEditedbyMariaKeloandLindseyKerber

Brussels,Belgium,2014

ThisprojecthasbeenfundedwithsupportfromtheEuropeanCommissionintheframeworkoftheLifelongLearningProgramme.

Thispublicationreflectstheviewsonlyoftheauthor,andtheCommissioncannotbeheldresponsibleforanyusewhichmaybemadeoftheinformationcontainedtherein.

5

TaBle oF conTenTsForeword……………………………………………………………………………………......................….……………..4

Chapter1: IntroductiontotheEQArepproject....................................................................................................5

Chapter2: Introductiontoqualityassurancereports.........................................................................................7

Chapter3: Understandingthestakeholders’perspectiveontheuseandusefulnessofexternalqualityassurancereports.................................................................10

Chapter4: Analysisofqualityassuranceagencies’currentpracticesinreportingtheoutcomesofthequalityassuranceprocedures.................................................................... 26

Chapter5: Europeanguidelinesforsummaryreportsofexternalqualityassuranceproceduresandgenericrecommendationsforcomprehensivereports................................42

Chapter6: Featuresofagoodexternalqualityassurancereport................................................................ 47

Chapter7: Conclusionsandrecommendations................................................................................................ 53

Acknowledgements.............................................................................................................................56

References.............................................................................................................................................. 57

ANNEXES...............................................................................................................................................58

6

ForeWorDTheevolutiontowardsamoreknowledge-basedsocietyandstrongcompetitioninthelabourmarkethasresultedinanincreasinginterestinhighereducationinEurope.Giventhelargenumberofhighereducationinstitutions(HEIs)andstudyprogrammesonthemarket,choosingaprogrammeandaninstitutionhasbecomemoredifficult.Asaresult,students,otherstakeholdersandthepublicatlargearelookingformoredetailed,reliableandcomparableinformationonthequalityofindividualstudyprogrammes,facultiesandHEIs.

AsstatedintheEuropeanCouncilconclusionsof12May2009,regardingastrategicframeworkforEuropeancooperationineducationandtraining(ET2020),oneofthefourstrategicobjectivesfortheframeworkisto“improvethequalityandefficiencyofeducationandtraining”.Qualityassurance(QA),whichplaysanimportantroleinfosteringthetransparencyofEuropeanhighereducation,canbeconsideredanimportantmeanstoachievethisobjective.Inresponsetothegrowingneedforaccessibleandreliableinformationoninstitutionsandstudyprogrammes,ENQAhascarriedouttheEU-funded“TransparencyofEuropeanhighereducationthroughpublicqualityassurancereports"(EQArep)”project.TheoverallaimoftheprojectistounderstandtheneedsoftherelevantstakeholdersandtoimprovetheQAreportsproducedbyQAagenciesasaresult.TheprojecthasbeenimplementedincollaborationwiththeSwissCenterofAccreditationandQualityAssuranceinHigherEducation(OAQ),Quality&QualificationsIreland(QQI),the(Croatian)AgencyforScienceandHigherEducation(ASHE)andtheEstonianHigherEducationQualityAgency(EKKA).

ThisreportpresentstheresultsoftheactivitiescarriedoutinthecontextoftheEQArepproject.Indoingso,thereportsetsoutthecurrentpracticesofQAagenciesinpublishingtheoutcomesoftheirevaluations(QAreports)andanoverviewoftheuseandusefulnessoftheseQAreportsfromastakeholderperspective.Bothanalysesarebasedonthefindingsofsurveysandworkshopscarriedoutaspartoftheproject.

TheprojecthasresultedinasetofEuropeanGuidelinesforsummaryQAreportsaddressedtoQAagenciesandasetofgenericrecommendationsforthedraftingofcomprehensivereports.TheseGuidelines,intandemwiththeEuropeanStandardsandGuidelines(ESG),aimtoprovideaframeworkfortheQAagenciestoworkwithin.TheGuidelinesspecificallyrecommendthatallsummaryreportsproducedbyQAagenciesprovidesimilarandcomparabletypesofinformation,tohelpthebeneficiariesbetterunderstand,compareandinformchoice,whilecontributingtothetransparencyofEuropeanhighereducation.

Onbehalfoftheprojectconsortium,Iwouldliketothankthecontributorstothisproject.Thesearetherespondentstothesurveys,participantsoftheworkshops,membersofthesteeringgroupandmembersoftheadvisoryboard.

IwouldalsoliketotakethisopportunitytoencourageallENQAmemberagenciestoconsidertheoutcomesofthisprojectfortheirownwork,thuscontributingtotheshapingofthefutureofQAreports.

Padraig WalshPresident of ENQA

7

Chapter 1:

inTroDUcTion To THe eQarep proJecTZeynep Olcen, European Association for Quality Assurance in Higher Education (ENQA)

responding to a need for better information

Asaresultofincreasingcompetitioninthelabourmarketandtheshifttowardsasocietywhereknowledgeisthekeytosuccess,highereducationinstitutions(HEIs)arethecenterofattentionforstudentsandotherstakeholdersmorethanever.TheincreasedinterestanddemandforHEIsraisesconcernsregardingthequalityandtheaccessibilityofinformationaboutinstitutionsandstudyprogrammes.Stakeholdersrequiretransparent,reliableandcomparableinformationaboutinstitutionsandprogrammesinordertomakewell-informeddecisions.

Amongothertoolsprovidingtransparentinformation,suchastheBolognatransparencytoolsorrankings,qualityassurance(QA)playsamajorroleincontributingtothetransparencyofEuropeanhighereducation.Withinthisperspective,QAreportsareconsideredanimportantsourceofreliableandcomparableinformation.AsmentionedintheStandards and Guidelines for Quality Assurance in the European Higher Education Area(ESG),externalQAreportsshouldbemadepublic.However,thecontent,structureandpublishingchannelsofthesereportsvaryconsiderably,andtherearenocommonguidelinesforQAagenciestofollow.ThissituationlimitsthepotentialofreportsincontributingtothecomparativedimensionandtransparencyofhighereducationattheEuropeanlevel.

InordertofostertheroleofQAreportsasasourceofreliableandcomparableinformationandtocontributetotheoveralltransparencyofEuropeanhighereducation,theEuropeanAssociationforQualityAssuranceinHigherEducation(ENQA)decidedtocarryoutaprojectentitled“TransparencyofEuropeanhighereducationthroughpublicqualityassurancereports(EQArep)”.Theprojectwassettoinvestigatethecurrentstateofthecontent,structureandpublicationofQAreportswhiletryingtounderstandtheexpectationsanddemandsofstakeholderswithregardtothesereports.Onthebasisoftheresults,theprojectdevelopedasetofrecommendationsforusebyQAagencies.

InadditiontoENQAastheleadpartner,theprojectconsortiumcomprisedfourQAagencies:theSwissCenterofAccreditationandQualityAssuranceinhighereducation(OAQ),Quality&QualificationsIreland(QQI),CroatianAgencyforScienceandHigherEducation(ASHE)andtheEstonianHigherEducationQualityAgency(EKKA).TheprojectwassuccessfulinobtainingfundingfromtheEuropeanCommissionLifelongLearningProgramme.

methodology

Theproject’sprimaryobjectivetodevelopcommon“EuropeanGuidelinesforexternalQAreports”requiredtwoimportantactions:1)analysingandunderstandinghowthe

8

stakeholdersperceivetheuseandusefulnessofexternalQAreportsproducedbyQAagencies,and2)mappingthecurrentpracticesofQAagenciesinpublishingexternalQAreportsintermsofcontent,structureandpurpose.

Withinthisperspective,thefirstpartoftheprojectmethodologyconsistedofconsultingvariousstakeholders-namelystudents,employers,governmentsandHEIs-throughanonlinequestionnairedevelopedbytheprojectworkinggroup.Theaimofthequestionnairewastwofold:1)tomapthecurrentuseofinformationregardingqualityinHEIsandstudyprogrammes,and2)toidentifytheexpectationsofthestakeholdersregardingthereports.Theanalysisoftheonlinesurveywaspresentedanddiscussedduringaninteractiveworkshopwithstakeholdersfromdifferentbackgrounds.

ThesecondactivityfocusedonmappingthecurrentreportingmethodsandpracticesofQAagencies.Followingthesamestructureasforpartone,anonlinesurveywasdevelopedbytheprojectworkinggroupandsenttoENQAmembersandaffiliates.Thesurveyquestionsfocusedonthepurpose,structure,contentandpublicationchannelsoftheQAreports.Inaddition,theprojectworkinggroupperformedanin-depthanalysisofaselectedsampleoftwentyQAreportsbasedonacommongridinordertoconsolidateandframethefindingsofthequestionnaire.ThefindingsoftheonlinesurveyandtheanalysisofthesampleofreportswerepresentedanddiscussedinaninteractiveworkshopwithQAagencies.

Theresultsofbothanalysessupportedtheprojectworkinggroupinidentifyingareaswhereamismatchexistedbetweenthecurrentpracticesandtheexpectationsofthebeneficiarygroups.BasedontheneedsandrequirementsexpressedbythestakeholderswithregardtoexternalQAreports,theprojectworkinggroupdevelopedasetofrecommendationsforQAreportsandspecificallyforsummaryQAreportsinsofarastheircontent,structureandaccessibilityareconcerned.

Theprojectconsortiumconsideredthatanimportantdistinctionhadtobemadebetweencomprehensive(“full”)reportsandsummaryreports.Infact,thegroupconsidersthatthemainpurposeofinstitutionalandprogrammecomprehensiveQAreportsistofacilitatearevieworaccreditationdecisionandtoserveasatriggerforenhancementatthereviewedinstitution.TheirprimaryusersarethereforetheHEIsorprogrammesunderreview.Thesummaryreports,ontheotherhand,aremainlyproducedtoprovideconciseandeasilyreadableinformationtothegeneralpublic.Theprojectconsortium,basedonthefindingsoftheprojectactivities,cametotheconclusionthatattemptingtouniformalisecomprehensivereportsmightresultinthelossofrelevantimportantinformationtotheinstitutionorareducedusefulnessofthereportstotheirmainusers.Therefore,whilethisreportprovidessomerecommendationsinsofarascomprehensivereportsareconcernedandenlistsgoodfeaturesofcomprehensivereports,theguidelinesthathavebeendevelopedconcentrateonsummaryreportsspecifically.Whilenotsuggestingfullstandardisationofsummaryreports,theprojectconsortiumconsidersthatinordertobettermeettheneedsofstakeholders,ahigherdegreeofcomparabilitybetweensummaryreportswouldbebeneficial.

ENQAinvitesitsmembersandaffiliates(whererelevant)tomakegooduseofthedevelopedrecommendationsinordertoenhancetheinformationvalueandaccessibilityoftheirreportsandthuscontributetotheincreasedtransparencyofEuropeanhighereducation.

9

Chapter 2:

inTroDUcTion To QUaliTY assUrance reporTsOrla Lynch, Quality and Qualifications Ireland (QQI)

Context and purpose of Qa reports

Theproductionandpublicationofreportsarecommonunifyingfeaturesofallexternalqualityassurance(QA)proceedingsintheEuropeanHigherEducationArea(EHEA).Standard2.5oftheStandards and Guidelines for Quality Assurance in the European Higher Education Area,20091(ESG)requiresthat“reportsshouldbepublishedandshouldbewritteninastylewhichisclearandreadilyaccessibletoitsintendedreadership.Anydecisions,commendationsorrecommendationscontainedinreportsshouldbeeasyforareadertofind.”TheproductionofthesereportsallowsforarecordofaQAreviewatprogrammeorinstitutionlevel.Thefrequentlyassertedcoreobjectiveforreportswhicharetobemadepublicistransparency.IntheEHEA,theoriginsofQAcanbetracedtotwomainsources:attemptstosolveproblemscentredonquestionsofqualityandconsiderationsrelatingtotheimprovementofthesystemsthatunderpinhighereducation.Consequently,QAhastraditionallyservedtwomainpurposesinhighereducation:enhancementandaccountability.ThequestionremainsastowhethertransparencyisapurposeofQAonparwithaccountabilityandenhancementorwhetheritisakeyprincipleorstandardforQA.Theexistenceofstandard2.5onQAreportsintheESGmayindicatethelatter.

IntheEHEA,whereexternalQAregimesareincompliancewiththeESG,varietyandsteadychangearekeyfeatures2.ThemostcommonexternalQAproceduresareaccreditationandevaluationofprogrammes,followedatasignificantdistancebyevaluationandaccreditationofinstitutions,thoughthegapisclosing.Ninetypercentofagenciesapplymorethanoneapproachand75percentofagencieshavechangedorarechangingtheirapproach.ThevariabilityofnationalagendasmeansthattheemergenceofafullyunifiedexternalQAsysteminEuropeisunlikely.

Overthecourseofrecentyears,inmoreeconomicallystraitenedcircumstances,thestakesaroundexternalQAhavebeenraised,andexternalQAintheEHEAhasbecomemorevisible.Duetocompetitionanddiversification,qualityhasbecomeacoresuccessfactorforinstitutionalsuccess.Guaranteeingacertainlevelofqualityorenhancingthequalityofaprogrammeoraninstitutionhasbecomeanintegralpartofregularmanagementandexternalmarketingofhighereducationinstitutions(HEIs).PossiblyderivingfromtherequirementforcompliancewiththeESG,thereisastrongeremphasis1 Thune,C.(2009).Standards and guidelines for quality assurance in the European Higher Education Area, 3rd edition. Helsinki:

EuropeanAssociationforQualityAssuranceinHigherEducation.2 Rauhvargers,A.,Deane,C.,&Pauwels,W.(2009).Bolognaprocessstocktakingreport2009.Report from working groups

appointed by the Bologna Follow-up Group to the Ministerial Conference in Leuven/Louvain-la-Neuve.Brussels:LifelongLearningProgramme,EuropeanCommission.

Westerheijden,D.F.,etal.(2010).The Bologna Process Independent Assessment-The first decade on working on the European Higher Education Area-Volume 1 Detailed assessment report. CHEPS/INCHER-Kassel/ECOTEC.Availableat:www.ond.

vlaanderen.be/hogeronderwijs/bologna/2010_conference/documents/IndependentAssessment_executive_summary_overview_conclusions.pdf;

Crozier,F.,Grifoll,J.,Harris,N.,Kekalainen,H.,&Malan,T.(2011).Evaluation of the reports on agency reviews (2005-2009). ENQA Occasional Papers 16.Helsinki:EuropeanAssociationforQualityAssuranceinHigherEducation.Availableat: www.enqa.eu/indirme/papers-and-reports/occasional-papers/ENQA_Occasional%20paper_16.pdf

10

onQAand,inparticular,onreportinginrelationtoQAasasourceofinformationtoinforminternalmanagementdecision-makingandtoinformanexternalpublicaboutthequalityofaninstitution.ThereisgrowinginterestindetailedandreliableinformationonthequalityofindividualstudyprogrammesandHEIs,sometimesforcomparisonpurposesratherthandescriptionofsingleprogrammesorinstitutions.

QAisahighlydynamicanddiversefieldwithdivergentrequestsfromstakeholders.HowcanQAandreportingonQAmeetthischallenge?InthefirsttenyearsoftheESG,thefocushasbeenonmethodology.Isittimenowforashiftinfocusfrommethodologytowardspurpose?AndarethepurposesofexternalQAsufficientlyclear?AstudybyDavidWoodhouse(2010)indicatesthataconfusingandheterogeneousarrayofstatedpurposeshaveemergedforexternalQAagencies3.ItisvalidtoassertthatasinglepurposeforexternalQAwouldnotbeappropriate.ArecentevaluationofexternalreviewsinIreland,theReviewofReviews4,inkeepingwiththeWoodhousereport,foundthatthepurposesofexternalQAreviewswerenotclear.Furthermore,theReviewofReviewspositedthatbeforemattersrelatingtoreviewmethodscanbeproperlyorevenadequatelyaddressed,theremustbeclaritywithrespecttopurpose.

WhilsttheexistingpurposesofexternalQAarenotclearorunified,theydoappeartoconvergearoundanumberofkeythemes,whichare:

• ToassistHEIsinassuring/enhancingtheirquality,indevelopingtheirinternalstructuresandproceduresandinachievingvariousaims

• Toinvestigatethe‘quality’ofprogrammesorinstitutions• Tocheckcompliancewithcertain(legal)requirements• Toassesseffectiveness/successofcertainpolicies/reforms• Toprovideindependentinformationforcomparingprogrammes/HEIs• Toprovideindependentinformationfordecision-making(funding,enrolment,

collaborativework)• Toprovideindependentinformationforcertainconstituencies• ToprovideindependentinformationaboutqualityofHEIs/programmes/

HEI-systems

Theintendedaudiencesforreportsarenothomogeneous.Reportscanbevariously,andsometimessimultaneously,addressedtoHEImanagement,teachers,students,employers,cooperationpartners,politicaldecision-makers,ormediaandsocietyatlarge.

InthesecontextsitisworthgivingdeliberateconsiderationtoreportingonexternalQA.Ifreportsaretobefitforpurpose,thenexternalQAreportsarerequiredtomeetvaryingpurposesacrosstheEHEA,alignedtodifferentnationalpriorities.Furthermore,evenwithinasinglestateoragencytherecanbearangeofpossiblycompetingpurposestowhichexternalQAisaligned.TheconsequenceofthisisthatvariouskindsofinformationarerequiredfromreportsacrosstheEHEA,andsometimestherearediverserequirements,orpurposes,forasinglereport.TheESGstandard2.5forreportingrequiresthat“reportsshouldbepublishedandshouldbewritteninastyle,whichis

3 Woodhouse,D.(November2010).Is there a globally common understanding to Quality Assurance?ESU Board Meeting 59 Seminar: Quality Assurance. PresentationconductedfromJerusalem,Israel.

4 QualityandQualificationsIreland(QQI)(2014).Review of Reviews: Report of the Independent Review Team. Availableat: www.qqi.ie/Downloads/Reviews/Review%20of%20Reviews/12639-QQI%20Review%20of%20Reviews-WEB.pdf.

11

clearandreadilyaccessibletoitsintendedreadership.Anydecisions,commendationsorrecommendationscontainedinreportsshouldbeeasyforareadertofind.”BasedonthecurrentdraftfortherevisedESG5,thenewversionwilllikelygofurther,statingthat“fullreportsbytheexpertsshouldbepublished.”Therevisedversionmayalsoinclude(intheGuidelines)arecommendationtoallowforthereporttoinclude“featuresofgoodpractice,demonstratedbytheinstitution”.Thecurrentversionoftherevisedguidelinesalsoindicatesthat“thepreparationofasummaryreportmaybeuseful.”

Asoutlinedabove,thediversityofpurposesforexternalQAthathaveemergedmeansthattherearevariedandheterogeneousneedsandaudiencesforQAreports.Allagenciesproducereportsasanoutcomeoftheirreviews.WhatarethekeyfeaturesofareportthatprovidesareliableaccountoftherevieweventwhileassuringthatthevariouspurposesofexternalQAhavebeenaddressed?Furthermore,howcanareportadequatelyinformadiverseaudience?Isonereportsufficient?Orshouldtherebedifferentreportsfordifferentaudiences?Thisprojectemergedasadirectresponsetoanecessitytodeliberateonreportingingeneral,thesequestionsinparticularandtheconsiderationofalloftheseinabroadercontextoftransparency.

5 E4Group,incooperationwithEI,BusinessEurope,andEQAR(2014).Revision of the ESG.Availableat:http://revisionesg.wordpress.com.

12

Chapter 3:

UnDersTanDinG THe sTaKeHolDers’ perspecTiVe on THe Use anD UseFUlness oF eXTernal QUaliTY assUrance reporTsMaiki Udam, Liia Lauri, and Tiia Bach, Estonian Higher Education Quality Agency (EKKA)

introduCtion and methodology

Thesurveyontheuseandusefulnessofexternalqualityassurance(QA)reportsfordifferentstakeholderswasdirectedatallmainstakeholdergroupsinhighereducationQA:students,potentialfutureemployers,governments,andhighereducationinstitutions(HEIs)themselves.Thepurposeofthesurveywastoidentifyandcomparetheexactinterestsofthevariousstakeholdersregardinginformationaboutthequalityofinstitutionsandprogrammes.

Thequestionnaire(Annex1)wassentbytheEQAreppartnerstostakeholderswithintheirrespectivecountriesinFebruary2013andtoadditionalstakeholdersbytheENQASecretariat.ThefirstpartofthequestionnairedealtwiththecurrentuseoftheinformationaboutqualityinHEIsandstudyprogrammes.Thesecondpartofthequestionnairefocusedontheexpectationsofstakeholders:whatinformationconcerningthequalityofaHEItheyneed,aswellaswhereandinwhatformattheinformationshouldbepresented.Thelastthreequestionsconcernedinformationabouttherespondent.AworkshopforstakeholdersorganisedinMay2013exploredthefindingsofthesurveyandcollectedfurtherinsightsontheexpectationsofdifferentstakeholdersandthepossibleformat/templateofanassessmentreport.

Stakeholdersweregroupedasfollows:

• representativesofHEIs• students• publicauthorities/governmentoffices• employers• funders/investors• other

13

Intotal,therewere127respondentsfrom15countries(Table1).Table 1. Number of respondents by country

CouNTry resPoNdeNTs

Ireland 37

Estonia 35

Switzerland 21

Croatia 15

Romania 3

Italy 3

France 3

Slovenia 2

Bulgaria 2

UnitedKingdom 1

Netherlands 1

Hungary 1

Germany 1

Bosnia-Herzegovina 1

Austria 1

ThemostactiverespondentswererepresentativesofHEIs,witharesponserateof70percent.Theresponserateofstudentsandpublicauthoritieswas28percentand21percent,respectively.Onlythreeemployerscompletedthequestionnaire,contributingtoa10percentresponserate(seeTables2and3). Table 2. Number of respondents, by stakeholder groups

GrouP resPoNdeNTs

Highereducationinstitution

78

Student 24

Publicauthorities 10

Other 12

Employer 3

Funder/Investor 0

Table 3. response rate, by stakeholder groups

GrouP Nº asked Nº resPoNded resPoNse raTe

HEI 112 78 70%

Student 86 24 28%

Publicauthority 48 10 21%

Employer 30 3 10%

14

results of the surVey

Below,theresultsofthesurveyarepresented.Allresponsesareincludedinthegeneralstatistics(e.g.Figure1);inthesegmentedstatistics(e.g.Figure2),onlytheresponsesfromstudents,publicauthoritiesandHEIsarepresented,astheresponseratefromemployerswastoolow.

reasons for searChing for information about the Quality of a hei and its study programmes

ThefirstquestionexploredthereasonswhystakeholderssearchforinformationaboutthequalityofHEIsandstudyprogrammes.Theanswersindicatethatthemainpurposeisdecidingonpossiblefurtherstudies(31%ofallresponses),butalsofindingpartnersamongotherHEIsandevaluatingthequalityofgraduatesforrecruitmentpurposesreceivedrelativelyhighscores–23percentand16percent,respectively(Figure1).Expectedly,mostofthestudentslookedforinformationonfurtherstudies,andthegreatestamountofHEIrepresentativeswereinterestedinfindingcollaborationpartners(Figure2).

However,24percentofrespondentsselected“otherpurposes”.Thesepurposeshavebeensummarisedandgroupedintothreecategories:

• TolearnaboutinternalQAsystemsinotherHEIs• Forcomparison/benchmarking(ofsimilarprogrammes,orQAprocedures)• Todoresearch

31%

16% 23%

6%

24%

For what purposes have you searched for information about the quality of a HEI and study programme?

To decide on possible further studies

To evaluate the quality of graduates for recruitment purposes

To find partners among HEIs

To decide on investments/funding/sponsorship to a HEI or its unit

Other

Figure 1. reasons for searching for information about the quality of higher education.

Forwhatpurposeshaveyousearchedinformationaboutthequalityofahighereducationinstitution(HEI)andstudyprogrammes?

15

34

2

21

22

4

5

40

2

1

6

3

2

29

6

5

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

HEIs

Public authorities

Students

For what purposes have you searched for information about the quality of a HEI and study programme?

To decide on possible further studies

To evaluate the quality of graduates for recruitment purposes

To find partners among HEIs

To decide on investments/funding/sponsorship to a HEI or its unit

Other

Figure 2. reasons for searching for information, by stakeholder groups.

Current sourCes of information

Therespondentswereaskedaboutthemainsourcesofinformationtheyusedtocollecttheinformationwhichtheysought.Byandlarge,themainsourcefordifferentstakeholderswerethewebsitesofHEIs(seeFigure3).Thesecondmostpopularsourceforinformationcamefromfriends,colleagues,parents,etc.Assessmentreportsappearedtobethethirdmostpopularsourceofinformationconcerningthequalityofaninstitutionorprogramme.Onlyveryfewrespondentsmarkedsocialmediaasasourceforthiskindofinformation,anditwasslightlymorepopularamongstudentscomparedtoothergroups(Figures4,5,6).

Intheadditionalcomments,NARICandalumniwerementionedasdistinctivesourcesofinformation.

56  

31  

31  

49  

5  

83  

51  

59  

69  

51  

41  

39  

11  

26  

16  

20  

63  

1  

0   20   40   60   80   100   120   140  

Information from/opinions of friends, colleagues, parents etc.

Various rankings/league tables

Government reports/publications

Assessment reports provided by QA agencies

Social media

Websites of HEIs

Sources to get information about the quality of HEI and study programmes

O,en   Once  in  a  While   Never  

Figure 3. sources for acquiring information about the quality of HeIs and study programs.

Forwhatpurposeshaveyousearchedinformationaboutthequalityofahighereducationinstitution(HEI)andstudyprogrammes?

WhichsourcesdoyouusuallyuseforgettinginformationaboutthequalityofHEIsandstudyprogrammes?

16

62

2

34

22

21

40

5

25

26

40

52

46

37

1

1

50

14

10

18

8

3

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Websites of HEIs

Social media

Assessment reports provided by QA agencies

Government reports/publications

Various rankings/league tables

Information from/opinions of friends, colleagues, parents etc.

Other (please name below)

HEIs: Which sources do you usually use for getting information about the quality of HEIs and study programmes?

Often Once in a while Never

Figure 4. sources for acquiring information: HeIs.

Figure 5. sources for acquiring information: Public authorities.

HEIs:WhichsourcesdoyouusuallyuseforgettinginformationaboutthequalityofHEIsandstudyprogrammes

PublicAuthorities:WhichsourcesdoyouusuallyusetogetinformationaboutthequalityofHEIsandstudyprogrammes?

17

Figure 6. sources for acquiring information: students.

reasons why not to use assessment reports

SeveralrespondentshadneverusedQAreportsasaninformationsource.Thesurveysoughtinsightfromrespondentswhoindicatedthattheyneverusedassessmentreportsastowhythesesourceswerenotutilised.Thirty-sixpercentofrespondentsindicatedtheyeitherdidnotknowwheretofindthereports,ortheydidnotknowaboutthem(Figure7),suggestingthatstakeholdersarenotsufficientlyawareofthereportsandthekindofinformationtheymayprovide.Incaseof20percentoftheresponses,respondentsfoundtheneededinformationelsewhere,andincaseof15percentofresponses,itwasclaimedthatthereportsdidnotcontainnecessaryinformation.One-fifth(21%)ofresponsesrevealedthatthereportsareeithertoolongortoocomplicatedtounderstand.Interestingly,thedifferentstakeholdergroupshadquitesimilarreasonsfornotusingtheQAreportsasasourceofinformation(Figure8).

Additionalcommentsindicatedthelackoftimetoconsultreportsasareason,butalsothefactthatthereportsarenotalwayspubliclyavailable.

Students:WhichsourcesdoyouusuallyusetogetinformationaboutthequalityofHEIsandstudyprogrammes?

18

20%

14%

3%

9% 15%

22%

14% 3%

If you do not use reports by QA agencies as a source of information, please explain why.

I found the needed information elsewhere

The reports are too long

There are no reports in English/language I understand

The reports are in a too complicated language

The reports do not contain the information I need

Did not know where to find them

Did not know about them

Other

Figure 7. reasons why not to use reports published by Qa agencies.

9  

1  

6  

16  

2  

6  

9  

2  

5  

7  

1  

3  

2  

0  

1  

9  

2  

5  

15  

0  

5  

2  

0  

0  

0%   10%   20%   30%   40%   50%   60%   70%   80%   90%   100%  

HEI

Public authorities

Students

Did not know about them Did not know where to find them

The reports do not contain the information I need The reports are in a too complicated language

There are no reports in English/language I understand The reports are too long

I found the needed information elsewhere Other (please name below)

Figure 8. reasons why not to use reports published by Qa agencies, sorted by stakeholder group.

helpfulness of assessment reports

IndividualswhouseassessmentreportspublishedbyQAagenciesasonesourceofinformationwereaskedwhethertheyfindthereportsandtheinformationprovidedinthemhelpful.

One-thirdofrespondents(32%)fullyagreedthatthereportsareahelpfulsourceofinformation,whileanadditional51percent“slightlyagree”withthestatement.Sixpercentofresponsesshowthatrespondentsdidnotfindthereportshelpfulatall(seeFigure9).Thestakeholdergroupspossess,onceagain,quitesimilarviews(Figure10).

Thisquestionraisedverymanycomments,55intotal.Themainideasexpressedinthecommentsaresummarisedbelow:

Ifyoudonotusereportsbyqualityassuranceagenciesasasourceofinformation,pleaseexplainwhy.

Ifyoudonotusereportsbyqualityassuranceagenciesasasourceofinformation,pleaseexplainwhy.

19

• Themainadvantageofthesereportsisthefactthatalltheinformationisgatheredinoneplace,andthesourceistrustworthy.Unfortunately,thereportsaresometimestootechnical.

• Somereportsarefartoostandardisedanddonotcontainsufficientinformationaboutwhattheevaluationteamactuallyfound.

• Thefinaldecisionoftheagencyonlyshowsifthestandardsarefulfilled–withoutanyrankingordegreeofperformanceoftheevaluatedprogrammes.

• Agencyreportsdifferinstyleandcontentconsiderably-amorestandardisedinternationalapproachwouldbeveryhelpful.

• Thelanguageusedistoocomplicatedandwouldneedtobechangedtomakethereportsmoreuser-friendly.

Ingeneral,thereportsarecomprehensiveandconsideredtocoverallrelevantareas.Atthesametime,easiercomparability(e.g.throughinternationalstandardsforreports)anduser-friendlinessisneeded(includinglanguage,lengthetc.).

32%

51%

6%

11%

If you use reports by QA agencies, do you find them helpful in providing information about HEIs/programmes?

Fully agree

Slightly agree

Fully disagree

Slightly disagree

Figure 9. Helpfulness of reports by Qa agencies.

Figure 10. Helpfulness of reports by Qa agencies, sorted by stakeholder group.

Ifyouusereportsby qualityassuranceagencies, doyoufindthemhelpfulinprovidinginformationaboutHEIs/programmes?

Ifyouusereportsbyqualityassuranceagencies,doyoufindthemhelpfulinprovidinginformationaboutHEIs/programmes?

20

plaCes to find reports published by Qa agenCies

Morethanhalf(56%)ofresponsesindicatedthatrespondentsfindthereportsonwebsitesofQAagencies.However,alargeproportionoftherespondents(36%)alsofindthereportsonwebsitesofHEIs(Figure11).Amongthestudents,nearlyhalfofthemusethewebsitesofHEIstofindthereports(Figure12).

36%

56%

8%

If you use reports by QA agencies, where did you find the assessment reports?

Websites of HEIs

Websites of QA agencies

Other

Figure 11. Places to find the reports by Qa agencies.

44  

4  

14  

67  

11  

16  

12  

1  

1  

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

HEI  

Public  authori0es  

Students  

If you use reports by QA agencies, where did you find/access the assessment reports?

Websites of HEIs Websites of QA agencies Other (please name)

Figure 12. Places to find the reports by Qa agencies, sorted by stakeholder group.

information expeCted by different stakeholders

Foronequestion,therespondentswereaskedtoindicatewhatkindofinformationtheynormallyneedtomakedecisionsregardingfurtherlearning,partnerships,comparisonswithotherinstitutions,etc.Themajorityofrespondents(96outof127)named‘contentofstudyprogrammes’asthesinglemostimportantpieceofinformation,followedby‘accreditationstatusofinstitutions/studyprogrammes’(80respondents)and‘strategicplanning,managementandgovernance’(78respondents)(Table4).Theoverall

Ifyouusereportsbyqualityassuranceagencies,wheredoyoufindtheassessmentreports?

Ifyouusereportsbyqualityassuranceagencies,wheredoyoufind/accesstheassessmentreports?

21

prioritiescorrespondwiththepreferencesofrespondentsfromHEIs,withtheexceptionofstudentsandpublicauthoritieswhoexpressedslightlydifferentneeds.Althoughstudentsalsoindicate‘contentofstudyprogrammes’asthemostimportantitem,theirsecondconcernis‘employabilityofgraduates’(9thimportantforHEIsandthemostimportantforpublicauthorities)followedby‘studentsupportsystem’and‘qualificationsofteachingstaff’.Publicauthoritiesvaluedequally‘contentofstudyprogrammes’,‘accreditationstatus’,‘qualificationsofteachingstaff’,‘studentsupportsystem’and‘financialresources’,placingthemallasthesecondmostimportantfollowing‘employabilityofgraduates’.Unimportantforallstakeholderswas‘institution’spositioninleaguetables’and‘historyandtraditions’(Table5).

Table 4. Information needed to make decisions regarding further learning, partnerships, comparisons with other institutions, etc.

INFormaTIoN resPoNdeNTs

Contentofstudyprogrammes 96

Accreditationstatusofinstitutions/studyprogrammes

80

Strategicplanning,management,governance 78

Qualificationsofteachingstaff 74

Internalqualityassurancesystem 73

Studentsupportsystem 67

Employability/employmentofgraduates 66

Reputationofteachingstaff 59

Numberofresearchgrants,publications,citations 57

Applicationandadmissionstatistics 57

Conditionofinfrastructure 54

Institution’sabilitytoresponddiversestudents’needs

46

Financialresources 45

Historyandtraditions 37

Institution’spositioninleaguetables 29

Other 4

22

Table 5. Information needed, by stakeholder group

HeI Public authorities

students

Contentofstudyprogrammes 69(1) 9(2) 27(1)

Accreditationstatusofinstitutions/studyprogrammes

62(2) 9(2) 13(9)

Strategicplanning,management,governance 61(3) 8(7) 14(8)

Internalqualityassurancesystem 56(4) 7(8) 12(11)

Qualificationsofteachingstaff 53(5) 9(2) 17(4)

Studentsupportsystem 47(6) 9(2) 18(3)

Numberofresearchgrants,publications,citations 46(7) 7(8) 9(13)

Reputationofteachingstaff 44(8) 4(13) 16(6)

Employability/employmentofgraduates 40(9) 10(1) 21(2)

Applicationandadmissionstatistics 40(10) 6(10) 15(7)

Conditionofinfrastructure 35(11) 6(10) 17(4)

Institution’sabilitytorespondtodiversestudentneeds

32(12) 6(10) 12(11)

Historyandtraditions 30(13) 1(15) 8(14)

Financialresources 28(14) 9(2) 13(9)

Institution’spositioninleaguetables 22(15) 2(14) 6(15)

Other(pleasenamebelow) 6(16) 1(15) 1(16)

preferable format of information

One-third(32%)ofrespondentsindicatetheywouldliketoreceiveinformationaboutthequalityofaHEIanditsstudyprogrammesina short, concentrated summary describingthemainstrengthsandareasforimprovement.Twenty-sevenpercentexpecttoseecomparativedatawithotherinstitutions,and25percentprefernumericaldatadesignatingthemostimportantaspectsoftheinstitutionanditsprogrammes.Only14percentareinterestedincomprehensivereportsprovidingextensiveinformationaboutstrengthsandareasofimprovementinmanagementandcoreprocessesofaHEI(seeFigure13).Nosignificantdifferencescouldbedetectedamongthedifferentstakeholdergroups(Figure14).Someadditionalcommentsreferredtotheuseofmultimediaasafacilitatedentrywaytoinformation.

Inthebreakoutgroupsduringthestakeholders’workshopinTallinnon6-7May,itwasdiscussedthatthenationalagencieswithinEuropeprioritisetheirownuniquenationalagendastherebymakingcomparisonbetweendifferentcountriesimpossible.Consequently,itseemsthatcomparisonscanreallyonlybemadebetweenHEIswithinonecountry.Thestudentsagreed,furthermore,thattheprovisionofcomparabledatawasnotataskofQAagencies:inQAprocedures,aninstitution’soraprogramme’sperformanceiscomparedagainstsetstandards,notagainstotherinstitutions.

23

Fortheemployers,theinstitutionalreportsareofnorelevanceandarethereforenotused,asemployersdonothaveanyspecificneedforacquiringinformationaboutinstitutions’internalQAmechanisms.Themostrelevantinformationfortheprofessionalworldis‘performance’,whichistranslatedintoquantitativeindicatorsthroughvariousrankings.

Allstakeholdergroupsagreedthatanassessmentreportshouldalsoincludeasummaryreportshowingtheoutcomeoftheassessment,themainstrengthsandweaknessesofaninstitution/programmeandtherecommendationsforfollow-upactivities.Thisisessential,asthecomprehensivereportsareconsideredtoolongandnotintendedfortheneedsofawiderreadership.Atthesametime,theinstitutionsandprogrammessubjecttoQAproceduresneedcomprehensivereports,andtheirneedsshouldnotbesacrificedforthesakeofotherpotentialreaders.

Employersrecommendedthefollowingcontentandformatbeconsideredforinclusioninthesummaryreportsofstudyprogrammes:

• Contextofthequalityassessment(voluntary/obligatory;accreditation/evaluation;periodofvalidity;qualitylabels?evaluatedbynational/internationalpanelagainstnational/internationalstandards;single/jointprocedure;accreditationstatusofofferingHEI;etc.);

• Syntheticprogrammedescription(specialfeatures,innovativecharacter,relevance,specificities);

• StatementsonachievementoftheintendedlearningoutcomesmatchingwithgivenQFlevel;

• Profileofstrengthsandweaknesses;• Linktocomprehensivereport;• Linktothewebsitewherethestudyprogrammecanbefound

Thegroupsagreedthatthelengthofasummaryreportshouldbekeptlimited,atapproximatelytwopages.Informationconcerningthecontextshouldbeprovidedinschematicform(ratherthanindiscursiveform),whiletheprogrammedescriptionshouldbediscursiveandnormallynolongerthanfivelines.Careshouldbetakenwiththeuseofquantitativedataorfiguresthatcouldbeeasilymisusedandmisinterpreted.Strengthsandweaknessesshouldbeprovidedinatablefocusingonthemainoutcomesoftheassessment.Inallgroupsitwasagreedthatatemplateorstandardreportingstructuremightbehelpful.

Thediscussionsinthestakeholders’workshopunderlinedtheimportanceofmakinginformationeasilyaccessible,readable,andevencomparable.Atthesametime,itwasunderlinedthatasdifferentQAprocesseshavedifferentsubjects(institutionsvs.programmes)anddifferentpurposes(accreditationvs.audit),asinglereporttemplatewouldnotbepossible,nordesirable.SomemaincharacteristicsofagoodcomprehensivereportcouldbeidentifiedattheEuropeanlevel,whilethedetailsshouldbediscussedanddeterminedinthenationalcontextinconsultationwiththestakeholders.Inaddition,thestakeholdergroupsagreedthattheinstitutionsaretheprimaryusersoftheQAreports,andthereportsshouldthusaddresstheirneedsfirst,beforeconsideringtherequirementsofotheraudiences.Areportsummarywasconsideredasapossiblewaytomeettheneedsofallgroups.

24

25%

32%

27%

14% 2%

In what format would you like to get this information?

Table(s) of numerical data indicating the most important aspects of a HEI/programme

A short, concentrated summary describing the main strengths and areas for improvement of a HEI/programme

Comparative data with other institutions

A comprehensive report providing extensive information about strengths and areas for improvement in management and core processes (study process, research and development), explaining also the possible reasons for a given situation

Other

Figure 13. Preferable format of information.

50  

7  

16  

70  

9  

16  

50  

10  

18  

28  

3  

10  

3  

1  

1  

0%   10%   20%   30%   40%   50%   60%   70%   80%   90%   100%  

HEIs  

Public  authori<es  

Students  

In what format would you like to get this information?

Table(s) of numerical data indicating the most important aspects of a HEI/programme

A short, concentrated summary describing the main strengths and areas for improvement of a HEI/programme

Comparative data with other institutions

A comprehensive report providing extensive information about strengths and areas for improvement in management and core processes (study process, research and development), explaining also the possible reasons for a given situation Other (please name below)

Figure 14. Preferable format of information, sorted by stakeholder group.

Inwhatformatwouldyouliketogetthisinformation?

Inwhatformatwouldyouliketogetthisinformation?

25

preferable sourCes of information

Similartotheanswerscorrespondingtothequestionwhichinquiresastocurrentlyusedsources,themostpreferablesourceforacquiringinformationconcerningthequalityofaHEIistheinstitution’sownwebpage(47%ofresponses)followedcloselybywebpagesofQAagencies(40%).Socialmediaisnotapreferredsourceforinformationonthequalityofinstitutions,withonly5percentofallrespondentsgivingitpriority(Figure15).

Whencomparingdifferentstakeholdergroups,itisevidentthatwhileabouthalfofallgroupspreferinstitutions’webpages,studentsdifferfromHEIsandpublicauthoritiesintheirexpectationsregardingothersources:onlyabout20percentofstudentsseekinformationfromthewebpagesofQAagencies.Atthesametime,theyname‘socialmedia’andeven‘printedreportsinlibraries’moreoftenthanotherstakeholders(Figure16).Somecommentssuggestthatitisnotnecessarytopublishthereportonmorethanonewebsite(e.g.thatofaQAagency),butthateventuallylinkstothepublishedreportshouldbemadeonotherrelevantwebpages(e.g.thatofaninstitution/programme).

8% 5%

40%

47%

Where would you like to find this information?

Printed reports in libraries/QA agencies/HEIs

Social media (please name the most preferred source, e.g. Facebook, Twitter, etc.)

Webpages of QA agencies

Webpages of HEIs

Figure 15. Preferable sources of information.

Wherewouldyouliketofindthisinformation?

26

81  

9  

27  

70  

11  

17  

5  

0  

9  

8  

0  

10  

0%   10%   20%   30%   40%   50%   60%   70%   80%   90%   100%  

HEIs

Public authorities

Students

Where would you like to find this information?

Webpages of HEIs

Webpages of QA agencies

Social media (please name the most preferred source, e.g. Facebook, Twitter, etc.)

Printed reports in libraries/QA agencies/HEIs

Figure 16. Preferable sources of information, sorted by stakeholder group.

ConClusions

Basedontheresultscollectedinthesurveyandatthestakeholders’workshopinMay2013,itispossibletodrawthefollowingconclusionsregardingtheinformationneedsanduseofQAreportsbythestakeholders:

• Thereisnosignificantdifferencebetweentheexpectationsanduseofinformationbetweenthedifferentstakeholders(i.e.HEIs,publicauthoritiesandstudents),exceptexpectationsregardingthetypeofinformationwhichissearched.

• VerylowresponseratefromemployersindicatestheydonotseethemselvesasatargetgroupforQAofhighereducationandareprobablynotfrequentusersofQAreports.

• QAreportsarethethirdsourceofinformationafterwebsitesofHEIsandfriends/colleagueswhenseekinginformationaboutthequalityofaninstitutionoritsprogrammes.

• AwarenessabouttheexistenceofQAreports,andlocationsofwheretheycanbefound,isnotwidespread,therebysignificantlyhinderingtheiruseandinformationalvalueforawidergroupofstakeholders.

• Reportsare,ingeneral,ahelpfultooltogetinformation,buttheycanbemorecomparableanduser-friendly,especiallyintermsoflengthandthelanguageused.

• Reportsshouldcontaineasilycomparabledataintheformatofshort,concentratedsummariesandtableswithquantitativedata.

• ReportsshouldbeaccessiblebothonthewebpagesoftheinstitutionsandtheQAagencies,preferablywithlinkstoeachother.

• SomemaincharacteristicsofagoodcomprehensivereportcouldbeidentifiedattheEuropeanlevel,whilethedetailsshouldbediscussedanddeterminedinthenationalcontextinconsultationwiththestakeholders.

Wherewouldyouliketofindthisinformation?

0

27

• InstitutionsaretheprimaryusersofQAreports,andreportsshouldthusaddresstheirneedsfirst,beforeconsideringtherequirementsofotheraudiences.Areportsummarywasconsideredasapossiblewaytomeettheneedsofallgroups.

28

Chapter 4:

analYsis oF QUaliTY assUrance aGencies’ cUrrenT pracTices in reporTinG THe oUTcoMes oF THe QUaliTY assUrance proceDUres Stephanie Hering, Swiss Center for Accreditation and Quality Assurance in Higher Education (OAQ)

baCkground, Context and methodology

Asoneoftheprojectactivities,theprojectteamsurveyedEuropeanqualityassurance(QA)agenciesontheircurrentpracticesinthepublication,purpose,structureandcontentofQAreports.ThesurveyoftheQAagenciesconsistedoftwoparts:

1. AquestionnaireforallENQAagencieswassentduringsummer2013,including41fullmembersand45affiliates.Atotalof50responseswerereceived(Annex2).

2. In-depthanalysisofasampleofapproximately20reports,conductedduringautumn/winter2013-2014byENQAandtheprojectpartneragenciesfollowingasharedanalysisgridinordertoconsolidateandframethefindingsofthequestionnaire.

Furthermore,theresultsofthesurveyandreportanalysiswerepresentedandfurtherdiscussedataworkshopforagencyrepresentativesinJanuary2014,inZurich,Switzerland.

Somequestionsinthequestionnaireturnedouttobeirrelevantornotsignificant.Inthefollowingsection,onlyrelevantandsignificantresultsareshowninordertokeepthereportcomprehensibleandreadable,highlightingthemostimportantfindings.Thesurveydistinguishedbetweeninstitutionalandprogrammeassessmentreports,andthefocuslayonpublished,publiclyavailablereports.

Awareofthedifferentuseandtargetgroupsforeachtypeofreport,thecurrentanalysishasfurthermoredistinguishedbetweencomprehensiveandsummary reports,whicharedefinedasfollows:

Comprehensive report:

AnextensivereviewreportwhichdocumentsthefullanalyticaloutcomesofagivenexternalQAassessmentprocedure,beitatinstitutionalorprogrammelevel,beitwrittenbyacademicexperts,agencyemployeesoranexternaltechnicalsecretary;anin-depthanalysisuponwhichthemainfindingsarebasedismadeexplicitandisakeycharacteristicofthistypeofreport.Oftenthisistheprimaryreport.

29

Q 4. How many institutional reviews / assessments does your agency conduct on average per year? (n = 46)

summary report:

AnysummarisingformofreportingtheoutcomesofanexternalQAassessmentprocedure,beitasummary,adescription,atableofcomparativedata,afinalproceduralreport,webtextorotherpossibletypesandformsofdescriptiveorschematicreporting.Allkindsofderivativeformsofaprimarycomprehensivereportareincludedhere(exceptexclusivelyyes/noassessmentresults).

results of the surVey

Theagencysurveycontained31questionscoveringthefollowingitems:

• Typeofassessmentsconductedandreportspublished• Publicationandintended/desiredreadership• Editingandpublicationpractices• Structure• Content• Usabilityandutility• Perspectives

Themainresultsofeachofthesearepresentedbelow.

type of assessments ConduCted and reports published

Theagencieswereaskedhowmanyinstitutionalreviewsorassessmentstheycarryout,onaverage,inayear.Asthefigurebelowshows(Figure1),thevariationbetweenagenciesissignificant.

Almostathird(30%)oftheagenciesconductbetween6and15institutionalreviewsorassessmentseachyear.Thesecondlargestgroupofrespondents,28percent,completenoreviewsinayear.Thiscanbeexplainedbythefactthatsomeoftheagenciesarespecialisedinprogrammeassessmentsand/orarenewagenciesjustabouttostarttheirwork.Thirteenpercentoftheagenciesconductmorethan31reviewsorassessmentsperyear.

Figure 1. amount of institutional reviews/assessments per year.

30

Q 5. What kind of report is issued for the institutional review / assessment? (multiple answers, n = 34)

Q 7. How many programme reviews / assessments does your agency conduct on average per year (span of 5-7 years, n = 46)?

Publishedcomprehensivereportsarethemostcommonformtopresenttheresultsofinstitutionalreviews(Figure2).Thisisthecasefor77percentoftherespondents.Forty-onepercentalsoreportontheresultsoftheinstitutionalreviewsinpublishedsummaryreports.Onlythreepercentofagenciesissuenon-publishedsummaryreports,whileasmanyassixagenciesreportedthattheydonotpublishthecomprehensivereport.

Figure 2. Types of reports issued for institutional reviews/assessments.

Programmereviewsaremorecommonthaninstitutionalreviewsamongtherespondents(Figure3).Morethan60percentoftheagenciesconductover30programmereviewsperyear,and35percentimplementbetweenoneand30programmereviewsannually.Onlyfourpercentdonotexecuteanyprogrammereviews.

Figure 3. amount of programme reviews/assessments per year.

31

Figure 4. Types of reports issued for programme reviews/assessments.

Three-quarters(76%)ofthoseconductingprogrammeassessmentspublishedthecomprehensivereports,asissimilartothe77percentforinstitutionalreviews(Figure4andFigure2,respectively).Ofall,31percentdisclosetheresultsofprogrammereviewsintheformofpublishedsummaryreports.

publiCation and readership

ItisevidentthatdifferentQAreportsmeetdifferentgoalsandareintendedfordifferentpurposes(Figure5).Overall,institutional comprehensive reportsareexpectedtofacilitaterevieworaccreditationdecisionsandtosupplyfeedbacktothehighereducationinstitution(HEI)tosupportitsinternalqualityenhancement.Programme comprehensive reportshavetheprimaryobjectivetofacilitatearevieworaccreditationdecision,whiletheymayalsobeusedforenhancementpurposesandtoassuretransparency.

Institutional summary reportsaswellasprogramme summary reports aremainlypublishedtosupplyinformationtothegeneralpublicandtoassuretransparency.Theinformationcontainedinthesummaryreportsisconciseandeasiertoread,butnecessarilylimitedindetailanddepth.

Figure 5. Purpose of reports.

Q 8. What kind of report is issued for the programme review / assessment? (multiple options, n = 45)

Q 10. What is the main purpose of the reports? (multiple options, n = 48)

32

Differentstakeholdersusedifferentreports(Figure6),andindeedthedifferentreportsaredesignedtomeettheneedsoftheidentifiedtargetgroups.AccordingtotheperceptionsoftheQAagencies,institutionsandofficialauthoritiesorgovernmentalbodiesusethereportsmostfrequently.Overall,itappearsthatcomprehensivereportsaremorefrequentlyusedthansummaryreports,andtheformerarefundamentallyimportantforHEIsandofficialauthorities.AccordingtothesurveyedQAagencies,programmecomprehensivereportsare,aftertheHEIsandofficialauthorities,themostimportantforstudents.

Figure 6. users of published reports.

Theagenciesagreethatthereportsshouldideallybeusedmore,byallstakeholders.Inparticular,theagencieswouldliketohavemorestudentsandprofessionalorganisationsoremployersamongtheirreadership(Figure7).Additionally,themediaisconsideredarelativelyimportantfuturetargetuserofQAreports.Agencieswouldalsoliketoseethecomprehensivereportsbeusedmorethantheyarenowandseemtogivelessimportancetothesummaryreports.Ontheotherhand,thestakeholders(inparticularthoserepresentingtheworldofwork)expressedastrongpleaforclearandcomparablesummaryreportsinthesurveyofstakeholders(seeChapter3).

Q 11. In your opinion, which stakeholder groups currently tend to use your published reports? (multiple options, n = 48)

33

Figure 7. Potential readership, sorted by stakeholder group.

DiscussingthequestionofthecurrentandpotentialfuturereadershipofpublishedQAreportsduringtheworkshopinZurichinJanuary2014,somerespondentspointedoutthatitmaynotbenecessarytohavedifferentreportstomeettheneedsofdifferentstakeholders,butthatthedifferentpartsofthesamereportmaybeofinteresttodifferentusergroups.Inordertoputthisintopractice,thereportswillneedtobewell-structuredandtheirpurposeandcontentclearlyexplainedtoallpotentialusers.Ontheotherhand,someotheragenciesareworriedthattryingtomeettheneedsofdifferentreaderswouldleadtothemodificationofthefullreports,arguingitisimportanttomaintainthemfortheirprimaryusergroups,i.e.theinstitutionitselfandtherelevantpublicauthorities.Manyagenciesexpressedawarenessofthedifficultyinmeetingthedifferentinformationneedsofstakeholdersandcommentedthattheyarecurrentlyconsideringdifferentcommunicationandpublicationstrategiestomeetthoseexpectationsbetter.Anotherimportantchallengeherederivesfromthefactthateventostakeholdersthemselves,itisnotalwaysnoryetclearwhatexactlytheywouldandcouldexpectfromaQAreport(seesectionentitled“Usabilityandutilityofreports”onpage37).

editing and publiCation of reports

Currently,agenciespublishtheQAreportstheyproducemainlyinthenationallanguage(s)(Figure8).Atthesametime,Englishhasbecomeanimportantworkinglanguageforagencieswhichutiliseinternationalexpertsandisgaininggroundasthepublicationlanguageofcomprehensivereportsinparticular.Languagesotherthanthenationalone(s)andEnglishseemnottoberelevantinthiscontext.Consideringthatforthemajorityofagenciestheinstitutionsthemselvesand/orofficialauthorities/governmentalbodiesareconsideredasthemostimportantreadershipofQAreports,itcomesasnosurprisethatthenationallanguageisthefirstandforemostlanguageinwhichthesereportsaredrafted.

Q 12. Who might be your potential or future readership? Which stakeholders would you like to use your reports more? (multiple options, n = 47)

34

Figure 8. Languages in which reports are made available.

Mostoftenthepanelofexperts,thepeerleaderandthesecretaryofthepanelofexperts(sometimesaQAagencyemployee)arethemainauthorsofthereports(Figure9).Forcomprehensivereports,theroleofthepanelisstronger,whilesummaryreportsmaybeeditedbythesecretaryofthepanelofexpertsand/oranagencyemployee.

Figure 9. Writers of reports.

Q 14. In which language(s) are your reports available? (n = 48)

Q 15. Who writes the reports? (n = 48)

35

PreparingQAreportscanbehighlytime-consuming.Thesurveyaskedtheagenciestoprovideanestimateontheaveragetimethatittakestoproduceareport(ofthetypeusedbytheagency),consideringthetotalinputoftimebetweenallthoseinvolved(panel,QAagencystaff,etc.).Unsurprisingly,thepreparationofthecomprehensivereportsrequiresthehighesttimeinvestment(Figure10).Mostagenciesestimatetheyspendatleast16hourstowriteaninstitutionalcomprehensivereport.Thecompletionofacomprehensiveprogrammereportisnearlyastime-consuming.Agenciesindicatedthatsummaryreportsaremostlywritteninlessthan5hours.Thismayneverthelessconstituteasignificantadditionaltimeexpenditure,inparticularforagenciesthatmaynotpublishsummaryreports.

Figure 10. Time spent to complete a report.

ThereportsaremostoftenpublishedonthewebsitesofQAagencies(Figure11).ThewebpagesofotherpublicauthoritiesandofHEIsareoflessinterest,andagenciesdonotoftenhaveinformationonwhetherinstitutionspublishthereportsontheirownwebsitesornot.Thehardcopiesofreportsareconsideredirrelevantbytheagencies,orarenotapriorityconsideringalsotheadditionalcostsassociatedwithprintingreports.Atthesametime,thestakeholdersconsiderthewebsitesoftheHEIsasthemainsourceofinformation,andthereforeitseemsthatinordertoreachawidergroupoftheintendedreadership,atleastlinkstotheagencywebsiteshouldbeprovidedontheinstitutions’websites(seeChapter3).

Q 16. How much time on average does it take to complete a report (summing up the time invested by all authors* )? (n = 48)

0-5

36

Figure 11. Location of published reports.

struCture of reports

Nearlyallagencieshavetemplatesavailableforwritingreports(Figure12),whichmeansthatapre-setstructureisfollowedthroughoutthedraftingprocessforallreportsofthesametypology.Templatesexistforallkindsofdifferentreportsbutareconsideredthemostimportantandrelevantfortheprogrammecomprehensivereports.

Figure 12. Templates for reports.

Thelengthofthereports(Figure13)variesaccordingtothetypeofreport,buttheyoftenfollowguidelinessetbytheagency.Themajorityofcomprehensivereportsfallintothe‘31pagesormore’category(55%ofinstitutionalreviewsand33%ofprogrammereviews).Noinstitutionalcomprehensivereportsareshorterthan6pages.

Q 17. Where are the reports published? (n = 47)

Q 19. Does your agency have a template for the writing of reports? (n = 47)

37

Forprogrammeassessments,thereportstendtobeslightlyshorter,withonly25percentfallingintotherangeof31ormorepagesinlength.Theshorterlengthofprogrammereportsisunderscoredbythefactthatthemajorityfallbetweensixand15pages.Summaryreportsareshort,astheirnameindicates,andtheyaremostoftenundersixpages.

Figure 13. Length of reports.

Reportsuseavarietyofformatstopresentinformation(Figure14).Discursive,detailedinformationismainlyusedforcomprehensivereports,andprovisionofkeydataisparticularlyimportantforprogrammecomprehensivereports.Summaryreportsusenearlynotablesofnumericaldataandschematiccomparativedata.

Figure 14. Formats of reports.

Q 20. What is the average length of the reports (please provide indicative number of pages)? (n = 48)

Q 21. In which format is the report information presented?

(Please choose as many options as applicable, n = 46)

38

Content of reports

Reportsprovideawiderangeofinformationandaddressdifferentrequirementsandinformationneeds.Principally,theagencieswanttoprovideinformationonthefinaloutcomesofanassessment,proposesuggestionsforqualityenhancementandpublishgeneralfindingsintheirreports(Figure15).Also,thedescriptionofformalcriteria,qualitystandardsandQAsystemsisofgreatimportance.Thesameappliestotheinformationaboutsitevisitsandthelearningenvironment.SomeagenciespointedoutthatthecontentofthereportsisconstantlyadjustedbasedonfeedbackandrecommendationsfromthereviewteamsandfromtheHEIs.Accordingtothefeedbackreceivedthroughtheagencyworkshop,insomecountries,theagenciesarenotallowedtopublishinformationordatathatcouldbeusedtocreaterankingsatthenationallevel.

Figure 15. Information provided in reports.

Q 23. What information does your agency systematically provide in the reports? (n = 47)

39

usability and utility of reports

Anoverwhelmingmajorityoftheagencies(46outof47)statedthatthecomprehensivereportsneedtobefirstandforemostusefulfortheHEIs(Figure16).HighimportanceisalsogiventotheroleofQAreportsinprovidingclearandtransparentinformationonthequalityofinstitutionsandprogrammes.Easyaccesstothereportsisalsocrucial.Agenciesdonotgive,overall,highimportancetotheusefulnessofthereportsforfundingdecisionsorforthemedia.

Figure 16. Comprehensive reports.

Themainpurposeofthesummaryreportsistoprovideclearandtransparentinformationinaneasilyreadableandaccessiblemanner(Figure17).Thesummaryreportsmayalsobeusefulforthepublicauthoritiesorthemedia.

Q 25. How would you best describe your comprehensive reports? (n = 47)

40

Figure 17. summary reports.

perspeCtiVe of agenCies on their reports

main strengths and weaknesses of the published reports

WhenaskedaboutthemainstrengthsandweaknessesrelatedtotheirQAreports,theagenciesidentifytransparencyandrecommendationsforqualityimprovementasthemainstrengthsofthepublishedreports.Ontheotherhand,theformatandconsistencyofthereports,aswellasthecomplicatedlanguageoftenused,areconsideredasimportantareasofimprovement.

Strengths:

• Transparency,opentothepublic(7)• Recommendationsforqualityimprovement(6)• Easilyreadable(5)• Qualityandlevelofinformation(4)• UsefulinformationforHEIs(4)• Format(3)• Easyaccess(3)

Otherpositivecharacteristicsmentionedinclude:clearstructure;coverageofawideareaofsubjects;availabilityinboththenationallanguageandinEnglish;fullinformationonthemembersofexpertpanelsandtheirexpertise;andthevisualpresentationofinformation.

Q 26. How would you best describe your summary reports? (n = 23)

41

Weaknesses:

• Formatandconsistencyofthereportschanges,whichdependsoftenontheexperts(4)

• Useofcomplicatedlanguage(3)• AvailabilityofreportsonlyinEnglish(2)

Othershortcomingsidentifiedbytheagenciesinclude:complexityofinformationprovided;notenoughinformationonevaluationprocedures;delaysinfinishingthereportsduetoexpertsbeinglateinsubmittingtheirparts;delaysbetweendecisionandpublicationofthereport;reportsareonlyavailableinthenationallanguage;reportsarenotusefulforstudentsandemployers;summariesdonotincludethekeydataoftheinstitution;andHEIsdonotpublishthereportsontheirwebpage.

elements that would inCrease the quality and usefulness of reports in the ehea

Therespondentswerealsoaskedwhichelements,accordingtotheirexpertise,couldincreasethequalityandusefulnessofreportsproducedbyQAagenciesintheEuropeanHigherEducationArea(EHEA).Sixrespondentsagreethatcomparabilityofreportswouldbemostwelcome.Inaddition,fiveagenciesmentionedtheuseofclearlanguageandtheavailabilityofaEuropeantemplate,orEuropeanGuidelinesforreportsasimportantimprovementstothereportsattheEuropeanlevel.

Inaddition,thefollowingpointsweremade:

• Clearterminology(4)• Availableforthepublic(4)• Goodstructureandeasilysearchable(4)• Englishreports(3)• Reportsshouldbemoreclearlytargeted(3)• OutliningbestpracticeintheHEIs;Opinionofforeignexpert/s(3)• Availabilityofrecommendations(2)

possible Challenges and risks in trying to inCrease tr ansparenCy and Compar ability of reports

Throughouttheproject,theprojectteamhasbeenawareofthepotentialriskoflimitingtheusefulnessofreportsbytryingtoadaptthemtowiderordifferentreaderships.Theagencieswherethusaskedtoexpresstheirviewsonthepotentialrisksorchallengesrelatedtotheimprovedaccesstomorecomparablereports.

Somequotesfromthesurveycollectingthevoiceswere:

• ComparisonofreportswillturnQAintoanotherformofranking.Reportsshouldneverbecomparable,onlytheprocedureand/orthedecisionmakingshouldbecomparable,i.e.consistent.

• Reportscontainanexcessiveamountofdatawhichmakestheirusedifficultandwhichmayleadtospeculationandunfaircompetition.

• Universitiescouldperceivetheprocessasanexerciseof“revealingitall".Itcanshowparticularweaknessesthatcouldmakesomestudentsthinktochooseanotheroption.

42

• GoodcomprehensivereportsinboththenationallanguageandEnglishrequirealotofworkandexpertiseandwell-establishedagencies.Publishingreportsrevealsnotonlythequalityoftheinstitutionorprogrammebutalsooftheagency.

• TransparencyisnotalwaysintheHEI’sbestinterest,asthereportsalsoshowtheirweaknesses.Thus,greatertransparencywouldleadtomoreprotestsagainstpublicationorlessopennessintheprocedures.

• Legalframeworksmightlimitthewayreportsarepublished.• Tomakereportsmorecomparative,allagencieswouldneedtochangetheircurrent

practices.• QAreportsdealwithverycomplexissues.Oversimplifyingthecontentisnot

appropriate.ItisimportanttodistinguishbetweenHEIsthemselvesandotherpotentialusersofthereports.

• Agenciesaredealingwithalargenumberofprocesseseveryyearandusealargenumberofexperts.Itwillbedifficulttoensurefullconsistencyofthereports.

• The(national)systemsareverydiverseandarealsoatdifferentdevelopmentstages,asaretheneedsandaimsofdifferentQAsystems.Thismakescomparisondifficultandrisky.

ConClusions

Basedonthefindingsofthesurveyandthein-depthanalysisofasampleofQAreports,conclusionscouldbedrawnasfollows:

key findings• Publishedcomprehensivereportsarethemostcommonformtopresenttheresults

ofinstitutionalreviews.• Thevariationinthenumberofpublicationsofinstitutionalreviewsperyearamong

agenciesishigh.• Reportsmeetdifferentgoalsandareusedbydifferentstakeholders:

– Comprehensivereportsfacilitaterevieworaccreditationdecisionsandsupplyfeedbacktotheinstitutionsforqualityenhancement.

– Summaryreportssupplyinformationtoageneralpublicandassuretransparency.

• QAreportsaremainlywrittenbyapanelofexpertsandrequiresignificanttimeinput.

• Webpages,primarilythoseoftheQAagencies,arethemostcommonmediumofpublication.

• Themajorityofagencieshavetemplatesavailableforthewritingofreports.• Comprehensivereportsareoftenlongerthan30pages.Programmereportsare,on

average,slightlyshorterthaninstitutionalreports.• Reportsprovideawiderangeofinformationandaddressdifferentrequirements.• QAreportsvarygreatlybetweencountriesandagenciesbutalsobetween

programmeandinstitutionalreports.Inaddition,comprehensiveandsummaryreports-andeveneditingpractice,structure,content,length,readability,accessibility,etc.-withinasingleagencydifferconsiderably.

• QAreportsare–atthemoment–notcomparable.

43

views on quality reports

EssentialqualitiesofagoodandusefulQAreport:

• Clearstructure• Introductiontotheframeworkandstandards/guidelines• Informationonprocedure/reviewandreviewpanel/author(s)• DetailedinformationaboutHEI/programme• Adequatecomplexity-reduction(withoutoversimplification)• Carefuluseofterminology/language• Pleasantlayout• Easilyaccessible

risks and Challenges

Possiblerisksandchallengesintryingtoincreasetransparencyandcomparabilityofreports:

• Transparencyitself:misuseormanipulationofinformation• Comparability/benchmarking• Losesightoftheactualpurposeofthereports(usefulnessfortheinstitution)• Overratingthepotentialvalueandinterestforwiderpublic/layreader• Oversimplification• Standardisation:comparingappleswithpears• Languageissues

44

Chapter 5:

eUropean GUiDelines For sUMMarY reporTs anD Generic recoMMenDaTions For coMpreHensiVe reporTs oF eXTernal QUaliTY assUrance proceDUresĐurđica Dragojević, Agency for Science and Higher Education (ASHE)

european guidelines for summary Quality assuranCe reports

Theresultsofthesurveyontheuseandusefulnessofqualityassurance(QA)reportsconductedbythestakeholdersshowthatthereisaclearneedformorecomparableandaccessibleinformationtobeprovidedbytheQAagencies.WhileoneoftheinitialaimsoftheprojectwastodevelopaEuropeantemplateforcomprehensivereports,thiswasdeemedbytheprojectconsortiumasunfeasibleduetothedifferencesinQAproceduresandsystemsofhighereducationandthespecificneedsofstakeholdersandauthoritiesrelatingtotheparticularpoliticalframeworkofexternalQAinhighereducation.Approximatelyone-thirdoftherespondentstakeholdersexpressedtheneedforsummaryreportstobepublishedontheagencies’websitesinordertoserveasanintroductiontothedatacollectedbytheQAagenciesinthecomprehensivereportsandtoprovidecomparableinformationtothewiderpublic.Thisisinlinewithstandard2.5ofthecurrent(and2.6ofthedraftrevised)ESGonthepublicationofQAreports.

Wehopethattheseguidelines,togetherwiththeESGstandardonthepublicationofQAreports,willencouragealargenumberofQAagenciestopublishsummaryreports.Whilefullstandardisationisavoidedbysuggestingtheuseofguidelinesratherthantemplates,theseguidelinesaimtoensurethatallsummaryreportsprovidesimilarandcomparableinformation,thusincreasingtheirusefulnessfortheintendedusergroupsandthetransparencyoftheagencies’work.

1. Typesofsummaryreports

Dependingonthetypesofevaluationsperformed,anagencymaywishtohaveacommontemplateforsummaryreportsforallofitsprocedures,or,inlinewiththecomprehensivereports,aseparatetemplateforinstitutionalandprogrammeevaluationsummaryreports.

Thetargetaudiencesofsummaryreportsarevarious–fromstudentsandprospectivestudentsandtheirparentstoemployers,otherhighereducationinstitutions(HEIs),governmentbodiesandmediarepresentatives.Asummaryreportshouldprovideclearandsuccinctinformationontheevaluation,whichshouldbeunderstandabletomembers

45

ofthegeneralpublicandsufficientasafirstsourceofinformation,withouttargetinganygroupspecifically.Targetingspecificgroupswouldrequiredevelopmentofanumberofdifferenttemplates,whichmightreducetheircomparabilityandsignificantlyincreasethetimenecessaryforproducingthem,withoutsufficientadditionalbenefits.However,agenciesarewell-advisedtosurveypotentialtargetgroupsontheuseofsummaryreportsandaddadditionalinformationifrequestedbyspecifictargetgroups.

2. Placement

Thesummaryreportsshouldbeaccessibletoaswideanaudienceaspossible.Forthisreason,asinthecaseofcomprehensivereports,itisrecommendedtheybeeasilyfoundbysearchenginesthroughtheuseofkeywords,suchasthenameoftheagencyandtheinstitution(s)and/orprogramme(s)andtermsconnectedwiththetypeofevaluation(e.g.accreditation,audit,etc.).

Thesummaryreportsshouldbeeasilyaccessibleontheagencies’websites,andtheagenciesmaywishtodisseminatethemfurther,forexample,byaskingtheinstitutionstoplacethemontheirownwebsitesorbyplacingthemonotherwebsitesthatprovideinformationonhighereducation(suchasthewebsitesofnationalENIC-NARICs,Studyin...websitesprovidinginformationtoforeignstudents,etc.).

3. Drafting

Toensurehomogeneityandconsistentuseofterminology,itisbestthatreportsarewrittenbyagencystaff.Asummaryreportshouldonlysummariseandpresentinaclearandsuccinctformsomeoftheinformationprovidedinthecomprehensivereport,withoutaddinganyadditionalinformation.Thesummaryreportshouldbesenttothepanelfortheircheckandapprovalsothatwhatishighlightedinthesummaryreportreflectstheoverallimpressionofthepanel.

4. Developingtemplates

Thecorporateimageoftheagencyshouldbeincludedinthesummaryreport,soitiseasilyrecognisableanddistinctive.Thetemplateshouldenablethesummaryreporttobepublishedasawebpage(i.e.thepartoftheinstitutionalwebsitewheretheconnectedcomprehensivereportcanbefound)withaprint-readyversion(e.g.pdf)whichcanbeeasilyprintedorevendistributedinaprintform(e.g.asaleaflet).Whenprinted,asummaryreportshouldnotbelongerthantwopages,inordertoprovideinformationinaconciseandstructuredway.

Giventhatthelayoutandthedesignarecrucialforthereadabilityanduser-friendlinessofthesummaryreports,theagencyshouldbecarefultoincludeheadings/textboxesandotherlayoutsegmentswhichprovidemaximumclarityandeaseofcommunication.Whendevelopingtemplates,theagencymaybewell-advisedtoworkwithcommunicationanddesignexperts.

5. Language

GiventhatoneofthefunctionsofthesummaryreportistoenablecomparabilityandprovideinformationacrosstheEuropeanHigherEducationArea(EHEA),forexample,toinstitutionslookingforpartnersinothercountriesorforeignstudents,summaryreportsshouldbeproducedinEnglish.However,dependingonthelanguage(s)usedby

46

theagency’sstakeholders,theagencymayalsowishtopublishsummaryreportsinthenationallanguage(s)oftheagencyortheinstitution.

Becausethesummaryreportsareaimedatthegeneralpublic,meaningarangeofstakeholdergroups,thelanguageshouldavoidspecialistjargonandprovideexplanationswhenitcannotbeavoided.ThisisespeciallytrueforQAandlegalterminologywhichcanoftengounnoticedbyexperts,whiledecreasingthetransparencyandtheunderstandabilityofthereportbythewideraudience.Inputfromlinguisticexpertsandstakeholderscanbehelpfulinthisaspect.

6. Content–minimumusefulinformationtobeprovidedinthesummaryreports

Summaryreportsshouldcontainbasicinformationontheagencyandtheinstitution/programmeunderevaluation.Thefactthatthesummarieswillbemadeavailableonlinemakesiteasytoinsertlinkstorelatedwebsitesandthecomprehensivereportonwhichitisbased.

Giventhatthesummaryreportisadocumentproducedbytheagency,butalsogiventhetypeofinformationthestakeholdersseekinthistypeofreport,thenamesofthepanelmemberswhohaveperformedtheevaluationneednottobementioned.Instead,thecompositionofthepanelshouldbenotedinordertogiveanideaofthegroupofstakeholdersrepresentedinthepanel(e.g.,academic,student,employer,etc.).

Asindicatedbystakeholdersinthesurvey,layreaderswouldalsoliketobeabletofindinformationwhichisoftenobviousforexpertreaders–individualsfromagenciesandinstitutions–andisthussometimesomittedfromcomprehensivereports.Suchinformationincludesthetypeofevaluation,whetheritisobligatoryorvoluntary,anditsfocus(ifany–e.g.,management,internationalisation,etc.).Hyperlinksshouldagainbeinsertedinthesummaryreporttomakethecomprehensivereporteasytofindforanyoneinterestedinit,possiblyaddingabriefdescriptionofthetypeofinformationthatcanbefoundthere.Othertypesofcontextualinformation–thedescriptionoftheQAsystemwithinthecountry,thecriteriausedintheevaluations,etc.shouldbeaccessibleontheagency’swebsiteandthusnotincludedinthesummaryreport.

TakingintoaccountthefactthatthegeneralpublicoftenlooksatQAreportssimplytoestablishiftheinstitutionand/orprogrammeisaccredited,thisinformationshouldbementionedinthesummaryreports.Inadditiontothis,becauseexternalQAdecisionsoftenhavean‘expirydate’,thedatewhenthereportwaspublishedorthefinaldecisionwasissuedshouldbenotedtogetherwiththevalidityperiod.Thiswillhelpthereadertoeasilyestablishwhentheevaluationwasconductedandwhetheritmaybenecessarytolookfurtherforreportsonmorerecentevaluations.

Althoughreportsshouldprimarilyprovideasummaryofthepanel’sfindingsratherthaninformationaboutevaluatedinstitutionsandprogrammes,somebasicinformationshouldbeprovidedtothereader.Thistypeofinformationwillvaryamongthedifferenttypesofreports,butshouldincludeinformationonthestatusoftheinstitution(e.g.,ifitisprivateorpublic,ifitisauniversityorsomeothertypeofHEI).Forprogrammereports,theBolognaframework(QF-EHEA),theNationalQualificationsFramework(NQF)and,ifapplicable,theEuropeanQualificationsFramework(EQF)leveloftheprogrammeshouldbeindicated,withotherinformationrelevantfortheevaluation(e.g.,ifitisanacademicoraprofessionalprogramme,ajointprogramme,etc.).

47

Content of summary reports: CheCklist

Information on the evaluation

• Thenameoftheagencywhichperformedtheevaluation.• Linktothewebsiteoftheagency.• Typeandfocusofevaluation(e.g.institutionalaudit,auditwithafocuson

internationalisation)andifitwasobligatoryorvoluntary(with,possibly,abriefmentionofthewayitfitsinwiththenationallegalframework/otheractivitiesoftheagency–e.g.,obligatoryaccreditationofstudyprogrammesaccordingtothenationalcriteria).

• Thedatewhenthereport/decisionwasissued,thevalidityperiodandthedateuntilwhichtherecommendationsshouldbeimplemented.

• Linktothecomprehensivereport(possiblywithanoteonitscontent,e.g.“recommendationsforimprovementanddetailedanalysiscanbefoundinthecomprehensivereport”).

• Abriefexplanationonthecompositionofthepanel.

Information on the institution/programme

• Nameoftheinstitution/programme.• Linktotheinstitutionalwebsiteand/orstudyprogrammewebsite.• Typeofinstitution(e.g.privateorpublic,professionaloruniversityforbinary

systems)orprogramme(BolognaFramework,NQFand,ifpossible,EQFlevel,whetheritisprofessionaloracademic,ifitisajointprogramme,etc.).

general reCommendations for ComprehensiVe Quality assuranCe reports

ThefindingsofthesurveyontheuseandusefulnessofQAreportsaddressedtothestakeholdersshowedthatQAreportscanbeattimestechnicalwithacomplicatedlanguageandtoodifferentinstyleandcontent.Thestakeholdersmentionedthatamorestandardisedapproachwouldbehelpful,butthepurpose(supportingtherevieworanaccreditationprocess)andtheverydiverseandcomplexnatureofcomprehensiveQAreportsdoesnotallowthedevelopmentofadetailedsetofcontents,ascouldbedoneinthecaseforthesummaryreports.Comprehensivereportsprovideimportantqualitativeinformationoninstitutionsorprogrammes.Givinganindicationofthecontentforcomprehensivereportswouldlimittheirusabilityandwouldthusresultinthelossofsomevaluableinformationortheusefulnessoftheinformationtothemainusers.Nevertheless,thereisspaceforsomegenericrecommendationsonthestructure,layout,andpublishingofthesereportstothosewhoproducethem.

Itisimportantthatcomprehensivereportsprovideclearandunderstandableinformationinordertomakethemmoreusableandaccessiblefordifferentgroupsofstakeholders,includingthosewithintheinstitutionssubjecttothereview.Inthisperspective,whenproducingcomprehensivereports,oneshouldconsiderthegenericrecommendationsbelow:

48

1. structure

• Thecomprehensivereportshouldbewell-structuredandeasytoread.• Alistofcontentsshouldbeprovidedatthebeginningofthereportinorderto

facilitatethefindingofspecificinformation.• Clearheadings,sub-headingsandnumberedparagraphsshouldbeusedinorderto

clearlyindicatethedifferentsectionsofthereport.

2. Content

• Thereportshouldclearlydescribetheprocess,recommendations,commendationsandconditionsclearlyidentifiableandthecontextofthereview.

• Thereportshouldusekeyterminologyconsistently.• Itisimportantthatthereportcontainsaglossaryexplainingtechnicaltermsfor

thosewhoarenotfamiliarwithQA.• Thereportshouldidentifyandpresenttheperson(s)whowrotethereportaswell

asthecompositionofthereviewteam.

3. Publication and dissemination

• Thereportshouldbemadeeasilyaccessible:itshouldbepublishedonlineandinaneasilyprintableformat.

• Thedecisions,institutionalresponsesandfollowupdocumentsshouldbepublishedtogetherwiththereport.

• ThereviewedHEI/programmeshouldbeencouragedtopublishthereportontheinstitutionalwebsite,oratleast,toprovidealinktotheagencywebsitewherethereportispublished.

Inadditiontothesefindings,thereflectionsonQAreportsbyPeterFindlayinthenextchapterprovidesomeinterestingpointstoconsiderinrelationtoQAreports.

49

Chapter 6:

FeaTUres oF a GooD eXTernal QUaliTY assUrance reporT – soMe THoUGHTsPeter Findlay, Quality Assurance Agency for Higher Education (QAA)

This collection of reflections and questions about the quality assurance (QA) reports produced by agencies is based on a presentation, part serious, part light-hearted and occasionally provocative, made at the EQArep workshop held in January 2014. While these reflections draw on some twenty years of working with QAA and its predecessor, the ideas here do not necessarily represent either the policy or practice of QAA as an agency.

qa questions for reports

Someoftheperennialandwell-triedquestionsforQAalsoapply,ofcourse,tothequalityofouragencyreports:

• Forwhatisitintended?(aims,readership)• Whyarewedoingitthatway?(methodandformatadopted)• Isthatthebestway?(evaluationofthemethod)• Howdoweknowitworks?(achievingtheaims,impactofreports)• Howcoulditbeimproved?(reviewing,evaluating,planning)

a report is a message

Areportis,intermsofstandardcommunicationtheory,justanotherkindofmessage.Itseffectivecommunicationwilldependonthreethings:thesender(theexpertteam,theagency),themedium(theformofthereport,themeansofitspublication)andthereceiver(theuniversityandotherstakeholderssuchasstudentsandemployers).

Messagesarealsogovernedbytherulethatthepurpose,thecontentandthereadershiporaudiencearethreeinter-relatedfactorswhichwilldetermineeachother.Anyreporthastotakeintoaccountitsmostimportantaims,itslikelyreadership,andthebestforminwhichtoaddressthese.

Communicationtheoryusestheterms‘encoding’and‘decoding’withregardtomessages.Forourconsiderations,‘encoding’means:

• Knowingthepurposeofthereport• Knowingthecontentofthereport• Knowinghowitwillbewrittenandthestagesofproduction• Knowingwhowill/shouldreadit• Knowinghowandwhentheywillreadit

50

and‘decoding’implies:

• thatthereportisreceivedandreadinamanagedprocess• thatthereaderunderstandstheprocessinvolvedandtheconventionsofthereport• thatthereaderknowswhatisexpectedfromthereportintermsofactionstobe

takenorinformationtobeused.

who is it for?

Anagencythereforeneedstobeveryclearindecidingtheaimsofitsreportsandtheirintendedreadershipandusers.Unfortunately,thatisoftennotasimplematter.Itwouldbereasonablystraightforwardifreportsweresimplywrittenfortheexclusiveattentionofhighereducationinstitutions(HEIs),butthereareotherinterestedparties–governments,students,employers,etc.whomayalsobeamongthetargetreadership.Areviewreportmayneedtoaddressalloftheseinterests,butcanasinglereportdothatsuccessfullywithsuchdifferentexpectationsandneedsinvolved?Amajorquestionforourreportsishowtomeettheneedsofthesedifferentstakeholders.

Answerstothisknottyproblemthathavebeentried:

• Separatereportsforseparatestakeholders• Asummarysectionofthereportwithclearoutcomes,separatedfromthemore

detailedmainbodyofthereport• Differentpartsofthereportfordifferentreaders• Ashortreportonthewebsiteandalongerreportfortheinstitution• A‘checklist’formatforthekeycriteria,withacommentsboxforthedetails,where

required.

the report is the result

Weneedtokeepinmindthatanagency’sreportistheendresult,oroutcomeofawholeprocess;thecharacterandqualityofthereportwillthereforedependonthequalityofmanyothercontributingaspects:

• Agencymanagement,policyandmethod• Agencyofficerandreviewteam• Documentationandinformationprovided,accesstoinstitutionalinformation• Visitandconfirmation• Draftingthereport• Editingthereport• Finalisingandagreeingonthereport

Itcaneasilybeforgottenthatcompletingthereportistheprimaryaimofallthesedifferentpartsofthereviewprocess.Itisboththeendproductandthefocusofthewholeprocess.Meetingotherreviewersormanagingtheteamisworthwhileandrewarding;thesitevisitislikelytobeexcitingandinteresting;thepreparationofthereport,ontheotherhand,isisolatedanddesk-based,andwilloftenbeademandingandeventedioustask.Itcanalltooeasilybeforgottenaboutduringtheearlierstagesoftheprocess.

51

Ateverystage,therefore,weneedtothinkaheadtothereport:keepingnotes,makingearlydrafts,checkingevaluationagainstthefacts,workingtowardsthejudgements.Agoodreportresultsfromaplannedprocessleadinguptoitsfinalproduction.

who owns what?

Theremust-fromtheverystartoftheprocess-beclarityaboutwho‘owns’thereport–thatis,whomakesthefinaldecisionaboutwhatisincludedinitornot,andhowjudgementsaremade.Sometimesownershipisshared,andthenthevariousresponsibilitiesmustbecarefullydefined.Doesownershipliewith:

• Theexpertpanel?• Thememberofthepanelwhowritesthereport?• Theagencyofficerwhoeditsthereport?• Theagencycommitteethatapprovesthereport?• Theagencyasalegalentity?• Theinstitution?• Mostimportantly,whatpersonorbodyhastherighttomodifythereportand/orits

judgements?

Candifferentparticipantsintheprocessowndifferentpartsofthereport?Forinstance,itmightbearguedthatthejudgementsinareportareownedbytheexpertsaspeerreviewers,buttheformandcontentofthereportareownedbytheagencythathasultimateresponsibilityforthequalityofpublication.

keep it simple , ******!

Itcanbearguedthatcomplicatedandopaquereportswhicharedifficulttoreadmayserveapurpose,whichwouldbetolimittheunderstandingandreceptionofthereporttoalimitedreadershipwhounderstandtheconventionsandcodesinwhichitiswritten.However,iftheaimistoreachaswideandvariousareadershipaspossible,thenthesimpler,thebetter.

Bysimplicityisnotmeantreducingthesignificanceorcomplexityofthecontent,butensuringthatitisconveyedviaaveryclearstructureandinlanguagewhichcanbeeasilyfollowedandunderstood.

Somedesirableandundesirablefeaturesofsuchareportmightbe:

To aim for:

• Length–asshortaspossible• Agoodsign-postedstructurewithclearheadings• Style–simple,precise,clear,focused• Well-structuredparagraphs‘beginning,middle,end’• Sentences,whicharesimpleandclear(neverlongerthantwolinesoftext!)• Tone–neutral,measured,objective• Content–minimumdescription,maximumanalysisandevaluation(itisdifficultto

seetheusefulnessoflargeamountsofdescriptionofinstitutionalprocedures–theinstitutionitselfknowsaboutthemalready,andotherreportreadersprobablydon’treallywanttoknowverymuchaboutthem,exceptwhethertheyareeffective)

52

• Argumentsclearlybasedonevidence• Regularremindersoftherelevancetothereviewquestions/agenda/framework• Regularsummariesofeachsection• Clearexplanationofbasisforjudgementsandrecommendations

To avoid:

• Toomuchreferencetotheprocess(theexpertteam,thereviewvisit,etc.)• Overlycomplexsentences• Jargonandcliché(especiallythejargonofQAexperts)• Emotiveorstronglyjudgmentalwords• Obscurewords• Ambiguity• Toomuchunnecessarydetail• Speculationaboutfuturedevelopmentsintheinstitution• Subjectivecommentsdrawnonexperts’personalexperience

the politiCs of l anguage

Nationallanguagesofsmallnationswon’thavethesamebreadthofimpactorallowinternationalcomparability(butwillstayintheculture),butthenationallanguagereflectsmoreexactlytheownershipandlocationofthereport.

Translationintothelinguafranca(English)bringsrisksofmistranslation,confusion,andmisunderstanding;therearemanyvarietiesofEnglish(a‘Euroglish’isindevelopmentinQAreports!).Wedon’trealisehowvariedourunderstandingofkeytermsinEnglishcanbe(‘accreditation,’‘expert,’‘validation,’‘assessment’and‘standards’aresomekeytermsthathavedifferentmeaningsindifferentcontexts),andyoucan’talwaysbesurewhatEnglishmeanstothereader.

sChedules

Atimelineforthedevelopmentandsubmissionofthereportisessential.Itshouldincludeaclearindicationofwhateachcontributorisexpectedtoprovideandtheiterationsneededforediting.

Deadlinesaretheplagueandsometimestheagonyofreport-writers.Alwaysmakeyourdeadlinesveryrealistic,andthenaddsomemoredayson,tomakeforgreaterflexibilityinmeetingtheproblemsthatwillcertainlyoccur.

tr ansparenCy… i Can see Clearly now

Transparencyisgenerallyregardedasavirtueandagoalofagoodreport,buttransparencycanharmaswellashelp–whatexactlyistheaimofcompleteclarity?Isthemainaimofthereporttoregulate,controlandcompare,orisittoimprove,enhanceanddevelop?Ifweseektoimprove,andtohelpaninstitutiontosucceed,thenisitalwaysagoodideatopublishthefullfacts,whichmightdamageitsreputation(topublishindetailforthepublictoreadallthatiswrongwithaninstitution?)?Isthealternativeacceptable-foranagency’spublicationpolicytobeadjusted,takingintoaccountthemessagedeliveredbythereport?Isthere,then,atensionbetweentransparencyandenhancement?

53

tight- rope walking – the tension between peer review and regul atory body

Mostagenciesuseareportingmethodwhichinvolvesanexpertteam.Eithertheywritetheearlydraftsofthereportthemselves,ortheirviewsandopinionsaregatheredandincorporatedintoareportbytheagencyofficer.Thisapproachfollowstheprincipleofpeerreview.Initspurestform,apeerreviewapproachusingtheinformedopinionoftheexpertswouldmeanthatthoseexpertsownthereviewandwritethereport.Insuchamethod,individualexpertsmightevenbeallowedtosubmitaminority,dissentingstatement.However,agencieswillusuallywantahigherlevelofcontroloverthereport.Theywillrequireexpertstoreachaconsensusonthereportjudgementsandmayimposeacommonframeworkforallreports.

Considerthepotentialspectrumofpossibilitiesfortheformatofthereport.Atoneextreme,purepeerreviewisclosetothecharacterofapersonalnarrativeaboutimpressionsgainedbyexperts;attheotherendofthespectrumissomethinglikeahighlycontrolledchecklist,pre-definedbytheagency,intowhichtheexpertsplacetheirviewsanddecisions.Aquestionduringallreportprocessesiswheretopositiontheformofthereportbetweenthesetwoextremes.Thesignsofagencycontrolwillbefoundintheamountofshapingstructuregiventothereport:specifiedstandardsandcriteria;requiredheadings;checklistsforcompletion;wordcounts;tables;pre-definedformsofjudgementstatements.

Thereisatensionhere,becausethehigherthelevelofcontrolthatisexertedbytheagency,thelesstheexpertsmayfeeltheirviewsandtheexpressionofthemaretakenseriously.Itisalsoquestionablewhetherasimple‘tick-box’conformityapproachcanaccuratelyreflectthecomplexityofinstitutionalsystemsandtheirdifferentapproaches.Soitisamatterofbalancingfreedomofexpression(ahighvalueinouracademiccontext)asagainstthelevelofbureaucraticcontrol,andthelevelsofconsistencyandcomparabilitybetweenreportsthatarerequiredbytheagency.

Itisalsoworthnotingthatthehigherthelevelofagencycontrol,themorewillbeneededathoroughtrainingandbriefingofexperts,sothattheyunderstandfullywhatisexpectedintheirworkonthereport.

show me the evidenCe ….

Wefollowtheprincipleof‘evidence-basedassessment’,butwhatexactlydowemeanby‘evidence’,andhowdoweuseit?

• Doweonlyuseevidenceprovidedbytheinstitution?• Doestheteamhavetherighttorequestotherevidence?• Canwerequestfromtheinstitutionsevidencethatdemonstratesorconfirmsthe

occurrenceofbadpractice?• Howdoweensurethatevidenceisrecordedandretained?(Photocopyingduring

thevisit?Accesstointernalwebpages?)• Whotakescareoftheevidenceandensuressafekeeping?Canindividualexpertsbe

reliedontodothis?• Istheevidencepublishedwiththereport(whichprobablymakesittoolong)orkept

inreserve?

54

• Maybetheevidence-basecanbeembeddedinearly,unpublishedversionsofthereportascomments,footnotesorparentheses.

Itisworthnotingtheinstitutionsusuallyonlyrequiresightoftheevidencewhenacriticismismade,notwhentheassessmentispositive:inthatcasedoweonlyneedtoretainevidencerelatingtocriticalpointsorproblematicmatters?

who reads it, who needs it?

ItseemsatleastarguablethatveryfewpeopleactuallyreadthroughafullinstitutionalQAreportfromendtoend.Thatis,ofcourse,withtheexceptionoftheever-resilientagencyofficerresponsibleforthereport,whoprobablyreadsitthroughfromendtoendmanytimes!

Whatevidencedoesanagencyhaveoftheeventualreadingandtheactualimpactofitsreports?Weshouldreflectcarefullyonthelikelyreadershipforreports.

Ifwereallyaredealingwithasmallnumberofreaders(morethanten,lessthan50?Orevenlessthanthat?),thensurelyweneedalsotothinkabouttheeconomyofourreportproduction:theexpenditureofeffortinproducingitrelativetotheactualpatternofreceptionanduse.Rememberalsothatresearchshowsthatgenerally,inreadingreportslikeours,theattentionspanofthereaderreduceswitheverypagethatisturned.

Maybemanymorepeoplewillreadthesummaryofthereportorashortversionpublishedonthewebsite.Isthereacaseforarelativelyshortpublishedreportandthenseparatelyanunpublished,moredetailedreportfortheinstitution?Usuallyevidenceofimpactcomesintheformoffollow-upreportsandactionplans.Maybe,then,allweneedforourreadershiparesummaries,judgements,recommendationsandactionplans?

Couldweimprovetheaccessibilityofreportsbyusingmorevisualimagesinthelayout(e.g.graphics,tables,highlighting,andevenpictures)?

Finally,weneedtorememberthatourreaderstoomayneedadviceandeventraininginreadingandusingreportsinthemostproductiveway.Guidancedocumentsabouttheprocedureandthemainfeaturesandaimsofreportsareveryworthwhile.

l ast word (it’ s always good to arrive at it!)

Theperfectreportismorelikelytobeanaspirationthanarealisation:thebestyoucando,inthetimeyouhaveavailable,hastobegoodenough!

55

Chapter 7:

conclUsionsZeynep Olcen, European Association for Quality Assurance in Higher Education (ENQA)

Withthegrowingpublicinterestinhighereducationinstitutions(HEIs)inEurope,accessingqualityandtransparentinformationontheinstitutionsortheprogrammesofferedbytheseinstitutionshasbecomeachallenge.AsoneofthemainBolognatransparencytools,qualityassurance(QA)reportsplayanimportantroleinfosteringthetransparencyofEuropeanhighereducationandprovidingreliableinformationtothestakeholders,butthereiscurrentlyagapbetweenthecurrentpracticesofQAagenciesinpublishingtheirexternalQAreportsandtheexpectationsofstakeholdergroupsinrelationtothesereports.

QAreportsarethethirdsourceofinformationforstakeholdersfollowingtheHEIs’websitesandfriendsorcolleagues.ThereisaclearneedtoraiseawarenessregardingtheexistenceofQAreportsamongthestakeholdersandtoensurethattheyareeasilyaccessible.WhiledifferenttargetgroupssuchasHEIs,studentsandpublicauthoritieshavesimilarexpectationsregardingQAreports,andutilisetheinformationwithinthem,employersdonotnecessarilyconsiderthesereportsasasourceofinformationontheprogrammeortheinstitution.TheoverallimpressionfromthestakeholdersisthatQAreportsareahelpfulsourceofinformation,butthetechnicality(jargonused)andthelengthofthesereportslimittheirpotential.Oneofthemainrequirementsofthestakeholdersisthatthereportsshouldcontaineasilyunderstandableandcomparabledata.However,summaryQAreportsdonotexistinordertoprovidequantitativedataonHEIs(whichcanbeeasilyfoundontheinstitutions’websites)butratherinformationontheevaluationprocessandtheoutcomes.Inaddition,stakeholdersunderlinedtheimportanceofeasyaccesstoQAreportsandrequirethattheybepublishednotonlyontheQAagencies’websitesbutalsoontheHEIs’websites.Withinthisperspective,HEIsshouldbehighlyencouragedtopublishtheirevaluationreportsontheirwebsites.

WhilecomprehensiveQAreportsareverydiverseandaddressprimarilyHEIsthemselves,summaryQAreportsmaybetterservetheneedsofallthedifferenttargetgroups.Thus,thisprojectaimedatdevelopingcommonEuropeanGuidelinesforthesummaryreportsandsomegenericrecommendationsforthestructureandthepublicationofcomprehensivereportsontheEuropeanlevel.Thedetailsandcontentofthecomprehensivereportsaretobedecideduponatthenationallevel.

TheEQArepprojectalsolookedintothecurrentpracticesofQAagenciesinpublishingtheirQAreportsandtherelatedstrengthsandweaknesses.Thesurveyedagenciesidentifiedtransparency(opennesstopublic)andtherecommendationsforqualityimprovementasthemainstrengthsofQAreports.Atthesametime,interestingly,someagenciesalsomentionedeasyreadability,clearformatandeasyaccessasstrengths.ThisisincontradictionwithwhatthestakeholdershaveidentifiedaslimitationswhenusingQAreports.Indeed,thereisadiscrepancybetweentheagencies’andthestakeholders’visionsregardingthereadabilityandtheaccessibilityofQAreports.ThismightbeduetothefactthatQAagenciesaremorefamiliarwiththestructureandterminologyofQAreportsandhavebetterknowledgeonwheretofindthem.Someoftheidentified

56

weaknessesincludetheinconsistencyconcerningtheformatofQAreports(whichverymuchdependsontheexpertsdraftingthem)andtheuseofcomplicatedterminology.TheavailabilityofreportsonlyinEnglishisalsoseenasaweaknessbysomesurveyedQAagencies.However,fromthestakeholders’pointofview,publicationofQAreportsinEnglishisessentialintermsofunderstandabilityandcomparabilityattheEuropeanlevel.Fromthisperspective,thecommonGuidelinesforsummaryreportssuggestthatthereportsshouldbepublishedinEnglishandanyotherlanguageswhichmightberelevant.

WhenquestionedabouttheelementsthatcouldcontributetotheimprovementofthequalityofQAreports,themajorityoftheagenciesemphasisedtheneedforclearterminology,accessibility(availabilitytopublic),goodstructure,cleartargetgroups,availabilityofreportsinEnglishaswellasotherlanguages,inclusionofinstitutions’bestpractices,andrecommendations/commendationsgivenbytheexperts.ThegeneralfindingoftheQAagencies’surveyindicatesthatQAreports(comprehensiveandsummary)varysignificantlyamongcountriesandagenciesbutalsobetweenprogrammeandinstitutionaltypes.Furthermore,theediting,structureandcontent,length,readabilityandaccessibilityofreportsalsodifferatthecountryandagencylevel.

Theconclusionsofthisproject,therefore,focusontheimprovementoftheQAreportsbybringingcloserthecurrentpracticesofQAagenciesinpublishingthesereportsandtheoverallexpectationsofthestakeholders.ThecommonEuropeanGuidelinesforsummaryreportsandthegenericrecommendationsforcomprehensivereportsaretheimportanttoolstoachievethis.

european guidelines for summary reports

Summaryreportstargetawideraudiencethancomprehensivereports,includingstudents,parents,employers,otherHEIsorgovernmentbodies,andthus,theyareexpectedtoprovideclear,understandableandconciseinformationabouttheevaluatedinstitutionorprogramme.Whenproducingsummaryreports,QAagenciesshouldensurethefollowingexpectationsaremet:

• Clearterminology,avoidingprofessionaljargon.• Easyaccesstothewideraudiencebypublishingreportsnotonlyontheagency’s

websitebutalsotheHEI’swebsite.Thetargetgroupsshouldbeabletofindthereportseasilythroughsearchenginesbytypingkeywords.

• Draftingofthereportbyanagency’sstaff,asitisimportanttoensurehomogeneityandconsistency.

• Developingauser-friendlyandeasy-to-communicatetemplatewhichincludestheagency’scorporateimageforvisibilityaswellasclearheadingsandotherlayoutsegments.

• ProducingthereportsinEnglishinordertoreachawiderpublic,especiallyforeignstudentsinothercountries,andforcomparabilitypurposesattheEuropeanlevel,butalsoinanyotherlanguage(s)thattheagencyorinstitutionuses.

• Providingusefulinformationon:1)theevaluationprocess;2)theinstitutionortheprogrammeunderevaluation;and3)themainconclusionsoftheevaluation.

57

reCommendations for ComprehensiVe reports

SincecomprehensivereportstargetmainlytheHEIsandcontainawiderangeofimportantinformationvaryingfromoneagencytotheother,therecommendationsforthesetypesofreportsislimitedtotheirstructure,draftingandthepublicationratherthanthecontent.Thecomprehensivereportsshouldpayattentiontothefollowing:

• Goodstructurewithclearheadings,sub-headingsandalistofthecontentstofacilitatesearchesforspecificinformation.

• Cleardescriptionoftheprocess,statingtherecommendations,commendationsandthecontextofthereview.

• Consistentuseofterminologyandaglossaryexplainingthetechnicalterms.• Easyaccesstothereportonlinethroughpublicationofthereportontheagency

andthehighereducationwebsites.• Publicationoftheinstitutionalresponsesandthefollow-updocumentstogether

withthereport.

ThroughtheanalysisoftheneedsofstakeholdersregardingQAreports,andtheQAagencies’currentpracticesonthepublishingofQAprocedures,theEQArepprojectshedlightontheexistinggapbetweenwhatisexpectedandwhatisreality.BasedonthefindingsandtheconclusionsderivedfromthefindingsoftheEQArepproject,ENQAhighlyrecommendsthattheQAagenciestakeintoconsiderationtheEuropeanGuidelinespresentedinthisreportwhenproducingandpublishingsummaryreports–orwhenconsideringthedevelopmentofsuchreports.Furthermore,theQAagenciesarealsoencouragedtocommunicatethegenericrecommendationsforcomprehensivereportstothoseinchargeofdraftingandtomirrortheircurrentpracticetothoserecommendations.

ThedetailedworkcarriedoutintheEQArepprojectbroughtintofocusdifferentviews,perspectivesandexpectationswithregardtoQAreports.AsQAreportsandtheoverallcontextofhighereducationisconstantlyevolving,QAagenciesarerecommendedtorequestfeedbackfromandperiodicallysurveytheirstakeholdersinordertofurtherimprovetheirreports.

TheoutcomesofthisprojectwillhaveanimportantimpactontheimprovementoftheroleofQAreportsasasourceofreliableandcomparableinformationfordifferenttypesofstakeholdergroupsiftherecommendationsmadearetakenintoaccount.QAagencieshaveanimportantroletoplayinthisprocess.

58

acKnoWleDGeMenTsENQAandtheprojectconsortiummembersaregratefultoalltheEuropeanQAagenciesthattookthetimetorespondtotheagencysurveyandallstakeholders(individualsandorganisations)thatprovidedtheirinvaluableinputthroughtheirresponsestothestakeholders’survey.Inaddition,wesendourwarmestthankstoalltheparticipantsofthetwoworkshops,whocontributedactivelybytheircomments,questionsandideas.

TheEQArepprojectbenefittedfromthesupportandadviceofanAdvisoryBoardcomposedofrepresentativesfromtheEuropeanUniversityAssociation(EUA),EuropeanStudents’Union(ESU),EuropeanAssociationofInstitutionsinHigherEducation(EURASHE)andBusinessEurope.Wearegratefulforthecontributionsandadvicereceivedthroughouttheprojectfromvariousexperts.WewouldliketothankGuyHaugforhisinputontheprojectastheexternalevaluator.TheprojectconsortiumisfurtherindebtedtoPeterWilliamsandAchimHopbachfortheirideasandinput.

59

reFerencesCrozier,F.,Grifoll,J.,Harris,N.,Kekalainen,H.,&Malan,T.,(2011),Evaluationofthereportsonagencyreviews(2005-2009),ENQAOccasionalPapers16.Helsinki:EuropeanAssociationforQualityAssuranceinHigherEducation.Availableat:www.enqa.eu/indirme/papers-and-reports/occasional-papers/ENQA_Occasional%20paper_16.pdf.

E4Group,incooperationwithEI,BusinessEurope,andEQAR,(2014),RevisionoftheESG.Availableat:http://revisionesg.wordpress.com.

QualityandQualificationsIreland(QQI),(2014),ReviewofReviews:ReportoftheIndependentReviewTeam.Availableat:www.qqi.ie/Downloads/Reviews/Review%20of%20Reviews/12639-QQI%20Review%20of%20Reviews-WEB.pdf.

Rauhvargers,A.,Deane,C.,&Pauwels,W.,(2009),Bolognaprocessstocktakingreport2009,Report from working groups appointed by the Bologna Follow-up Group to the Ministerial Conference in Leuven/Louvainla-Neuve. Brussels:LifelongLearningProgramme,EuropeanCommission.

Thune,C.(2009),Standards and guidelines for quality assurance in the European Higher Education Area, 3rd edition. Helsinki:EuropeanAssociationforQualityAssuranceintheEuropeanHigherEducation.

Westerheijden,D.F.,etal.(2010),The Bologna Process Independent Assessment-The first decade on working on the European Higher Education Area-Volume 1 Detailed assessment report, CHEPS/INCHER-Kassel/ECOTEC.Availableat:www.ond.vlaanderen.be/hogeronderwijs/bologna/2010_conference/documents/IndependentAssessment_executive_summary_overview_conclusions.pdf.

Woodhouse,D.(November2010),IsthereagloballycommonunderstandingtoQualityAssurance?ESU Board Meeting 59 Seminar: Quality Assurance. PresentationconductedfromJerusalem,Israel.

60

anneX i

Questionnaire: the use and usefulness of external Quality assuranCe reports for different stakeholders

TheEuropeanAssociationforQualityAssuranceinHigherEducation(ENQA)iscarryingoutaEuropeanCommissionLLPco-fundedprojectTransparencyofEuropeanhighereducationthroughpublicqualityassurancereports(EQArep)togetherwithfourEuropeanQAagencies:OAQ(Switzerland),QQI(Ireland),ASHE(Croatia)andEKKA(Estonia).

TheaimoftheprojectistodevelopEuropeanstandardsforqualityassurancereportsinordertoensurethattheinformationcontainedinthequalityassurancereportsmeettheexpectationsofthestakeholders.Theoutcomesoftheprojectshallresultinarecommendationtoqualityassuranceagenciesonthecontentandformofinformativeandapproachablequalityassurancereports.

The surveyisdirectedatallmainstakeholdergroupsinhighereducationqualityassurance:students,potentialfutureemployers,governmentsandthehighereducationinstitutionsthemselves.Thepurposeofthesurveyistoidentifyandcomparetheexactinterests of the various stakeholdersasregardsinformationaboutthequalityofinstitutionsandofprogrammes.

Wewouldbemostgratefulifyoucoulddedicatesometimetorespondingtothissurvey.Theapproximatetimeneededis10 minutes.Theresponseswillbeanonymous.

Pleaserelyonyour personal experienceanduseofinformationsourcesonHEI.Pleasenotethatnotallpartsofaquestionwillberelevanttoalldifferentrespondergroups.

Yourcommentswillbeofhighvalueforus.Pleaseprovideasmanyofthemasyouwishinthe comment boxes provided,toexplainyourexperiencebetter.

Pleaseleaveyouremailaddressifyouwishtobeinformedoftheworkshopwhichwillbeorganiseasafollow-upofthesurvey,andwillprovideanopportunitytodiscusstheinformationneedsofdifferentusergroupsfurther.

The deadline for replying is 28th of February.

Warmthanksinadvanceforyourtimeandcontribution

The eQarep consortium

61

i - use of the information about Quality in higher eduCation institutions and study programmes

1. Forwhatpurposeshaveyousearchedinformationaboutthequalityofahighereducationinstitution(HEI)andstudyprogrammes?(Multiple answer)

a. Todecideonpossiblefurtherstudies

b. Toevaluatethequalityofgraduatesforrecruitmentpurposes

c. TofindpartnersamongHEIs

d. Todecideoninvestments/funding/sponsorshiptoaHEIoritsunit

e. Other(pleasename)

2. WhichsourcesdoyouusuallyusetogetinformationaboutthequalityofHEIandstudyprogrammes?

a. WebsitesofHEIsoften – once in a while – never

b. Socialmedia(e.g.Facebook)often – once in a while – never

c. Assessmentreportsprovidedbyqualityassuranceagenciesoften – once in a while – never

d. Governmentreports/publicationsoften – once in a while – never

e. Variousrankings/leaguetablesoften – once in a while – never

f. Informationfrom/opinionsoffriends,colleagues,parentsetc.often – once in a while – never

g. Other(pleasename)

3. Ifyoudonotusereportsbyqualityassuranceagenciesasasourceofinformation,pleaseexplainwhy(pleasechooseallthatapply):

a. Didnotknowaboutthem

b. Didnotknowwheretofindthem

c. ThereportsdonotcontaintheinformationIneed

d. Thereportsareinatoocomplicatedlanguage

e. TherearenoreportsinEnglish/languageIunderstand

f. Thereportsaretoolong

g. Ifoundtheneededinformationelsewhere

h. Other(pleasename)

62

If you do not use reports by quality assurance agencies as a source of information, please continue to Question 8.

4. Ifspecificinformation,pleaseindicatewhatkindofinformationyouwerelookingfor(pleasechooseallthatapply)

a. Generalfindings(e.g.generalrecommendations,overallassessment)

b. Strategicplanning,management,governance

c. Internalqualityassurancesystem

d. Historyandtraditions

e. Applicationandadmissionstatistics

f. Numberofresearchgrants,publications,citations

g. Employability/employmentofgraduates

h. Contentofstudyprogrammes

i. Reputationofteachingstaff

j. Qualificationsofteachingstaff

k. Financialresources

l. Conditionofinfrastructure

m. Studentsupportsystem

n. Institution’sabilitytoresponddiversestudents’needs

o. Accreditationstatusofinstitutions/studyprogrammes

p. Institution’spositioninleaguetables

q. Other(pleasename)

5. Theassessmentreportsprovidetheinformationyouarelookingfor

Fullyagree Slightlyagree Slightlydisagree FullydisagreePlease comment what exactly is good, what is missing, how it can be improved

6. YoufindtheassessmentreportsbyqualityassuranceagencieshelpfulinprovidinginformationaboutHEIs/programmes

Fullyagree Slightlyagree Slightlydisagree FullydisagreePlease comment what exactly is good, what is missing, how it can be improved

7. Wheredidyoufind/accesstheassessmentreports?

a. WebsitesofHEIs

b. Websitesofqualityassuranceagencies

c. Other(pleasename)

63

ii - expeCtations of the information proVided by the Quality assuranCe reports

8. Whatinformationdoyouneedtomakedecisionsforfurtherlearning,partnerships,comparisonswithotherinstitutionsetc.?(Upto5choices)

a. Strategicplanning,management,governance

b. Internalqualityassurancesystem

c. Historyandtraditions

d. Applicationandadmissionstatistics

e. Numberofresearchgrants,publications,citations

f. Employability/employmentofgraduates

g. Contentofstudyprogrammes

h. Reputationofteachingstaff

i. Qualificationsofteachingstaff

j. Financialresources

k. Conditionofinfrastructure

l. Studentsupportsystem

m. Institution’sabilitytoresponddiversestudents’needs

n. Accreditationstatusofinstitutions/studyprogrammes

o. Institution’spositioninleaguetables

p. Other(pleasename)

9. Inwhatformatwouldyouliketogetthisinformation?(chooseone)

a. Table(s)ofnumericaldataindicatingthemostimportantaspectsofaHEI/programme

b. AshortconcentratedsummarydescribingthemainstrengthsandareasforimprovementofaHEI/programme

c. Comparativedatawithotherinstitutions

d. Acomprehensivereportprovidingextensiveinformationaboutstrengthsandareasforimprovementinmanagementandcoreprocesses(studyprocess,researchanddevelopment),explainingalsothepossiblereasonsforagivensituation

e. Other(pleasename)

64

10.Wherewouldyouliketofindthisinformation?(pleasechooseallthatapply)

a. WebpagesofHEIs

b. Webpagesofqualityassuranceagencies

c. Socialmedia(pleasenamethemostpreferredsource,e.g.Facebook,Twitter…)

d. Printedreportsinthelibraries/qualityassuranceagencies/HEIs

e. Other(pleasename)

iii - information about the respondent

11. Whatisyourcountryofresidence?(ChoiceofthecountriesofEHEA)

12.Whichofthefollowinggroupsdoyourepresent?

a. Highereducationinstitution

b. Publicauthorities/Governmentoffice

c. Funder/Investor

d. Employer

e. Student

f. Other(pleasename)

13.TheresultsofthissurveywillbepresentedandfurtherfocusgroupinterviewsconductedinaworkshopinTallinn,Estonia,onmay 6-7, 2013.Ifyouareinterestedinparticipatinginthisevent,pleasewriteyournameande-mailaddressbelow.Theprojectwillcoverthetravelcostsoftheselectedparticipants.

• Name(optional)• E-mailaddress(optional)

thank you for your Contribution !

65

anneX ii

Questionnaire: the publiCation of Qa results: purpose, struCture and Content

survey on the use and usefulness of quality assur anCe reports

ENQAiscarryingouttheprojectTransparency of European higher education through public quality assurance reports (EQArep),togetherwithfourmemberagencies:ASHE(Croatia),EKKA(Estonia),OAQ(Switzerland)andQQI(Ireland).TheaimofthisprojectistoenquireoncurrentneedsandpracticesaboutthepublicationofexternalQAreportsandevaluatethepossibilityofdevelopingEuropeanstandardsforthesereportsinordertoensurethattheinformationprovidedbythemmeetstheneedsofstakeholders.

ThissurveyhasbeendevelopedforQualityAssuranceAgencies(ENQAmembersandaffiliates).ItsoutcomeswillbecomplementedwiththeanalysisofasampleofpublishedqualityassurancereportsandfurtherdiscussedinaworkshopinSwitzerlandduringwinter2013-14.

Wewouldbegratefulifyouwoulddedicatesometimetorespondthissurvey.Theapproximatetimeneededis20minutes.

Pleasenotethatyouranswerscanbesavedandreviewedanytimebeforeyourfeedbackissubmitted.Youranswerswillexclusivelybeusedfortheprojectpurposesandtreatedwithdiscretion.

Thesurveywillbeopenuntil[XX.XX.XX]

Thankyouinadvanceforyourtimeandpreciouscontribution.

The eQarep consortium

6661  

 

Respondent Quality Assurance Agency (QAA): …please insert weblink to the agency

Name and role of the person of reference for this survey: …

Please note that for the purpose of the present survey:

- a distinction between institutional and programme external QA reports is made; - the focus is on published, publicly available reports;

There is a diversity of practices in writing and publishing reports across the EHEA, accordingly, in this survey, we distinguish between

- “comprehensive report” and “summary report(s)”:

comprehensive report: An extensive review report which documents the full analytical outcomes of a given external QA assessment procedure, be it at institutional or programme level, be it written by academic experts, agency employees or an external technical secretary; an in-depth analysis upon which the main findings are based is made explicit and represents a key characteristic of this type of report. Often this is the primary report.

summary report(s): Any summarising form of reporting the outcomes of an external QA assessment procedure, be it a summary, a description, a table of comparative data, a final procedural report, web text or other possible types and forms of descriptive or schematic reporting. All kinds of derivative forms of a primary comprehensive report are included here.

Section 1 - Type of assessments conducted and reports published

1. How many institutional reviews/ assessments does your agency conduct on average per year (covering a time span of 5-7 years)?

0

1-5

6-15

16-30

31+

2. What kind of report is issued for the institutional review/ assessment? (Please choose as many options as applicable)

Non published comprehensive report Published comprehensive report

Non published summary report(s) Published summary report(s)

3. Additional comment/ further explanation:

respondent Quality assuranCe agenCy (Qaa)weblink to the agency Name and role of the person of reference for this survey: …

67

62    

4. How many programme reviews/ assessments does the agency conduct on average per year (covering a time span of 5-7 years)?

0

1-5

6-15

16-30

31+

5. What kind of report is issued for the programme review/ assessment? (Please choose as many options as applicable)

Non published comprehensive report Published comprehensive report

Non published summary report(s) Published summary report(s)

6. Additional comment/ further explanation:

Section 2 – Publication and Readership

7. What is the main purpose of the reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Facilitate a QA/ accreditation decision

Supply feedback to a HEI for quality enhancement

Supply information to the general public

Assure transparency Agency accountability Other (specify):

63    

8. In your opinion, which stakeholder groups currently tend to use your published reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

HEIs Students (and their parents) Official authorities/ Governmental bodies

QA agencies Funding authorities Professional organisations/ Employers/ Recruitment agencies

Media Other (specify):

9. Who might be your potential or future readership? Which stakeholder groups would you like to use your reports more?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

HEIs Students (and their parents) Official authorities/ Governmental bodies

Other official authorities QA agencies Funding authorities Professional organisations/ Employers/ Recruitmet agencies

Media Other (specify):

10. Additional comment/ further explanation

Section 3 – Editing and publication of reports

11. In which language(s) are your reports available?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

National language(s) English Other (specify):

68

63    

8. In your opinion, which stakeholder groups currently tend to use your published reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

HEIs Students (and their parents) Official authorities/ Governmental bodies

QA agencies Funding authorities Professional organisations/ Employers/ Recruitment agencies

Media Other (specify):

9. Who might be your potential or future readership? Which stakeholder groups would you like to use your reports more?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

HEIs Students (and their parents) Official authorities/ Governmental bodies

Other official authorities QA agencies Funding authorities Professional organisations/ Employers/ Recruitmet agencies

Media Other (specify):

10. Additional comment/ further explanation

Section 3 – Editing and publication of reports

11. In which language(s) are your reports available?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

National language(s) English Other (specify):

63    

8. In your opinion, which stakeholder groups currently tend to use your published reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

HEIs Students (and their parents) Official authorities/ Governmental bodies

QA agencies Funding authorities Professional organisations/ Employers/ Recruitment agencies

Media Other (specify):

9. Who might be your potential or future readership? Which stakeholder groups would you like to use your reports more?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

HEIs Students (and their parents) Official authorities/ Governmental bodies

Other official authorities QA agencies Funding authorities Professional organisations/ Employers/ Recruitmet agencies

Media Other (specify):

10. Additional comment/ further explanation

Section 3 – Editing and publication of reports

11. In which language(s) are your reports available?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

National language(s) English Other (specify):

64    

12. Who writes the reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Panel of experts Peer leader of the panel of experts

Secretary of the panel of experts (QA employee)

Secretary of the panel of experts (external subcontractor)

Other (specify):

13. How much time on average does it take to complete a report?

approximate number of working hours

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

-5 6-15 16-30 31+

14. Where are the reports published?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Webpages of the agency Webpages of HEIs Webpages of other public authorities

Social media (specify): Hard copy Other (specify):

15. Additonal comment/ further explanation

Section 4 – Structure of reports

16. Does your agency have a template available for the writing of reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

yes no Other (specify):

64    

12. Who writes the reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Panel of experts Peer leader of the panel of experts

Secretary of the panel of experts (QA employee)

Secretary of the panel of experts (external subcontractor)

Other (specify):

13. How much time on average does it take to complete a report?

approximate number of working hours

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

-5 6-15 16-30 31+

14. Where are the reports published?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Webpages of the agency Webpages of HEIs Webpages of other public authorities

Social media (specify): Hard copy Other (specify):

15. Additonal comment/ further explanation

Section 4 – Structure of reports

16. Does your agency have a template available for the writing of reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

yes no Other (specify):

69

64    

12. Who writes the reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Panel of experts Peer leader of the panel of experts

Secretary of the panel of experts (QA employee)

Secretary of the panel of experts (external subcontractor)

Other (specify):

13. How much time on average does it take to complete a report?

approximate number of working hours

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

-5 6-15 16-30 31+

14. Where are the reports published?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Webpages of the agency Webpages of HEIs Webpages of other public authorities

Social media (specify): Hard copy Other (specify):

15. Additonal comment/ further explanation

Section 4 – Structure of reports

16. Does your agency have a template available for the writing of reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

yes no Other (specify):

65    

17. What is the average length of the reports?

number of pages

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

1 2-5 6-15 16-30 31+

18. In which format is the report information presented? (Please choose as many options as applicable)

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Discursive detailed information Discursive consolidated information

Presentation of key data Schematic comparative data Table(s) of numerical data Synthetic web text Other (specify):

19. Additional comment/ further explanation

Section 5 – Content of reports

20. What information does your agency systematically provide in the reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Assessment procedure/ national legal framework

Information about site-visit Feedback on self-evaluation References/ documentation used/ annexes

Accreditation status General information and key data of institution

Profile and content of programme

Fulfillment of formal criteria or quality standards

General findings

65    

17. What is the average length of the reports?

number of pages

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

1 2-5 6-15 16-30 31+

18. In which format is the report information presented? (Please choose as many options as applicable)

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Discursive detailed information Discursive consolidated information

Presentation of key data Schematic comparative data Table(s) of numerical data Synthetic web text Other (specify):

19. Additional comment/ further explanation

Section 5 – Content of reports

20. What information does your agency systematically provide in the reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Assessment procedure/ national legal framework

Information about site-visit Feedback on self-evaluation References/ documentation used/ annexes

Accreditation status General information and key data of institution

Profile and content of programme

Fulfillment of formal criteria or quality standards

General findings

70

65    

17. What is the average length of the reports?

number of pages

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

1 2-5 6-15 16-30 31+

18. In which format is the report information presented? (Please choose as many options as applicable)

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Discursive detailed information Discursive consolidated information

Presentation of key data Schematic comparative data Table(s) of numerical data Synthetic web text Other (specify):

19. Additional comment/ further explanation

Section 5 – Content of reports

20. What information does your agency systematically provide in the reports?

Institutional comprehensive report

Institutional summary report(s)

Programme comprehensive report

Programme summary report(s)

Assessment procedure/ national legal framework

Information about site-visit Feedback on self-evaluation References/ documentation used/ annexes

Accreditation status General information and key data of institution

Profile and content of programme

Fulfillment of formal criteria or quality standards

General findings

66    

(commendations, recommendations) Profile of strengths and weaknesses

Management, strategic planning Financial resources Quality assurance system Learning environment Employability of graduates Suggestions for quality enhancement

Other benchmarking information, please specify:

Other (specify): Other (specify):

21. Additional comment/ further explanation

Section 6 – Usability of reports

22. How would you best define your comprehensive reports?

strongly agree

agree neutral disagree strongly disagree

don’t know

easily accessible

easily readable

provides clear and transparent information on the quality of an institution/ programme

useful for HEIs

useful for public authorities

useful for funding decisions

useful for students

useful for employers

71

66    

(commendations, recommendations) Profile of strengths and weaknesses

Management, strategic planning Financial resources Quality assurance system Learning environment Employability of graduates Suggestions for quality enhancement

Other benchmarking information, please specify:

Other (specify): Other (specify):

21. Additional comment/ further explanation

Section 6 – Usability of reports

22. How would you best define your comprehensive reports?

strongly agree

agree neutral disagree strongly disagree

don’t know

easily accessible

easily readable

provides clear and transparent information on the quality of an institution/ programme

useful for HEIs

useful for public authorities

useful for funding decisions

useful for students

useful for employers

67    

useful for media

23. How would you best define your summary reports?

strongly agree

agree neutral disagree strongly disagree

don’t know

easily accessible

easily readable

provides clear and transparent information on the quality of an institution/ programme

useful for HEIs

useful for public authorities

useful for funding decisions

useful for students

useful for employers

useful for media

24. How do you get external feedback on the quality and usefulness of your reports?

25. Additional comment/ further explanation

Section 7 – Perspectives

26. In general, what do you think are the main strengths and weaknesses of the published reports of your agency?

72

67    

useful for media

23. How would you best define your summary reports?

strongly agree

agree neutral disagree strongly disagree

don’t know

easily accessible

easily readable

provides clear and transparent information on the quality of an institution/ programme

useful for HEIs

useful for public authorities

useful for funding decisions

useful for students

useful for employers

useful for media

24. How do you get external feedback on the quality and usefulness of your reports?

25. Additional comment/ further explanation

Section 7 – Perspectives

26. In general, what do you think are the main strengths and weaknesses of the published reports of your agency?

73

68    

27. Please name some elements that, from your professional perspective , would increase quality and usefulness of reports.

28. What do you think are possible challenges and risks in trying to increase transparency and comparability of reports?

Thank  you  for  your  contribution!      

74

anneX iii

list of respondents to the surVey on the publiCation of Qa results: purpose, struCture and Content

• AccreditationAssociationforMedicalEducation,Turkey• AccreditationCommission(ACCR),CzechRepublic• AccreditationOrganisation(NVAO),TheNetherlandsandFlanders• Akkreditierungs-,Certifizierungs-undQualitätssicherungs-Institut(ACQUIN),Germany• AgenciaparalaCalidaddelSistemaUniversitariodeCastillayLeón(ACSUCYL),Spain• Agenced'évaluationdelarechercheetdel'enseignementsupérieur(AERES),France• Agencepourl'EvaluationdelaQualitédel'Enseignementsupérieur(AEQES),Belgium• AgênciadeAvaliaçãoeAcreditaçãodoEnsinoSuperior(A3ES),Portugal• AgencianacionaldeEvaluacióndelaCalidadyAcreditación(ANECA),Spain• AgencyforDevelopmentofHigherEducationandQualityAssurance(HEA),Bosnia

&Herzegovina• AgencyforQualityAssuranceintheGalicianUniversitySystem(ACSUG),Spain• AgencyforScienceandHigherEducation(ASHE),Croatia• AgenziaperlaCertificazionediQualitàel’AccreditamentoEUR-ACEdeiCorsidiStudioinIngegneria(QUACING),

Italy• AgencyforHigherEducationQualityAssuranceandCareerDevelopment(AKKORK),Russia• AgenturfürQualitätssicherungdurchAkkreditierungvonStudiengängen(AQASe.V.),Germany• AgenţiaRomânădeAsigurareaCalităţiiînÎnvăţământulSuperior(ARACIS),Romania• AgènciaperalaQualitatdelSistemaUniversitarideCatalunya(AQUCatalunya),Spain• AssociazioneItalianaperlaFormazioneManageriale-ItalianAssociation forManagementDevelopment(ASFOR),Italy• AkkreditierungsagenturfürStudiengängederIngenieurwissenschaften,derInformatik,

derNaturwissenschaftenundderMathematike.V.(ASIINe.V.),Germany• AustrianQualityAssuranceandAccreditationAgency(AQ),Austria• CenterforQualityAssessmentinHigherEducation(SKVC),Lithuania• CentralEvaluationandAccreditationAgency(ZEvA),Germany• Commissionforaccreditationandqualityassurance(CAQA),Serbia• TheCommissionforTechnologyandInnovation(CTI),France• EstonianHigherEducationQualityAgency(EKKA),Estonia• EuropeanAccreditationAgencyfortheLifeSciences(EAALS)• EuropeanAssociationforPublicAdministrationAccreditation(EAPAA)• EuropeanCouncilforBusinessEducation(ECBE)• EuropeanCouncilonChiropracticEducation(ECCE)• EvaluationsagenturBaden-Württemberg(evalag),Germany• FoundationforInternationalBusinessAdministrationAccreditation(FIBAA),Germany• FinnishHigherEducationEvaluationCouncil(FINHEEC),Finland• HungarianAccreditationCommittee(HAC),Hungary• InstitutionalEvaluationProgramme(IEP)• KosovoAccreditationAgency(KAA),Kosovo• NationalAccreditationAgencyoftheRussianFederation(NAA),Russia• NationalCentreforPublicAccreditation(NCPA),Russia• NationalCenterforProfessionalEducationQualityAssurance(ANQA),Armenia• NationalEvaluationandAccreditationAgencytotheCouncilofMinisters

oftheRepublicofBulgaria(NEAA),Bulgaria• NorwegianAgencyforQualityAssuranceinEducation(NOKUT),Norway• OrganfürAkkreditierungundQualitätssicherung(OAQ),Switzerland• PolishAccreditationCommittee(PKA),Poland• PublicAgencyforAccreditationofHigherEducation(PAAHE),Albania• TheQualityAssuranceAgencyforHigherEducation(QAA),UK• QualityandQualificationsIreland(QQI),Ireland• TheCouncilforHigherEducation(CHE),Israel• TheDanishAccreditationInstitution(AKKR),Denmark

76

the present publiCation is based on the findings of the “Transparency of european higher education through public quality assurance reports (eQarep)” project. This report investigates the current state of content, structure and publication of quality assurance reports while trying to understand the expectations and demands of stakeholders with regard to these reports. This report sheds light on the existing gap between what is expected by the stakeholders and the reality of the situation. in addition, the report provides Qa agencies with a set of european Guidelines for summary reports and some generic recommendations on the structure and publication of the comprehensive reports in order to support these agencies in producing reports with understandable and comparable information.

oCCasional papers 21

isBn 978-952-5539-74-5 (paperbound)isBn 978-952-5539-75-2 (pDF)issn 1458-1051