Reflection Report (IO6) and Assessment of Pilot Peer ... · Assessment of Pilot Peer Reviews and Reflection Report Peer Review VNFIL Extended 2018 7 2. Aims, topics and methods of

Universität für Bodenkultur Wien University of Natural Resources and Life Sciences, Vienna

ReflectionReport(IO6)andAssessmentofPilotPeerReviews(O7)Finalreport

MariaGutknecht-Gmeiner,Susanna-MariaHenkel,SophieKroiss,ChristinaPaulus

EuropeanPeerReviewAssociation(EPRA),UniversitätfürBodenkultur

Vienna,May2018

Project"TransnationalPeerReviewforqualityassuranceinValidationofNonFormalandInformalLearning(VNFIL)Extended"(Projectno.2015-1-NL01-KA204-009004)

AssessmentofPilotPeerReviewsandReflectionReport

PeerReviewVNFILExtended2018 2

AssessmentofPilotPeerReviews(IO7)andReflectionReport(IO6)

MariaGutknecht-GmeinerVienna,May2018Imprint:EuropeanPeerReviewAssociationDr.-Josef-Resch-Pl.14/3,1170WienTel.:+436642365980E-Mail:[email protected]://www.Peer-review-network.euhttp://www.Peer-review-vnfil.eu

ThisprojecthasbeenfundedwithsupportfromtheEuropeanCommission.Thispublicationreflectstheviewsonlyoftheauthors,andtheCommissioncannotbeheldresponsibleforanyusethatmaybemadeofthein-formationcontainedtherein.



CONTENTS

1. BackgroundandaimsoftheprojectPeerReviewVNFILExtended.................................................61.1 EuropeanPeerReview........................................................................................................61.2 TheprojectPeerReviewVNFILExtended...........................................................................6

2. Aims,topicsandmethodsofthequalitativestudyandassessment...............................................72.1 Formativeandsummativeaimsandapproach..................................................................72.2 Researchquestions.............................................................................................................82.3 Designandmethods...........................................................................................................82.4 Availabledocumentationandresponserates....................................................................9

3. PeerReviewPilots.........................................................................................................................113.1 Organisationofpilots.......................................................................................................113.2 ParticipationinPeerReviewpilots...................................................................................133.3 QualityAreas....................................................................................................................143.4 Peers.................................................................................................................................153.5 PeerTrainingandotherformsofsupport........................................................................19

3.5.1 Trainingandsupportoffered..................................................................................................193.5.2 Participationintrainingsandotherformsofsupport.............................................................203.5.3 Feedbackontrainingsandsupportduringpreparatoryphase...............................................223.5.4 Levelof“preparedness”ofVNFILprovidersandPeers...........................................................23

3.6 PreparationofthePeerReview(Phase1)........................................................................233.6.1 PreparationoftheVNFILProviders.........................................................................................233.6.2 Self-Report..............................................................................................................................233.6.3 PreparationinthePeerTeams................................................................................................24

3.7 PeerVisit(Phase2)...........................................................................................................253.7.1 PeerVisitAgendas...................................................................................................................253.7.2 ExperiencesduringPeerVisits................................................................................................27

3.8 PeerReviewReport(Phase3)..........................................................................................293.8.1 Feedbacksessionandusefulnessoffeedback........................................................................293.8.2 Timelinessofreports...............................................................................................................293.8.3 Writingthe“PeerReviewReport”..........................................................................................293.8.4 PeerConsulting.......................................................................................................................29

3.9 Puttingplansintoaction(Phase4)...................................................................................303.9.1 Communicationofresults.......................................................................................................30

3.10 Cooperation,rolesandtransnationalaspects..................................................................313.10.1 QualityofcooperationduringPeerReviews.........................................................................313.10.2 BenefitsofmultipleparticipationofPeersandfacilitators...................................................313.10.3 LessonslearnedfromnationalandinternationalcooperationduringPeerReview.............313.10.4 Transnationalaspects:TheimportanceofinternationalPeers............................................32



4. EffectsofthePeerReview.............................................................................................................334.1.1 Institutional„returnoninvestment“ofthePeerReview.......................................................334.1.2 RelevantchangesintermsofinstitutionaldevelopmentsincethePeerReview...................334.1.3 GoodpracticesinVNFILduringPeerReviewtransferredtootherinstitutions/countries.....344.1.4 LessonslearnedduringthePeerReviewintermsofinstitutionalQAandQM......................34

5. GeneralassessmentofPeerReview..............................................................................................355.1 ApplicabilityandaddedvalueofPeerReviewforVNFIL..................................................355.2 Potentialof(transnational)PeerReviewforthedevelopmentofVNFILandquality

assuranceinVNFIL............................................................................................................355.3 FurtherdevelopmentofinstitutionalqualityassurancethroughPeerReview................355.4 Challengesforimplementing(transnational)PeerReviewinVNFIL................................36

6. FeedbacktoManualandToolboxandrecommendationsforfinalisation....................................376.1 Manual..............................................................................................................................376.2 QualityAreas....................................................................................................................376.3 Toolbox.............................................................................................................................37

7. Documentsand(re)sources...........................................................................................................397.1 Basicdocuments...............................................................................................................397.2 Documentationofpilotphaseandmonitoringdata........................................................397.3 Reportsandpublications..................................................................................................397.4 Listofinterviewpartners..................................................................................................397.5 Participantsreflectionworkshop......................................................................................407.6 Additionaltables...............................................................................................................41



Tableofexhibits

Figure1: PhasesoftheEuropeanPeerReview...................................................................................7Table2: OverviewofdocumentsdeliveredbyVNFILproviders.........................................................9Table3: Surveyresponses................................................................................................................10Table4: ParticipatingVNFILprovidersandPeerVisitdates.............................................................13Table5: NumberofdeploymentsperPeer......................................................................................13Figure6: NumberofPeerReviewswith…Peers...............................................................................14Table7: QualityareaschosenbyVNFILproviders............................................................................15Figure8: CountriesofPeers...............................................................................................................16Figure9: FunctionsofPeersduringreviewofowninstitution..........................................................16Figure10: InstitutionalbackgroundsofactivePeers...........................................................................17Figure11:PreviousQualityAssuranceTrainingofPeers....................................................................17Table12: ReviewskillsofPeers(self-assessment).............................................................................18Figure13:Participationin(typesof)PeerTrainings............................................................................20Figure14:NºofPeerTrainingsperPeer.............................................................................................21Figure15:OtherformsofsupportforPeers(multipleanswerspossible)..........................................21Table16: FeedbacktoEuropeanPeerTraining..................................................................................22Table17: Levelof“preparedness”ofthePeers.................................................................................23Table18: PeerVisitAgendas:Numberofsessions,numberofinterviews

withdifferentinterviewgroups..........................................................................................25Table19: RepresentationofcandidatesinPeerVisit.........................................................................26Table20: QualityofcooperationinPeerTeam..................................................................................27Table21: QualityofcooperationbetweenProviderandPeers.........................................................31Table22: PeersperPeerReview........................................................................................................41Table23: Peers:distributionbysex...................................................................................................41Table24: Peers:distributionbycountry............................................................................................41Table25: InstitutionalbackgroundofPeers.......................................................................................42Table26: TypesofPeerTrainingcompletedbyPeers.......................................................................42Table27: NºofPeerTrainingsperPeer.............................................................................................42Table28: PreviousQualityAssuranceTrainingofPeers....................................................................43Table29: Review/evaluationskillsofPeers(self-assessment)...........................................................43Table30: ParticipationofPeersinPeerReviewofowninstitution...................................................43Table31: Howwasthecooperationwithinyourinstitution

inthepreparationandconductofthePeerReview?.........................................................44Table32: WasthetimeforthePeerVisitsufficient?.........................................................................44



1. BackgroundandaimsoftheprojectPeerReviewVNFILExtended

1.1 EuropeanPeerReview

PeerReview is a formofexternalevaluationwith theaimof supporting the reviewededucationalinstitution in its quality assurance and quality development efforts. An external group of experts,calledPeers,isinvitedtoassessthequalityofdifferentfieldsoftheinstitution,suchasthequalityofeducationandtrainingprovisionofindividualdepartmentsoroftheentireorganisation.Duringtheevaluationprocess,thePeersvisitthereviewedinstitution.Peersareexternalbutworkinasimilarenvironmentandhavespecificprofessionalexpertiseandknowledgeoftheevaluatedsubject.Theyareindependentand"personsofequalstanding"withthepersonswhoseperformanceisbeingre-viewed.PeerReviewhasinthepast10yearsbeenadaptedfromhighereducationtovocationaledu-cationandtraininginaseriesofEuropeanprojectsthatwerecloselylinkedtoEQAVET.

TheEuropeanPeerReviewprocedureprovidesaqualityassuredprocedureandcommonstandardforconductingPeerReviewsacrossEurope.TheprocedureisdocumentedintheEuropeanPeerRe-viewManualandToolbox.Itwasoriginallydevelopedintheareaofvocationaleducationandtrain-ingintheyears2000(PeerReviewininitialVET,2004-2007;PeerReviewExtended,2007;PeerRe-viewExtendedII,2007-2009).After2009,theEuropeanPeerReviewwasintroducedasanexternalinstrumentforqualityassuranceinthevocationaleducationandtrainingsystemsinvariousEurope-ancountries,amongwhichareFinland,Italy,Hungary,CataloniaandAustria.

OnaEuropeanlevel,transferofPeerReviewwasbuttressedbysubsequentcallsforprojectsintheLifelongLearningProgramme(priority for transferof innovationprojects)whichresulted in furtherexperimentationofPeerReview invocationalguidanceandcounselling (EuroPeerGuid,2010-2012)and,from2014onwards,alsointheareaofrecognitionofpriorlearninginPortugal,France,andtheNetherlands(EuroPeerguid-RVC,2014-2015).1

1.2 TheprojectPeerReviewVNFILExtended

The'EuropeanPeerReviewVNFILExtended'projecttakesupfromthesepreviousprojects.ItseekstointroducePeerReviewasaninstrumentofqualityassuranceandqualitydevelopmenttothevali-dationofnon-formalandinformal learning(VNFIL). Inparticular, itbuildsupontheresultsofEuro-Peerguid-RVCwhich itaims toextend– tonewcountries (Austria, Lithuania,Slovakia), to transna-tional useof PeerReviewand to a sustainable integrationof PeerReview into thenational – andpotentiallyalsoEuropean–qualitystrategiesforVNFIL.

Adecisiveelementof theprojectwasto test theapplicabilityof theEuropeanPeerReviewproce-durein11pilotPeerReviewsin6oftheparticipatingcountries,adoptingatransnationalapproach,i.e.withhalf thePeer Teamcoming fromanother country (O5). The latterwas intended to fostermutuallearningbetweenVNFILprovidersacrossborders,enhancingtransferofinnovationinVNFILand enhancingmutual trust in the quality of VNFIL provision in Europe. The pilots took place be-tweenJune2016andMay2017andwereheadedoffbyajointEuropeanPeerTraining(June2016).

Acomprehensiveassessmentandin-depthqualitativeinvestigationofthepilotphasewascrucialfordeterminingtheusefulnessofPeerReviewforthequalitydevelopmentofVNFILprovision.Itprovid-edabasisforthedevelopmentorfinalisationofthemajorityofprojectoutcomes:§ thefinalisationoftheManual(O2),theToolbox(O3),andtheQualityAreas(O3)§ thedevelopmentof thebusinessmodel (O7), thenational strategies for PeerReview inVNFIL

(O8)aswellasthePeerReviewReader(O9)andtheEuropeanpolicypaper(O10).

1Informationonthe„history“oftheEuropeanPeerReviewandpastprojectscanbefoundonthewebsiteoftheEuropeanPeerReviewAssociationwww.Peer-review-network.eu.



2. Aims,topicsandmethodsofthequalitativestudyandassessment

2.1 Formativeandsummativeaimsandapproach

Theoverallaimoftheassessmentandqualitativestudyofthepilotphasewastoappraisethefeasi-bilityandsuitabilityoftheuseoftheEuropeanPeerReviewprocedureinVNFILwiththeprospectofapossibleexpansiontootherVNFILinstitutions.ThisalsoincludedaninvestigationoftheusefulnessandaddedvalueofPeerReviewforthePeersandtheVFNILinstitutionsinvolved.Assessingthesuc-cess of the transfer of PeerReview toVNFIL thuswas the summativepartof the investigation. Inaddition,theassessmentandqualitativestudywantedtocontributetofurtherimprovementoftheorganisation,preparationandconductofEuropeanPeerReviewsintermsofaformativeassessment.

Atwo-stepapproachwastakentoproducevalidanddependableresults:§ Anin-depthexaminationoftheelevenpilotPeerReviewsascertainedthequalityoftheprocess

(includingtrainingandsupportbythecoordinatingbody)andthelevelofimplementationfidelityintermsofadherencetotheEuropeanPeerReviewprocedureaslaiddownintheManualandtheToolbox(O7).

§ Basedonthis,thequalitativestudy(i.e.thereflectionreport)wascarriedout(O6).

Sincethetwostepsareinseparablyinterlinked–thelatterpartcannotbedoneinaseriousmannerwithoutthepriorprocessassessment–thisreport isbasedonacoordinateddatacollectioneffortbetweenthetworesponsibleinstitutionsandcomprisestheoutcomesofbothO6andO7.Thefideli-ty assessment also constitutes integral part of the Peer Review label award (O7). It followed thephasesofthePeerReviewasdescribedintheManual.

Figure1: PhasesoftheEuropeanPeerReview

Source: EuropeanPeerReviewManualforVNFIL

Phase1:Prepara'on(min.3months)•  Ge4ngstarted•  Invi'ngPeers•  Self-Evalua'onsandSelf-Report•  PreparingthePeerVisit

Phase2:PeerVisit(1.5-2days)•  Collec'ngdata•  Analysingdata•  OralFeedback

Phase3:Peerreport(4weeks)•  DraOReport•  CommentsoftheVNFILprovider•  Finalreport

Phase4:Pu4ngplansintoac'on(6-12months)•  Formula'ngtargets•  Clarifyingresources•  Ac'onPlanandImplementa'on•  PlanningthenextReview

NextPeerReview



2.2 Researchquestions

Part1:Processevaluationandassessmentofimplementationfidelity§ How(well)werePeersandVNFILproviderspreparedforthepilotPeerReviews?Whatwerethe

contributions of the European Peer Training?What other support did Peers and providers re-ceive?Howcouldtheprovisionoftrainingandsupportbeimproved?

§ Howwere thePeerReviewsconducted?Were thequality standardsof theEuropeanPeerRe-viewprocedure(Manual,Toolbox)adheredto?Howcantheprocessbeimproved?

Part2:Qualitativestudy§ IsthePeerReviewmethodologysuitableforVNFIL?Isthetrainingprogrammeappropriate?Are

thequalityareasandindicatorssuitableforVNFILproviders?HowcanthePeerReviewmethod-ology(Manual,Toolbox)stillbeimprovedandfine-tunedtoVNFIL?

§ WhataretheprosandconsofinternationalPeerReview?HowdoyouassessthecontributionofinternationalPeers?HowdidtheprocessworkwithtwointernationalPeers?WhatwerebenefitsordrawbacksofmultipleparticipationofPeersandfacilitatorsandmutualPeerReviews?

§ WhatkindofimpactdidthePeerReviewshaveontheinstitutionsandonthePeers?

§ Whatistheaddedvalueof(transnational)PeerReviewsforinstitutionalQAandQMandistheintegrationofPeerReviewapossiblefurtherdevelopmentofinstitutionalqualityassurance?

§ What are the possibilities and challenges in further implementation of transnational Peer Re-viewsforVNFIL?Istherepotentialforimplementing(transnational)PeerReviewasaformativeexternalevaluationforVNFILonnationaland/orEuropeanlevel?

2.3 Designandmethods

Thecombinedassessmentandstudyfollowedamixed-methoddesignwithatriangulationofdiffer-entstakeholders,sourcesandmethodsincludingbothquantitativeandqualitativeelements.Awiderangeofdataandsourceswereusedwith

§ “hard data” from the documentation of the Peer Reviews (documents, monitoring data) andonlinesurveysgatheringinformationandfeedbackfromparticipatingPeersandProviders(O7)

§ complementedbyaworkshopwithpartnersandqualitative interviewsaswell asaqualitativeanalysisofsurveyresponses(openquestions)andselectedmonitoringdocuments,inparticular,themeta-evaluationsofPeers(O6).

1. MonitoringdataonthepilotPeerReviews

§ PeerReviewdatabase:overviewofpilotPeerReviews

§ PeerdatabasecontainingallinformationfromPeerapplications

§ PeerReviewdocuments(Toolbox)§ Self-Reports§ PeerReviewAgendas§ Meta-evaluationsofPeers§ PeerReviewReports§ OtherdocumentationofPeerReview,ifexistent:presentations,documentationoffeed-

backsession,interviewandobservationguidelines,interviewprotocols,photos



2. OnlinequestionnairesforparticipantsinpilotPeerReviews

InSeptember2017,anonlinesurvey(usingtheonlinetoolLimeSurvey)wasconducted,encompass-ingquantitativeandqualitativequestions.Theyincludedseparatequestionnairesfor

§ Peersand

§ VNFILproviders(PeerReviewfacilitators/coordinators,managersofreviewedcentres).

3. AworkshopwiththeprojectpartnersintheformofaWorldCaféaspartoftheprojectmeetinginMay/June2017

4. Qualitativeinterviewswith7projectpartnersbasedonaninterviewguide

Cf.Listofinterviewpartnersinannex

2.4 Availabledocumentationandresponserates

Databasesandpilotdocumentation(monitoring)

Thepurposeof thedocumentationof thePeerReviewsand theestablishmentofapilotdatabasewastosupporttheplanning,monitoringandsteeringofthepilotPeerReviews.

Table2: OverviewofdocumentsdeliveredbyVNFILproviders

Code

Initialinform

ation

sheet

Selfrepo

rt

Agen

dafo

rPeerV

isit

Otherdocum

entatio

nPe

erVisit

Metaevalua

tion

ofPeers

PeerReviewRep

ort

PeerApp

lications

16_01_NL ✓ ✓ ✓ minutesof3interviews ✓ ✓ ✓16_02_AT ✓ ✓ ✓ ample ✓ ✓ ✓17_03_LT ✓ ✓ ✓ attendancelist ✓ ✓ ✓17_04_FR ✓ ✓ ✓ ample ✓ ✓ ✓17_05_PT ✓ ✓ ✓ minutesof11interviews,

finalpresentation ✓ ✓ ✓

17_06_AT ✓ ✓ ✓ ample ✓ ✓ ✓17_07_AT ✓ ✓ ✓ questionsmatrix,findingsof

PeerTeam ✓ ✓ ✓

17_08_PT ✓ ✓ ✓ minutesof6interviews,findings ✓ ✓ ✓

17_09_AT ✓ ✓ ✓ ample ✓ ✓ ✓17_10_NL ✓ ✓ ✓ minutesof6interviews,

QAAssessment ✓ ✓ ✓

17_11_SK inSelf-Report ✓ ✓ attendancelist ✓ ✓ ✓except1

Source: PilotDatabase;documentationofpilots

Allmaindocumentswereprovided inEnglish. The coordinatingbodyalso collectedPeer contractsandphotosdocumentingthePeerReviews.



Onlinequestionnaires

ThequestionnaireswereonlinefromSeptember1sttoOctober16th,2017(providers)andfromSep-tember4thtoOctober31st,2017(Peers).Responseratesforthequestionnaireswere100%forVNFILprovidersand89,3%forPeers. (2Peersfilled inonequestionnairetogether,whichwascountedastworesponsesintheresponseratesaswellasintheoverallanalysis.)

Table3: Surveyresponses

CountryVNFILproviders Peers

totalnumber responses total

number responses

Austria(AT) 4 4 9 9

France(FR) 1 1 1 1

Lithuania(LT) 1 1 5 4

Netherlands(NL) 2 2 6 4

Portugal(PT) 2 2 4 4

Slovakia(SK) 1 1 3 3

Total 11 11 28 25

Source:OnlinesurveyofVNFILProvidersandPeers

Up to twodirect reminderswere sent to respondents via LimeSurveyduring theonline period (incaseofPeersupto3remindersforsome).Inaddition,emailsweresentdirectly,or(incaseofPeers)viamediatingprojectpartners.



3. PeerReviewPilots

3.1 Organisationofpilots

Organisationalstructure

Organisationina largernetworkrequiresanactivecoordinatingbodyasaskedfor intheEuropeanPeerReviewManual.Thetaskofthecoordinatingbodyisto§ managethewholeprocessofconductingthePeerReviews,§ ensureacoherentapproach,§ providesupportandguidancetoProvidersandPeers,§ checkimplementationfidelity§ conductconstantmonitoring§ andsupportevaluation.

In theproject, thepartof the coordinatingbodywasassumedby thepartner Libereaux (aidedbyprojectcoordinatorErikKaemingk).LibereauxwasalsothepartnerresponsibleforIntellectualOut-come5.Generalmanagementandsupportremainedwiththeprojectcoordinator.EPRAwasrespon-sibleforpreparingtheassessmentofthepilots,actedas“sparringpartner”forthecoordinatingbodyandassumedanadvisoryroleprovidinggeneralorientationandexpertise.

Anonlineplatform(Shareboard,www.share-board.nl)wasinstalledfortheprojectandmanagedbyErikKaeminkC.V.AlthoughindividualspacesweresetupforallPeerReviews,theplatformwasnotwidelyused in thepilotphase since findingmaterialaswellasuploading/downloadingdocumentswasnotaseasyasthepartnershadexpected.Documentswerethuspassedontothecoordinatormainlyviaemail.EPRAprovidedaDropbox folder forsharing informationbetweenthe threepart-nerssteeringtheprocess,whichwasthenalsousedtostorecomprehensiveinformationonthepilotphaseandmakeitaccessibleforanalysis.

Process

ThePeerReviewswerescheduledforthesecondprojectphase,June2016–May2017andconduct-edbetweenNovember2016andMarch2016.

RegistrationforthePeerTrainingwasavailablefromendofJanuary2016,theonlinePeerapplicationtoolwasopenstartingmid-May2016.Thecoordinatingbodydevisedaplan(“Peerpuzzle”)formatch-ingPeersand institutionsforthePeerReviews.ThechallengewastoorganisethePeerReviews inawaythatallteamsincludenationalandatleast2internationalPeers.ThecoordinatingbodyalsoaskpartnerstostartpreparingthepilotphasebeforethePeerTrainingsothattheindividualplanscouldberevised,ifnecessary,andalignedwithanoverallactionplanuponduringthetraining.

ThetoolboxwasreadybeforethePeerTraining.However,anupdatedversionoftheQualityAreaswitharevisedstructureofcriteriawasonlypresentedduringthePeerTraining.SincepartnershadalreadystartedthepreparationoftheinitialphaseofthePeerReview(decidinguponQualityAreas,settingupaninternalstructure,requirementsforPeers),thisrequiredsomechanges.Aftersomeinitialconfusiontheadaptationtothe“new”QualityAreasoveralldidnotposeaproblemforpartners.

AcommonEuropeanPeerTrainingprecededthepilotPeerReviews:ItwasheldbetweenJune13-17,2016, inVienna, in theNetherland, anadditionalnational training followed (cf. III.6) inwhichalsonationalPeersandotherstakeholdersparticipated.ThetrainingwasalsousedtopreparethepilotphaseandcomeupwithamasterplanforthePeerReviewsaswellasmatchingPeersandinstitu-tions.ContrarytopreviouspilotsadduetothefundingrequirementsofErasmus+,atleasttwoin-ternationalPeersparticipated ineveryPeerReviewandthePeerpoolwas largelymadeupofper-sonneloftheprojectpartnersandonlyafewexternalPeers.ThisresultedinahighmutualityofPeerReviewswithparticipatingprovidersexchangingPeers.



Amemberof thecoordinatingbodyalsoparticipated inallPeerReviews (MarloesSmit fromLibe-reauxorErikKaemingk)ensuringhighconsistencyofapproach,direct support forPeerTeamsandProviders andongoing secondorder learning concerning thePeerReviewprocess during thepilotphase.

Documentationandmonitoringofthepilotphasewascarriedoutbythecoordinatingbodyfromthestartandaccompaniedthewholepilotphase. Itallowedforaconstantmonitoringandsteeringofthepilotsandprovidedaverygooddatabasefortheassessmentandthestudy.

AninternalmentoringschemewassetupaccordingtoplanduringthePeerTraining.Itwas,howev-er, not usedmuchduring the initial stages of the PeerReviews sincepartners had ample supportthroughthecoordinatingbody(cf.Mentoringreport).

Projectteam“PeerReviewVNFILExtended”,June2016,Vienna(duringEuropeanPeerTraining)



3.2 ParticipationinPeerReviewpilots

ThefollowingoverviewshowstheparticipatinginstitutionsandthetimetableforthePeerVisits:

Table4: ParticipatingVNFILprovidersandPeerVisitdates

Code VNFILprovider Date

16_01_NL LibereauxBV,TheNetherlands Nov.24-25,2016

16_02_AT Weiterbildungsakademie(wba),Austria Dec.6-7,2016

17_03_LT VilniusVocationalTrainingCentreforServiceBusi-nessSpecialists,Lithuania Jan.18-19,2017

17_04_FR CentreInterinstitutionneldeBilandeCompétences(CIBC),BourgogneSud,France Feb.13-14,2017

17_05_PT CITEFORMA,Portugal Feb.2-3,2017

17_06_AT UniversityofNaturalResourcesandLifeSciences(BOKU),Unitoflifelonglearning,Austria March22-23,2017

17_07_AT FrauenstiftungSteyr,Austria Apr.4-5,2017

17_08_PT ISLASantarém,Portugal March9-10,2017

17_09_AT AKSalzburg/BFI,Austria May8-9,2017

17_10_NL EVCCentrumVigor,TheNetherlands March29-31,2017

17_11_SK NárodnýústavceloživotnéhovzdelávaniaNÚCŽV,Slovakia Jan.10-12,2017

Source: PilotDatabase

The11PeerVisitswerecarriedoutinAustria,France,Lithuania,theNetherlands,Portugal,andSlo-vakia.Thedurationofthevisitswasbetween2and2,5days;thedurationofthePeerReviewswasbetween1.75daysand2days(2daysformostPeerReviews)

Allinall,28Peersbecameactiveduringthepilotphase(seealsobelow).All13Peersthatonlypartic-ipatedinonePeerReviewwereemployedasnationalPeers.MorethanhalfofthePeersparticipatedinmorethanonePeerReview(54%),mostofwhomin2PeerReviews(39percentagepoints).15%wereactivein3ormorePeerReviews(amongwhomonepersonfromthecoordinatingbodywith7participations).ThesePeerswithmultipleengagementsallcamefrompartnerinstitutionsandwereactiveduringthewholeproject(participationintransnationalmeetingsandinPeerTraining,respon-sibilityforIntellectualOutcomes).Overall,therewere52deploymentsofPeers;onaverage,aPeerwasactive1.8timesduringthepilotphase.

Table5: NumberofdeploymentsperPeer

Deployments NºPeers %1 13 462 11 393 2 74 1 47 1 4

Total 28 100Source: PilotDatabase



The Peer Reviewswere conducted in teams of 2 to 7 Peers,with an average of 4.7 Peers per re-view/institution. The standard number of Peers according to the Manual is four, but larger PeerTeamsarepermissible if theorganisationandthePeersagree. Inthecaseofourproject italsoal-lowedmorepeopletoexperiencetheprocessfirst-hand.

Figure6: NumberofPeerReviewswith…Peers


ThenumberofPeersinthepilotphaseseemedtobesuitablenomatterwhatthesizeoftheteam,andnorecommendationsweremadetoadjustthenumberofPeersintheManual:Ofthoseprovid-erswith4Peersontheirreview’sPeerTeam,3answeredinthesurveythatitwastheoptimalgroupsize.FromthePeers’perspective,41.7%alsorated4theoptimalgroupsize,another41.7%judgingitworkablebutthatothergroupsizesarepossibleaswell.Theotherratingsofgroupsizedonotdeliv-erstatistical/usefulinformationonadequacyofgroupsize.

All PeerReviewswere carriedout in a transnationalmannerwith at least twoPeers coming fromanothercountry;inallPeerReviewsbutone,theteamconsistedofatleast50%transnationalPeers.Thus inthispilotphasethedeploymentoftransnationalPeers(29or56%ofallPeers) forthefirsttimeoutnumberedthenationalPeers(23or44%).ForanoverviewofthenumbersofPeers(nationalandtransnational)perPeerReviewpleaseseetheAnnex.

3.3 QualityAreas

InallPeerReviews, theEuropeanQualityAreaswereused.As recommended in theManual,mostpilotsreviewedtwoQualityAreas,exceptforonepartnerwhoreviewedthreeandonepartnerwhoreviewedonlyoneQualityArea.

Exceptforonecase,allPeerReviewscoveredatleastone“core”qualityarea(i.e.oneoftheQualityAreas1,2,3,4).Qualityarea8(“QualityManagement”)wasnotchosenbyanyinstitution.FeedbackfrompartnerssuggeststhatthiswasduetothefactthattheQualityArea(whichwascalled“QualityAssurance”)wasmisunderstoodtomeanonlyexternalevaluation(subtitleofearlierversionofthisQualityAreawasmisleading: “Evaluationproceduresand reportson tacticandoperational level”),whiletheindicatorsclearlypointtoamorecomprehensiveviewofqualityassurance.QA1(Identifi-cation)andQA5(Information,GuidanceandCounselling)werereviewedmostoften.Allotherquali-tyareaswereselectedbyatleastoneinstitution.Coverageofthequalityareasisthusoverallsatis-factoryandshowswhichissuesthepartners’institutionsarecurrentlymostinterestedin.



Table7: QualityareaschosenbyVNFILproviders

Code

NºofQAs

QA1Iden

tification

QA2Do

cumen

tatio

n

QA3Assessmen

t

QA4Ce

rtificatio

n

QA5Inform

ation,Guida

nce,

Coun

selling

QA6Stakeh

olde

rCo

ordina

tion

QA7Co

mpe

tencesof

Practitione

rs

QA8Qua

lityMan

agem

ent

QA9Organ

isatio

nofVNFIL

Provision

Special

evalua

tionqu

estio

ns

16_01_NL 2 1 1 ✓16_02_AT 2 1 1 ✓17_03_LT 2 1 1 --17_04_FR 2 1 1 ✓17_05_PT 2 1 1 ✓17_06_AT 2 1 1 ✓17_07_AT 2 1 1 ✓17_08_PT 2 1 1 ✓17_09_AT 3 1 1 1 ✓17_10_NL 2 1 1 ✓17_11_SK 2 1 1 ✓

Total 2,1 6 2 1 3 5 1 4 0 1 10


Allbutone institutiontookadvantageofthepossibilitytoposespecialevaluationquestionstothePeers. Some institutions even did so extensively. This contributed to tailoring the Peer Review tospecific information needs and making its outcomes useful to the institution. In some instances,however, the special evaluation questionswere in fact asking for expert consultancy and/orwereimpossibletoanswerintheframeworkofaPeerReview.ThisputstressonsomePeerTeamswhothoughtittheirdutytoanswerthesequestionseventhoughtheManualspecifiesthatrecommenda-tionsandconsultingareonlypartof thePeerReview inexceptionalcases (and thiswasalsohigh-lightedduringthetraining).

3.4 Peers

Peerapplications

In total, 32 people applied online as Peers, 20women and 12men, fromdifferent countrieswithvarying institutionalbackgrounds.Aboutone thirdof theapplicants (10)work forVNFILproviders.About50%ofallapplicantshadexperienceinVNFIL.5oftheapplicantsdidnotparticipateinPeerReviews,4ofwhichhadalsoparticipated(at leastpartly) inthePeerTraining. Intheend28PeersbecameactiveduringthepilotPeerReviews.OneapplicationwasonlypartlyfilledoutandonePeerdidnotfilloutaPeerapplicationatallanddidnotparticipateinthetrainingeither.



BackgroundandcharacteristicsofPeers

Of the Peers, about two thirds (18)were female, one third (9)male. They came from all partnercountries:9fromAustria(almostathird–duetothehighnumberofAustrianpartners),6fromTheNetherlands (including the coordinating body), 5 from Lithuania (with a high number of nationalPeers,4fromPortugal,3fromSlovakiaand(only)1fromFrance.

Figure8: CountriesofPeers

Source: PilotDatabaseandPeerRegister(merged)

Of the28peoplewhoparticipated in theroleofaPeer,19were froman institutionthatwasalsobeingreviewedintheproject(67,9%).TheotherswerePeersexternaltotheprojectandrecruitedbytheProvidersforthePeerReview.

WhetherPeerswereinvolvedinthePeerReviewoftheirowninstitutionwasalsoaquestionaskedinthesurvey;15Peers(64%)answeredinthesurveythattheydidso(whichcorrespondstotheanaly-sisabove)2,twothirdsoftheseasFacilitators,onethirdas interviewee,onePeeralsoasadirectorandtwoPeersalsoinotherfunctions:onesupportedtheFacilitator,theothertrainedPeers(multi-pleanswerspossible).

Figure9: FunctionsofPeersduringreviewofowninstitution

Source: OnlineSurveyofPeers(N=15),multipleanswerspossible

2NotallPeersansweredtheonlinesurvey,seeaboveresponserates.



ThetablebelowshowsthedifferentinstitutionalbackgroundsoftheactivePeers:

Figure10: InstitutionalbackgroundsofactivePeers

Source: PeerRegister,N=27

15ofthe28Peershadexperienceinvalidationofnon-formalandinformallearning,manyofthemalsolong-temexperience(median:7years;mean:9years),6Peersevenfor10ormoreyears.

ThemajorityofthePeershad(some)experienceinevaluation/qualityassurance.Accordingtothein-formation in the Peer application, 20 of the 28 Peers had previously done review/evaluationwork.AboutonefourthofthePeers(7)hadhadtrainingasISOauditor(2alsoasexternalauditor),2PeersweretrainedEFQMassessors.4Peershadhadsomeotherrelevanttraining(otherQMsystems,train-ingduringuniversitystudies),abouthalfofthePeershadhadnotraininginqualityassurancebefore.

Figure11: PreviousQualityAssuranceTrainingofPeers


18werefromaninstitutionthathadbeenreviewedbefore,mostlyinISOaudits(orrelatedassess-ments),someinPeerReviewsduringpreviousEuropeanprojects.

Peerswere also asked for a self-assessment of their expertise in different skills and competencesnecessarytocarryoutaPeerReview.



Table12: ReviewskillsofPeers(self-assessment)

Source:PeerRegister,N=26

Accordingtotheirself-assessment,Peersbringaveryhighlevelofrelevantreviewskillstothejob:

§ Beingabletogiveoralfeedback,conductinterviewsandmoderategroupsaretheareaswherethePeersratetheircompetenceshighest,followedbyanalysingqualitativedata,writingreportsandgeneralskillslikeconflictmanagementandtimemanagement.

§ Conductingobservations,analysingquantitativedataanddoingreviewworkinforeignlanguagesareskillswherePeersfeellesscompetent.

§ ThePeershavecomparablylowexpertiseinscientificevaluationsofvalidation.

Thepreviousexperiencesofreviewprocessesorevaluationskillsdidnotinfluencethecompositionof thePeerTeamssinceotherconsiderations tookprecedencewhenthePeerTeamswereput to-gether(seeabove“Peerpuzzle”).ItturnedoutthatsomePeerTeamsdidnotencompassPeerswithreportwritingskills,althoughtheyhad indicatedotherwise in theirapplication,or that insome in-stancesPeerCoordinatorsfeltoverwhelmedbytheirrole.SoperhapssomeofthePeerswereover-confidentintheassessmentoftheirownskills.

(→FormoredetailsonprofilesofPeers,seetablesinAnnex)



3.5 PeerTrainingandotherformsofsupport

3.5.1 Trainingandsupportoffered

EuropeanPeerTraining,June2016,Vienna

TheEuropeanPeer Training tookplace in theChamberof Labour („AKBildungszentrum“, Theresi-anumgasse 16-18, 1040 Vienna) and at the site of theWeiterbildungsakademie between 13th and17thofJune2016.

The Peer Review Training aims to impart to prospective Peers all necessary knowledge, skills andcompetencestoconductaPeerReviewinaprofessionalmannerandaccordingtothequalitycriteriasetoutintheEuropeanPeerReviewManual.ThetrainingfollowedtheEuropeanPeerTrainingcur-riculumthathadoriginallybeendevelopedinanearlierprojectandhadsubsequentlybeenadaptedtoVNFIL in thecurrentproject.Thetrainingcoveredallphasesof thePeerReviewandputspecialemphasisonthequalityofthePeerReviewprocessandtheprofessionalroleandtasksofthePeers.It isdescribed indetail ina separatedocument.3The trainingalso includedan internationalWork-shopwithAustrianexperts(afternoonofJune15)4andanextensiveplanningphasefortheupcom-ingpilotphase.

Uponcompletionofthetraining,allparticipantsreceivedaEuropeanPeerTrainingCertificatefromEPRAandwereincludedintheEuropeanPeerRegister.

Adetailedagenda, apowerpointpresentationandothermaterialweremadeavailable for furthertrainingsonthenationallevel.

EuropeanPeerTraining,June2016,Vienna

23peopleparticipatedintheEuropeanPeerTraining(ofwhichonepersonwasthetrainerandoneperson participated only partly).5The majority of the active Peers had thus undergone the sametraining(seealsobelow).Sincemostparticipantshadmultiplefunctionsinthepilotphase,thisalsomeantthattherewere1to2peoplepresentfromeveryreviewedinstitution(atleastoneperinsti-tution).3participantsoftheEuropeanPeerTrainingsdidnotparticipateinPeerReviewsafterwards.

3Gutknecht-Gmeiner,Maria(2018):PeerTrainingProgrammefortheProject„PeerReviewVNFILExtended“,Vienna.4Workshop“ValidationofNon-formaland InformalLearning(VNFIL) inaEuropeanPerspective.ExchangeofExperiencesandPeerReviewasanInstrumenttoStimulateQualityDevelopment”organisedwithsupportoftheChamberofLabour.39Austrianandinternationalparticipants.5Theattendancelistthusshows22participants.



NationalTrainingNetherlands,October2016

AnationaltrainingwascarriedoutonOctober5thand6th,2016,intheNetherlands.Thetrainingwasfullydocumented. Itdealtwith themost importantelementsofPeerReview.TheseconddaywasusedtopreparethefirstDutchPeerReview(Libereaux,November2016).

12People,amongwhomallDutchPeerswhobecameactiveinthepilotphase,tookpartinthistrain-ing.ThreeDutchPeersthushadtwotrainingexperiences–ontheEuropeanandthenationallevel.Thetrainingalsoinvolvedfurtherpeoplewhowerenotdirectlyinvolvedinthepilotphase.

DutchPeerTraining,October2016

Otherformsofsupport

Other forms of support camemainly from the coordinating body (see above: 3.1.Organisation ofPilots)andthroughmentoring/learningactivitiesbetweenpartners(cf.MentoringReport).

3.5.2 Participationintrainingsandotherformsofsupport

MonitoringdataandtheresponsesfromthePeerSurveyshowthatalmostallPeersunderwentsomesortoftraining.

Figure13: Participationin(typesof)PeerTrainings

Sources: PilotDatabase,DocumentationofEuropeanandNationalPeerTrainings;OnlineSurveyofPeers;

datamerged;N=28;multipleanswerspossible



Almosttwothirds(18res.64%)ofthePeershadtakenpart intheEuropeanPeerTraining.6Peers(21%)participatedinaNationaltrainingintheNetherlands,3ofwhichhadalsoparticipatedintheEuropean Peer Training. Therewere seven active Peerswho neither participated in the EuropeanPeerTrainingnortheNationalTrainingintheNetherlands,fourofthesehadhadotherPeerTrain-ings.Allinall,6peoplehadparticipatedinotherPeerTrainingsprevioustothecurrentproject,twoofwhichhadtakenpart intheEuroPeerguidtraining,whichfollowedthesamebasicqualitystand-ardsandcurriculum.

ThreePeershadreceivednotrainingatall (11%),buttheydidreceiveotherformsofsupport.Theresthadatleast1PeerTraining(71%)oreventwo(18%),mostofwhichparticipatedintheEuropeanPeerTraininginJune2016inVienna.ThisshowsacomparablyveryhighleveloftrainingofPeers.

Figure14: NºofPeerTrainingsperPeer

Sources: PilotDatabase,DocumentationofEuropeanandNationalPeerTrainings;OnlineSurveyofPeers;

Datamerged;N=28

ThemostcommonandmostimportantformofsupportforPeerswassupportfromthecoordinatingbody(2/3), followedbypreparatorymeetings(50%)withtheotherPeers intheirteam,4ofwhichmentioneddoing itoverSkype. In2of thesecases, theVNFILproviderand/or the facilitatorwereinvolved.ThecoordinatingbodyreceivedsupportfromtheEuropeanPeerReviewAssociation.

InternalmentoringwasreportedasaformofsupportforathirdofthePeers(allofwhichworkedinaninstitution thatwasalsobeing reviewed).Only twoPeers (8%)hadnot receivedany furthersupportduringthepilotphase,theyhad,however,participatedintheEuropeanres.DutchPeerTraining.

Figure15: OtherformsofsupportforPeers(multipleanswerspossible)

Source: OnlineSurveyofPeers(N=24,multipleanswerspossible)



Except for the LithuanianVNFILprovider (whowas, however, supportedby the Lithuanianprojectpartner), all reviewed institutionshad staff trainedduring theEuropeanPeerTraining.9of the11providers (82%) had support from the coordinating body; 4 did some internal mentoring/mutuallearningactivitiesduringthepreparatoryphase(seealso→Mentoringreport)and4providersmen-tionedhavinghadapreparatorymeeting.Altogether,allproviders received somekindof support;54,5%receivedmorethanoneformofsupport,2onaverage.

3.5.3 Feedbackontrainingsandsupportduringpreparatoryphase

FeedbackonEuropeanPeerTraining

TheEuropeanPeerTrainingwasconsideredeither“helpful”(38,5%)or“veryhelpful”(61,5%)byall13Peerswhoansweredthisquestion.

Providerswerealittlelesssatisfied–butthetrainingwasalsonotgearedtowardsthem,buttothePeers.Still,almostallproviders(91%)foundtheEuropeanPeerTrainingeither“useful”or“veryuse-ful”aswell,exceptforonewhofoundit“notsouseful”,thereasonbeingthatthestaffofthispro-viderwerealreadyveryexperienced.

Table16: FeedbacktoEuropeanPeerTraining

Howhelpful? Peers Providers

Veryhelpful 61.5% 50%

Helpful 38.5% 40%

Notveryhelpful -- 10%

Nothelpfulatall -- --

Source: OnlineSurveyofPeers(N=13)OnlineSurveyofProviders(N=10;oneproviderdidnotsendstafftotheEuropeanPeerTraining)

BeingtakenthroughallthestepsofthePeerReviewand/orpracticingtheminsimulations/exerciseswerementioned as particularly helpful aspects. 5 participants (out of 9who commentedonwhattheymissedorwhatshouldbechanged)expressedthewishforevenmorepracticalexamples/casestudies.One Peer commented: „Imissed information on how towrite a Peer report. The formatprovidesalotofinformation,butdoesnotprovideanyexamples.[...]Ithinkitwouldbeagoodideatoadd'writingthePeerreport'tothetraining(mainlythepartwhereyoudescribeyourfindings,therestisprettyclear).“GettingtoknowtheotherPeerswasalsoapositiveaspectoftheEuropeanPeerTraining(asintended).

The(mostlyactivatingandexperience-oriented)trainingmethodsusedwereconsideredeitherhelp-ful(onethird)orveryhelpful(twothirds)bythePeers.

TheEnglishlanguageaffectedparticipationfor5participantsinsomeinstances,therest(12)experi-encednoproblemsatall.

Feedbackonandothersupport(includingNationalTrainingintheNetherlands)

13Peersalsogavefeedbacktotheothersupport(includingtheDutchnationaltraining):thesupporttheyreceivedwasratedeither“helpful”(38,5%)or“veryhelpful”(53,8%)byallparticipantsbutone,whoratedit“nothelpfulatall”butdidnotfurthercommentontheirrating.CommentsinthePeersSurvey suggest that for some participants, a littlemore time for further elaboration and practicewouldhavebeenhelpful(especiallyforthosewhohadnotbeenintheEuropeanPeerTraining),butoverallsatisfactionseemedhigh.



3.5.4 Levelof“preparedness”ofVNFILprovidersandPeers

91%(10outof11)ofprovidersratedtheirinstitutionas“wellprepared”or“verywellprepared”forthePeerReview.Theoneinstitutionthatfelt"notsowell"preparedansweredthatthiswasduetoconductingthefirstPeerReviewintheproject.

ProvidersfeltthatPeerswerewellprepared(45,5%)orverywell(54,5%)preparedfortheirtask.ThePeers themselves felt slightly lessprepared (44%verywell preparedand44%well prepared) thanindicatedbytheproviders6:Butonly3Peersfelt“notsowellprepared”,twoofwhichwerenewtoPeerReview,andhadsomeexperiencebuthadto“jump into” the firstPeerReview in theroleoffacilitator(andactuallyfeltbetterpreparedwhenfirstactiveasPeerlateron).Oneresponsepointedout that level of “preparedness” also has to dowith howmuch time and energy the Peers them-selvesinvestedinthepreparationofthePeerVisit(byreadingandanalysingtheSelf-Report,prepar-ingtheVisitetc.).

Table17: Levelof“preparedness”ofthePeers

HowwellpreparedwerethePeers?

PerspectiveofPeers PerspectiveofProviders

Nº % Nº %

Verywell 11 44 6 54.5

Well 11 44 5 45.5

Notsowell 3 12 -- --

Notwellatall -- -- -- --

Sources:OnlineSurveysofPeers(N=25)andProviders(N=11)

3.6 PreparationofthePeerReview(Phase1)

3.6.1 PreparationoftheVNFILProviders

TasksofpreparationofthePeerReviewwereclearlydistributedattheVNFILProvidersastheopenanswersofthesurveyshow.ThemainpreparationworkhadtobedonebythePeerReviewFacilita-tor.Mostly s/heworked closelywith a teamwhere e.g. themanager or the qualitymanagerwasinvolved. S/hewas responsible for doing the agenda,writing the self-reportwith definitionof thequestionsandthechoiceofinterviewees.Severalpeoplewereinvolvedinthepreparation,especiallythedirectorindrawinguptheself-report.FortheconcretepreparationofthePeerReviewmeetingswereheldwiththepersonsinvolved.Sotopicsandquestionsofthecounsellorswereconsideredintheself-report.

Asmanyaspossiblewereinformedabouttheactivitieswithintheinstitution,aboveallthecounsel-lorsandassessors. Informationtookplaceindifferentformslikepresentationsindenregularteammeetingsorviae-mail.

3.6.2 Self-Report

All Self-Reports followed theprescribed format. Peerswere asked in thequestionnaire if the Self-ReporthadbeenagoodbasisforthePeerReview(i.e.ifitwasunderstandableandprovidedallrele-vantinformation);60%answered“yes”,40%answered“partly”;nooneanswered“no”.

The qualitative answers showed that some Self-Reportsweremore, some lessmeaningful for thePeers:Someanalyseswerenotveryextended,sothePeerTeamhadtoaskquestionswhichcouldhavebeenansweredalreadyintheSelf-Report;insomecases,thereportwasdeliveredverylate(oreventoolate)tobeusefulforthePeersinthepreparationoftheReview.Itwasstatedininterviewsthat the Self-Report should includemoredetaileddescriptionof theprovider’sVNFILprocessor a

6ThisisaphenomenonthathasbeenobservedalsoinpreviousPeerReviewpilots,intheirself-assessmentsPeerstendtobemorecriticalthantheproviders.



descriptionof the rolesof involvedpersons inVNFIL-processes.Thequalitativeanswers show thatthePeersrealisedduringtheexperiencehowimportanttheSelf-Reportis(cf.alsoManual)andthatitshouldbewritten inawaytocatertooutsiderswhodonotknowthe institutionorthenationalsetting. As always, therewere somewhowantedmore detail and otherswho thought the reportshouldbeshorter.

TheextentandthecontentofthePeerreportswasveryvarious.Youcouldreallytellthatsomepro-vidersmadea lotofeffortandthatsomeprovidersdidnothavetimetopreparethe[self]report.One of the reportswas a one-on-one copy from another provider.Most of the time, thereweresomequestionsafterreadingtheSelf-Reports.DuringaSkypemeetingbeforethevisit,theseques-tionswerealwaysansweredbythefacilitator.Ingeneral,Inoticedthatthe'simplestuff'isforgottenbythefacilitator.Justbecause(s)heisveryintotheprocess,andforgetstomentionthesmallthingsthat are very crucial for thePeers tounderstand theprocess.Agoodand completeSelf-Report isveryimportant,especiallywhenitconcernstransnationalPeerReviews.WithoutagoodSelf-ReportandagooddescriptionoftheVNFIL-provisionintheparticularcountry,asaPeerTeam,youneedalotof(valuabletime)tounderstandthisandbeabletocarryoutagoodPeerVisit.(experiencedPeerandFacilitator)

3.6.3 PreparationinthePeerTeams

SeveralpartnersemphasisedintheinterviewsandattheworkshopthatjointpreparationinthePeerTeamwasveryimportant.ItwasconfirmedasgoodpracticethattheteammeetsintheafternoonofthefirstdayofthestaytodiscusstheSelf-Reportandtoprepareinterviews.Atthispointoftime,thetasksmustbedividedaswell.Inaddition,intensiveexchangebye-mailandSkypemeetingswereusedforpreparationbeforethePeerVisit.ItwasidentifiedasbestpracticeforthePeerTeamstoworkouttheinterviewquestionsinadvancesothereislesstimepressureontheteamimmediatelybeforethePeerVisit.

ThefindingsofthepartnerworkshopalsoconfirmedthatatraininginPeerReviewforallprospectivePeersisaprerequisiteforasuccessfulPeerReview.Peersmustbeabletoconductinterviews,ana-lyse,drawconclusionsandwriteareportaboutthefindings.

AsrequiredbytheManual,thePeerVisitagendashouldbediscussedinadvancebetweenproviderandPeerssothattheycanmakesomesuggestionsforimprovement.



3.7 PeerVisit(Phase2)

3.7.1 PeerVisitAgendas

The structureof theagendasof thePeerVisits largely followed the recommendationsof thePeerReviewManual;mostadaptedthemodelagendafromtheToolbox.ThePeerVisitsonlyvariedslight-ly in length,followingtheexperiencesoftheprojectEuroPeerguid-RVCthevisitstooktwodays(oralmosttwodays).TheintensityofthePeerVisitshowevervariedwith4PeerReviewshavingfewerthan10datacollectionsessions,andtheothersupto12oreven14sessions.

Table18: PeerVisitAgendas:Numberofsessions,numberofinterviewswithdifferentinterviewgroups

PeerVisit Nºofinterviews

Other(e

.g.Presentation)

Observatio

ns

Tourpremise

s

Documen

tana

lysis

Totalnºofse

ssions

Overalltim

efor

analysis(hou

rs)

Qua

lityAreas

Code

Duratio

nda

ys

Prep

arationda

ys

Man

agem

ent/coo

r-dina

tors

Coun

sellors/A

sses-

sors(Traine

rs)

Cand

idates

Form

ercan

dida

tes

Otherstakeh

olde

r

Otherstaff

Nºofinterviews

16_01_NL 2 0,25 2 3 1 0 2 4 12 1 1 0 0 14 4,5 4;516_02_AT 2 0,5 3 2 1 2 1 0 9 0 1 1 1 12 6,25 6;117_03_LT 1,75 0 2 2 ? 1 0 0 5 1 1 1 0 8 5,5 1;717_04_FR 2 0 2 2 ? 1 1 0 6 0 0 1 0 7 3 1;517_05_PT 2 0,5 2 7 2 0 0 0 11 0 0 1 0 12 7 4;717_06_AT 2 0,25 4 0 2 0 0 1 7 3 0 1 1 12 5,75 3;517_07_AT 2 0,5 2 2 2 2 2 0 10 0 1 1 0 12 6,75 1;217_08_PT 2 0,16 2 7 2 0 0 0 11 0 0 1 0 12 6,75 4;717_09_AT 2 0 2 5 2 0 0 1 10 0 0 1 0 11 6 1;2;517_10_NL 1,75 0,25 2 1 ? 1 0 2 6 1 0 0 1 8 4,5 7;917_11_SK 1,75 0,4 1 1 2 1 1 1 6 2 0 1 0 9 0 1;5Average 1,9 0,2 2,2 2,9 1,3 0,7 0,6 0,8 8,5 0,7 0,4 0,8 0,3 10,6 5,1 2,1

Source: PeerReviewVNFILExtendedPilotDatabase,informationfrompilotdocumentation,esp.Agendas,PeerReviewReports

Inclusionofstakeholders

AnanalysisofthePeerVisitAgendasshowsthattherelevantstakeholdergroupswerebyandlargeincluded in the Peer Visits. A very positive result is that partners alsomanaged to involve formercandidates.Inonecase,anonlinesurveyonthePeerReviewtopicwascarriedoutwithformercan-didatesbeforethePeerVisitandtheresultsweremadeavailableforthePeers.

Thereare,however,somePeerVisits,forwhichtheofficialagendashowednointerviewswithcan-didates and in one instance also not with practitioners. During the partnermeeting in Bratislava,partnerspointedout that in fact thesegroupshadbeen included (e.g. throughan interviewtapedbeforethePeerVisit).Partnerswereaskedtofurnishfurtherinformation,buttonoavail.Withdatamissing, there canbeno final assessmenton thequestionof full inclusionof candidates from thedocument analysis. Yet the feedback from Peers and providers in the survey suggest that by andlargecandidateswereinvolvedsufficiently(seebelow).



Datacollectionandtimeforanalysis

Themost frequentdata collectionmethodwas interviews,manyof themwithone intervieweeoronlyaverysmallgroupofinterviewees.Insomecases,staffmembersoftheprovidersgavepresen-tations.Inatleast4casesobservationstookplaceandin3casesalsoadocumentanalysis.

TherewasmoretimeallottedtoanalysisthanintheEuroPeerguid-RVCprojectwiththeAustrianandPortugueseagendasclearlyinthelead,whileintheNetherlandsandinFrancetimeforanalysisdoesnotseemasimportant.Perhapswealsoseeculturalpreferencesinthisrespect.

AllVNFILprovidersanswered in thequestionnaire that thePeerVisitproceededasplanned in thePeerVisitAgenda.But itwaspointedout in theworkshop, that in somecases therewas too littletimescheduledforreflectionandforinterviews.

Involvementofcandidates

Thepilotphaseshowed that itwaspossible to recruit candidates for the Interviewsaspartof thePeerReview.ProviderswerecontentwiththerecruitmentofcandidatesforthePeerVisit,thePeerswerealittlemorecritical.TheresultsshowthatallProvidersmanagedtoincludecandidates.

Table19: RepresentationofcandidatesinPeerVisit

Howwellwerecandidatesrepresented?

PerspectiveofPeers PerspectiveofProviders

Nº % Nº %

Fullyrepresented 9 37.5 5 45.5

Sufficientlyrepresented 15 62.5 6 54.5

Notsufficientlyrepresented -- -- -- --

Notatallrepresented - -- -- --

Sources:OnlineSurveyofPeers(N=24)andProviders(N=11)

Accordingtotheprovidersitwaspartlyachallengetofindcandidateswhohavetimeduringtheday,sincemost are in employment. It is thereforenecessary to show flexibility and fix interviewswithcandidatesintheprocedureaccordingtotheirtimepossibilities.

Anotherissueisthatusuallyonlycontentandcommittedcandidateswhoarecommittedtothevali-dationprocessareavailableforinterviews.This,however,doesnotpromoteabalancedviewofthevalidationoffer.Most important,possibledrawbacksandstumblingblocksmaynotbedetectedbyonlytalkingtothesuccessfulcandidates.

Itwouldbehelpfultomeet"dissatisfied"participants,or"dropouts"fromtheprocess.Itisdifficulttoorganise,butitwouldpreventpotentialbiasfromself-selection.(OnlineSurveyofPeers)

Inordertoreachthosecandidateswhohave(temporarily)discontinuedtheVNFILprocess,apartnerinstitutiondevelopedanonlinequestionnaireandsent it tothosecandidateswitharequesttore-turnitcompleted.ThequestionsconcernedtheirexperienceswithVNFIL.Thisapproachworkedwellandcouldbe includedinthePeerReviewManualtoprovidethePeerswithadditional informationaboutthislesssuccessfulgroupofcandidates.Thisis,however,adatacollectionactivitythatshouldtakeplaceduringself-evaluation(aswasthecaseintheexamplecited).

Anotheravenueforcapturingtheexperienceoftheunsuccessfulcandidatesisthroughthepercep-tionoftheVNFILpractitioners:

Someonewho reportsonhisown failure is certainlydifficult to find. It is alwayseasier toget thesuccessful ones.But through the coaches and theassessors you canapproach thedifficulties too.Theyhaveagoodoverviewofwhatworksandwhatdoesnot.(Source:Interview)



Opennessandprotectionofinterviewees

Protectionofanonymityofintervieweescanbeaproblemifinterviewgroupsare(very)smallaswasthecaseinmostPeerReviewsinVNFILand/orifonlyonerepresentativeofagroupofstakeholdersisinterviewed.Thelattershouldbeavoidedifpossible.Generally,ProvidersundergoingPeerReviewaswellasPeersshouldbemadeawareofthisproblem.Duringthepilotphase,theopinionsofindi-vidualweregiven in thePeerReviewReportsnaming the source.This isprincipallynotadmissibleandwouldrequireofficialconsentbytheintervieweeconcerned.Obviously,thegeneralfeelingthateverybodyis“friends”inaninstitutionobfuscatesthefactthatthisbehaviourisaseverebreachofprofessionalconduct.

Whenaskedaboutpossiblereticenceofinterviewees(especiallystaffwhomightbeafraidofreper-cussions),Peersmostlyreportedthatallinterviewshadbeenconductedinaveryopenmannerandthat intervieweeshadbeen forthcoming. In rare cases,where interviewparticipantshadhad littleinformationonthePeerReviewbeforehand,therewassomereluctanceatfirst.Inthesecases,thepurposeoftheinterviewandtheconfidentialityoftheinterviewswereexplainedandintervieweesbecamemorerelaxed.

3.7.2 ExperiencesduringPeerVisits

PeerssaidininterviewsandattheworkshopthatitisveryimportantthatallrelevantdocumentsareinplaceatthestartofthePeerVisit.InadditiontothosethatwerepreviouslypreparedliketheSelf-Report,uponrequestofthePeersadditionaldocumentsmaybefurnished,likeanexampleofaport-foliooranoverview from feedbackof thecandidates.Peers shouldask for it ingood timeso thatthesedocumentsandalsonecessaryinformation-e.g.abouttheproceedingsortheinterviewees-canbeprepared.

InatransnationalPeerTeam,it isessentialtoreceiveinformationabouttheeducationsystemandthe VNFIL system in the country beforehand. Thiswaswhy country ficheswere developed in theproject(O1),butitseemsthatpilotphaseparticipantsdidnotextensivelyusethem.Iftheproviderdoes not deliver the necessary documents in time or if they do not contain enough information,problemswithtimeandschedulemayariseduetoopenquestionsandmisunderstandings.

ItwasstressedbythePeersthataconstructiveworkingatmospherebetweenallpersonsinvolvedisveryimportant,whichwasthecaseinallTeams:Morethan83%ofthePeersratedthecooperationinthePeerTeamas“verygood”,therest(17%)as“good”.Noconflictswerementioned.

Table20: QualityofcooperationinPeerTeam

CooperationinPeerTeam Nº %

Verygood 20 83.3

Good 4 16.7

Someconflicts 0

Notgood 0

Total 24 100

Source: OnlineSurveyofPeers(N=24)

ThegoodcooperationbetweenPeersis,ifnothingelse,theresultofagoodjointpreparation.Aposi-tive effect is achieved by communicating openly and transparently between Peers and the repre-sentativesof the institution. Itmustbe clear from thebeginningwhowill takeonwhich tasks.Asconfirmed in the interviews, Skypemeetings for arrangements in advance andpreparationon thefirst evening serve this purpose.During thesemeetings, the agendaand interviewquestionsmustalsobediscussed.



According to theprojectpartners, timepressurehasanegativeeffecton theatmosphereand theresultsof thePeerVisit.ProviderandPeershavetotakecareof thetimemanagementduringthePeerVisit. Iftoomuchtimeisused,forexample,ontheinterviews,thistimewillbemissingintheimportant phases of reflection and drawing conclusions, which are paramount. A lack of time forreflectionwascriticisedbyseveralPeers.Theproblemseemstohavebeenaminorone,though:Inthesurvey,50%ofthemsaidtheyhadexperiencedsomeproblemsbecauseoftimepressure,thoughnoneofthemwereseriousproblems.Mentionedissueswerethat ittookmoretimethanplanned,sometimesduetoextratranslationtime.Theother50%foundthetimeforthePeerVisitsufficient.

ThePeershadtheexperiencethatitisnecessarytoworkdigitallywithlaptopsduringthePeerVisit.Therefore,PeersshouldalsobegrantedaccesstoWi-Fiandaprinter.OnePeerdescribedthattherewasadelayduetoanunclearroomsituationfortheinterviews.

SomePeerTeamsexperiencedlanguageproblems.Over50%ofprovidersansweredinthequestion-nairethatEnglish/languageproblemswereanissue;thoughtheywerenotnecessarilyalwaysprob-lematic.AllPeerswerefluentinEnglishaswellasintheirmothertongue.Aprobleminthiscontextwas that somemanageror interviewees in the institutionsdidnot speakEnglish.Notonlydid thiscausealotoftimebeingspentontranslations;insomecases,thesummarisingnatureofthetransla-tions,whichweredonebynationalPeers,alsoleadtolossofcontent.Also,spontaneousquestions,comments andnotesbecamemoredifficult.Additionally, consecutive interpreting canbe veryex-haustingforthepersoninroleofthetranslator.Consequently,somePeerswhodidnotunderstandtheoriginaltextdidnotfeelasinvolvedintheinterviews.Forsuchcasesatimely,directtranslationisrecommendedintheinterviews,whichmeansnottoomuchshorteningoftheoriginaltext.PossiblelanguagedifficultiesaretobedeterminedalreadyaheadofthePeerVisitduringandhavetobetak-enintoaccountfortheschedulingofthePeerVisit.

Peers said that for successful interviews well prepared questions are crucial, as well as the rightchoiceof interviewees. In somePeerReviews, thequestionswere compiledat shortnoticeat thebeginningofthePeerVisit.

Both the preparation of interview guidelines in advance and comprehensive information of inter-vieweesatthebeginningofinterviewsarepartofthePeerReviewprocedureandweretaughtdur-ing thePeerTraining.Balancing thenarrative flowof interviewswhilenot losing sightof themainquestionsoftheinterviewisaskillPeersneedtodevelopfortrulyexploratoryinterviews.Peerswhodonothavetheexperienceareadvisedtostickmorecloselytotheinterviewguidelines.

Peersalsopointedoutthatinsomecasestheinterview(s)withthemanager(s)is/aresocrucialtothePeerReviewfindingsthatit/theyshouldbeattendedbythewholePeerTeam.However,thisshouldnotbegeneralisedsince itwilldependon theconcretesituationand theworkingapproachof thePeerTeam,whichcanhavemoreorlessdistributionbetweenPeerTandems.

AttheendofthePeerReview,apresentationoftheresultsistobeplannedtogetherwiththepro-vider. It is important that this presentation is scheduled bymanagement and that other relevantemployeeslikeacounsellororthequalitymangerarepresentaswell.

IntheexperienceofthePeers,thispresentationfulfilstwomainfunctions:First,itisamotivationforPeerstopresenttheresultsoftheanalysisinmutualappreciation.Secondly,thereviewedinstitutioncanusetheresultsandtheprocessofimplementationcanbegin.



3.8 PeerReviewReport(Phase3)

3.8.1 Feedbacksessionandusefulnessoffeedback

63.6%ofproviders foundthe feedbacktheyreceived in the feedbacksessionandthePeerReviewReport“useful“;theother36.4%founditatleast"partlyuseful”.

3.8.2 Timelinessofreports

Result orientationandgood cooperation inPeer Teamduring the visit is aprerequisite for a solidfeedbacksessionduringthevisitandameaningfulPeerReviewReport.

SomePeerReviewReportsweredeliveredaftertheduedate,butallweredeliveredbeforeprovidershadtofilloutthequestionnaire.

3.8.3 Writingthe“PeerReviewReport”

ThewritingofthePeerReviewReportwasseenasakeyelementincompletingaPeerVisit.Howev-er,somePeersexperiencedproblemswritingthereport.

OneparticularproblemwasthatinsomePeerReviewsitwasdifficulttodealwiththereportstruc-turewhich focusesonQualityAreas. Thiswasdue to the fact that thePeerReviewprocessoftenveeredfromtheoriginalQualityAreasand/orthatfindingswererelatedtomoregeneralissuesandtoucheduponmorethanoneQualityArea.Inaddition,inmanyPeerReviewsalotofemphasiswaslaid on providing direct recommendations and advice for the reviewed institution (which is not acommonpartinaEuropeanPeerReview).ItwasthereforediscussedintheprojectteamtoexpandthePeerReviewReportandadd(sub)chapters forgeneral feedbackandrecommendations/advice.Sincethisis,however,notaconstituentpartofthePeerReviewandwouldmisleadfutureusersinbelieving that giving general feedback and providing advice is amandatory task itwas decided toleavetheformas it is. InfutureroundsofPeerReviewsitcould,however,bepointedoutthattheformcanbeadapted, ifnecessary, as longas thecorepartsare retained. In theend,experiencedreportwriterswillfindawayofincorporatingallimportantinformation.

AnothersuggestionwastoprovideyetanotherguidelineonhowtowriteaPeerReviewReport.Yet,thereportformalreadyprovidesampleguidanceanditisquestionablewhetherlackingreportwrit-ingskillscanbebuiltbyprovidingsuchaguideline.Intheend,thisfeedbackaswellasinformationonthereportingprocessesforsomeofthePeerReviewspointstoselectingPeerTeamsmorecare-fullyandmakingsurethatat leastonmemberhassufficientexperience inreport-writing.Thesug-gestiontoschedulemoretimeduringthePeertrainingtodiscussthewritingofthePeerVisitreportcanbeheeded in futuretrainings. Itwill,however,notremedythesituationof lackofexpertise inthePeerTeam if itdoesnot includean intensive (andtime-consuming)practicalexercise in reportwriting.ThisinturnwouldnecessitatesimulationofanentirePeerVisitandreportwriting(Phase2and3ofPeerReview).

3.8.4 PeerConsulting

Peershadengagedinaconsiderableamountofconsulting–tryingtobeashelpfulaspossibleandanswering“evaluationquestions”ofprovidersasking foradviceon“tricky”problems.Thiswas forthemostpartbeyondthescopeofaPeerReview(seeabove).Duringthepilotphase,waysfordeal-ingwiththissituationdevelopedwiththehelpofthecoordinatingbody–delineatingtheboundarybetweenthekindofadviceor“suggestions”Peerswereablytofurnishandtherequestsforconsult-ing thatPeerscouldnotprovide. In futurePeerReviews it shouldbe (evenmore)highlighted thatspecific recommendations (consulting) are not the aim of a Peer Review (which is an evaluation).Insteadthereviewedinstitutionshouldfinditsownsolutionstoproblems–andnotposeevaluationquestionsthatareimpossibletoanswerinaPeerReview.



3.9 Puttingplansintoaction(Phase4)

Inthesurvey,90.9%oftheprovidersstatedthattheywillactatleastpartlyonthefeedbackreceived(including4providerswhosaidwouldactonalmostallfeedback).Onlyoneinstitutionanswerednotactingupon feedback, the reasongivenwasnothaving receivedany feedbackduring themeetingbut only with the report – so this answer clearly resulted from amisinterpretation of the surveyquestion.

TheresultsofaPeerReviewarefindingsondifferentsub-aspectsofthechosenQualityAreas.Whichtopics are quickly implemented in the institutions depends onwhether or not they are urgent orsilentquestions.Somecannotbeimplementedimmediatelyduetomissingstructuralrequirements.However, thereare someexamplesof findings thatwerequickly implemented, inparticularwhentheywereintherealmoftheVNFILprofessionalsanddidnottouchuponlargerstructures:

§ The recommendation to prepare a set of standard keywords in the context of labourmarketpolicyorinthecontextofacompanyforusingitduringvalidationorcompetenceassessment.

§ Another recommendation that has been implemented concerns the sustainable effect of thecandidate's validation: So that the participants should also later be reminded of their compe-tencestheparticipantswritinge-mailstothemselvesduringthevalidationthattheyreceiveatalaterpointintime.

§ Inonecase,itwasrealisedthattheportfolioshouldnotonlybeusedasatoolforvalidation,butthatitshouldalsobeusableforpreparingcandidates’curriculumvitae

§ Inanothercase,anopenproblemfromapreviousevaluationprocesswassolved:Thisquestionwaswhymanycandidatesdonotcompletethevalidation.AsaresultofthePeerReview,aclos-ersupportfortheparticipantsandanoptimisedschedulewillbeoffered.

Inadditiontothe implementationofconcretemeasureswithinthe institutions,theprojectalsoal-lowed theexchangeofexperience,evenacrossborders. In the interviews, the followingmeasureswereidentifiedexemplary,thatcouldbetransferredconcretely:

§ Itseemedinteresting,thatinonecountryexisttheroleoftheeditorwhoformulatestheportfo-liosoftheparticipantsinsuchawaythattheybecomecompatibletotheNationalQualificationFramework.

§ Itwasalsonoticedthattheroleofinitialconsultationisveryimportantinonecountry.Theap-propriatedegreesare identifiedfromacatalogueof18.000possiblequalifications. Inthisway,thecandidates’competenciesandwhatispossiblealongthequalificationstandardsarematched.

3.9.1 Communicationofresults

At the end of each Peer Visit, the Peers presented their key findings to the involved staff of theVNFIL-provider.Thefeedbackwasverballycommunicated,mostly inthecontextofapresentation,tothedirectorandtheinterviewedstaff.Insomecases,additionalmembersofthestaff,whowerenotinvolvedinthePeerVisit,werepresent,e.g.thequalitycoordinator.Theotheremployeeswereinformedaboutthefeedbackandafollow-upcarriedoutindifferentwaysofcorporateinformationand communication like websites, newsletters or personal information. In single cases it was notpossible that the optimal selection of employees could be informed in the context of the finalpresentation.

Abouthalfofprovidershadeitheranextrapresentationorinvitedotherstafftothefinalmeeting(orboth).Othersonlysharedorpublishedthereport(sometimestranslated).Inthreeinstitutions,therewasnofurtherdisseminationofresultswithintheinstitutionexceptforsharingitwiththedirector.



3.10 Cooperation,rolesandtransnationalaspects

3.10.1 QualityofcooperationduringPeerReviews

TheonlinesurveysforVNFILprovidersandPeersandthequalitativeinterviewsprovideinformationonthequalityofcooperationduringthewholePeerReviewprocess.Cooperationwasgenerallyrat-edverypositively:

Allprovidersansweredthatcooperationwithintheinstitutionhadbeeneither“verygood”(>60%)or“good”.RegardingtheCooperationbetweenprovidersandPeerTeams,bothsidesratediteither“verygood”(>60%)or“good”aswell.Providerswerenoticeably(even)morecontentthanthePeers.

Table21: QualityofcooperationbetweenProviderandPeersCooperationbetweenProvidersandPeers

PerspectiveofProviders PerspectiveofPeersNº % Nº %

Verygood 9 81,8 14 60,9Good 2 18,2 9 39,1

Someconflicts -- -- -- --Notgood -- -- -- --

Total 11 100 22 100

Source: OnlineSurveyofProviders(N=11)andPeers(N=22)

CooperationwithinthePeerTeamswasalsoratedhighlypositive:83.3%found it“verygood”, therestfoundit“good”(seealsoabove).Noconflictswerementioned.

3.10.2 BenefitsofmultipleparticipationofPeersandfacilitators

MultipleengagementswereregardedhighlypositivebythePeers(85%rateditas“mostlypositive”)andnonegativesideeffectswerementioned.CommentssuggestthattheexperienceofPeerswhohadalreadyparticipated inotherPeerReviewswashelpful to theprocessandtothosePeerswhodidnothaveanypreviousexperience.

Experiencingbothperspectives-intheroleasaPeerandasfacilitator-wasalsojudgedaspositiveinthequalitativeinterviews.Inparticular,itwasseenasanadvantageiftherolePeerisfirsttaken.TheinsightintothePeervisitprocessmakesiteasiertomakepreparationsintheowninstitution–alsoonaverypracticallevel:

“Basedonour initialexperience, Ihavedevelopedamatrix thatwaspracticableduringthesubse-quentPeerVisits.”(Source:Interview)

AmutualPeerReview(i.e.providersexchangePeers)isnotdeemedadecisiveelementofthemeth-od, but was rather due to organisational and financial considerations. Thematic expertise of thePeersremainsthemaincriterionforchoosingPeers.AsapeculiarityofthemutualPeerReview,anintervieweementioned that there is a special commitment between the institutions and that thisraisesthemotivationandconstructivecooperationofPeersandprovidersandsoalsothequalityoftheresults.

3.10.3 LessonslearnedfromnationalandinternationalcooperationduringPeerReview

The Peer Reviews offered the institutions the opportunity to get in touchwith other experts in asimilar areaofwork,whichwashighlyappreciatedbothnationally and internationally:On thena-tional level, Peers gained experience in other educational subsystems; internationally, completelydifferent education systems could be explored. The special feature of Peer Review is to gain verydeepinsightintothesystems.Thiswasmainlysupportedbypersonalcontact,especiallythroughthePeerVisits,whereyouintensivelydealwithdetailedquestionsoftheinstitutioninvestigated.



Astransnationalparticipants,thePeersexperiencedthatacloserelationshipwiththevisitedinstitu-tionwas felteven though theywere inadifferentenvironment.7The reasonwas thatyoualreadyknewa lotabout thepartnerand thatyousharedgoalsandworkexperienceswith the institutionyouvisited.Thiscomparabilitysurprisedwhengettingtoknowothereducationalsubsystems.AvisitwithoutcooperationinaPeerReviewwouldnotrendersuchadeepinsight.However,withregardtotransnationalPeerReviews,itisnecessarytomaketherestrictiveobservationthatasaninternation-alPeer,youmusteitherhavepriorknowledgeof the foreigneducationsystemand the institutionyouarevisitinginordertounderstandtheinnerworkingsofthereviewedinstitutionoryourfellowPeersmustbringyouuptoparduringpreparationtime.

Inthiscasetheinternationalpartwasveryinteresting.ButbeforeyoucandoaninternationalPeerVisit, youhave toknowabout thecountry, the systemand the institutionyougo to.Otherwise itcostsalotoftimereally[learning]aboutthatsysteminthewholecountryandthesysteminthein-stitution. It is a lot of information and you need a clear goal with the international partners.(Source:Interview)

3.10.4 Transnationalaspects:TheimportanceofinternationalPeers

In the surveys, the transnational aspects were generally rated positively by all VNFIL providers –morethan80%foundthempositive,therest“partlypositive(withsomeproblems)”.Requirementsres.recommendationsforincludingatransnationalPeerwere

• awarenessoflanguageissues(56%),• sufficientknowledgeofnationalsystem(33%),• sufficientfunding(22%).

Theimportanceofinsightintoeducationsystemsofthecountryvisitedwasemphasisedalsointheinterviews(seeabove).

The opinionwas expressed in some interviews, that Peer Review supports the implementation ofVNFILwellandindifferentways.EspeciallyinatransnationalPeerReview,itbecomesclearthataninternationalPeerhasnoconflictofinterestandcanthusgiveveryopenfeedbackwithouthavingtodealwithissuesofcompetition.Intheviewofonepartner,workingintheforeignlanguageEnglishalsocreatesthenecessarydistancefromeverydaybusiness.The internationalityalsobringswith itan“auraofexpertise”(thosewhocomefromfarawaymusthavesomespecialexpertise…)thatcanstrengthenthemotivationintheinstitutiontotakethefeedbackseriously.Youalsoslipintoanotherrolewiththejourneyandleavethefocusonyourownsystembehind.TheinternationalPeerbringsnoprejudicestothetask,whichhelpstotakeastepbackandlookatthesituationinthevisitedinsti-tutionwithaprofessionaldistance.

Bycontrast,atthenationalPeerReview,especiallyinasmallcountry,theselectionofPeersmustbemadecarefully.AnationalPeerisratherpartofhis/hersystemandconstantlytakesintoconsidera-tionhis/herownpositionwithinthenationalsystem.Cross-connectionsandrelationshipsareauto-maticallytakenintoaccount.

ThecompositionofTeamswithtwointernationalPeerswasgenerallyevaluatedpositively:Itseemshelpfultobeabletoexchangeviewswithanother internationalPeer,especiallywithregardtotheexternalviewofthesystem.ItisalsopossibleinthiswaythattheinternationalPeerssplitintodifferenttandemsduringthePeerVisit.Fromtheprovider'spointofview,bringinginexperiencefromtwoothereducationsystemsand/orVNFILprocessesisbeneficial.ThiscanonlybeachievediftheinternationalPeersarenotfromthesamecountryorareemployedindifferenteducationalsubsystems.

7ThishasalsobeenarecurringphenomenoninprevioustransnationalPeerReviewprojects.



4. EffectsofthePeerReview50%ofVNFILprovidersansweredinthesurveythattherewereadditionaleffectsofthePeerReviewapartfromfollowinguponthefeedbackreceived.Effectsmentionedinthesurveywereimplemen-tationof special interest groups (1 answer),more involvement inpolicymaking (1), organisationallearningonqualityassuranceandnewevaluationmethods(1),impactonprofessionaldevelopmentofstaff(1)andvalidationbecomingpartoftheaimsoftheinstitution(1).

Additionally, having staff of one’s own institutionparticipate in the PeerReview(s) of otherVNFILprovidersalsohadadditionaleffectsonthatinstitution,becauseitisagreatlearningopportunityforthePeersaswell.

AsaPeeryoubringbackalotofimplicitandexplicitknowledgetoyourownorganisation.The insight into the validation procedures of other institutionsmade the viewmore clear for thepossibilitiesthatexistatone'sowninstitution.(Provider)

4.1.1 Institutional„returnoninvestment“ofthePeerReview

ThePeerReviewmethodwas judged tobebeneficial to the institution in severalways. Formanyinstitutions, this formof quality development has been a new experience and is regarded a goodcomplementtosummativequalityassuranceprocedures.Itwasseenasabenefittohavespecialistsinthe institutionwithdifferentperspectivesandvisionsandaviewfromoutside.Thestudyofthespecific institutional issues by Peers offered the examined institutions anoutsideperspective. Theinstitutionalself-imagecouldbecomparedwiththeexternalviewofthePeers.

PeerReviewallowsaverydeepinsightintothevisitedorganisationthatcannotbeachievedbymerevisitsorexchanges.TheopennessandhonestyofthePeersandthetrustingandcollegialrelationshipareparticularlyappreciated.TheroleofPeersistohelptheinstitutiondevelopfurther(andwasalsorecognisedassuch)whichenhancestheusefulnessofthePeerReviewanditsfindings.Theprocessofdealingwithconcreteproblemsof theworkhadamotivatingand inspiringeffectonallpartici-pants.Positive feedback fromthePeers isperceivedasconfirmation.Someof theresultscouldbeimplementedinaconcretewayorencouragedalonger-termdevelopment.ParticipationintheEu-ropeanprojectitselfandtheopinionoftheinternationalPeersisoftenusedinthecontextofmar-ketingandpublicrelations.Asanexampleofoptimaldissemination,itcanbementionedthat“Frau-enstiftungSteyr”receivedthe"AustrianNationalAwardforAdultEducation"fortheirparticipationinthisprojectinNovember2017.

ThePeerReviewsledtovariousfindingswithintheinstitutions.WritingtheSelf-Reportasaproviderrequiredacriticalexaminationofone'sownprocessesintherun-uptothePeerReview,sotheim-portanceofregularself-evaluationbecameevident.ThefindingsandrecommendationsofthePeerswereanimportantbasisforinstitutionaldevelopment.TheadvantageofPeerReviewisthatitexam-inesexactlythosetopicsthattheinstitutionhaschosentoanalyse.Thelearningprocesswasfurtherpromotedby takingon thedifferent rolesasaPeerandas representativeofaprovider.Thisway,onecanget toknow institutionswithinone’sownareaofprovisionvery thoroughlyand fromtheinside. Itwaspossibletocomparewhatissimilar,what isdifferentbetweeninstitutions,andtoin-corporatethisknowledgeintoone'sowninstitution.

4.1.2 RelevantchangesintermsofinstitutionaldevelopmentsincethePeerReview

During the Peer Review, the participating partner institutionsmostly discussed those areaswhereweakpointsweresuspectedand theneed for changewasalreadyevident.The focusof theques-tionswasbasedontheQualityAreas,alsowithregardtothedelineationoftheproblemareas.Basedontheissuesanalysed,thePeerReviewdidnotresultinanyoverallstructuralchangeintheinstitu-tion.ThepurposeofthePeerReviewwasspeciallytoclarifychangesconcerningcertainissuesortodevelopdifferentscenariosforchangeinmoredetail.Inthiscase,thePeerReviewReportservestounderpintheneedforchanges–alsoforthemanagementoftheinstitution.



Insomeorganisations, institutionalchangeswereimmediately implementedasaresultofthePeerReview;inothercases, itwillbealong(er)processuntilchangescanactuallybeimplemented.Theexistenceofaneffectivequalitymanagementsystemwillhelptofollow-throughonchanges.

Ingeneral,PeerReviewhasapositiveeffectontheawarenessforqualityissuesandthewillingnesstoreflectone’sownwork.OneexampleforthiswastheclarificationofrolesbroughtaboutbythePeerReviewinoneinstitution.Thiswaspossiblebecausetheresponsiblepersonsinvolvedasmanycolleaguesaspossibleinthepreparation,implementationandfollow-upofthePeerReview.

Yes,we had changes and development. The Peer Reviewwas like a confirmation of the develop-ment-plans, that they are o.k. Now we have a new structure of the portfolio-procedures. A fewweeksagowehadtheofficialstartofthenewones.(Interview)

4.1.3 GoodpracticesinVNFILduringPeerReviewtransferredtootherinstitutions/countries

ThefeedbackfromprojectpartnersshowsthatPeerReviewoffersgoodconditionsforthetransfergoodpracticesofVNFILtootherinstitutionsand/orothercountries.Thebasicideaofvalidationinthe institutions is always the same, but there are differences between systems that often do notallowacompletetransfer.Ingeneral,itwillbesub-aspectsthataretransferredfromonesystemtoanother.However,itisoftenhardtoseefromwheretheinfluencesthatcausechangesintheVNFILprocess of the institutions ultimately originated. Effects are most noticeable when changes takeplaceimmediatelyafteraPeerReviewandaredocumentedinthePeerReviewReport.

Isawalotofnicemethodsandinstrumentsthatarebeingused.Theycanbetransferred.(Interview)

4.1.4 LessonslearnedduringthePeerReviewintermsofinstitutionalQAandQM

For somepartners, thePeerReviewmethodwascompletelynew.For them, the learningeffectatthe first visits as Peer, but also in the role as a providerwas the biggest. Some already knew themethodfromthenationalcontext.TheVNFILprovidersconsideredhowthePeerReviewmethodforVNFILcouldbeintegratedintothequalitymanagement.PeerReviewwasseenasagoodmethodofqualitydevelopment,complementingotherexternalevaluationsandauditsthatrathersupportqual-ityassurance.

Aboveall,thecollegialaspectwashighlightedaspositive.TheeffectisdifferentinthePeerReviewthan inanauditbecause it isnotabouthavingtoobtainacertificate. It isalsoadvantageousthat,unlike certifications, oneor two individual questions can be examined very intensively. Therefore,thePeerReviewasasupplementtoexistingQAproceduresmakessense.

Whether concrete activities for implementing Peer Review will follow depends on the extent towhichVNFILisalreadyanchoredonthepoliticalandlegallevels.ConcretediscussionswitheducationpolicystakeholdersaboutPeerReviewforqualitydevelopmentofVNFILhavealreadybeenheld intheNetherlands. IfVNFIL isnotyetanchored,PeerReviewcanhelptobuildupor furtherdevelopthesector.

Aprovidersaid,thatthemethodofPeerReviewisparticularlysuitableforbuildingupalotofknow-how in a short time,which is especially helpful at the timeof establishment of VNFIL.OnlywhenVNFILisalreadyfirmlyestablished,qualityassurancemethodssuchasauditsseemappropriate.Thedevelopment-oriented featureofVNFIL couldbeamotivation for government toprovide financialresourcesforPeerReviewforVNFIL,especiallyincountrieswhereVNFILisnotimplementedyet.ThefindingsfromVNFILthroughPeerReviewcanalsobeusedonthepoliticallevelforfurtherplanning.



5. GeneralassessmentofPeerReview

5.1 ApplicabilityandaddedvalueofPeerReviewforVNFIL

PeerReviewhasprovedtobeverywellapplicableinVNFILandhasmetwithveryhighapprovalbyVNFILprovidersandPeers:

§ According to the answers of the providers’ online survey, 54.5%will certainly conduct a PeerReviewagain, theotherswill considerdoing so.Noproviderprecluded implementingPeerRe-viewinthefuture.Only27.3%,however,plantodoatransnationalPeerReview,with itsaddi-tionalchallenges,again.

§ When asked if they would recommend other VNFIL providers to use Peer Review, 81.8% an-swered„yes“,theother18.2%answered„yes,butwithreservations“,thereservationsconcern-ingtime(2outof11)andfinancialresources(1outof11).

§ (Almost)allPeersrecommendbecomingaPeertootherprofessionals,withtheexceptionofonePeer(outof28),whodidnotspecifyanyreasons.

PeerReviewandVNFILhaveparallelgoalsandprocessesandarethereforeconsideredtobea“goodmatch”:VNFILisalwaysaboutdiscoveringormakingawareofcompetenciesofaperson.Thus,thepersonupgradesherorhisqualificationsandbecomesmoreself-confident.PeerReviewhassimilargoalsandeffects:Itisaboutevaluatinguponrequestandateyelevelandtogivefeedbackinacon-structiveandappreciativeway.

ThepilotphasehasalsoshownthatPeerReviewshaveapositiveeffectonthecontinuedorganisa-tionalandprofessionaldevelopmentofVNFILprovidersandPeers.

5.2 Potentialof (transnational)PeerReviewforthedevelopmentofVNFILandqualityassuranceinVNFIL

Theaddedvalueof transnationalPeerReview forVNFIL is thatdifferences innational and institu-tionalprocessesbecomevisible.Exemplaryaspects canbeadoptedbyother systems thatare lessdevelopedintermsofVNFIL.Forexample,ifVNFILislessdevelopedinonecountryduetounfavour-ablepolicypriorities, impulses fordevelopmentcanbegleanedby lookingat thepracticeof thosecountries and/or providerswho aremore advanced. These experiences can then possibly also bepassedontothepoliticallevel.

BecausetheredoesnotexistaninternationalbasicstructureforVNFILyet,itisimportantthattrans-national Peer Review supports the development of an international standard at a higher level ofVNFILprocess:BestpracticesofallinstitutionsprovideanimprovedstandardofVNFIL.

5.3 FurtherdevelopmentofinstitutionalqualityassurancethroughPeerReview

IntheinterviewspartnersconfirmedtheirinterestincontinuingtoapplyPeerReviewintheirinstitu-tion.ThepossibilitytocombinePeerReviewwithotheralreadyestablishedqualityassurancemeth-odsisassesseddifferently.Ontheonehand,PeerReviewisflexibleenoughtointegrateitintoinsti-tutionalqualityassurance.Ontheotherhand,theadditionalexpenditureoftimeandresourcesisconsideredanobstacle.Someinstitutionsmustmeetqualitycertificationrequirements(e.g.ISOcer-tificate),forthemPeerReviewcanbeanadditionalactivitytomandatoryqualityassurance.Con-ductingaPeerReviewisespeciallyconsideredwhennewissuesanddevelopmentsneedtobetack-led.Theinternationalfeatureisexpectedtoprovideadditionalperspectivesforsolvingveryspecificquestions.Additionally,thePeerReviewisalsoseenasanopportunitytoevaluatequalitymanage-mentsystems.

Currently,ISOiscompulsoryandPeerReviewthe“freestyle”section.FromthepointofviewofapartnerPeerReviewcouldbeintegratedintoISOcertificationsothat(some)auditscanalsobere-



placedbyPeerReviews.8ThiswouldoffertheopportunitytooptforPeerReviewwhilestillcomply-ingwithISO,whichisanecessityformanyeducationaleducationswhoneedaqualitylabeltobeabletocompeteinpublictenders.

5.4 Challengesforimplementing(transnational)PeerReviewinVNFIL

There are several challenges for implementing Peer Review as a formative external evaluation forVNFILonthenationalandinternationallevel.PublicfundingforPeerReviewsisseenbytheprojectpartnersasthemostimportantrequirement.

Also the specificity of the process- and development-orientation of Peer Reviewwas emphasised.This means that the system has to be flexible enough for further development. The question iswhether there is (still) a need for this kindof evaluationprocedure ifVNFIL is (more) established,especiallywhenPeerReviewiscompetingwithother(obligatory)standardisedaudits.

Fromaninternationalperspectiveitshouldberemembered,thatlegalanchoringisnotimplementedineverycountry. Inthosecountries increasedeffortsareneededfordevelopingand implementingVNFIL.Peerreviewcouldmakeanimportantcontributiontothis–iffundingforacoordinatingstruc-tureandfortransnationalactivitiesisavailable.

8Actually,researchonPeerReviewshowedthatastrandofPeerRevieworiginallydevelopedfromspecialtypesofsecond-partyauditsor“dress-rehearsals”forthird-partyauditsinISO(cf.Gutknecht-Gmeiner2008).



6. FeedbacktoManualandToolboxandrecommendationsforfinalisation

6.1 ManualCompliancewiththeManualisthepreconditionforagoodPeerReview–thiswasalsocorroboratedbypartnersintheirfeedback.

The Manual was evaluated positively by the partners and deemed suitable for the evaluation ofVNFIL.AllPeerReviewsbyandlargefollowedtheprocedureaslaiddownintheManual.TheMeta-evaluationrevealedonlyproblemsthatwereduetodeviationsfromtheManual.

Nevertheless,whereassomesaidthatnoadjustmentsneededtobemade,someproposeda“simpli-fication”oftheManualinordertomeetthetimeandresourceconstraintsofthepeopleinvolvedinthePeerReview.Othersaskedformoredetailandmorepractical instructionsandexamples.Withtheseconflictingexpectations,itwasdecidedduringtheBratislavathatthecarefulbalanceofacon-cisebutsufficientlydetaileddescriptionofPeerReview in theManual shouldbemaintained.Part-nersarefreetoaddchecklistsandflowchartstosupportinexperiencedcolleagues.

6.2 QualityAreas

TheQualityAreasand the indicatorswereusedsuccessfullyduring thepilotphase, therewerenoindicationsofproblems.Theoverallassessmentwasthattheywerewell-suitedtostructurethePeerReviewandthattheyhelptofocusonimportantissues:

§ 79.2%ofPeersfoundtheQualityAreasandoutcomes/indicatorsforVNFILproviderseithersuit-ableorverysuitableinthecurrentformat.20.8%foundthemnotsosuitable,allofwhichmen-tionedthattheQualityAreasaretoolonganddetailedand"trytoachieveeverything,incorpo-rateeverything".SuggestionsweretoeitherdownsizetheQualityAreasortocreateanaddition-aloverview(which,infact,alreadyexists).

§ Over90%oftheparticipatingVNFILprovidersfoundtheQualityareasandoutcomes/indicatorsforVNFILprovidersinthecurrentformateithersuitable(63,6%)orverysuitable(27,3%).

ThatthestructureoftheQualityAreasfollowstheprocessofVNFILwashighlightedasaparticularlypositiveaspect.Theywereusedto reflecton thedifferentphasesof thevalidationprocessandtoidentifythoseareasthatshouldbeconsideredinthePeerReview.Thereportingofoutcomes,exam-plesofcriteriaandsourcesofevidencefacilitatedthewritingofboththeSelf-ReportandthePeerReview Report. The Quality Area “Quality Assurance” (now “QualityManagement”) wasmisinter-pretedbysomepartners.

TheQualityAreasaresocomprehensivethattheinstitutioncanpickoutaspectsofcurrentinterestandtotakeintoaccountthestageofdevelopmentofVNFILintheinstitution.ThebreadthalsoofferstheopportunitytocloselyexaminedifferentareasinsuccessivePeerReviews.

SincetheQualityAreasmustbekeptcomprehensive inorder to1)cover thewholeVNFILprocessand 2) be applicable across countries and requests for changeswere contradictive itwas decidedduringtheBratislavatokeeptheQualityAreasastheyare–withsomeimprovementstobemadeintheformulationoftheQualityAreaonQualityAssurancetopreventfurthermisinterpretations.

6.3 Toolbox

The Toolbox was considered very good and helpful. Partners appreciated that tools were flexibleenoughbutclearwhenitcomestogivingguidancefortheirwork.Twoforms(GenderMainstream-ingtool,QualityAreaAssessmentForm)werenotusedbutasrecognisedusefulforthePeerReview.EspeciallytheGenderMainstreamingChecklistwasseenasanecessityforaEuropeanproject.



Critical feedback to the toolbox largely oscillated between the forms being “too detailed” or “notdetailed enough” so that, overall, the tools seem to strike a good balance between the two ex-tremes.

NocommentsweremadeontheredundantbasicinformationintheInitialInformationSheettotheSelf-ReportandPeerReviewReportwhich isnecessaryformaintainingasatisfactory levelof infor-mation in all documents. It seems that the informationon thepossibility of copy-pasting identicalinformationfromoneformtothenextwaspassedonsuccessfullyduringtrainingandthroughdirectadvice.

Insomeinstances,partnerschangedformsthatwereonlysupportivedocumentsandnotmandato-ry.Thiswasthecaseespeciallyfortheinterviewminutes–Peersmodifiedtheformorfounddiffer-entwaysofdocumentinginterviews.ThisisinlinewiththeEuropeanPeerReviewprocedureaslongassufficientdocumentation is takingplace.Thesameholdsgoodfor themodelPeerVisitAgenda,whichcanbeadaptedorsupplantedbyownforms.TheQualityAreasAssessmentformwasnotusedatall.ItseemsthatitspurposeasaninstrumentofpreparingthecentralpartofPeerReviewReportduringthePeerVisit(withouttheadditionalchaptersandinformationaskedforinthePeerReviewReport)wasnotclear.

Partners suggested incorporating theGenderMainstreaming form into the Self-Report in order tostressitsimportancefromtheverybeginning.Thiswould,however,resultinamuchmorecompre-hensive and time-consuming report – which is contrary to the aim of making documentation as“light” as possible. Central information on gender splits (candidates/participants; staff) is alreadyincludedintheSelf-Reportform.

Themeta-evaluation toolwas ratedasenrichingsince itpromotes theexchangeamongPeersandpartners.Itwasalsodeemedveryusefulforthecoordinatorwhoneedstowritethereport.

ThedifficultiesPeersexperiencedinfillingoutthePeerReviewReporthavebeendiscussedabove.NochangesareneededsincetheproblemsdidnotlieinthereportstructurebutratherinthewaythePeerReviewwassetupand/ordeveloped.Insomeinstances,PeerTeamsorCoordinatorsalsomissedcrucialanalyticandreportwritingskills.Itis,however,recommendedtopointouttofutureusersthatthereportstructurecanbeadapted(e.g.addingageneralfeedbacksectionorasectionforrecommendations,ifnecessary)aslongasthecorepartsremain.

Inthesameveinitcouldbehelpfultopointoutmorecandidly• whichpartsofthetoolboxaremandatoryforensuringahigh-qualityandtransparentPeerRe-

view–namelytheInitialInformationSheet,theSelf-Report,thePeerReviewReportaswellasthe“GroundrulesforPeers”asafundamentalsetofrulesgoverningthePeerReviewprocessfromtheperspectiveofPeers–

• andwhichformsandchecklistsprovidehelpandguidancebutmayalsobechangedorinsomecasesalsoomittedorreplaced(allothers).



7. Documentsand(re)sources

7.1 Basicdocuments

EuropeanPeerReviewManualforVNFIL

PeerReviewToolboxforVNFIL

PeerTrainingProgramme

7.2 Documentationofpilotphaseandmonitoringdata

PeerReviewpilotdatabase:overviewofpilotPeerReviews

PeerregistercontainingallinformationfromPeerapplications

Documentation of trainings: Training programmes in English (aims, contents, agenda/schedule,methods),listofparticipants

PeerReviewdocuments(Tool-box)§ Self-reports§ PeerReviewAgenda§ Meta-evaluationofPeers§ PeerReviewReport§ OtherdocumentationofPeerReview,ifpossible:presentations,documentationoffeedback

session,interviewandobservationguidelines,interviewprotocols,photos

7.3 Reportsandpublications

Gutknecht-Gmeiner,Maria(2008):ExterneEvaluierungdurchPeerReview.Qualitätssicherungund-entwicklung inderberuflichenErstausbildung.Wiesbaden:VSVerlag. (Doctoral thesis,Uni-versityofKlagenfurt2006).

Sprlak,Tomas(2017):SynthesisReportonVNFILinpartnercountries,Bratislava.

Gutknecht-Gmeiner,Maria; Kroiss, Sophie (2018):Mentoring Report for the Project „Peer ReviewVNFILExtended“,Vienna.

Gutknecht-Gmeiner,Maria(2018):PeerTrainingProgrammefortheProject„PeerReviewVNFILEx-tended“,Vienna.

Gutknecht-Gmeiner,Maria(2018):PeerReviewLabelandQualityAssuranceforPeerReview,Vienna.

7.4 Listofinterviewpartners

BurtscherKlaudia,FrauenstiftungSteyr

Fuchs-WeiklFranz,AKSalzburg

GonçalvesSusana,Citeforma

GuimaraesTeresa,Citeforma

OsterhoutKeesvan,VigorTransitions

PaulusChristina,BOKU

SmitMarloes,Libereaux

WagnerGiselheid,WBA



7.5 Participantsreflectionworkshop

AllparticipantsoftheprojectmeetinginLisbon

ErikKaemingkMarloesSmitMadelineEichnerEvaBrazdilovaFrancescaOpertiFranzFuchs-WeiklHeidiWagnerChristinaPaulusIngaPuisaIsabelMiguelMichaelaFreimüllerSabineFischerSusanaGoncalvesTeresaGuimares



7.6 Additionaltables

Table22: PeersperPeerReview

Code VNFILproviderPeers

NationalTrans-national Total

16_01_NL LibereauxBV,TheNetherlands 3 3 6

16_02_AT Weiterbildungsakademie(wba),Austria 2 2 4

17_03_LT VilniusVocationalTrainingCentreforServiceBusinessSpecialists,Lithuania

5 2 7

17_04_FR CentreInterinstitutionneldeBilandeCompétences(CIBC),Bour-gogneSud,France

0 2 2

17_05_PT CQEPCITEFORMA,Portugal 2 2 4

17_06_ATUniversityofNaturalResourcesandLifeSciences(BOKU),Unitoflifelonglearning,Austria 2 2 4

17_07_AT FrauenstiftungSteyr,Austria 2 3 5

17_08_PT ISLASantarém,Portugal 2 3 5

17_09_AT AKSalzburg/BFI,Austria 1 3 4

17_10_NL EVCCentrumVigor,TheNetherlands 2 4 6

17_11_SK NárodnýústavceloživotnéhovzdelávaniaNÚCŽV,Slovakia 2 3 5

Totalpilotphase 23 29 52

Source:PilotDatabase

Table23: Peers:distributionbysex

Sex NºPeers %oftotal

F 18 64

M 10 36

Total 28 100

Source:PilotDatabaseandPeerApplications(merged)

Table24: Peers:distributionbycountry

Country NºPeers %

Austria(AT) 9 32%

France(FR) 1 4%

Lithuania(LT) 5 18%

TheNetherlands(NL) 6 21%

Portugal(PT) 4 14%

Slovakia(SK) 3 11%

Total 28 100%

Source:PilotDatabaseandPeerRegister(merged)



Table25: InstitutionalbackgroundofPeers

Typeoforganisation NºofPeers %

VNFILprovider 8 30%Highereducation/researchinstitu-tion 4 15%

Adulteducation 4 15%

Educationalauthority 3 11%

Socialpartners 3 11%

Non-Profit-Organisation 2 7%

Consultingfirm 1 4%

Enterprise 1 4%

Vocationaleducation 1 4%

Total 27 100%


Table26: TypesofPeerTrainingcompletedbyPeers

TypeofPeerTraining NºofPeers %

EuropeanPeerTraining 18 64%

NationalTraining 6 21%

OtherPeerTraining 6 21%

Notraining 3 11%Sources:PilotDatabase,DocumentationofEuropeanandNationalPeerTrainings;OnlineSurveyofPeers;

datamerged;N=28;multipleanswerspossible

Table27: NºofPeerTrainingsperPeer

NºofPeerTrainings NºofPeers %

1Training 20 71%

2Trainings 5 18%

NoTraining 3 11%

Total 28 100%

Source:PilotDatabase,documentationofTrainings,OnlineSurveyofPeersmerged,N=28



Table28: PreviousQualityAssuranceTrainingofPeersTypeofTraining NºofPeers %ISOauditor 7 27%

ISOinternalauditor 7 27%ISOexternalauditor 2 8%

EFQMassessor 2 8%EFQMinternalassessor 1 4%EFQMexternalassessor 2 8%

Other 6 23%Nonesofar 13 50%Total 26 100


Table29: Review/evaluationskillsofPeers(self-assessment)

Expertisein... excellent good fair basic nonesofar

Conductinginterviews 12 8 5 1 0

Conductingobservations 5 11 6 3 1

Analysingquantitativedata 6 6 11 2 1

Analysingqualitativedata 9 9 7 1 0

Givingoralfeedback 13 9 4 0 0

Writingreviewreports 8 12 4 2 0

Reviewworkinaforeignlanguage 2 10 5 4 5

Moderatinggroups 11 10 3 1 1

Conflictmanagement 7 12 4 3 0

Timemanagement 8 10 5 3 0ScientificevaluationsintheareaofVNFIL 1 8 2 4 11

Source:PeerRegister,N=26

Table30: ParticipationofPeersinPeerReviewofowninstitutionParticipationinPeerReviewofowninstitution Nº %

Yes 16 64

No 9 36

Total 25 100




Table31: HowwasthecooperationwithinyourinstitutioninthepreparationandconductofthePeerReview?

CooperationinProvider NºofProviders %Verygood 7 63,6

Good 4 36,4Someconflicts -- --

Notgood -- --Total 11 100

Source:OnlineSurveyofProviders(N=11)

Table32: WasthetimeforthePeerVisitsufficient?

Sufficienttime NºofPeers %Yes 12 50

Overallyes,butsomeproblemsbecauseoftimepressure 12 50

Seriousproblemsbecauseoftimepressure -- --Total 24 100
