Evidence Review 8 Apprenticeships - What Works Growth...Evidence Review: Apprenticeships - September 2015 8 Introduction This review looks at the impacts of apprenticeship programmes

Evidence Review 8

Apprenticeships

September 2015

Contents

Preface 3

ExecutiveSummary 4

Introduction 9

Impactevaluation 13

Methodology 16

Definition 19

Findings 23

Summaryoffindings 33

References 36

AppendixA:Findingsbyoutcome 38

AppendixB:EvidenceReviewed 39

00

Evidence Review: Apprenticeships - September 2015 3

01

Preface

Thisreportpresentsfindingsfromasystematicreviewofevaluationsoftheimpactofapprenticeshipsonfirmsandworkers.

ItistheeighthreviewproducedbytheWhatWorksCentreforLocalEconomicGrowth.TheWhatWorksCentreisacollaborationbetweentheLondonSchoolofEconomicsandPoliticalScience,CentreforCitiesandArupandisfundedbytheEconomic&SocialResearchCouncil,TheDepartmentforCommunitiesandLocalGovernmentandTheDepartmentforBusinessInnovation&Skills.

Thesereviewsconsideraspecifictypeofevidence– impact evaluation–thatseekstounderstandthecausaleffectofpolicyinterventionsandtoestablishtheircost-effectiveness.Toputitanotherwaytheyask‘didthepolicywork’and‘diditrepresentgoodvalueformoney’?Bylookingatthedetailsofthepoliciesevaluatedwecanalsoassesswhattheevidencetellsusaboutdeliveryissues–forexample,isthereanyevidencethatschemeswithaparticularsectoralfocusdobetterthanotherschemes?.

Processevaluation–lookingindetailathowprogrammesoperate–providesavaluablecomplementtoimpactevaluation,butwedonotfocusonthis.Werecognisethatmaysometimescausefrustrationforpractitionerswhoareresponsiblefordelivery.

We see these impact-focused reviews as an essential part of more effective policy making. Weoftensimplydonotknowtheanswerstomanyofthequestionsthatmightreasonablybeaskedwhenimplementinganewpolicy–notleast,doesitwork?Figuringoutwhatwedoknowallowsustomakebetterdecisionsandtostartfillingthegapsinourknowledge.This also helps us to have more informed discussions and to improve policy making.

Thesereviewsthereforerepresentafirststepinimprovingourunderstandingofwhatworksforlocaleconomicgrowth.Inthemonthsahead,wewillbeworkingwithlocaldecisionmakersandpractitioners,usingthesefindingstohelpthemgeneratebetterpolicy.

HenryOverman;Director,WhatWorksCentreforLocalEconomicGrowth


Executive Summary

Thisreportpresentsfindingsfromasystematicreviewofevaluationsoftheeconomicimpactofapprenticeships,focusinginparticularontheimpactonworkersandfirms.ItistheeighthreviewproducedbytheWhatWorksCentreforLocalEconomicGrowth.

Thereviewconsideredmorethan1,250policyevaluationsandevidencereviewsfromtheUKandotherOECDcountries.Itfound27impactevaluationsthatmettheCentre’sminimumstandards.

Approach

TheCentreseekstoestablishcausalimpact–anestimateofthedifferencethatcanbeexpectedbetweentheoutcomeforworkersundertaking,orfirmsoffering,apprenticeshipsandtheaverageoutcometheywouldhaveexperiencedwithouttheapprenticeship(seeFigure1).OurmethodologyforproducingourreviewsisoutlinedinFigure2.

02

Figure 1: Evaluating impactEvaluating impact

VS

Change inoutcome for thoseon the programme

Change inoutcome for those

not on the programme


Findings

Thisreviewconsiderstheimpactsofapprenticeshipsonworkersandfirms.Thissectionsummarisesthedetailedfindings.Weemphasisethatmanyofthesefindingsdependonasmallnumberofstudies.Theyare,however,consistentwithotherresearchonapprenticeshipsandonemploymenttrainingmorebroadly(seeourfirstevidencereview).

What the evidence shows• Thereissomeevidencethatapprenticeshipsimproveskilllevels,andstimulatefurthertrainingor

study.

• Apprenticeshipscanincreasewages,althoughintwoevaluationseffectsarenegative.Impactsalsovarybytypeofparticipant.

• Apprenticeshipstendtohaveapositiveeffectonparticipants’subsequentemployment(andalsoreducesunsequentunemployment).

• Level3orhigherapprenticeshipsdeliversubstantiallyhigherlifetimewagegainsrelativetolowerlevelapprenticeships(basedonthelimitedUKevidenceavailable).

• Thereissomeevidencethatapprenticeshipsaremorelikelytoincreaseemploymentthanotherformsofemploymenttraining(unlessthattrainingalsoinvolvesanin-firmelement).Theevidenceofimpactonwagesismoremixedandappearstovarybygender.

• Thereissomeevidencethatidentifiesmechanismsthatmayincreaseentryintoapprenticeshipsandattendanceduringtheprogramme(e.g.pre-qualifications,higherwagesandsubsidiestoindividuals).However,wehavelessevidenceonwhatworkstoensurepeoplecompleteapprenticeships.

Figure 2: Methodology

government

34

5

1 & 2

user panelacademic panel

1scope

2search

3sift

4score

5

synthesis

To identify what works, each policy review finds and evaluates the evidence which is robust and demonstrates clear outcomes in a 5 stage process

Evaluation evidence is collected using a wide range of sources

Each study is scored based on the quality of

method and quality of implementation

The full set of evidence is refined based on its relevance and the robustness of the research method

Conclusions drawn are based on a combination of these findings and existing literature

academiathinktanks

call forevidence

Existing literature and evidence is reviewed on the basis of an agreed review question,

specific search terms, and a set of inclusion criteria

http://www.whatworksgrowth.org/policies/employment-training/


Where the evidence is unclear • Itisunclearwhetherthedurationoftheapprenticeshipmattersforeffectsonwagesor

employment(althoughlongerapprenticeshipsthatdeliverhigherqualificationsmayhavemorepositiveeffects)

Where there is a lack of evidence • Thereissomelimitedevidencethatfirmsparticipatinginapprenticeshipsexperience

economicgains,suchashigherproductivityorprofits.Thisfitswithsurveyevidence,butmoreimpactevaluationsareneeded.

• Thereistoolittleevaluationevidencetodrawclearconclusionsonwhetherapprenticeshipsworkbetterinsomesectorsthanothers.

• Thereissomeevidencethatpost-apprenticeshipmovescanincreasewagesalthougheffectsdependoncircumstances.

• Thereisnoimpactevaluationevidencelookingattheeffectofapprenticeshipsonagivenlocalarea(ratherthanindividualparticipantsorfirms).

• Thereisnoimpactevaluationevidencecomparingtheeffectsofnationallyrunprogrammesversuslocallyrunprogrammes.

• Existingex-antemodellingsuggeststhattheeconomicbenefitsofapprenticeshipscomfortablyoutweightheircosts.However,onlyoneoftheimpactevaluationsprovidescostdatainaformwhichallowsustocalculateex-postbenefit-costratiosforthatprogramme.

• Noneoftheshortlistedstudieslookattheeffectsofsubstantiallyscalingupapprenticeshipprovision,asiscurrentlyhappeningintheUK.Weneedmoreevidenceonwhetheridentifiedbenefitsalsoholdinalargerprogrammes.GiventheothersubstantialchangestotheUKapprenticeshipsysteminthepastdecadeandahalf,moreuptodateUKimpactevaluationevidenceisalsoneeded.

How to use these reviews

Apprenticeshipsarecurrentlyveryhighonthepolicyagenda,andtheevidencereviewhighlightsanumberoffactorsforpolicymakerstobeawareofwhenconsideringapprenticeships:

• Whiletheevidencesuggeststhathigherlevelapprenticeships(specifically,Level3andabove)mayofferbetteroutcomes,itdoesnotcurrentlytelluswhetherthisisbecausestrongercandidatesgravitatetowardsmoredemandingprogrammes.Ifthisisthecase,policymakersneedtoconsiderhowtoaddresstheneedsofthose‘leftbehind’bythistypeofapprenticeshipoffering.

• Anypolicyshouldcarefullyconsiderhowtorecruitfirmstoprovideapprenticeships,andtraineestofillthem.Abetterunderstandingofthecostsandbenefitstofirmswillhelpinthis(seebelow),aswillabetterunderstandingofwhichpolicydesignaspectsincreasetake-upandreducedrop-out.

To determine policy priorities

TheCentre’sreviewsconsideraspecifictypeofevidence–impactevaluation–thatseekstounderstandthecausaleffectofpolicyinterventionsandtoestablishtheircost-effectiveness.Inthelongerterm,theCentrewillproducearangeofevidencereviewsthatwillhelplocaldecisionmakers


decidethebroadpolicyareasonwhichtospendlimitedresources.Figure3illustrateshowthereviewsrelatetotheotherworkstreamsoftheCentre.

Helping to fill the evidence gaps

Asshouldbeclearfromthisreview,therearemanythingsthatwedonotknowaboutthelocaleconomicimpactofapprenticeships.

Ifachievinglocaleconomicimpactisanimportantpartofthecaseforapprenticeshipprovision,thenthereneedtobemoreevaluationsthatexplicitlyexploretheseimpactsandhowtomaximizethem.Centralandlocalpolicymakers–andprivatesectorpartners–should:

• Looktoundertakesystematiccomparisonsthatcoverdifferentkindsofapprenticeshipmodel-forexample,theGermansystemversusamoredecentralisedsystem.

• Conductfurtherresearchlookingatoutcomesforfirms.Surveysoffirmswhoofferapprenticeshipssuggestthosefirmsseeclearbenefits,buttheymaynotberepresentativeofallemployers.

• Setupevaluationsofschemedesignanditseffectontake-up,completionandoutcomes.ThisisparticularlyimportantgivendevolutionofskillsbudgetstocitiessuchasLondonandManchester.Centralandlocalpolicymakersshouldworktogethertodesignrobustevaluationthatincreasesourunderstandingofhowtoimprovethedesignofapprenticeships.

• Makeschemecostdataavailabletoresearcherssothatrobustbenefit-costratioscanbecalculated.

TheCentre’slongertermobjectivesaretoensurethatrobustevidenceisembeddedinthedevelopmentofpolicy,thatthesepolicesareeffectivelyevaluatedandthatfeedbackisusedtoimprovethem.Toachievetheseobjectiveswewantto:

• Workwithlocaldecisionmakerstoimproveevaluationstandardssothatwecanlearnmoreaboutwhatpolicieswork,where.

• Setupaseriesof‘demonstrationprojects’toshowhoweffectiveevaluationcanworkinpractice.

Interestedpolicymakerspleasegetintouch.

Evidence reviews

Demonstrationprojects

You are here

Capacitybuilding

Understanding what works

More effective policy

Capacitybuilding

Capacitybuilding

Figure 3: What Works Centre work programme


Introduction

Thisreviewlooksattheimpactsofapprenticeshipprogrammesonworkersandfirms.Apprenticeshipsinvolvepaidemploymentinafirm,alongsidetrainingprovidedbythatemployer(andsometimesothers),typicallyleadingtoaformalqualificationortitle:

Apprenticeships involve a program of courses, work-based learning, and productive employment in which workers achieve occupational mastery and industry-recognized credentials.Unlike school-based vocational education, apprenticeships involve extensive work-based learning and practice; real jobs involving production, pay, and the discipline of work; and close mentoring by professionals. Unlike on-the-job training contracts, apprenticeships include related courses and the development of occupational mastery, not simply the ability to do a particular job.1 [our emphasis]

Asthe2012RichardReviewpointsout,'occupationalmastery'iscrucial:apprenticeshipsinvolveparticipantslearningajobrolethatisnewtothemandrequiressubstantialstudyandpracticalcommitment.2

Theuseofapprenticeshipsbyemployersdatesbackhundredsofyears.Morerecently,theperceptionthatapprenticeshipsarealsoeffectiveinimprovinglabourmarketoutcomeshavemadeapprenticeshipsincreasinglypopularwithpolicymakersandemployers.IntheUK,forexample,governmenthasrunapprenticeshipprogrammessince1994.3TheUKhashistoricallyfollowedamarket-basedapproach,inwhichfirmstakethelead;apprenticeshipsareaformoftemporaryemploymentcontract,lastingbetweenoneandfouryears,andthereisnoformallinktotheeducationsystem.

Followingthe2012RichardReview,theprevioussystem,whichinvolvedover250differentapprenticeshipframeworks,isnowbeingsimplifiedandformalized,withsomegraduate-levelapprenticeshipsintroducedthataredesignedtoofferaformalalternativetoafull-timedegree.4Thenumberofapprenticeshipsisalsoincreasing.Forexample,since2010,thenumberinEnglandhasdoubled,withaccompanyingincreasesinfunding;however,therearestillfewerapprenticesasa

1 Lerman(2014).2 Thisisthedefinitionadoptedbythe2012RichardReview.3 CEBR(2013).English,WelshandScottishgovernmentsruntheirownprogrammes.4 Richard(2012),CEBR(2013).

03


shareoftheworkforceinEngland(11per1,000employees)thansomeothercountries(39/1,000inAustralia,40/1,000inGermanyand43/1,000inSwitzerland).5

TheGovernmentiscommittedtofurtherincreasingthenumberofapprenticeships.The2015SummerBudgetannouncedthecreationof3millionnewapprenticeshipsby2020,fundedbyalevyonlargeemployers.Theexactdetailsofthislevy,includingratesandimplementation,haveyettobeestablishedbutfirmswillhaveaccesstothefundsraisedtocoverthecostsofpost-16ApprenticeshipsinEngland.Moneytopayforthetrainingwillcontinuetogostraighttoprovidersratherthanemployers,withemployersaccessingthefundingviaanapprenticeshipvouchersystem.Theoverallaimistoincreaseemployerinvestmentintraining.

Inadditiontothesenationallevelchanges,devolutionwillalsohaveimplicationsforpolicy,withGreaterManchesternowhavingcontrolovertheemployerapprenticeshipgrantsbudget,andothercitiespotentiallyfollowingsuitinthefuture.6

Thisreviewfocusesontheeconomicimpactsofapprenticeshipprogrammesonparticipatingindividualsandfirms(andbyimplication,thelocalareasinwhichtheyarebased).

Largeclaimsaresometimesmadefortheseeconomicimpacts.7Forexample,arecentBISreportsuggeststhatthenetpresentvalueofLevel2andLevel3apprenticeshipsbegunin2013/14are£12bnand£10bnrespectively.8Thesearesubstantialnumbers.Inadditiontohelpingwithprogrammedesign,theimpactevaluationevidencereviewedinthisreportisoneelementindeterminingwhetherornotsuchpredictionsareplausible.

Whatcanweexpectapprenticeshipprogrammestoachieve?Thereisalargeexistingliteratureontheeconomicreturnstotrainingandeducation,includingsomestudiesthatlookdirectlyatapprenticeships.Theseevaluationstypicallyfindlarge,significantpositivewageandemploymentgainstoindividualswhohaveparticipatedinapprenticeships,versusindividualswhodidnot.9Notallofthesestudiesmeetourqualitythresholds,butweincludeanysuitablyrobustevaluationsinourshortlist.

Whatmightdrivetheseeffects?Apprenticeshipsaimtoimproveindividuals’employmentandwageoutcomesbyraisingtheirhumancapital.Apprenticesshouldbemoreemployableaftertheprogramme,andhavemasteredspecificskilledroles.Thisshouldraiselifetimewages,reducetheriskoffutureunemploymentandimproveprospectsofcareerprogression.

Apprenticeshipsmayalsohelpemployers,byprovidingthemwithmoreskilled(andthusmoreproductive)employees.Apprenticesmayalsobemoreloyalandcontributemoreeffortthanregularentry-levelworkers.Ofcourse,aftercompletingtheprogrammeapprenticesmaybepoachedbyotheremployers,whothenreapthebenefits.Forthisreasonfirmswilltendtounderprovideapprenticeships,whichiswhygovernmentsprovidesupporttofirmstocoversomeofthecostoftheprogramme(throughgrants,wagesubsidiesorsomecombinationofthese).Thesepolicyfeatureshavetheadditionalbenefit(toemployers)ofmakingapprenticescheaperthanother,equivalentlyqualified,membersoftheirworkforce.10

Asexplainedfurtherbelow,figuringoutthe‘causal’impactofapprenticeshipsonfirmsandworkersisnotstraightforward.Issuesincludeagenciesorfirms‘cherrypicking’themostableapprentices;

5 Todd(2014).6 HMTreasuryandGMCA(2014).7 CEBR(2013),Todd(2014).8 BIS(2015).9 SeeMcIntoshandGarrett(2009)forarecentreview.10 PicchioandStaffolani(2013).


the‘best’or‘worst’firmschoosingtoparticipateinaprogramme;identifyingthereasonsforpost-apprenticeshipmovesinthelabourmarket,sincethemostableapprenticesmaybethemostlikelytomove;andtrackingparticipants(andcontrolgroupmembers)overtime,particularlyforstudiesthatseektopickoutthelongtermeffectsofapprenticeshipprogrammes.

Figuringouttheimpactsofapprenticeshipsonalocaleconomyisevenharder.Evenifweknowtheindividualbenefits,worker(andfirm)locationdecisionsaffectthegeographicaldistributionofthosebenefits.Forexample,totheextentthatapprenticesinagivenareastayinthatareaaftercompletingtheprogramme,workforcequalityinthearearises;however,ifparticipantsmovetoanotherareathathumancapitalislosttothenewlocation.Ofcourse,humancapitalacrosstheUKriseseitherway.

Thesecomplexitiesalsomakeitmoredifficulttoestimatethenetbenefitsofapprenticeshipschemes,evenifcostdataisavailable(whichisonlythecaseinoneofourshortlistedstudies).

Apprenticeship systems across countries Afinalcomplicationarisesfromthefactthatnotallapprenticeshipsystemsarethesame.AlotofdiscussionintheUKfocusesontheGermansystem,inwhichapprenticeshipsareanintegralpartofthenationaleducationsystem,withcentralizedprovisionandchambersofcommercecloselyinvolvedinregulatingcontentandquality.Over40%ofgeneralsecondaryschoolleaversinGermanycompleteanapprenticeship,whichcanlastuptothreeyears.11OtherEuropeancountriessuchasSwitzerland,Austria,NorwayandDenmarkalsofollowthisapproach.CountriessuchastheUStakeahybridapproach,withsomecentralizedprogrammes,buttheseareverysmallandfocusedonsectorssuchasconstructionormanufacturing.12Otherprogrammesaredevolvedanddesignedbyindividualstates,orthroughpublic-privatepartnerships.

Asaresultofthiscomplexity,itisunsurprisingthattheevaluationsthatweendupconsideringcovercountriesthatapproachapprenticeshipsinaslightlydifferentway.Asnoted,thismakesthembothhardertodefineandmoredifficulttocompare.Thatsaid,theprogrammeswereviewcanbegroupedintothreeoverarchingcategories:thedualeducationalsystem,employmentcontractsandacombinationapproach.

Dual Educational SystemThisapproachisparticularlystronginGermany,andhasbeenfollowedbysomeofitsEuropeanneighbours,includingSwitzerland,Austria,NorwayandDenmark.13Thekeyfeatureofthismodeliscentrallyco-ordinated,government-ledprovision,withapprenticeshipsformalisedaspartofthenationaleducationsystem(Cooke,2003–Paper847).Provisionistypicallyfacilitatedthrougha‘dualeducationalsystem’whichoffersdifferentschoolingroutesforyoungpeople:theacademicroute,typicallyleadingtouniversityandhighereducation(MaturainAustria;GymnasiuminGermany)andthevocationaltrack,whichcomprisescompulsoryvocationaltrainingandleadstoapprenticeship(HauptschuleinGermanyandAustria),althoughthisformatvariesfromcountrytocountry(Cooke,2003-Papers847;Ferstereretal.,2004-Paper878).Apprenticesdividetheirtimebetweenon-the-jobtraining(65-70%)andeducationaltraining(1to2daysaweek),providedtypicallybythestate(Parey,2009-Paper986).Inthismodel,thegovernmentisnormallyresponsibleforcoveringthecostofallapprentices’classroomtraining.Akeyelementofthisapprenticeshipmodelistheformalised,

11 REF-TBC12 REF-TBC13 Lerman(2014).


institutionalisednatureofapprenticeships,andtheregardwithwhichapprenticeshipsareheldnationally;asamarkofthepopularityoftheapprenticeshiprouteinthesecountries,over40%ofgeneralsecondaryschoolleaversinAustriacompleteanapprenticeship(Ferstereretal.,2004-Paper878).

Employment ContractTheapproachinotherEuropeancountries,suchastheUK,isverydifferentfromthatoutlinedabove.Ratherthanagovernment-ledsystem,theonusisontheprivatesectortoprovideapprenticeships.14Inthesesystems,theapprenticeshipisseenmoreasaformoftemporaryemploymentcontract,whichallowsthefirmstopaylesssocialcontributions,andinreturnasksthattheyprovideapprenticeswithcertifiedon-the-jobandclassroomtraining(Capellarietal.,2012–Paper897)..Thisistraditionallynotlinkedtotheeducationsystem;astudentmaychoosetoleaveschoolandcommenceanapprenticeship,butthisisnotaformaltracksetbygovernment(ibid).Thereisnocentrallycontrolledapprenticeshipprogrammeandassuchtheretimesplitbetweenworkandtrainingvariesfromapprenticeshiptoapprenticeship.Trainingisoftenprovidedinfirm,butexternalcoursescanalsobeused–thiscouldlinktocoursesprovidedbygovernmenteducationbodies(ibid).Fundingfortrainingistypicallysplitbetweengovernmentandbusinessesinthismodel;intheUK,Governmentonlycontributes50%ofthecostoftrainingforapprenticesaged19–24(SkillsFundingAgency,2015)15.

Combination approachCountriessuchastheUS–whichhashistoricallyhadaverysmallapprenticeshipsystemwithlittlecoverageoutsideconstructionandmanufacturing–operateahybridapproach.16

TheUShasagrowingapprenticeshipsystem,withPresidentObamaannouncinganew$100milliongrantprogramtosupportthedevelopmentofinnovativeapprenticeshipprogramsacrossthecountryin2015(WhiteHouseOfficeofthePressSecretary,2015)17.Centralisedapprenticeshipprogrammesexist,whicharegovernment-led,suchastheRegisteredApprenticeshipwhichisadministeredbytheOfficeofApprenticeshipandStateApprenticeshipAgenciesatanationallevel.Inaddition,theUShaveimplementedSchooltoWorkreformswhichfocusonhighschoolstudents,withsomeelementsoftheacademic/vocationaltrackingseeninGermany(NeumarkandRothstein,2005–Paper964).Inaddition,however,thereisalsoanemphasisonfacilitatingmoreprivatesectorprovisionofapprenticeships.TheUSdepartmentoflabourrunsan‘AmericanApprenticeshipGrant’whichinvitespublic-privatepartnershipstoimplementapprenticeshipsystems(USDepartmentofLabour,2015)18.Programmesforapprenticesarethereforeorganisedonastatebystatebasisbyprivate,publicandthirdsectorbodies,aswellasatanationalgovernmentlevel.(Veum,1995–Paper976).

Cross-country comparisons of the evaluation evidence Clearly,thiscomplexitycreateschallengeswhencomparingtheresultsofevaluationsofschemesindifferentcountries.Aswithourotherevidencereviews,westillthinkthatitispossibletofurtherourunderstandingofapprenticeshipsbylookingat,andcomparing,theavailableimpactevaluationevidencefromacrosstheOECD.Butthisdiscussionhighlightssomeofthecaveatsthatmustapplytothefindingsemergingfromourreview.Wherenecessary,wehighlightspecificissuesfurther,below.

14 Lerman(2014).15 https://www.gov.uk/funding-rules-2015-to-2016-the-adult-skills-budget-including-apprenticeships,accessed20August2015.16 Lerman(2014).17 https://www.whitehouse.gov/the-press-office/2015/03/09/fact-sheet-president-obama-launches-new-techhire-initiative,

accessed20August2015.18 http://www.dol.gov/apprenticeship/grants.htm,accessed20August2015.

https://www.gov.uk/funding-rules-2015-to-2016-the-adult-skills-budget-including-apprenticeships

https://www.whitehouse.gov/the-press-office/2015/03/09/fact-sheet-president-obama-launches-new-techhire-initiative

http://www.dol.gov/apprenticeship/grants.htm


Impact evaluation

Governmentsaroundtheworldincreasinglyhavestrongsystemstomonitorpolicyinputs(suchasspendingonpublicsubsidiesforapprenticeships)andoutputs(suchasthetotalnumberofapprenticestrained).However,theyarelessgoodatidentifyingpolicyoutcomes(suchastheimpactsofapprenticeshipsonparticipatingindividualsorfirms).Inparticular,manygovernment-sponsoredevaluationsthatlookatoutcomesdonotusecrediblestrategiestoassessthecausal impact ofapprenticeshippolicies(henceforth,werefertotheseas‘programmes’).

Bycausalimpact,theevaluationliteraturemeansanestimateofthedifferencethatcanbeexpectedbetweentheoutcomeforindividuals,firmsorareasimplementingaprogramme(inthiscase,takingpartinanapprenticeshipscheme)andtheaverageoutcometheywouldhaveexperiencedwithouttheproject.Pinningdowncausalityisacruciallyimportantpartofimpactevaluation.Estimates of the benefits of a project are of limited use to policy makers unless those benefits can be attributed, with a reasonable degree of certainty, to that project.

Thecredibilitywithwhichevaluationsestablishcausalityisthecriteriononwhichthisreviewassessestheliterature.

Using CounterfactualsEstablishing causality requires the construction of a valid counterfactual –i.e.whatwouldhavehappenedtoanarea(orpartofanarea)iftheprojecthadn’thappened.Thatoutcomeisfundamentallyunobservable,soresearchersspendagreatdealoftimetryingtorebuildit.Thewayinwhichthiscounterfactualis(re)constructedisthekeyelementofimpactevaluationdesign.

A standard approach is to create a counterfactual group of similar places not undertaking the kind of project being evaluated.Changesinoutcomescanthenbecomparedbetweenthe‘treatmentgroup’(peopleorfirmsparticipatinginapprenticeships)andthe‘controlgroup’(peopleorfirmsnotparticipating).

A key issue in creating the counterfactual group is dealing with the ‘selection into treatment’ problem.Selectionintotreatmentoccurswhenpeopleorfirmsparticipatinginapprenticeshipsdifferfromthosewhodonotdoso.

04


Anexampleofthisissueforapprenticeshipswouldbewhentheagencyimplementinganapprenticeshipprogrammeselectsthemostcapableemployersand/orapprenticesfortheprogramme(‘cream-skimming’).Ifthishappens,estimatesofpolicyimpactmaybebiasedupwardsbecauseweincorrectlyattributebettereconomicoutcomestotheprogramme,ratherthantothefactthattheparticipantsarealreadylikelytoperformwell.

Selectionproblemsmayalsoleadtodownwardbias.Forexample,ifalocalauthorityprogrammeexplicitlytargetsthe‘hardesttohelp’foraprogramme–suchastheverylongtermunemployed,orthosewithchaoticlifestyles,thenwemaymistakenlyattributepooreconomicoutcomestotheprogrammeratherthantheunderlyingchallengesfacingthosetakingpart.

Thesefactorsareoftenunobservabletoresearchers.So the challenge for good programme evaluation is to deal with these issues, and to demonstrate that the control group is plausible.Iftheconstructionofplausiblecounterfactualsiscentraltogoodpolicyevaluation,thenthecrucialquestionbecomes:howdowedesigncounterfactuals?Box1providessomeexamples.

Box 1: Impact evaluation techniques

Onewaytoidentifycausalimpactsofaprojectistorandomlyassignparticipantstotreatmentandcontrolgroups.Forresearchers,suchRandomised Control Trials (RCTs) areoftenconsideredthe‘goldstandard’ofevaluation.Properlyimplemented,randomisationensuresthattreatmentandcontrolgroupsarecomparablebothintermsofobservedandunobservedattributes,thusidentifyingthecausalimpactoftheproject.However,implementation of these ‘real world’ experiments is challenging and can be problematic.RCTsmaynotalwaysbefeasibleforlocaleconomicgrowthpolicies–forexample,policymakersmayunderstandablybeunwillingtorandomisethelocationofprojects.19

Whererandomisedcontroltrialsarenotanoption,‘quasi-experimental’approachesofrandomisationcanhelp.Thesestrategiescandealwithselectiononunobservables,by(say)exploitinginstitutionalrulesandprocessesthatresultinsomelocationsquasi-randomlyundertakingprojects.

Evenusingthesestrategies,though,thetreatmentandcontrolgroupsmaynotbefullycomparableintermsofobservables.StatisticaltechniquessuchasOrdinary Least Squares (OLS) andmatchingcanbeusedtoaddressthisproblem.

Notethathigherqualityimpactevaluationfirstusesidentificationstrategiestoconstructacontrolgroupanddealwithselectiononunobservables.Thenittriestocontrolforremainingdifferencesinobservablecharacteristics.Itisthecombinationthatisparticularlypowerful:OLSormatchingaloneraiseconcernsabouttheextenttowhichunobservablecharacteristicsdeterminebothtreatmentandoutcomesandthusbiastheevaluation.

Evidence included in the review We include any evaluation that compares outcomes for people or firms taking part in apprenticeship programmes (the treated group) after the programme with outcomes in the treated group before the programme ; relative to a comparison group used to provide a counterfactual of what would have happened to these outcomes in the absence of the programme.

19 Gibbons,NathanandOverman(2014).


Thismeanswelookatevaluationsthatdoareasonablejobofestimatingtheimpactoftheprojectusingeitherrandomisedcontroltrials,quasi-randomvariationorstatisticaltechniques(suchasOLSandmatching)thathelpmaketreatmentandcontrolgroupscomparable.Weviewtheseevaluationsasprovidingcredibleimpactevaluationinthesensethattheyidentifyeffectsthatcanbeattributed,withareasonabledegreeofcertainty,totheprojectinquestion.AfulllistofshortlistedstudiesisgiveninAppendixA.

Evidence excluded from the reviewWeexcludeevaluationsthatprovideasimplebeforeandaftercomparisononlyforthosepeopleorfirmstakingpartinapprenticeshipprogrammes,becausewecannotbereasonablysurethatchangesforthetreatedgroupcanbeattributedtotheeffectoftheproject.

Wealsoexcludecasestudies,cross-sectionalparticipantsurveysorevaluationsthatfocusonprocess(howtheprojectisimplemented)ratherthanimpact(whatwastheeffectoftheproject).Suchstudieshaveusefulrolestoplayinhelpingformulatebetterpolicybuttheyarenotthefocusofourevidencereviews.


Methodology

ToidentifyrobustevaluationevidenceonthecausalimpactofapprenticeshipsweconductedasystematicreviewoftheevidencefromtheUKandacrosstheworld.Ourreviewfollowedafive-stageprocess:scope,search,sift,scoreandsynthesise.

Stage 1: Scope of Review WorkingwithourUserPanelandamemberofourAcademicPanel,weagreedthereviewquestion,keytermsandinclusioncriteria.Wealsousedexistingliteraturereviewsandmeta-analysestoinformourthinking.

Figure 1: Methodology

government

34

5

1 & 2

user panelacademic panel

1scope

2search

3sift

4score

5

synthesis

To identify what works, each policy review finds and evaluates the evidence which is robust and demonstrates clear outcomes in a 5 stage process

Evaluation evidence is collected using a wide range of sources

Each study is scored based on the quality of

method and quality of implementation

The full set of evidence is refined based on its relevance and the robustness of the research method

Conclusions drawn are based on a combination of these findings and existing literature

academiathinktanks

call forevidence

Existing literature and evidence is reviewed on the basis of an agreed review question,

specific search terms, and a set of inclusion criteria

05


Stage 2: Searching for EvaluationsWesearchedforevaluationevidenceacrossawiderangeofsources,frompeer-reviewedacademicresearchtogovernmentevaluationsandthinktankreports.Specifically,welookedatacademicdatabases(suchasEconLit,WebofScienceandGoogleScholar),specialistresearchinstitutes(suchasCEPRandIZA),UKcentralandlocalgovernmentdepartments,andworkdonebythinktanks(suchastheOECD,ILO,ipprandPolicyExchange.)Wealsoissuedacallforevidenceviaourmailinglistandsocialmedia.Thissearchfoundjustover1250books,articlesandreports(thefulllistofsearchtermscanbefoundonlinehere:whatworksgrowth.org/policies/apprenticeships/search-terms)

Stage 3: Sifting EvaluationsWescreenedourlong-listonrelevance,geography,languageandmethods,keepingimpactevaluationsfromtheUKandotherOECDcountries,withnotimerestrictionsonwhentheevaluationwasdone.WefocusedonEnglish-languagestudies,butwouldconsiderkeyevidenceifitwasinotherlanguages.Wethenscreenedtheremainingevaluationsontherobustnessoftheirresearchmethods,keepingonlythemorerobustimpactevaluations.WeusedanadjustedversionoftheMarylandScientificMethodsScale(SMS)todothis.20TheSMSisafive-pointscalerangingfrom1,forevaluationsbasedonsimplecrosssectionalcorrelations,to5forrandomisedcontroltrials(seeBox2).WeshortlistedallthoseimpactevaluationsthatcouldpotentiallyscorethreeoraboveontheSMS21.Inthiscasewefoundoneevaluationscoringfive:forexamplesofimpactevaluationsthatscorethreeorfourontheSMSscaleseethecasestudiesandourscoringguideavailableatwww.whatworksgrowth.org.

Stage 4: Scoring EvaluationsWeconductedafullappraisalofeachevaluationontheshortlist,collectingkeyresultsandusingtheSMStogiveafinalscoreforevaluationsthatreflectedboththequalityofmethodschosenandqualityofimplementation(whichcanbelowerthanclaimedbysomeauthors).Scoringandshortlistingdecisionswerecross-checkedwiththeacademicpanelmemberandthecoreteamatLSE.Thefinallistofincludedstudiesandtheirreferencenumbers(usedintherestofthisreport)canbefoundinAppendixA.

Stage 5: Synthesising EvaluationsWedrewtogetherourfindings,combiningmaterialfromourevaluationsandtheexistingliterature.

20 Sherman,Gottfredson,MacKenzie,Eck,Reuter,andBushway(1998).21 Shermanetal.(1998)alsosuggestthatSMSLevel3istheminimumlevelrequiredforareasonableaccuracyofresults.


Box 2: Our robustness scores (based on adjusted Maryland Scientific Methods Scale)

Level 1: Either (a) a cross-sectional comparison of treated groups with untreated groups, or (b) a before-and-after comparison of treated group, without an untreated comparison group.Nouseofcontrolvariablesinstatisticalanalysistoadjustfordifferencesbetweentreatedanduntreatedgroupsorperiods.

Level 2:Use of adequate control variables and either (a) a cross-sectional comparison of treated groups with untreated groups, or (b) a before-and-after comparison of treated group, without an untreated comparison group.In(a),controlvariablesormatchingtechniquesusedtoaccountforcross-sectionaldifferencesbetweentreatedandcontrolsgroups.In(b),controlvariablesareusedtoaccountforbefore-and-afterchangesinmacrolevelfactors.

Level 3:Comparison of outcomes in treated group after an intervention, with outcomes in the treated group before the intervention, and a comparison group used to provide a counterfactual (e.g. difference in difference). Justificationgiventochoiceofcomparatorgroupthatisarguedtobesimilartothetreatmentgroup.Evidencepresentedoncomparabilityoftreatmentandcontrolgroups.Techniquessuchasregressionand(propensityscore)matchingmaybeusedtoadjustfordifferencebetweentreatedanduntreatedgroups,buttherearelikelytobeimportantunobserveddifferencesremaining.

Level 4:Quasi-randomness in treatment is exploited, so that it can be credibly held that treatment and control groups differ only in their exposure to the random allocation of treatment.Thisoftenentailstheuseofaninstrumentordiscontinuityintreatment,thesuitabilityofwhichshouldbeadequatelydemonstratedanddefended.

Level 5: Reserved for research designs that involve explicit randomisation into treatment and control groups, with Randomised Control Trials (RCTs) providing the definitive example.Extensiveevidenceprovidedoncomparabilityoftreatmentandcontrolgroups,showingnosignificantdifferencesintermsoflevelsortrends.Controlvariablesmaybeusedtoadjustfortreatmentandcontrolgroupdifferences,butthisadjustmentshouldnothavealargeimpactonthemainresults.Attentionpaidtoproblemsofselectiveattritionfromrandomlyassignedgroups,whichisshowntobeofnegligibleimportance.Thereshouldbelimitedor,ideally,nooccurrenceof‘contamination’ofthecontrolgroupwiththetreatment.

Note:TheselevelsarebasedonbutnotidenticaltotheoriginalMarylandSMS.Thelevelsherearegenerallyalittlestricterthantheoriginalscaletohelptoclearlyseparatelevels3,4and5whichformthebasisforourevidencereviews.


Definition

Apprenticeshipsarenoteasytodefine,anddeliverymodelsvaryacrosscountries,makingdefinitionharder.However,wecanidentifycertaincommonfeatures.Forthepurposesofthisreport,apprenticeshipsaredefinedas:

Paid employment within a firm, alongside theoretical training that is usually provided by government, the employer, or a trade union, targeted specifically at school leavers (level of education varies by type of apprenticeship scheme). The apprentice often acquires a formal qualification by the end of the apprenticeship.

Apprenticeshipsarenotsimplya)programmeswhichtakeplaceeitherentirelyintheclassroom(i.e.vocationaleducation);orb)programmesthattakeplaceentirelyinthefirm22(asdiscussedintheEmploymentTrainingReview).Rather,itisthecombinationofthesetwoelementsthatmatters.Althoughapprenticeshipschemesdonotnecessarilyneedgovernmentsupport,suchsupportisverycommon.Inlinewiththis,forallthoseprogrammesforwhichtheevaluationprovidesdetails,thereisalwaysanelementofpublicsectorsupport.

Impact evaluation for apprenticeshipsEvaluatingthecausalimpactsofapprenticeshipprogrammesisnotstraightforward.Ideallywewouldwanttorandomizeparticipantsintoaprogramme,andthencomparechangesintheirpost-programmelabourmarketoutcomeswithchangesforsimilarpeoplewhodidnotparticipate.Inprinciplerandomisationisfeasibleforprogrammeslikeapprenticeships-however,weonlyfoundoneexampleinthisreview(wefoundrathermoreinourreviewofotheremploymenttrainingprogrammes(T01).

Apprenticeshipsare,inmanyways,apartnershipbetweenapprenticeandemployer,andunlikemanyemploymenttrainingprogrammes,employerschoosetogetinvolved.Asnotedintheintroduction,surveyevidencesuggeststhatparticipatingfirmsareverypositiveaboutapprenticeships.23Thismayreflectrealbenefitsofsuchprogrammes,butalsosuggeststhatparticipatingemployersmaynotberepresentativeofallemployers(i.e,they‘selectinto’providingapprenticeships).Inprinciple

22 Theseprogrammesarediscussedinouremploymenttrainingreview:http://www.whatworksgrowth.org/policies/employment-training/.

23 BIS(2013).

06



randomisingthefirmswhoparticipateinapprenticeshipsispossible,butwefoundnoreal-worldexamplesofprogrammeswherethishappens.

Intheabsenceofrandomization(ofparticipantsand/orfirms),weworryaboutthe‘selectionintotreatment’problem.Aswithactivelabourmarketprogrammesasawhole,selectionintotreatmentoccurswhenindividuals–orfirms–participatingintheprogrammeddifferfromthosewhodonotparticipateintheprogramme,inwaysthatcanbehardforresearcherstoobserve.

Forexample,employersorprogrammedeliveryagenciesmay‘cherrypick’participantswiththemostskillsormotivationtosucceedinanapprenticeship.24Agenciesmayalsoselectthefirmswhoprovidethebestopportunities.Thismeansthattheaverageeffectsoftheapprenticeshiparebiasedupwards,sincetheprogrammewouldnotdeliverthesamebenefitstootherparticipantsorinotherbusinesses.Conversely,wemayseedownwardbias,ifprogrammesaretargetedatthe‘hardesttohelp’–suchaspeoplewhoareverylongtermunemployedorwhohavechaoticlifestyles,whichmakeitharderforthemtocompletetheapprenticeship.

Asecondsetofchallengesarisespost-programme.Anumberofthestudiesheresetouttoexploretheimpactof‘post-apprenticeshipevents’,suchasmovingjobsorcareerpaths.Justasparticipantsmayselectintoanapprenticeship,however,theirdecisiontostayormovejobsafterwardsmayberelatedtounobservablecharacteristicsofthoseindividuals.Forinstance,firmsmayactivelyseekto‘lose’the‘worst’apprentices.

Athirdsetofchallengesismoreprosaic.Apprenticeshipscanhavelongtimescales–sometimesrunningforthreeyears–whichmeansthatsomeparticipantsmaydropoutduringtheprogramme.Keepingtrackofparticipantspost-programmeraisessimilarissues.Itisnotclearhowlonganyeffectsofapprenticeshipsmightlast–inourshortlistwehavestudiestrackingimmediateimpacts(suchasthefirstjobgained),aswellaslifecycleeffects(wagegainsoverthefollowing40years).Understandingthelonger-termimpactsofapprenticeshipsiscrucialforpolicymakers–butischallengingtodoinpractice.Non-participants–incontrolgroups–maybeevenhardertotrack,especiallyoverlongtimeperiods.

Themostrobuststudiesinourshortlistadoptimaginativestrategiestodealwiththesechallenges,andtoestablishtreatmentandcontrolsettings.Onepaper(study994)usesaRandomisedControlTrialapproachandscores5ontheMarylandScale.ThispaperlooksattheUSCommunityRestitutionApprenticeshipFocusedTraininginitiative(CRAFT),a6-monthemploymentprogrammedesignedtotrainandplacehighriskyouthsandjuvenileoffendersinemploymentintheconstructionindustry.

ThreepapersscorefourontheMarylandScale,andusearangeofapproachesforidentification.Oneusesinstrumentalvariables(IVs).Study878looksatwages,investigatingtheeffectoftheAustrianapprenticeshipsystem,whichisaone-yearvocationaltrainingschemeaimedatsecondaryschoolleavers.Itfocusesonfirmswhogobankruptaftertakingonapprentices,sothatapprentices’trainingtimevariesinwaysparticipantscannotcontrol.Study897usesspatialandtimevariationinpolicyrolloutinItalytoidentifyeffects:apprenticeshipslegislationwasintroducedindifferentregionsatdifferenttimes,unrelatedtoregions’underlyingeconomictrends.Asaresult,theevaluationisabletocomparethechangeinperformanceofsimilarfirms‘treated’withapprenticeshipsatdifferentpointsintime.Study939,anotherItalianevaluation,usesaregressiondiscontinuitydesignthatexploitsdifferentregionalagecutoffsineligibilityforapprenticeships.Asaresultofthesevariationsineligibilitysimilarpeopleundertheageof(say)27willgetapprenticeshipsinsomeregionsandnotothers.

24 Soskice(1994),quotedinFerstereretal(2004).


Studieslookatpost-apprenticeshipjobmovestendtofocusonparticularclassesofmovers,wherethedecisiontomoveis(moreorless)random.Forexample,Study867,forGermany,looksonlyatapprenticeswhomovejobwhentheiremployershutsdownorconductsamasslayoff.Insomecasesthesemoversalsohavetolookforworkindifferentindustriestowhichtheytrained,whichhelpstestthetransferabilityoftheirskillsandhumancapital.

TheseexamplesshowanumberofideasthatUKpolicymakerscouldadopttoevaluatetheapprenticeshipsysteminthiscountry.Inaddition,accesstoadministrativedatasetsthattrackindividualsovertime–suchasdataheldbytheHMRC,DWPandDfE–willbeimportantinidentifyingthelongtermimpactsofapprenticeshipprogrammes.

The returns to education literatureThepaperswereviewarethosewhereapprenticeshipsarethemainfocusoftheevaluation.25Thishelpsensurethatthestudiesweconsiderexplicitlytacklethechallengeoftryingtoidentifythecausalimpactofapprenticeshipsonoutcomessuchasemploymentandwages.

Theapprenticeshipsstudieswereviewherecomplementthewiderliteratureonthereturnstoeducation.Thelattertypicallyregresseswagesoneducationalachievementcontrollingforpersonalcharacteristics(i.e.estimatesaMincerianwageequation).Aswiththestudiesweconsideronapprenticeships,thesestudiesstrugglewithselectionbias:thereisselectionintobothtypeandlengthofeducationbasedonunobservableindividualcharacteristicssuchasmotivationandability.Inordertodealwiththeselectionproblem,alargebodyof‘returns’studiesuseinstrumentalvariableapproaches.Card(1999)summarisessomeoftheseearlierIVstudies,findingthewagereturntoayearofschoolingtotypicallyliebetween8and13%.

Incontrasttotheextensivegeneralreturnstoeducationliterature,therearefarfewerstudieswherethemainfocusisapprenticeships.Totheextentthatthestrategiesusedinthereturntoeducationstudiesconvincinglydealwithselectionintoapprenticeshipstheywouldprovideadditionalevidenceonwageeffects.Butassessingthiswouldbeatimeconsumingtask–henceourdecisiontofocusonstudiesexplicitlylookingatapprenticeshipsandtocoverawiderrangeofoutcomesthaninsomeofourotherstudies.

Mostoftheavailableevidencethatwereviewusesabefore-and-aftercomparisonagainstacontrolgroup(SMS3).Abefore-and-afterstudyofthewagereturnstoapprenticeshipischallengingfortworeasons.First,itisdifficulttofindapre-apprenticeshipwageasmostapprenticesareschoolleaversanddidnotworkbefore.Second,thebefore-and-aftercomparisonshouldcoveralongstudyperiodsinceapprenticeshipsarealongterminvestmentinwhichthepayofmaterialiseoverthelifecycleofearnings.

Forthesereasonsweonlyfindrelativelyfewstudies(fiveintotal)thatestimatethewagereturnstoapprenticeships.Threeoftheseuseaninstrumentalvariableapproach,asinthewiderreturnstoeducationliterature(837,878,986).Sinceapre-apprenticewageisnotavailable,thetwostudiesthatuseabefore-and-afterapproach(847,976)usewagechangesatalaterstageintheapprentice’scareer.Thisisnotastrictbefore-and-aftercomparisonbutdoeseliminatetheeffectoffixedunobservables.Thedrawbackhereisthatitidentifiesonlytheeffectonwagechangesnotlevels;thusonlyapartialimpact.Thesewagereturnstudiestypicallystudyalongerperiodthanevaluationofotheroutcomes.Forexampleadatasetcoveringover20yearsisusedinstudy837.

Somestudiesexamineadifferentaspectofwagesbylookingatthechangeinwagesfollowingajob

25 SeeAppendixBforthekeywordsusedtoidentifystudies.


moveforapprenticeswhomoveoutoftheirprofessionagainstthosewhostayintheprofession.Thisprovidesameasureoftransferabilityofskillsratherthangeneralreturnstoapprenticeships.

Manyofthestudieswereview,however,donotexaminewagesatallbutinsteadconsideroutcomessuchasemploymentandfurtherstudy.Thesearemoreeasilyimplementedasabefore-and-afterpolicyevaluationwithshorterevaluationperiods.Morethanperhapsanyofourotherreviews,theexistenceofasubstantialevidencebaseonthereturnstoeducationsuggeststhatthefocusinthispolicyareaneedstoshifttoassessingtheeffectivenessofdifferentelementsofpolicydesign.Wereturntothisissuebelow.


Findings

Thissectionsetsoutthereview’sfindings.Webeginwithadiscussionoftheevidencebase,andthenexploretheoverallpatternofresults.Afterthisweconsiderspecificoutcomesinmoredetail.

Quantity and quality of the evidence baseThereviewinitiallyconsideredover1,250articlesandevaluationsfromtheUKandotherOECDcountries,whichwerepickedupundertheinitialkeywordsearch.Onfurtherhighlevelreview,almost1,000weresiftedoutasnotultimatelyrelevant(becausetheyweretheoreticalratherthandata-based,becausetheywerecomparativeordescriptiveratherthananalytical,orbecausetheyreviewednon-OECDcountries,werewritteninaforeignlanguageorbecauseofsubjectrelevanceforexample).74werediscardedaspurelyqualitativeevaluationsleavingsome171articleswereshortlistedfordetailedreview.

Afurther61ofthese171wereultimatelydiscountedongroundsofrelevance,and83werediscountedonthegroundsofnotmeetingtheCentre’sminimumstandardofevidence.Theremaining27evaluationshavebeenincludedinthisreview.Relativetoourotherreviews,thisisroughlyinthemiddleintermsofthequantityofimpactevaluationevidenceavailable.26Table1showsthedistributionofthestudiesrankedbySMSscore.

Table 1:Implementation Quality Scores

SMS Score No. of studies Evaluation reference numbersSMS3 23 837,838,847,859,862,867,876,880,900,

904,937,942,950,960,964,976,979,985,986,987,990,997,1045

SMS4 3 878,897,939

SMS5 1 994

Total 27

26 Atthetimeofwritingareascoveredandnumberofimpactevaluationsconsideredareasfollows:employmenttraining(71);sportsandculture(36);roadandrail(29);accesstofinance(27);businessadvice(23);estaterenewal(21)andbroadband(16).Seehttp://www.whatworksgrowth.org/policies/foranup-to-datecomparison.

07


Country coverage Thepapersevaluatedinthisreportcoveranumberofdifferentcountries,mostlyEuropean.Table2providesacountrybreakdown.

OverathirdofthepapersinvestigatetheGermanApprenticeshipSystem.AfurthersevenpapersexamineapprenticeshipsintheUKandUnitedStates,withotherslookingatavarietyofdifferentEuropeancountries,plusonestudyforAustralia.

Asdiscussedintheintroduction,variationbycountrycomplicatescomparisonsoftheevaluationevidence.Wherenecessary,webreakoutresultsbybroadtypeofschemeaccordingtothethreewayclassification–outlinedintheintroduction.

Table 2:Evaluations by country

Country No. of

evaluations Study numbers Austria 1 878

Australia 1 937

Denmark 2 942,990

France 1 838

Germany 11 837,847,867,876,880,904,950,979,986,987,997

Hungary 1 960

Italy 2 897,939

Norway 1 1045

UnitedKingdom

3 862,900,985

UnitedStates 4 859,964,976,994

Total 27

Focus of the evaluationsWeconsiderevaluationsthatcoverthreeaspectsoftheimpactofapprenticeships.Somestudieslookattheimpactofapprenticeshipsonindividualorfirm-leveleconomicoutcomes–thesehavebeenthemainfocusinourotherevidencereviews.Asecondsetofstudiesexploretheeffectofindividuals’post-apprenticeshipdecisionsontheirlabourmarketprospects.Finally,athirdgrouplookatthefactorsinfluencinguptakeofapprenticeships.27

Allthreeoftypesofevaluationprovideusefulinformationforpolicymakers–sowedistinguishbetweentheminwhatfollows.Specifically,weclassifyevaluationsintothreetypesasfollows:

• ‘Returns to apprenticeships’:Studies that compare the impact of undertaking an apprenticeship vs not undertaking an apprenticeship on economic outcomes for individuals or firms.Thisisthetypeofevaluationmostcommonlyusedinourreviewswherechangesinoutcomesforacleartreatmentgroupwhoundertookanapprenticeshiparecomparedtoasimilarcontrolgroupwhodidnot.

27 Thisbreadthofimpactevaluationevidenceisunusualinourreviewstodate,andreflectsthelargeandlong-standinglabourandeducationeconomicsevaluationliterature.


• ‘Post-apprenticeship’: Studies that examine the effects on apprentices of ‘post-apprenticeship’ events, e.g. differences in economic outcomes for stayers vs movers.Inthevastmajorityofcases,thesepapersevaluatetheeffectsofachangeinemployeroroccupationpost-apprenticeship,andevaluateeffectsonthoseprogrammeparticipantsthathavemovedfromtheirtrainingfirmvsthosethatstaywiththeirtrainingfirm.

• ‘Take-up’:Papers that evaluate the effects of specific policy interventions upon apprenticeships.Thesepaperstakeapprenticeshipsasthedependentvariable,andnormallyinvolveanexaminationoftheeffectofvariouspoliciesonapprenticeshipsupplyanddemandforexample.

Aswithotherevidencereviews,intermsofestablishingwhetherpolicyiseffective,wearemostinterestedinthefirstgroupofevaluations–lookingatthereturnstoapprenticeships.Westartwiththesestudies,beforereviewingtheevidenceonpost-apprenticeshipandtake-up.

Returns to ApprenticeshipsNineteenevaluationsconsiderthe‘returnstoapprenticeships’–i.e.lookattheeffectofapprenticeshipsononeormoreeconomicoutcome.Mostoftheseconsidertheimpactonindividualsintermsofemploymentorwages.Asmallernumberconsidertheeffectonindividualskillsortheeffectonfirms.Giventhediversityofoutcomeswethinkthattheresultsbyoutcomearemoreinformativethananoveralljudgementonwhetherapprenticeships‘work’.28

AbreakdownofthestudiesbyoutcomeisprovidedintableA1intheappendix.Wehaveseparatedoutthefirmandindividuallevelstudiesandgroupedthelaterintothreebroadcategories-humancapitalaccumulation,wages,andemploymentandlabourmarketeffects.

Individual outcomes: skills and further training

There is some evidence that apprenticeships improve skill levels, and stimulate further training / study.

Apprenticeshipsaredesignedtoraiseparticipants’formalskillsandhumancapital,itishoped,stimulatefurthertrainingorstudy.Onlythreestudiesconsidertheseeffects(studies838,964,994).Thetwostudiesthatlookatformaltrainingfoundsomepositiveeffects–one(study838)onlikelihoodofundertakingfurtherstudypost-apprenticeship,theother(study994)forattendanceandlengthofparticipationinformaltraining,butwithnoeffectoncompletion.Study964alsolooksatthelikelihoodofundertakingfurthertrainingpost-apprenticeship,findingmixedresults.

Intermsofskilllevels,study994,whichevaluatestheUSCRAFTprogrammedescribedabove,looksatformaltrainingduringtheprogramme.UsingaRandomisedControlTrial,thestudyfindsthatCRAFTparticipantsaresignificantlymorelikelytohaveattendedaGeneralDiplomaprogramme.ApprenticesattendingtheGeneralDiplomaprogrammealsotendtoparticipateforlonger(i.e.iftheydrop-out,thedrop-outlater).However,highschoolgraduationratesdidnotdifferbetweentreatmentandcontrolgroups.Study838,ontheFrenchapprenticeshipsystem,findsthatprogrammeparticipantsaresignificantlymorelikelytogetahighschooldiploma,andalsohavea42%higherprobabilityofstayingineducationafterfinishingschool.

28 Insomeofourotherreviews,wherethereislessdiversityinoutcome,butmorestudiesthatlookatmultipleoutcomes,wehavefounditusefultosummarisetheoverallpatternoffindings(toaccountforpossiblecorrelationacrossoutcomesforevaluationsofaspecificprogrammethatconsidersmultiplerelatedoutcomes).


Paper964alsolooksatlikelihoodofundertakingpost-apprenticeshiptraining.ItevaluatestheUSSchooltoWorkProgramme,whichisaimedat‘hardtohelp’groupsTheauthorsfindmixedresultsonthelikelihoodthatparticipantspartakeinfurthertraining.Formenwhoarelikelytoattendcollegethereisanegativeeffectontwo-yearcollegeattendance,whileformenunlikelytoattendthereisapositiveeffect(althoughanegativeeffectonfour-yearcollegeattendance).29Forwomen,thereisapositiveeffectonbothtwo-yearandfour-yearcollegeattendance.Unfortunately,theauthorsdonotprovidefurtheranalysisonwhatmightbedrivingthisresult.

Evenifanapprenticeshipdoesnotleadtoformalqualificationsorfurthertrainingitisstillpossiblethattheyimproveworkerproductivity(indeed,thisisacentralaimofapprenticeships).Inturn,thisincreasedproductivitycouldtranslateintohigherwages.Weconsiderwageeffectsnext.

Individual outcomes: wages

Apprenticeships can increase wages, although in a couple of evaluations some effects are negative. Impacts also vary by type of participant.

Ifapprenticesaremoreproductivethiscouldtranslateintohigherwages,eitheratthefirmwhichtrainsthemormorebroadlythroughtheirfuturecareers.Offsettingthis,apprentices’wagebargainingpowermaybelimitedallowingfirms,ratherthanworkerstobenefitsfromanyproductivityimprovements.Boththeextentofanyproductivityeffectsandvariationsinbargainingpowermaymeanthatwageeffectsvaryacrossapprenticeships.

Fivestudiesexplorewageeffects,withonlyone(study837)findingconsistentlypositiveeffects.Paper837findsthatapprenticeshipsleadtowagegainsinthe40yearsaftertheprogramme.30

Study878,usingAustriandata,findsawagegainofaround2.6%foraone-yearapprenticeshipscheme,andonlyforparticipantsinfirmswithlessthan10employees;wageeffectsforapprenticesinallotherfirmsarestatisticallyinsignificant.Study847alsofindsmixedeffectsofapprenticeshipsuponwages,dependentontheschoolingsystemundertakenpriortoapprenticeship;forapprenticestakingtheAbiturroute(pre-universityroute)theeffectsarenegative,whilstforthosetakingtheHauptschuleroute(pre-apprenticeshiproute)theeffectsarepositive.

Studies976and986findthatapprenticeshipshaveastatisticallyinsignificanteffectonparticipants’wages.Study986alsousesGermandata,andcompareswagesfor23-26yearoldsonapprenticeshipsversusthoseonothervocationaltrainingschemes,findingnosignificantdifferencebetweenthetwogroups.Study976findsnorelationshipbetweenthedurationofanyformofapprenticeshipsandwagelevelsintheUS.

29 Thelikelihoodofattendingcollegeisestimatedusingareducedformmodelforcollegeattendanceestimatedexcludinginformationonschool-to-workparticipation.Individualsare‘likelytoattend’iftheyareintheupperhalfofthedistributionofpredictedprobabilitieswithrespecttotheirgender.

30 Thepaperactuallydemonstratesnetpositiveeffectsonameasureor‘welfare’(inthesenseusedbyeconomists,ratherthanwelfarepayments).TheGermanapprenticeshipsystem,whichlaststhreeyears,paysparticipantsnotablylowerwagescomparedtoothercountries,andtheselifecyclegainsin‘welfare’takeintoaccounttheshorttermopportunitycostsofparticipatingintheprogramme.Given,however,thattheafter-programmegainslargelyreflectincreasesinwages,weincludethisstudyinthissection.


Individual outcomes: employment and labour market

Apprenticeships tend to have a positive effect on participants’ subsequent employment (and also reduce unemployment post-programme).

Incontrasttotheeffectonwages,the11evaluationsthatconsiderpost-programmeemployment,generallyfindpositiveeffects.Nineofthesestudieslookattheeffectofapprenticeshipprogrammesonparticipants’subsequentlabourmarketoutcomes.Fivestudies(900,964,965,985,994)lookdirectlyatemploymentaftertheapprenticeshipiscompleted,withthree(960,985,994)findingpositiveeffectsandtwo,findingmoremixedresults.

Anotherfourstudieslookatunemploymentpost-apprenticeship(studies862,880,986and1045);allfourfindthatapprenticeshipsreducethechancesbeingunemployed.31

Thefinaltwostudiesconsiderspecificaspectsofemployment.Study939focusesonapprentices’subsequententryintothejobmarket,findingpositiveeffectsofprogrammeparticipationonlandingafirstjobwhichisapermanentposition(eitherinthesamefirmorsomewhereelse).

Study997looksatcareermobilitypost-apprenticeship,specificallyapprenticeshipparticipants’movementsfromlow-skilledtohigher-skilledoccupations.UsingGermandata,itfindspositiveeffects.

Firm level outcomes

There is some evidence that firms participating in apprenticeships experience economic gains, such as higher productivity or profits.

Thereareonlytwostudiesintheshortlistthatlookattheimpactsofapprenticeshipsonparticipatingfirms.

Paper897investigatestheimpactoftheBiagiLaw(2003)whichsawchangestotheItalianapprenticeshipsystemthroughraisingthemaximumageto30andintroducingtheoptionforworkplacetraining.Thisevaluationfindspositiveeffectsofthislegislativechangeonfirmlevelemployment,withtheapprenticeshipcontractreformcausinganincreaseinthelevelofapprenticeshipemploymentof5.2%.

Paper987,ontheotherhand,whichlooksattheSüdwestmetallemployerassociationcoachingprogrammeinGermanythattargetsdisadvantagedyoungadultsandcommissionedatrainingprovidertoadministertheprogrammeandmatchfirmsanddisadvantagedyouths,hadnoimpactuponfirmlevelemployment.

Theseconclusionsbroadlymatchwithevidencefromsurveysofparticipatingemployers,almostallofwhichsuggeststrongsupportforprogrammes(albeitonaselectedsampleofemployers).32

Returns to apprenticeships: breakdown by countryCantheseresultstellusanythingaboutwhethereffectsdifferacrossapprenticeshipsystems?Tohelpanswerthisquestion,weusetheinformationonTable2onthebreakdownofstudiesbycountry.Theevaluationofwageeffectsandoflabourmarketeffectsaretheonlyoutcomesforwhichwehavesufficientstudiestocontemplatedifferencesacrosscountries.

31 Study862looksatexitsintoarangeoflabourmarketstatesincludingemploymentandunemployment,butthefocusofthepaperisonthelatter.

32 ForarecentUKexample,seeBIS(2013).Lerman(2014)providesareviewoftheinternationalsurveyevidence.


Forwages,fouroutofthesevenevaluationscomefromGermany.Twoofthesefindunambiguouslypositiveresults,butonereportsmixedresultsandonenoeffect.Incontrastofthethreeremainingstudiesoneshowsnoeffect(fortheUS)andtwoshowmixedeffects(forAustriaandGermany).Overall,giventhesmallnumberofstudiesavailableandthesplitacrosscountriesitwouldbehardtoconcludethatthereisstrongevidenceofmarkeddifferencesintermsoftheeffectonwages.

Turningtootherlabourmarketeffects,wecangrouptogethertheninestudieslookingatemploymentandunemployment.TheseevaluationshaveamorebalanceddistributionacrosscountrieswithtwocoveringGermany(bothfindpositiveeffects),threecoveringtheUK(twofindpositiveeffect,onefindsnoeffect)andoneeachcoveringHungary(positive),Norway(positive)andtheUS(noeffect).Again,giventhesmallnumberofstudiesavailableandthesplitacrosscountries,itwouldbehardtoconcludethatthereisstrongevidenceofmarkeddifferencesintermsoftheeffectonemployment.

Whilethedistributionofevaluationsacrosscountriesmakesithardtoreachanyfirmconclusionondifferencesbetweensystems,thegeographicalspreaddoesgiveussomeconfidencethatthefindingsaregeneralizableoutsideasinglecontext.

Programme designThecountrylevelcomparisonsthatwehaveconsideredsofarinvolvecontrastingresultsfromdifferentsystems.Inthissection,wetakeadifferentapproachandconsiderwhetherthereareanyspecificaspectsofprogrammefeaturesthatarecorrelatedwithprogrammesuccess.Itisimportanttorecognizethatthesmallamountofevidencerestrictstheextenttowhichwecancontrolforotheraspectsofthesystemwhichmayalsoaffectoutcomes.Asbefore,giventhenumberofstudiesavailable,wefocusourdiscussiononlyoneffectsonwagesandlabourmarketoutcomes(employmentandunemployment).

Duration

The effect of the duration of the apprenticeship on wages or employment is unclear

Thereportedlengthsofeachapprenticeshipprogrammevaried.Insomecases,wherenospecificdurationofapprenticeshipwasgiven,weassignprogrammedurationbasedoncountry(e.g.weassumethatGermanapprenticeshipslast3-4years).

Aswithcountrycomparisonsasawhole,therearenostrongpatternofdifferencesacrossapprenticeshipsofdifferentduration.Forwages,apprenticeshipslasting3-4yearsshowamixtureofpositive(studies837);mixed(studies847and878)andnoeffects(study986).33

Foremploymentresultsaresimilarlyinconclusive.Bothstudiesthatlookatapprenticeshipslasting3-4yearsshowpositiveeffects(studies880and986).Butsodotwooutofthethreestudieslookingatapprenticeshipslasting2-3years(studies960and1045)andtheonestudythatlooksatapprenticeshipsofshorterthanayear(study994).34Consistentwiththis,study985whichconsidersmultiplelengthapprenticeshipsalsofindspositiveeffectsonemployment.

Thisfinalevaluation(study985)alsoprovideadirectcomparisonofapprenticeshipsofdifferentlengths–specificallytheintermediate(NVQLevel2)andadvanced(NVQLevel3)apprenticeshipsfortheUK.Thepaperreportsgreaterwageandemploymentreturnsforadvancedapprenticeships,

33 Apprenticeshiplengthcouldnotbedeterminedforthefinalevaluationthatlooksatwages(study976)34 Forcompletenessnotethatthethirdstudyforapprenticeshipslasting2-3yearsshowsnoeffect(study964).asdoes

onestudywhereapprenticeshiplengthcouldnotbedetermined(study976).


whichtendtohavelongerdurationsthanintermediateprogrammes.Thepapersetsoutthebenefitstotheindividualandtotheexchequerofbothprogrammes.TheLevel2apprenticeshipcosts£5,202totheexchequer,whichis£2,013lessthanLevel3(whichcosts£7,215unitcost).Thebenefitsareseentogreatlyoutweighthesecostsinbothcases,withareturntotheexchequerofbetween£31,000and£48,000atLevel2,andbetween£56,000and£81,000forLevel3apprenticeships.AttheindividuallevelthelifetimebenefitsassociatedwiththeacquisitionofApprenticeshipsatLevel2and3areverysignificant,standingatbetween£48,000and£74,000forLevel2andbetween£77,000and£117,000forLevel3Apprenticeships.Itshouldbenotedthatthisfindingisonlyforonestudyandshouldbetreated,thereforewiththerelevantcaution.

Different skill levels

On the (limited UK) evidence available, Level 3 and higher apprenticeships deliver substantially higher lifetime wage gains than lower level apprenticeships.

Itisimportanttonotethatfindingsondurationneedtobeinterpretedwithcautionbecauselongerprogrammesarelikelytoleadtohigherskilllevelsanddifferentformalqualifications.Sodurationmaytellussomethingaboutultimatequalifications,ratherthansimplytheeffectofprogrammelength.Unfortunately,veryfewstudiesinvestigateskillslevelsofindividualapprenticeshipschemesandcomparedifferentqualificationsobtained.However,oneoftheUKevaluations(study985)doesinvestigatethedifferencesbetweendifferentNVQlevelqualifications.

Article985,whichlooksattheimpactofallBritishApprenticeshipsuponearnings,findsthatLevel3apprenticesandLevel2apprenticesseeearningreturnscomparedtosimilarindividualswhodon'tgainanapprenticeshipof22%and12%respectively.Italsofindsthattheeffectsonemploymentcandifferdependingonthetypeofapprenticeship.Wereportedontherelativecost-benefitsabove.

Article900alsolooksatskillslevels,investigatingtheYouthTrainingSchemewhichprovidesarangeoftrainingtounemployed16-17yearolds.ItlooksattheimpactofundertakingtheYTSalongsideanapprenticeships,versusundertakingYTSwithoutanapprenticeship.Theresultssuggestthatthereisnosignificanteffectofanyoftheseeducationprogrammesonunemployment.

Twootherpapersspecifythataqualificationisobtainedasaresultoftheirapprenticeshipscheme:howevertheydonotlookattheimpactofachievingdifferentlevelsofqualificationuponeconomicoutcomes.35

Sectoral targeting and outcomes

There is too little evaluation evidence to draw clear conclusions on whether apprenticeships work better in some sectors than others.

Veryfewpapersprovidedetailsontheextenttowhichschemestargetparticularsectors.Wehaveoneevaluation(study994)thatlooksatemploymenteffectsforaschemeaimedattheconstructionindustry(itfindspositiveeffects;alsoforcourseattendance).However,onlyoneevaluation,forHungary,looksatthereturntoapprenticeshipsindifferentsectors.Itfindsthattheprobabilityofbeingemployeddiffersalotbetweenindustries,butthatthepositiveeffectofworkplace-basedtrainingcomparedtoschool-basedvocationaltrainingisstableacrossindustries.36

35 Paper838forFrance;paper937forAustralia.36 Paper904alsoconsidersdifferenceacrossmanufacturingandserviceindustryforpost-apprenticeshipmoves–seebelow.


Apprenticeships versus other forms of training

There is some evidence that apprenticeships are more likely to increase employment than other forms of employment training (unless that training also involves an in-firm element). The evidence on wages is more mixed and appears to vary by gender.

Ouremploymenttrainingreviewfound71evaluationsthatlookatthereturnstoemploymenttraining.Wehaveaconsiderablysmallerevidencebaseforapprenticeships(19evaluations).Forapprenticeships,ofthe9evaluationsthatlookateitheremploymentorunemployment,7havepositiveeffects.Takenatfacevaluethisis77%ofevaluationsandcomparesfavourablytothe60%ofemploymenttrainingstudiesthatfoundpositiveeffectsonemployment(40outof67).Interestingly,however,whenwespecificallyfocusedonemploymenttrainingthatinvolvesawithinfirmelementwefoundonly4studies(outof17)reportingnoornegativeeffects,while13reportedpositivefindings.37Asintheemploymenttrainingreview,wethinkthatthemainmessagetoemergehereconcernstheimportanceofinvolvingfirmsinthetrainingprocess.In-firm/onthejobtrainingprogrammesoutperformclassroom-basedtrainingprogrammes.Employerco-designandactivitiesthatcloselymirroractualjobsappeartobekeydesignelements.

Threepapersmakespecificcomparisonsbetweenapprenticeshipsandotherformsofemploymenttraining.Study900comparesemploymenteffectsfromUKapprenticeships,theYouthTrainingScheme(YTS),apprenticeshipsandYTStogether.Thestudy,however,findsnosignificanteffectonunemploymentforanyofthesevariants..

Study964looksattheUS‘SchooltoWork’Programme,whichwasaimedatdisadvantagedyoungpeopleandincludedanumberofstrandsincludingjobshadowing,mentoring,combinedacademicandvocationaleducation;workplacements,singlecareer-focusedstudyandapprenticeships.Thestudyfindsthatformaleparticipants,anumberofstrandsareeffective,includingapprenticeships,whichraiseemploymentanddecreaseinactivityafterleavingsecondaryschool.Forwomen,apprenticeshipsarethemosteffectivestrandintermsofraisingwages.

Study986comparesearningsandlabourmarketparticipationforthosetakingGermanapprenticeshipsversusotherformsoftraininginthecountry.Apprenticeshipparticipationleadstosubstantiallylowerunemploymentratescomparedtootherformsofvocationaltraining,althoughthiseffectdiminishesovertime.Interestingly,thestudyfindsnodifferenceinwagesbetweenparticipantsinapprenticeshipsversusthosewhodidothertypesofvocationaltraining.Thisisconsistentwiththebiggerpictureonapprenticeshipsandwages,discussedabove,whichfindsmixedeffectsonpost-programmeparticipantearnings.

Post Apprenticeship decisions

There is some evidence that post-apprenticeship moves can increase wages although effects depend on circumstances (e.g. whether the individual chooses to move, stays in the same occupation, or moves within the manufacturing sector)

37 Forsakeofcomparisonwehavetreatedemploymenttrainingstudy234(classifiedasmixed)aspositive:itfindspositiveeffectsontransitiontoemploymentposttraining,negativeeffectsduringtraining.Similarlyforemploymenttrainingstudy236(classifiedasmixed):itfindsnoeffectontransitiontoemploymentduringtraining,butpositiveeffectsfordurationoffutureemploymentspells.Finallywetreatedemploymenttrainingstudy243aszeroornegative(classifiedasmixed)becauseeffectsonemploymentarenegativeforfouryearsposttrainingbeforereducingtozero.


Thisgroupofevaluationsinvestigatestheimpactsofchoicesoreventstakenpost-apprenticeshipuponvariouseconomicindicators.Forexamplethewageeffectofanindividualstayingwiththetrainingfirmpost-apprenticeshipvs.movingtoanewfirm.Wefoundfourstudiesofthiskind–allforGermany–thatmetourminimumstandards,anddiscusseachbrieflybelow.Ineachcase,thestudiesfindwaystotacklethefactthatmoving/stayingmightbeinfluencedbyunobservablecharacteristics(suchasambition)orcontextualfactors(suchasbargainingpowerorotherconditionswithinafirm).

Study876looksatthefactorsinfluencingwhetherapprenticestaketheirfirstjobinthefirmthattrainedthem,orinsteadmovetoadifferentfirm.Higherwagesduringtheapprenticeship,highertrainingintensityandlongerapprenticeshipprogrammesincreasethelikelihoodofstayingwithafirmpost-programme.Stayers’jobstendtolastlongerthanthoseofmovers.

Threestudies,allGerman,lookatthewageeffectsofmoving/staying.Study950providesthemostdetailondifferenttypesofmoves.Overall,thewagepenaltyfrommovingjobisclosetozero,butthisvariesdependingonwhythemovehappens:forthosewhodecidetoquitthereisapositivesignificantwageeffect,butforthosewholosetheirjobsthereisnosignificantimpacteitherway.Study867picksuponthelatterissueandlooksatwageoutcomesforapprenticesforcedtomovefirmbecauseofclosureorlayoffs,versusstayers.Itfindsthatparticipantswhostaywithinthesameprofessionintheirnewjobhavehigherwages,whilethosewhoshifttradesendupwithlowerwagesthanbefore.Finally,study904,alsoforGermany,findssomesectoraldifferencesinwageoutcomes;apprenticesinmanufacturingwhomovejobexperiencewagegains,whilethoseworkinginconstruction,crafts,commerceortradingoccupationsexperiencewagelosseswhenmovingjobs.

Take-up of Apprenticeships

There is some evidence that identifies mechanisms that may increase entry into apprenticeships and attendance during the programme (e.g. pre-qualifications, higher wages and subsidies to individuals). However, we have less evidence on what works to ensure people complete apprenticeships.

Thisgroupofpapers,ratherthanlookingattheimpactsofapprenticeships,investigatestheimpactofvariouspoliciesuponapprenticeshipentryandcompletion.

Twostudieslookattoolsthataimtoraiseentryintoapprenticeships.Study937,onAustralia,findsthatCertificateI/IIpre-qualificationsintroducedunderthecountry’snationalqualificationsincreasedthelikelihoodofparticipatinginanapprenticeshiportraineeship,bothformenandwomen.Study942,onaDanishpre-apprenticeshipprogramme,foundapositiveandsignificantimpactoftheprogrammeonparticipantsenteringanapprenticeshipafterward.

Twostudieslookatfactorsaffectingattendanceduringanapprenticeshipprogramme.Study876looksatthelevelofwagespaidtoGermanapprentices,aswellasthelengthoftheapprenticeship;higherwagesandlongerprogrammesbothhavepositive,significanteffectsonapprenticeretention.Study990evaluatedtheDanishApprenticeshipSubsidyscheme(AAS),findingasignificantpositiveeffectonthevocationalattendancerateamonglow-schooled25-year-oldmeninitsfirstfullyearofoperationbutnosignificanteffectthereafter.

Threestudiesexploredrop-outfromprogrammes.Study859compares‘regular’apprenticeshipsintheUSwith‘joint’apprenticeshipswhichareco-sponsoredbythefirmandaunion.Itfindsmixed


results:thoseonjointprogrammestakelongertocompleteanapprenticeship(intermsofaveragemonthstaken)buttheyarealsolesslikelytodropoutandhavehighercompletionrates.

Study979,forGermany,evaluatesanapprenticeshipbonusforemployers,whichrewardsfirmsfortakingonparticipantsandwhentheycompletethecourse–itfindsnoeffectofthisonparticipationorcompletionrates.Thisstandsincontrasttostudy876,whichsuggestspaymentstoindividuals(ratherthanfirms)canbeeffective.

Finallystudy838,forFrance,comparestheratesofschooldropoutforapprenticesversusregularstudents,findingnostatisticallysignificantdifferencebetweenthetwo.


Summary of findings

Thisreviewconsiderstheimpactsofapprenticeshipsonworkersandfirms.Thissectionsummarisesthedetailedfindings.Weemphasisethatmanyofthesefindingsdependonasmallnumberofstudies.Theyare,however,consistentwithotherresearchonapprenticeshipsandonemploymenttrainingmorebroadly(seeourfirstevidencereview).

What the evidence shows• Thereissomeevidencethatapprenticeshipsimproveskilllevels,andstimulatefurther

training/study.

• Apprenticeshipscanincreasewages,althoughinacoupleofevalutionseffectsarenegative.Impactsalsovarybytypeofparticipant.

• Apprenticeshipstendtohaveapositiveeffectonparticipants’subsequentemployment(andalsoreduceunemploymentpost-programme).

• Higherlevelapprenticeships(Level3andabove)deliversubstantiallyhigherlifetimewagegainsrelativetolowerlevelapprenticeships(basedonthelimitedUKevidenceavailable).

• Thereissomeevidencethatapprenticeshipsaremorelikelytoincreaseemploymentthanotherformsofemploymenttraining(unlessthattrainingalsoinvolvesanin-firmelement).Theevidenceofimpactonwagesismoremixedandappearstovarybygender.

• Thereissomeevidencethatidentifiesmechanismsthatmayincreaseentryintoapprenticeshipsandattendanceduringtheprogramme(e.g.pre-qualifications,higherwagesandsubsidiestoindividuals).However,wehavelessevidenceonwhatworkstoensurepeoplecompleteapprenticeships.

Where the evidence is unclear • Itisunclearwhetherthedurationoftheapprenticeshipmattersforeffectsonwagesor

employment(althoughlongerapprenticeshipsthatdeliverhigherqualificationsmayhavemorepositiveeffects)

08



Where there is a lack of evidence • Thereissomelimitedevidencethatfirmsparticipatinginapprenticeshipsexperience

economicgains,suchashigherproductivityorprofits.

• Thereistoolittleevaluationevidencetodrawclearconclusionsonwhetherapprenticeshipsworkbetterinsomesectorsthanothers.

• Thereissomeevidencethatpost-apprenticeshipmovescanincreasewagesalthougheffectsdependoncircumstances(e.g.whethertheindividualchoosestomove,staysinthesameoccupation,ormoveswithinthemanufacturingsector).However,theseresultsareeachbasedonasinglestudy.

• Thereisnoimpactevaluationevidencelookingattheeffectofapprenticeshipsonagivenlocalarea(ratherthanindividualparticipantsorfirms).

• Thereisnoimpactevaluationevidencecomparingtheeffectsofnationallyrunprogrammesversuslocallyrunprogrammes.

• Existingexantemodellingsuggeststhattheeconomicbenefitsofapprenticeshipscomfortablyoutweightheircosts.However,onlyoneoftheimpactevaluationsprovidescostdatainaformwhichallowsustocalculateex-postbenefit-costratiosforthatprogramme.

• Noneoftheshortlistedstudieslookattheeffectsofsubstantiallyscalingupapprenticeshipprovision,asiscurrentlyhappeningintheUK.Weneedmoreevidenceonwhetheridentifiedbenefitsalsoholdinalargerprogramme.GiventheothersubstantialchangestotheUKapprenticeshipsysteminthepastdecadeandahalf,moreuptodateUKimpactevaluationevidenceisalsoneeded.

How to use these reviews• Apprenticeshipsarecurrentlyveryhighonthepolicyagenda,andtheevidencereview

highlightsanumberoffactorsforpolicymakerstobeawareofwhenconsideringapprenticeships:

• Whiletheevidencesuggeststhathigherlevelapprenticeships(level3andabove)mayofferbetteroutcomes,itdoesnotcurrentlytelluswhetherthisisbecausestrongercandidatesgravitatetowardsmoredemandingprogrammes.Ifthisisthecase,policymakersneedtoconsiderhowtoaddresstheneedsofthose‘leftbehind’bythistypeofapprenticeshipoffering.

• Anypolicyshouldcarefullyconsiderhowtorecruitfirmstoprovideapprenticeships,andtraineestofillthem.Abetterunderstandingofthecostsandbenefitstofirmswillhelpinthis(seebelow).Aswillabetterunderstandingofwhichpolicydesignaspectsincreasetake-upandreducedrop-out.

Helping to fill the evidence gapsAsshouldbeclearfromthisreview,therearemanythingsthatwedonotknowaboutthelocaleconomicimpactofapprenticeships.

If achieving local economic impact is an important part of the case for apprenticeship provision, then there need to be more evaluations that explicitly explore these impacts and how to maximize them. Centralandlocalpolicymakers–andprivatesectorpartners–should:


• Looktoundertakesystematiccomparisonsthatcoverdifferentkindsofapprenticeshipmodel-forexample,theGermansystemversusamoredecentralisedsystem.

• Conductfurtherresearchlookingatoutcomesforfirms.Surveysoffirmswhoofferapprenticeshipssuggestthosefirmsseeclearbenefits,buttheymaynotberepresentativeofallemployers.

• Setupevaluationsofschemedesignanditseffectontake-up,completionandoutcomes.ThisisparticularlyimportantgivendevolutionofskillsbudgetstocitiessuchasLondonandManchester.Centralandlocalpolicymakersshouldworktogethertodesignrobustevaluationthatincreasesourunderstandingofhowtoimprovethedesignofapprenticeships.

• Makeschemecostdataavailabletoresearcherssothatrobustbenefit-costratioscanbecalculated.

TheCentre’slongertermobjectivesaretoensurethatrobustevidenceisembeddedinthedevelopmentofpolicy,thatthesepolicesareeffectivelyevaluatedandthatfeedbackisusedtoimprovethem.Toachievetheseobjectiveswewantto:

• Workwithlocaldecisionmakerstoimproveevaluationstandardssothatwecanlearnmoreaboutwhatpolicieswork,where.

• Setupaseriesof‘demonstrationprojects’toshowhoweffectiveevaluationcanworkinpractice.

Interestedpolicymakerspleasegetintouch.


References

Adda,J,Dustmann,C,Meghir,CandRobin,JM(2010).CareerProgressionandFormalversusOn-the-JobTraining.IFSWorkingPaperW10/13,London,IFS.Alet,EandBonnal,L(2011).VocationalSchoolingandEducationalSuccess:ComparingApprenticeshiptoFull-TimeVocationalHighSchool,ToulouseSchoolofEconomics,WorkingPaper.Bilginsoy,C(2007).DeliveringSkills:ApprenticeshipProgramSponsorshipandTransitionfromTraining.IndustrialRelations,vol46(4),p738-765BIS(2015).MeasuringtheNetPresentValueofFurtherEducationinEngland.BISResearchPaper228.London,DepartmentofBusiness,InnovationandSkills.BIS(2013).ApprenticeshipsEvaluation:Employer.BISResearchPaper123.London,DepartmentofBusiness,InnovationandSkills.BIS(2011).Intermediateandlowlevelvocationalqualifications:economicreturns,BISResearchpaper53,London,DepartmentofBusiness,InnovationandSkills.Booth,ALandSatchell,SE(1994).ApprenticeshipsandJobTenure.OcfordEconomicPapers,vol.46(4)p676-695.Buchel,F;Pollman-Schult,M(2002).OvercomingaPeriodofOvereducatedWork--DoestheQualityofApprenticeshipMatter?AppliedEconomicsQuarterly—Konjunkturpolitik(SpecialIssue),Vol48(3-4),p304-316.Capellari,L,Dell'Aringa,CandLeonardi,M(2012).TemporaryEmployment,JobFlowsandProductivity:ATaleofTwoReforms,TheeconomicJournal,vol122(562),pF-188-F215.Card,David."TheCausalEffectofEducationonEarnings,"inOrleyAshenfelterandDavidCard,eds.,HandbookofLaborEconomics,Vol.3A,pp.1801-63.Amsterdam:ElsevierScience,1999.CEBR(2013).ProductivityMatters:Theimpactofapprenticeshipsontheeconomy,London,CEBR.Clarke,D(2000).HowTransferableisGermanApprenticeshipTraining?IZASeminarPaper,http://bit.ly/1E77kkv.Clarke,DandFahr,R(2002).ThePromiseofWorkplaceTrainingforNon-CollegeBoundYouth:TheoryandEvidencefromGermanApprenticeship.CentreforEconomicPerformanceDiscussionPaper518,London:CEP.Cooke,LP(2003).AComparisonofInitialandEarlyLifeCourseEarningsoftheGermanEducationandtrainingSystem.EconomicsofEducationReview,vol.22,p79-83.Dolton,PJ,Makepeace,GHandGannon,BM(2001).TheEarningsandEmploymentEffectsofYoungPeople'sVocationalTraininginBritain,TheManchesterSchool,vol69(4),p387-417.Euwals,RandWinkelmann,R(2004).TrainingIntensityandFirstLaborMarketOutcomesofApprenticeshipGraduates,InternationalJournalofManpower,vol25(5),p447-462.Fersterer,J,Pischke,J.S.andWinter-Ebmer,R.(2004).ReturnstoApprenticeshipTraininginAustria:EvidencefromFailedFirms,TheScandinavianJournalofEconomics,vol.110(4),p733-753.Franz,W,Inkmann,J,Polmeier,WandZimmermann,V(2000).YoungandOutinGermany(OnYouths?ChancesofLaborMarketEntranceinGermany),UniversityofKonstanz,CenterforInternationalLaborEconomics(CILE)DiscussionPapers40.

http://bit.ly/1E77kkv



Fries,J,Gobel,CandMaier,MF(2014).Doemploymentsubsidiesreduceearlyapprenticeshipdropout?,JournalofVocationalEducationandTraining,Vol66(4),p433-461Gibbons,S,Nathan,MandOverman,H(2014).EvaluatingSpatialPolicies.SERCPolicyPaperSERCPP0012,London,LSE.Goeggel,KandZwick,K(2012).HeterogeneousWageEffectsofApprenticeshipTraining,TheScandinavianJournalofEconomics,Vol114(3),p759-779HMTreasuryandGMCA(2014).GreaterManchesterAgreement:DevolutiontotheGMCAandtransitiontoadirectlyelectedMayor.http://bit.ly/1tT3Rim.Holm,A(2002).Theeffectoftrainingonsearchdurations:arandomeffectsapproach,LabourEconomics,Vol9,p433-450.Horn,D(2013).School-basedVocationalorWorkplace-basedApprenticeshipTraining?EvidenceontheSchool-to-WorkTransitionofHungarianApprentices,MWPworkingpaperMWP2013/10,SanDomenicodiFiesole,EIU.Lerman,R(2014).Dofirmsbenefitfromapprenticeshipinvestments?.IZAWorldofLabor2014:55.McIntosh,SandGarrett,R(2009).TheEconomicValueofIntermediateVocationalEducationandQualifications.EvidenceReport11,London,UKCES.Mohrenweiser,JandPfeiffer,F(2009).Coachingdisadvantagedyoungpeople:Evidencefromfirmleveldata,ZEWDiscussionPaper14-054,ZEW.Neumark,DandRothstein,D(2005).DoSchool-to-WorkProgramsHelpthe"ForgottenHalf"?IZADiscussionPaper1740,Bonn:IZA.Oliver,D(2012).LowerLevelQualificationsasaSteppingStoneforYoungPeople,NationalCentreforVocationalEducationResearchOccasionalPaper,Adelaide:NCVER.Parey,M(2009).VocationalSchoolingversusApprenticeshipTraining:EvidencefromVacancyData,monograph.Picchio,MandStaffolani,S(2013).DoesApprenticeshipImproveJobOpportunities?ARegressionDiscontinuityApproach,IZADiscussionPaper7719,Bonn:IZA.Richard,D(2012).TheRichardReviewofApprenticeships.London,SchoolforStartups.Schaeffer,CM,Henggeler,SW,Ford,JD,Mann,M,Chang,RabdChapman,JE(2014).RCTofapromisingvocational/employmentprogramforhigh-riskjuvenileoffenders,JournalofSubstanceAbuseTreatment,Vol46,p114-148Sherman,LW,Gottfredson,DC,MacKenzie,DL,Eck,J,Reuter,P,&Bushway,SD(1998).PreventingCrime:WhatWorks,WhatDoesn’t,What’sPromising.WashingtonDC:USDepartmentofJustice.Storen,LI(2011).Keyfactorsbehindlabourmarketmarginalizationofyoungimmigrants:limitedaccesstoapprenticeships,‘statedependence’orlowqualifications?Young,vol19(2),p129-158.Todd,J(2014).UptotheJob.London,Demos.Veum,JR(1995).SourcesofTrainingandTheirImpactonWages,IndustrialandLaborRelationsReview,Vol.48(4),p812-826.Weatherall,CD(2009).DoSubsidizedAdultApprenticeshipsIncreasetheVocationalAttendanceRate?AppliedEconomicsQuarterly,Vol.55(1),p60-81

http://bit.ly/1tT3Rim


Appendix A: Findings by outcome

Table A1: Findings by outcome ‘returns to apprenticeship

Outcome Evaluated Number

Evaluation reference numbers Positive

No effect Mixed Negative

Individual LevelHumanCapital Total 4 838,964,994 838,994 964

Courseattendance

1 994 994

Coursecompletion

1 838 838

Furthereducation

1 838 838

FurtherTraining 1 964 964

Labourmarketoutcomes

Total 11 862,880,900,939,960,964,985,986,994,997,1045

862,880,960,985,939,986,994,1045

900,964,964

997

Movingintoemployment

5 900,960,964,985,994

960,985,994

900,964

Avoidingunemployment

4 862,880,986,1045

862,880,986,1045

Movingintopermanentwork

1 939 939

Movingtoaqualifiedjob

1 997 997

Wages Total 7 837,847,878,976,986

837, 976,986 847,878,

Firm LevelFirmLeveloutcomes

3 897,897,987 897,897 987

Employment(firm)

2 897,987 897 987

Productivity 1 897 897

09


Appendix B: Evidence Reviewed

Ref no. Reference837 Adda,J;Dustmann,C;Meghir,CandRobin,JM(2010).CareerProgressionandFormal

versusOn-the-JobTraining.IFSWorkingPaperW10/13London:IFS.

838 Alet,EandBonnal,L(2011).VocationalSchoolingandEducationalSuccess:ComparingApprenticeshiptoFull-TimeVocationalHighSchool,ToulouseSchoolofEconomics,WorkingPaper.

847 Cooke,LP(2003).AComparisonofInitialandEarlyLifeCourseEarningsoftheGermanEducationandtrainingSystem.EconomicsofEducationReview,vol.22,p79-83

859 Bilginsoy,C(2007).DeliveringSkills:ApprenticeshipProgramSponsorshipandTransitionfromTraining.IndustrialRelations,vol46(4),p738-765

862 Booth,ALandSatchell,SE(1994).ApprenticeshipsandJobTenure.OcfordEconomicPapers,vol.46(4)p676-695.

867 Clark,D(2000).HowTransferableisGermanApprenticeshipTraining?IZASeminarPaper,http://bit.ly/1E77kkv.

876 Euwals,RandWinkelmann,R(2004).TrainingIntensityandFirstLaborMarketOutcomesofApprenticeshipGraduates,InternationalJournalofManpower,vol25(5),p447-462.

878 Fersterer,J;Pischke,JSandWinter-Ebmer,R(2004).ReturnstoApprenticeshipTraininginAustria:EvidencefromFailedFirms,TheScandinavianJournalofEconomics,vol.110(4),p733-753

880 Franz,W;Inkmann,J;Polmeier,WandZimmermann,V(2000).YoungandOutinGermany(OnYouths?ChancesofLaborMarketEntranceinGermany),UniversityofKonstanz,CenterforInternationalLaborEconomics(CILE)DiscussionPapers40.

897 Capellari,L;Dell'Aringa,CandLeonardi,M(2012).TemporaryEmployment,JobFlowsandProductivity:ATaleofTwoReforms,TheeconomicJournal,vol122(562),pF-188-F215.

900 Dolton,PJ;Makepeace,GHandGannon,BM(2001).TheEarningsandEmploymentEffectsofYoungPeople'sVocationalTraininginBritain,TheManchesterSchool,vol69(4),p387-417

904 Goeggel,KandZwick,K(2012).HeterogeneousWageEffectsofApprenticeshipTraining,TheScandinavianJournalofEconomics,Vol114(3),p759-779

937 Oliver,D(2012).LowerLevelQualificationsasaSteppingStoneforYoungPeople,NationalCentreforVocationalEducationResearchOccasionalPaper,Adelaide:NCVER.

939 Picchio,MandStaffolani,S(2013).DoesApprenticeshipImproveJobOpportunities?ARegressionDiscontinuityApproach,IZADiscussionPaper7719,Bonn:IZA.

942 Holm,A(2002).Theeffectoftrainingonsearchdurations:arandomeffectsapproach,LabourEconomics,Vol9,p433-450.

950 Clarke,D;Fahr,R(2002).ThePromiseofWorkplaceTrainingforNon-CollegeBoundYouth:TheoryandEvidencefromGermanApprenticeship.CentreforEconomicPerformanceDiscussionPaper518,London:CEP.

960 Horn,D(2013).School-basedVocationalorWorkplace-basedApprenticeshipTraining?EvidenceontheSchool-to-WorkTransitionofHungarianApprentices,MWPworkingpaperMWP2013/10,SanDomenicodiFiesole,EIU.

964 Neumark,D;Rothstein,D(2005).DoSchool-to-WorkProgramsHelpthe"ForgottenHalf"?IZADiscussionPaper1740,Bonn:IZA.

976 Veum,JR(1995).SourcesofTrainingandTheirImpactonWages,IndustrialandLaborRelationsReview,Vol.48(4),p812-826.

979 Fries,J;Gobel,C;Maier,MF(2014).Doemploymentsubsidiesreduceearlyapprenticeshipdropout?,JournalofVocationalEducationandTraining,Vol66(4),p433-461



985 BIS(2011).Intermediateandlowlevelvocationalqualifications:economicreturns,BISResearchpaperNumber53,pp169

986 Parey,M(2009).VocationalSchoolingversusApprenticeshipTraining:EvidencefromVacancyData,monograph.

987 Mohrenweiser,J;Pfeiffer,F(2009).Coachingdisadvantagedyoungpeople:Evidencefromfirmleveldata,ZEWdiscussionpaperNumber14-054,pp27

990 Weatherall,CD(2009).DoSubsidizedAdultApprenticeshipsIncreasetheVocationalAttendanceRate?AppliedEconomicsQuarterly,Vol.55(1),p60-81

994 Schaeffer,CM;Henggeler,SW;Ford,JD;Mann,M;Chang,R;Chapman,JE(2014).RCTofapromisingvocational/employmentprogramforhigh-riskjuvenileoffenders,JournalofSunstanceAbuseTreatment,Vol46,p114-148

997 Buchel,F;Pollman-Schult,M(2002).OvercomingaPeriodofOvereducatedWork--DoestheQualityofApprenticeshipMatter?AppliedEconomicsQuarterly—Konjunkturpolitik(SpecialIssue),Vol48(3-4),p304-316

1045 Storen,LI(2011).Keyfactorsbehindlabourmarketmarginalizationofyoungimmigrants:limitedaccesstoapprenticeships,‘statedependence’orlowqualifications?Young,vol19(2),p129-158

Findthefulllistofsearchtermsweusedtosearchforevaluationsonourwebsitehere:whatworksgrowth.org/policies/apprenticeships/search-terms.

The What Works Centre for Local Economic Growth is a collaboration between the London School of Economics and Political Science (LSE), Centre for Cities and Arup.

www.whatworksgrowth.org


ThisworkispublishedbytheWhatWorksCentreforLocalEconomicGrowth,whichisfundedbyagrantfromtheEconomicandSocialResearchCouncil,theDepartmentforBusiness,InnovationandSkillsandtheDepartmentofCommunitiesandLocalGovernment.ThesupportoftheFundersisacknowledged.TheviewsexpressedarethoseoftheCentreanddonotrepresenttheviewsoftheFunders.

Everyefforthasbeenmadetoensuretheaccuracyofthereport,butnolegalresponsibilityisacceptedforanyerrorsomissionsormisleadingstatements.

Thereportincludesreferencetoresearchandpublicationsofthirdparties;thewhatworkscentreisnotresponsiblefor,andcannotguaranteetheaccuracyof,thosethirdpartymaterialsoranyrelatedmaterial.

September2015

WhatWorksCentreforLocalEconomicGrowth

[email protected]@whatworksgrowth


©WhatWorksCentreforLocalEconomicGrowth2015

mailto:info%40whatworksgrowth.org%0D?subject=

https://twitter.com/whatworksgrowth

http://whatworksgrowth.org/

Documents

Evidence Review 8 Apprenticeships - What Works Growth...Evidence Review: Apprenticeships - September 2015 8 Introduction This review looks at the impacts of apprenticeship programmes