Embed Size (px)
Citation preview
(UNJIUReportDecember2017AnnexII)-AdaptedfromJohnMayne,“Resultsmanagement:canresultsevidencegainafootholdinthepublicsector?”inTheEvidenceBook:Concepts,GenerationandUseofEvidence,OlafRieper,FransLeeuwandTomLing,eds.(NewBrunswick,TransactionPublishers,2010))Planning(results-based)
• Planning(results-based)• Delivery• Monitoring,analysis+evaluation• Learning+action• Improvingdesign• Improvingmeasurementstrategy
Whatdoesmanagementconsiderinoutcome-focusedresults-based
management?
KeyProblemswithRBM
• RBMmadetoappearstraightforward/simple• Useforaccountabilitydrownsoutlearning• Notunderstoodwithinorganizations• NoincentivestouseRBM• Basedonindicators• ITsystemscan’thandleresultsinformation• BeliefthatRBMdoesn’trequireadditionalresources
BUT
• RBMisnotstraightforward,notsimple• RBMrequiresthinking,understanding,experienceandprofessionalexpertise
• RBMrequiresongoingsupport,incentives,resources:aresult-orientedculture
• RBMforlearningisasignificantchallenge• IndicatorsareNOTenough:
Youcannotlearnandmanagewithindicators
Disappointingly
Wearen’tlearningwellaboutwhatisneededtogetRBMto
workeffectively
Butmaybethereishope
PrinciplesforRBMImportant• ThinkingaboutanddoingRBMrequiresPrinciplesforguidance• NeedtounderstandwhatRBMisallabout,whatispossibleandhowresultsinformationcanbeused
• UNandOECDhavebeendevelopingRBMprinciplesformanyyears.• JIU2017reportsthemostrecentversionandthemostinsightful
KeyNewFeaturesofJIUPrinciplesRBMPrinciplesrecognizetheneedto:• Focusonoutcomes• Appreciatethecomplexityofinterventions• DevelopanduseToCs• Understandhowinterventionsareinfluencingresults,howtheymakeacontribution
• Linkmonitoringandevaluation• Thinkbeforesettingoutmeasurementapproaches
NickKowbel,currentevaluatorintheUnitedNationsSecretariat,andformerevaluatorintheGovernmentofCanada“viewspresentedaremyownanddonotnecessarilyreflecttheofficialviews
oftheOfficeofInternalOversightServices”
Who conducts Evaluation in the UN?
Source:OIOS-IEDManual2014edition
Donors
OIOS evaluations find challenges with fully implementing RBM
Implications for JIU
principles of RBM
Source: OIOS-IED internal assessment of 2018-2019 reports
• Need to focus more
on theory of change
(validity of results chain);
• use of strategies
that acknowledge
niche roles; • resolve challenges
with M&E and use of
evidence in decisions
ManagingForResults:UNIDO’sIntegratedResultsand
PerformanceFrameworkPPXSymposiumOttawa,Canada
15May2019
GiuseppeDeSimoneOfficeofStrategicPlanning,Coordination
andQualityMonitoring
11
Content
• Theimperativeofmanagingforresults• TheIRPFandtheUNIDOstrategicplanningframework• OurrenewedtakeonRBM:anactor-based,behavioralchangemodel
• TellingtheUNIDOperformancestory• Benefitsandchallengesgoingintoimplementation
12
Theimperativeofmanagingforresults
• Inthe2030Agendaera,OrganizationsfordevelopmentmustbeabletoshowtheircontributiontoSDGs
• SDGs:country-ownedtransformational,systemicchange!• TheUNDSreformgoesinthesamedirection:Organizationstodemonstratecontributionàbutattributionisachallenge!
• Aresultsframeworkismuchmorethanareportingtool:itmeasuresprogresstowardsmanagementforchange,atscale
• UNIDO’sintegratedresultsandperformanceframework(IRPF):directemanationofUNIDO’sstrategicplan
13
Thestrategicplanningframework:integrationandscaleup• Betterintegrationacrosscorefunctionstoenablescalingupofresults
• Strengtheningknowledgeandinstitutionsaskeyenabler
• Strengthenresultsorientation• BettershowcontributiontotransformationalchangeandSDGs
• BetterequipUNIDOforUNDSreformandSDGs
14
MakingRBMworkformorethanindicatorsWefollowthesequenceofRBMprinciples(JIU2017):1. Visionandclarityofdesiredendproductorimpact2. Causallinkageinahierarchyofresults(inputs,activities,outputs,
outcomes,impact)basedontheoryofhowchangehappens3. System-operationthatgoesbeyondlinearlogicandconsiderscontext,
espousesequifinality,andaddressesrisksandconditionsforsuccessforachievinghigher-levelresults
4. Performancemeasurementfortransparency,consensusbuilding,andhavingacommonperspectiveonresults,aswellasforaccountability
5. Performancemonitoringforsinglelooplearning,andevaluationfordouble-looplearning
15
OurrenewedtakeonRBM
• Anactor-based,behavioralchangemodeltocrediblycapturecomplexity
• Importanceofkeyactorsinthesystematthemicro,mesoandmacrolevel
• Focusonchangeofbehaviors,whichtriggerhigher-levelresults• Amixofindicatorsandnarrativestotelltheperformancestory• EmbedsaTheoryofChangeapproachwithrisksandexternalfactors
16
Telling the UNIDO performance story
UNIDOisthekeyOrganizationthatprovidesanintegratedapproachtosupportinclusiveandsustainableindustrialization,toaddressthecomplexityoftheentailedsystemictransformation
UNIDOaddressesthecomplexityofISIDbycloselyworkingwithallrelevantactors:individuals&firms;intermediaryinstitutions;policymakersandglobalplayers
UNIDOsupportstheseactorsinstrengtheningtheirtechnical,institutionalandpolicy knowledge&understandingofISIDissues,andassociatedskills&capacities
Withavailableresourcesandhighlyspecializedstaff,UNIDOsupportsby:TC;Policyanalysis&adv.;Norms&standards;Convening&partnerships
UNIDOthereforecancrediblyclaimitscontributiontoISIDandtheSDGsbysupportingcountriesin:
UNIDOsupportstheseactorsinchangingkeybehaviors:
Knowledge&Capacities
Reach&Reactions
Sex,ageandcountrydisaggregateddatawhereverpossible
17
Level3:Programmaticoffer&programmemanagement
effectiveness
Level4:Organizationalresourcesandefficiency
Level2:CountryandglobalresultwithUNIDOsupport
Level1:Globalindustrialdevelopmentcontext
UNIDO-relatedSDGindicators(UNIDOascustodianagency)
18
Howdoesitwork?Clearinghouse
anddataanalysis
SPQand
IRPFTaskForce
Data*sourceandqualityassurance
Researchandstatisticsdepartment
TCDirectorate(projectmanagers,alldepartments,fieldoffices)Policy,ResearchandStatisticsOfficeofindependentevaluation
TCDirectorate(projectmanagers,alldepartments,fieldoffices)CorporateservicesOfficeofDirectorGeneral(strategy,quality)
Corporateservices(procurement,finance,HR,etc.Externalrelations/CommunicationInternalOversight/evaluation
Level1
Level2
Level3
Level4
*Note:KPIdatafromnewPPMorin-housesurveyswhenRBbased
19
Benefits1. BettertellUNIDO’sperformancestory,whereittrulyis
• Ownournarrative,changetheconversation(from$$$toimpact)• SpelloutUNIDO’scontributiontotransformationalchange,ISID,theSDGs• Provideconvincingevidencefornonproject-based,lessearmarkedfunding
• MeetthechallengeofUNDSreform(crediblyshowcontributiontoSDGs)
2. Harmonizescorporateresultsindicatorswithprojectresultstoavoidduplicationandfacilitatealignment Noadditionalburden!
4. Aligntostrategy:• Bemorethanthesumoffragmentedprojects• LinkprogrammaticKPIstoorganizationalresults
6. Theprocessitself:• developedandagreedwiththewholehouse(top-downmeetsbottom-up):groundedonacomprehensiveprogrammemapping
20
Amanagementtool,withalreadyvisibleresults
• IRPFispartofthebiggerpolicyarchitecture• AtooltocomplementandreinforceothereffortstoreformUNIDO:
• QualityAssuranceFramework• Adaptivemanagement• Standardization• Reengineeredprojectmanagement• Alignmenttostrategicdocuments• Integrationandscaleup(MTPF)
Changeisalreadyhere:newprogrammesbeingdevelopedarealreadyusingtheIRPFframework:sustainablecities,migration,youth,MICs,GEEWstrategy,AGRservicemodules,PCPEthiopia…
21
Showingourcontributionstory
• WearedevelopingavisualforUNIDO’sISIDandSDGcontribution
• Scalableforaproject,aprogramme,acountryprogramme/PCP,UNIDOasawhole.
The width of the arrows corresponds to the data in the IRPF framework
ConsideraTC-heavyUNIDO• WhatifUNIDOgoesallupstream?UNIDO’sapproach:Integrationandscaleup
The impact contribution arrows will depend on theoretical linkages and empirical evidence from UNIDO interventions
22
Goingintoimplementation• Changes are being introduced in a gradual manner, after testing and
validation. • Consistent signals from policies, indicators and narratives to
facilitate the required shift in mindsets and processes • Challenges:
• Change management – (not again!!!) • Capacity – training needs • Financial investments – but not massive • Timing – UNDS reform, donor assessments etc. • Data availability: time lag in achieving impact
• The consistent leadership of senior and middle management will continue to be crucial during implementation
• Constant engagement with Member States is also be necessary
23
Conclusions1. Acomprehensivesystemtodrivechange,basedonanactor-
basedbehavioralchangemodel,embeddingaToCapproach–scalablefromprojectstocorporate
2. StrengthenstheabilityofUNIDOtorealizeandshowitsuniquecontributiontoISIDandSDGsanddrivefurtherintegrationandscaleupofresults
3. TheIRPF(indicatorsandnarratives)providesmeasuresofprogress–amanagementtoolforUNIDOandareportingtoolforexternalparties
25
AnnexesTesting
withprojects,programmesandcountryreports
26
IRPFtesting–itworks!• AtestingexercisewasconductedwithPTCdepartmentstesttherelevance,robustnessandusefulness.Examples:
• TII–SMARTFishproject
• ENV–Eco-industrialparkprogramme• ENE–EnMSprogramme,SustainableCities• AGR–Servicemodules
• RequestfromPCP–EthiopiatosupportPCPresultsframeworkalignedtoIRPF
• Conclusions:• GreatmatchofIRPFindicators–relevance,robustnessandusefulnessconfirmed• RequestsformoregranularitytocapturedetailsandmorevalueofUNIDOwork• Eagernesstomoveawayfromvolumesapproachandfindwaystobettershowresults• WealthofknowledgefromUNIDOstaff(incl.Field!)–sofarpartlyuntapped
27
ExampleoftheIRPFtestingatprojectlevel
28
Indicators tell the UNIDO performance story at PROJECT level (e.g. Smart Fish project)
UNIDOisthekeyOrganizationthatprovidesanintegratedapproachtoISIDservices,toaddressthecomplexityofthesystemictransformationitentails
• Farmersandfirmsalongthreevaluechains• Privatesectorrepresentatives• National,provincialanddistrictlevelgovernments• Multilateralandbilateralagencies
• 5401participantsin127capacitybuildingrelatedactivities
• 3valuechainroundtabledialogues• EstablishmentofTropicalSeaweedInnovationNetwork• Improvedmanagementpracticesalongthevaluechains(ISPIREDtool,standardoperating
procedures,branding);e.g.590farmersadoptedtheSOP• 4Women-ledhomebasedunitsestablishedregisteredtoproducesea-weedbasedfoods• Morethan3000farmsand13processingcompaniesjoinedthetraceabilityplatform• iPride4Fishcentreestablishedasasemi-autonomouscentrewithintheuniversityoffisheries• 18seaweedbasedfood/drinkproductscommerciallyproducedvaluedatUSD350,000peryear• USD6,426,000newinvestmentraised
• 20,460USDpermonthestimatedvalueofproductionfornewmicroenterprises(undercomponent2:seaweedqualityandproductivitywhichincludedMSMEdevelopment)
• 1.2millionUSDestimatedannualsalesofexistingSMEs• Productionofqualityfishincreasedfrom30to69-80%inTulungagun• Participatingcompanieshaveover80%ofmarketshare
• 127capacitybuildingactivities(trainings,workshops,seminars,roundtablesorganized,etc.)• TrainingonINSPIREDtoolandonstandardoperatingprocedures• Policyrecommendationsandpolicydialogues
3.1 Nr. of trainings/workshops provided
IRPFindicators
2.1 Nr. of firms reporting economic gains 2.5 Nr. of MSMEs reporting increased market inclusion
2.14 Nr of firms introducing improved management practices (incl. business models/business plans, products) 2.15 Nr. of firms reporting adoption of new technologies 2.20 Value ($) of new investments leveraged 2.21 Nr. of firms participating in enhanced collaboration settings
2.25 Nr. of actors gaining awareness/ knowledge/skills on entrepreneurial/industrial skills, management practices, standards,etc.
2.26 (a) Nr. of firms/entrepreneurs/individuals reached 2.26 (b) Nr. of intermediary institutions reached 2.26 (c ) Nr. of governments/policymakers reached 2.26 (d) Nr. of international actors reached
29
Indicators tell the UNIDO performance story at PROGRAM level (e.g. Eco-Industrial Parks)
UNIDOisthekeyOrganizationthatprovidesanintegratedapproachtoISIDservices,toaddressthecomplexityofthesystemictransformationitentails
• Industrialparksmanagementstaff• Relevantnationalinstitutionsandgovernmentalagencies• Residentfirms• SMEstafftrained
• onEIPpolicyandmanagement
• ImprovementintermsofEIPinternationalframework• Policiesreviewedandamended• Synergyopportunitiesimplemented• InvestmentsonRECP/EIP
• Target:Atleast25%oftotalfirmworkersinindustrialparkareemployedthroughdirectemploymentandpermanentcontracts.
• Target:Atleast25%ofresidentfirmsuselocalsuppliersorserviceprovidersforatleast80percentoftheirtotalprocurementvalue.
• Target:Atleast90%oftotalprocurementvalueofparkmanagemententityaresuppliedbylocalfirmsorserviceproviders
• Ratioofrenewableenergyused• Ratioofwaterreused/recycled• GHGemissionsandreductiontCO2eq./year
• Capacitybuildingworkshopsorganized(EIPpolicy,industrialsynergy)• Studytoursorganized• Policyreview• RECPassessment• Conductindustrialsynergyfeasibilitystudy• Outreachandawarenessraisingcampaigns
3.1 Nr. of trainings/workshops provided 3.7 Nr. of analytical publications released
2.26 (a) Nr. of firms/entrepreneurs/individuals reached 2.26 (b) Nr. of intermediary institutions reached
2.1 Nr. of firms reporting economic gains 2.5 Nr. of MSMEs reporting increased market inclusion 2.6 Number of people reporting improved access to manufactured goods 2.8 Indicator on reduced emissions 2.14 Nr of firms introducing improved management practices (incl. business models/business plans, products)
IRPFindicators
2.25 Nr. of actors gaining awareness/ knowledge/skills on entrepreneurial/ industrial skills, management practices, standards, innovation, gender, etc.
30
Indicators tell the UNIDO performance story at COUNTRY level (e.g. Egypt)
UNIDOisthekeyOrganizationthatprovidesanintegratedapproachtoISIDservices,toaddressthecomplexityofthesystemictransformationitentails
• Ministries,policymakersandnationalinstitutions• Youthandwomen• MSMEsandproductiveclusters• Publicandprivatebusinessdevelopmentserviceproviders• Training/Educationalinstitutions• Farmers• Europeanmarketsforexport• CivilsocietyandNGOs• DevelopmentpartnersandIOs
• 3,500youthwithskillstorealizeopportunitiesfordecentemploymentandentrepreneurship• 7ministries,80policymakers,and39nationalinstitutionswithstrengthenedcapacitiesonISID
• Over700women-ledstart-upsestablished• AdoptionofISO50001standardonenergymanagement• 20,000farmersimprovedproducequality,foodsafetypracticesandaddedvaluetocrops• EGP67millionmobilizedforanindustrialzoneinLuxor• 23linesofnewleatherandhabitatproductsdevelopedandexported
• $852,000additionalincomeforruralfarmers• Accesstocleanwaterfor150,000peopleinUpperEgypt• 10,000womencanaccessEGP2millionincommunitysavingsandloans• 1560OPDtonsreducedsince1993• 3.58MtCoeqlesscarbonemissionsemittedbyindustry
• Trainingofyouthandwomenentrepreneurs• Trainingoftrainersatthemesolevel• Publicationsproducedwithlessonslearned(MENApublication)• Holdeventsatnationalandregionalleveltoinformpolicydialogue
3.1 Nr. of trainings/workshops provided 3.7 Nr. of analytical publications released 3.12 Nr. of events held
2.26 (a) Nr. of firms/entrepreneurs/individuals reached 2.26 (b) Nr. of intermediary institutions reached 2.26 (c ) Nr. of governments/policymakers reached 2.26 (d) Nr. of international actors reached
2.25 Nr. of actors gaining awareness/knowledge/skills on entrepreneurial/industrial skills, management practices, standards, innovation, gender, etc.
2.14 Nr of firms introducing improved management practices (incl. business models/business plans, products) 2.15 Nr. of firms reporting adoption of new technologies 2.17 Nr. of new standards adopted 2.20 Value ($) of new investments leveraged 2.21 Nr. of firms participating in enhanced collaboration settings
2.1 Nr/Rate of firms reporting economic gains 2.3 Nr/Rate of firms reporting an increase in exports 2.5 Nr/Rate of MSMEs reporting increased market inclusion 2.8 Indicator on reduced emissions
IRPFindicators
