Draft Monitoring and Evaluation Framework · 2018. 7. 26. · Draft Monitoring and Evaluation Framework 1 Introduction This document outlines a Monitoring and Evaluation (M&E) Framework

Draft Monitoring and Evaluation Framework

Draft Monitoring and Evaluation Framework i

TABLE OF CONTENTSIntroduction 11.0 What does MAMPU aim to achieve and how? 12.0 How will MAMPU be monitored? 5 2.1 Partner Quarterly Progress Reporting 5 2.2 MAMPUSecretariatfieldmonitoringandverification 7 2.3 RoutinereflectionbyPartnersandtheMAMPUSecretariat 73.0 How will MAMPU be evaluated 8 3.1 KeyEvaluationQuestions(KEQs) 8 3.2 KeyEvaluationQuestion1:HowandtowhatextenthasMAMPUaffectedthe partnersandnetworks’capacitytoinfluencegovernmentreform? 9 3.3 KeyEvaluationQuestion2:Howandtowhatextenthavethepartnersand networksinfluencedgovernmentreforminrelationtotheneedsandpriorities ofpoorwomen? 12 3.4 KeyEvaluationQuestion3:HowandtowhatextenthasMAMPUcontributed toimprovedaccesstoessentialgovernmentservicesandprograms? 14 3.5 JudgingsuccessattheEnd-of-Programstage(Targets) 18 3.6 Typesofevaluation 194.0 How will the information reach decision - makers? 20

Draft Monitoring and Evaluation Framework 1

Introduction

ThisdocumentoutlinesaMonitoringandEvaluation(M&E)FrameworkfortheAustralia-IndonesiaPartnershipforGenderEqualityandWomen’sEmpowerment(MAMPUPhaseII).Thetargetaudi-enceforthisdocumentaremembersoftheMAMPUTechnicalCommittee(TC)andSteeringCom-mittee(SC).IthasbeendevelopedtoenablememberstoappraisetheproposedoverallapproachtoM&EforPhaseIIofMAMPU.ThisM&EFrameworkshouldbereadinconjunctionwiththeMAMPUPhaseIIStrategicForwardPlanandStandardOperatingProcedures(SOPs).ThefinalapprovedversionofthisM&EFrameworkwillbethebasisofarevisedM&EPlan,whichelaboratesMAMPU’sM&Esystemingreaterdetail.

Thisdocumentisstructuredinfoursections.Insection1itbrieflyexplainsthe‘changeprocess’thatneedstobemonitoredandevaluatedandsummariseswhatMAMPUexpectstoachieve.Section2outlineshowMAMPUwillroutinelymonitorimplementationprogress.Section3setsoutarrange-mentsforevaluation,includingKeyEvaluationQuestions,datatoaddressthese,andthebasisforjudgingsuccess.Finally,section4describesmechanismsforensuringthatfindingsfromtheseM&Eprocessesreacheskeydecision-makersintheMAMPUgovernancestructure.

1.0 What does MAMPU aim to achieve and how?

TheSubsidiaryAgreementbetweentheGovernmentsofIndonesiaandAustralianotesthattheul-timategoalofMAMPUistocontributeto“genderequalityandwomen’sempowermentinselectedareas in Indonesia.”Achieving thishigh levelgoalwillbe theresultof thecomplex interactionofwidersocio-cultural,politicalandeconomicforces,manyofwhichlieoutsidethedirectinfluenceofMAMPU.

Nevertheless,by2020MAMPUwillmakeacontributiontowardsthisgoalintwoways.Firstly,MAM-PU expects to have “improved access to essential government services and programs for poorwomenin target locations”.This is theEnd-of-Program-Outcome(EOPO).The ‘essentialgovern-mentservicesandprograms’referredto in thisEOPOstatementreflect thefiveMAMPUthemesdescribedintheSubsidiaryAgreement:

• Social protection programs, particularly publically-funded health insurance providedthroughtheNationalHealthInsuranceScheme(JaminanKesehatanNasional)adminis-teredbyBPJS(theme1);

• Workplaceprotections,particularlyhealthinsuranceforwomenhomeworkers(theme2);• Servicesthatimprovemigrationconditionsforwomenmigrantworkers(theme3);• Servicesthataddresswomen’ssexual,reproductivehealth,andnutritionalneeds(theme

4);and• Counsellingandsupportservicesthataddresstheneedswomenvictimsandsurvivorsof

violence(theme5).

Secondly, by 2020MAMPU expects to seepositivechange in the ‘voice’and ‘influence’ofwomenatmultiplelevels.Invillages,wom-enwill working collectively, expressing theirviewsinpublicandprivate(‘voice’),shapingdecision-making and influencing the alloca-tionofstateresources(‘influence’) forwiderbenefit, including improved access to ser-vices. By doing so they will be challengingnormsthatconstrainwhatissociallyaccept-ableforwomenandgirlstodo.Thesechangesin‘voice’and‘influence’areaprocessaswellasanexpectedoutcome.Asa‘process’theydescribeapathwaythroughwhichMAMPUimproveswomen’saccesstoservices.Theyarean ‘outcome’ofMAMPUin that theydescribeanexpectedendstatethatinitselfhasintrinsicvalue.Criticaltobothisaviewofpoorwomenasagents,notonlyusersofservicesprovidedbyothers.Thisisthe empowerment agenda that is central toMAMPU.

Draft Monitoring and Evaluation Framework2

How does capacity increase ‘voice’ at the grassroots?

In 2017 a qualitative study of eight local women’s groups established by MAMPU partners explored the effects of membership. The analytical framework centred on five types of empowerment ‘assets’. Applying this framework to women’s experiences as members shed light on how and in what circumstances individual and collective capacity leads to increased ‘voice’: “In terms of an empowerment pathway development of human assets, particularly confidence, self-belief, and as called by many interviewees, ‘courage’, appears to be a pre-requisite for other forms of empowerment. Members of each example of collective action studied reported a progression from growing confidence and knowledge (changes in human or individual assets), to speaking out and participating in or presenting to community forums (agency assets), and then some expectation that this will lead to either finance and resource assets or enabling assets, and these will reinforce each other.” These findings are consistent with a wide variety of evidence about women’s empowerment drawn from other contexts.

(MAMPU, (2017) Women’s Collective Action for Empowerment in Indonesia: A study of collective action initiated by partners of the MAMPU program. Yogyakarta: Migunani and MAMPU)

MAMPUisbasedontheideathatnetworksofselectedcivilsocietyorganisations–theMAMPUpart-ners–canplayapivotalroleinshapinggovernmentreformtobenefitpoorwomenonasignificantscale.TheprogramhaselectedtobuildontheworkoforganisationswithanestablishedtrackrecordofinfluencingreforminIndonesia.Thetheoryisthatwiththerightkindofsupportattherighttime,theseorganisationswillactmorecollectively,andincreasinglyinconcertwithalliesingovernment,parliament,andprivatesector.Atthesametime,partnerswillworkwithanddrawfromtheprioritiesandexperiencesofpoorwomeninvillagesacrossIndonesia.Bysupportingthesemultilevelpro-cesses,MAMPUexpectsmomentumforchangetogrow,influencinghowthegovernmentmakesandcarriesoutpolicies,improvingwomen’saccesstoessentialservicesonawidescale.

Figure 1: Outcomes and timeframes


By theconclusionof2017MAMPUexpects the resultsof thisprocess tobevisible in increased‘voice’ and ‘influence’. In the villages where MAMPU works, women will be more involved indecision-making and their ongoing participation will be acknowledged and enshrined in formalvillageregulations.At thedistrict,provincialandnational levels,MAMPU’scontributionwillbere-flectedinpolicydecisionsthatcanplausiblycontributetoimprovingwomen’saccesstoservicesonawidescaleby2020.Thisistheexpectedmedium-termoutcomeofMAMPU.

However, a series of preconditions are needed if these outcomes are to contribute to betterservicedelivery, improvedaccess to services, andempowerment by2020.Firstly, theymust beaccompaniedbypositivechangesintheself-belief,knowledgeandconfidenceofwomenwithwhomMAMPUworksatthevillagelevel.EvidencefromMAMPUaswellasinternationally,suggeststhattheseareimportantprerequisitesforvoiceandempowerment(seetextboxabove).Secondly,theauthorizingregulationsandpoliciesforreformsthataddresswomen’sprioritiesneedtobeinplaceatthenational,provincial,anddistrictormunicipalitylevels.Withoutthese,furtheractionsuchasal-locatingbudgetstowomen’sprioritiesisdifficult.Thirdly,resources(financialandhuman)needtobemobilizedandallocatedtowardstheparticularservice-relatedissueaddressedinthepolicy.Fourthly,governmentserviceprovidersneedsufficientcapacitytodeliverthepolicyintentattheservicelevel.Thesefourpreconditionsarenotenoughintheabsenceofafifth:evidentcommitmentandsupportfromacriticalmassofleaders–insidegovernmentandparliamentaswellasincommunities.

MAMPUiscautiousaboutgeneralizingacrossthediversityofcontextswheretheprogramaimsforchange.Theopportunitiesandconstraintswilldependtoagreatextentontheuniquesocial,politicalandculturaldynamicsineachvillage,districtandprovince.Nonetheless,togetherthesefiveprecon-ditions–invaryingconfigurations–representthe‘causalpackage’thatMAMPUbelievesisrequiredforpolicyinfluencetoleadtoimprovedaccesstoservicesandempowerment.

Increasing thecapacityand readinessofpartners forcollectiveaction– including their linkswithgovernmentandprivatesector– laysthefoundationforachievingmedium-termandEnd-Of-Pro-gram-Outcomes.MAMPUhelps toaccelerate thisbydevelopingpartnerorganisational capacity,bolstering the focus and intensity of collective action, and enabling them to expand their reachamongwomenat thegrassroots.Theprogramincentivizespartnerstowork innetworksthat linkup localandnationalorganisationsandprovides themwithgrant funds to test ideas inselectedlocationsacrossIndonesia.Alongsidethis,MAMPUlinkspartnerstotechnicalexpertise,highqual-ityevidence,andresources,toenablethemtobetterseizeopportunitiesthatemergeinthecontext.

AsMAMPUmoves intoasecondphase, itsroleasanactive ‘connector’and ‘bridgebuilder’be-tween partners, government agencies, and other strategically significant actors will increase.The new governance structure for MAMPU – which opens space for routine partner-nationalgovernmentinteraction–willbecriticaltothis.TheincreasedinvolvementofBappenasinaguidingandfacilitatingrolewillboosttheprogram’scapacitytolinkwithwiderreformsacrossthegovern-ment.


Table 1: Summary program logic for MAMPU


2.0 How will MAMPU be monitored?

Thepathfrominfluencinggovernmentpolicytoimprovedaccesstoservicesisneitherstraightnorpredictable.Insuchcontextsitisespeciallyimportantthatimplementationteamshaverapidfeed-backtogaugeprogressandmaketimelyadjustments.MAMPU’smonitoringsystemwilladdressthisneed.

Broadly,therewillbethreecomplementarycomponents:(i)quarterlyprogressreportingandanaly-sis;(ii)fieldmonitoringandverification;and(iii)regularopportunitiesforstructuredreflectionandadaptation.

2.1 Partner Quarterly Progress Reporting

TheworkofpartnersisattheheartofMAMPU’schangeprocessandfrequentfeedbackontheirperformanceisessentialforresponsiveandadaptivemanagement.

Every3months,eachpartnerwillsubmitashort,structuredreporttoMAMPUthroughtheonlinereportingsystem,‘MANISKita’.ThequarterlyprogressreportwillcontaininformationanddatasothatpartnersandtheMAMPUSecretariatcananswer5keymonitoringquestions:

1. Did we do what we expected to do?Appliesa‘trafficlight’scaleandbriefnarrativetoproduceasnapshotofperformanceagainstannualworkplanovertheprevious3months.TheexistingtemplatewillbemodifiedtoensurealignmentwiththeGovernmentofIndone-sia’sBASTfinancialreportingobligations.

2. Were the costs in line with what we expected?Providesasummarizedpictureofex-penditureagainstbudgetforeachimmediateoutcomeintheannualworkplan.

3. What challenges and risks are affecting progress? Provides information to explaindivergence between planned and actual implementation, and identifies risks that haveemergedinthecontextduringtheprevious3months.

4. Are we reaching and engaging the right people and groups in sufficient numbers? Qualitativeandquantitativedata–includingsex-disaggregatedstatistics–enablingarapidassessment of whom andwhere partners are engaging, including other organisations,menandwomenatthevillagelevel,religiousandcommunityleadersandpolicymakersatmultiplelevelsofgovernment.

5. What changes and benefits are being experienced by direct participants and stake-holders?InformationonoutcomesincludingshortnarrativeofprogresstowardspartnerEnd-of-Project-Outcomes,progresstowardspolicyinfluence,andquantitativedataonse-lectedindicators.

Takentogether,thisinformationisdesignedtogenerateapictureofperformanceacrosskeydimen-sionsofastylizedprogramlogic,asillustratedinfigure1overleaf.


InputsBudget

Resources

InputsBudget

Resources

InputsBudget

Resources

ImmediateOutcomes

ReachEngagement

ImmediateOutcomes

ReachEngagement

ImmediateOutcomes

ReachEngagement

IntermediateOutcomes



Partner Endof ProjectOutcomes



Partner 3Partner 2

Partner 1

Theme

Monitoring

4 & 5

Monitoring

1, 2 & 3 Dashboards that visualize databy partner, aggregated by

Figure 2: Using data from Quarterly Progress Reports to monitor performance by partner, theme, or portfolio

Keydata ineachreportarevisualizedautomatically inaseriesof interactive ‘dashboards’whichareaccessibletopartnersandtheMAMPUSecretariatthroughtheonlineMANISKitasystem.Thisassistsanalysisandhelpstomonitor trends inkey indicatorsat thethematic level,oracrossthewhole-of-MAMPUportfolio.


2.2 MAMPU Secretariat field monitoring and verification

Monitoring dan verifikasi lapangan akan menjadi komponen penting yang melengkapi sistempelaporankemajuan triwulanan formal ini.BagiMAMPU, ‘monitoring lapangan’diartikansebagaikegiatanditingkatdesaataukelurahan.Bersamadenganmitra,SekretariatMAMPUmelaksanakanrata-rata10kunjunganmonitoringlapanganpertriwulananuntukmengamatipelaksanaankegiatan,mendengarsecaralangsungdarilaki-lakidanperempuanyangterlibatdiakarrumput,sertaberin-teraksidenganparatokohmasyarakatsetempatdanaktorpemerintah.Salahsatutujuanpentingdarimonitoringlapanganadalahmentriangulasidanmemverifikasihasildanisudalamlaporanke-majuantriwulanan.Temuan-temuanpentingyangdidapatdariseluruhmonitoringlapanganmaupundatakuncikemudiandimasukkankedalamLaporanBack-to-OfficediSistemInformasiNasionalMAMPU(MANIS).HaliniakanmembantustafMonevuntukmelakukananalisistemuansecararu-tin,memperkuatkepercayaanterhadaphasilyangdilaporkan,sertamembantuSekretariatMAMPUuntukmelacakfrekuensidancakupanmonitoringlapangandiseluruhportofolio.

Figure 3: Field trip coverage dashboard, MAMPU National Informa-tion System

2.3 Routine reflection by Partners and the MAMPU Secretariat

OpportunitiestomakesenseofmonitoringinformationandplanfollowupactionswillbeessentialtoMAMPU’sapproach.TheMAMPUSecretariatwillfacilitatetwotypesofregularstructuredreflectionprocessduringPhaseII.

Firstly, theMAMPU Secretariat will facilitate regular 6-monthly reflection sessions (Participatory


AnalysisandReflection)witheachpartnercoveringwhatworkedwell,whatwaschallenging,andwhatactionneedstobetaken–byboththepartnerandtheMAMPUSecretariat.Thesehalftofulldaysessionsaredesignedtobesimple,interactiveexercisesthatareopenandflexible.Facilitationwillmakeuseofavarietyofparticipatorytechniquesincludingranking,voting,andWorldCafétoelicit reflectionandencouragediscussion.Thisapproachacknowledgesthatwrittenreportingfol-lowingastructuredtemplatecanmissinterestingandvaluableinformation.Highlights,keyfindingsandagreedactionsfromeach6-monthlyreflectionarerecordedonMAMPU’sinternalManagementInformationSystemwheretheprocessandfollowupcanbemonitored.

Asecondtypeofroutinereflectionwill takeplaceevery3monthswithintheMAMPUSecretariat.FacilitatedbyinternalM&Estaff,thiswillbringtogetherfindingsfromarapidanalysisofquarterlyprogressreports,fieldmonitoring,andfinancialmonitoringtoconsiderprogressineachofMAMPU’sfivethematic‘hubs’.Usingamixofpresentationsandinteractivediscussion,thesewillaimtofostergreaterstrategiccoherencewithinandbetweenhubsandensureMAMPUisresponsivetoemergingdevelopments.

3.0 How will MAMPU be evaluated?

ThissectionexplainshowMAMPUwillbeevaluatedatkeypointsoverPhaseII.ItdescribestheKeyEvaluationQuestions(KEQs)thatneedtobeanswered,whichdataandinformationwillbeused,howjudgementswillbemade,andwhattypesofevaluationexercisewillbeundertaken.

3.1 Key Evaluation Questions (KEQs)

Allevaluativeactivitywilladdressasetof4KeyEvaluationQuestions(KEQs)thatlinktotheout-comesexpectedatkeytimesintheprogramlife.ProposedKEQsforMAMPUsetinthedesignare:

KEQ1. HowandtowhatextenthasMAMPUaffectedthepartnersandnetworks’ capacitytoinfluencegovernmentreform?

KEQ2. Howandtowhatextenthavethepartnersandnetworksinfluenced governmentreforminrelationtotheneedsandprioritiesofpoorwomen?

KEQ3. MAMPUEnd-of-Program-Outcome(EOPO):Howandtowhatextenthas MAMPUcontributedtoimprovedaccessforpoorwomentoessential governmentservicesandprograms?

KEQ4. WhatchangedinthecontextandhowdidMAMPUrespond?

AllfourKEQswillbeaddressedduringphaseIItogenerateacompletepictureofMAMPU’scon-tributiontotheEOPO.However,therewillaspecialfocusonKEQ3reflectingexpectedprogramachievementinthisareaby2020.


Contribution Analysis (CA): MAMPU’s approach to addressing KEQs 1, 2 and 3 will draw from ‘contribution analysis’ (CA)1. CA can be distinguished from traditional approaches to evaluation which typically attempt to attribute an outcome to a particular intervention. This often involves isolating (using statistical techniques) the role of the intervention from other factors that could be responsible for causing the outcome. By doing this it is possible to say unequivocally whether or not X intervention ‘caused’ Y outcome. Such an approach is well suited to highly defined interventions with largely predictable types of effect.

MAMPU will need a different approach. Influencing policy, fostering empowerment in a dynamic political, economic and social context is complex and non-linear and it is widely acknowledged that the outcomes of interventions in this space are more challenging to predict. Generally MAMPU will be only one of many factors that contribute to an observed change. In this situation it is more reasonable to establish a credible case that MAMPU contributed rather than attempt to tease out the effects of large numbers of variables that are often interdependently related.

CA is well suited to this task. This approach boils down to four ingredients. First, set out the ‘logic’ or ‘theory’ that shows how an intervention is expected to work. Secondly establish whether or not the expected outcomes have happened. Thirdly, map out the contribution by an intervention to that outcome using the theory or logic to structure the evidence. Fourthly, acknowledge and account for the relative contributions of other factors. This approach is reflected in the sections below addressing KEQs 1, 2 and 3.

3.2 Key Evaluation Question 1: How and to what extent has MAMPU affected the partners and networks’ capacity to influence government reform?

KEQ1addressestheshort-termoutcomeofMAMPUthatisexpectedtoemergewithinyears2to3oftheprogramlife:positivechangesinthecapacityandreadinessofpartnersandnetworks.To

Figure 4: Domains of organisational capacity assessed through the OCPAT

1 Mayne,J.(2008)ContributionAnalysis:Anapproachtoexploringcauseandeffect,ILACmethodologicalbrief,availableathttp://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf


answerthisMAMPUmustfirstestablishifcapacityischangingandinwhatways.Assuch,thefirstsub-questionis:How and to what degree has the capacity of MAMPU partners changed?

Datatoaddresssub-question1willdrawfromtwosources:(i)longitudinalcapacityassessmentsofMAMPUnationalpartnerorganisations;and(ii)monitoringdataonpartnercollaborationwithotherorganisations.

Longitudinalassessmentsapplyastructuredmethodologycalledthe‘OrganisationalCapacityandPerformanceAssessmentTool’(OCPAT)developedbyIndonesianCSOYAPPIKA.TheOCPATisbasedonexistingevidenceofhowcapacitydevelopsinorganisations,includingtheexperienceofotherCSOprogramsinIndonesia2.Thetoolassessescapacityinsix‘domains’(seefigure4).Theprocesstakestwodayswitheachorganization and combines a highly participatory approach withspecificmeasurestobolstervalidityandreliability.TheassessmentisconductedbyindependentfacilitatorsandMAMPU’sroleislim-itedtoobserving,andonlywiththeapprovalofthepartner.

A ‘baseline’ OCPATwas facilitated withMAMPU partners in late2012/2013,asecondroundin2015,andathirdroundin2017.Ateachround,theassessmentmethodappliedamixtureofscoring,

Scale for tracking level of

partners and other types of

1 –

2 –

3 –

4 –

rankinganddiscussiontoidentifywhichdomainsofcapacityhadchanged,towhatextent,andwhatneedstobedonedifferently.AfinalroundofOCPATassessmentswillbecompletedin2019.

AsidefromtheOCPAT,additionaldatawillbeneededtoassessthestrengthofthenetworksamongpartnersandbetweenpartnersandothertypesoforganisations.ThiswilldrawfromMAMPU’squar-terly reportingsystem.Each threemonths,partner’sprovidedataon(i)organisationswithwhichtheyhadmostcontactduringthereportingperiod;(ii)describewhattheycollaboratedon;and(iii)usesasimplescale(seetextboxontheright)toassessthelevelofcollaboration.Thesedatawillbeanalysedtodrawoutwhichtypesoforganisationspartnersdevelopedlinkswith,whattypesofactivitytheyworkedtogetheron,andhowthischangedovertime.

Specifically,thisanalysiswilladdresstwoindicators: •Numberofinstancesofincreasedpartnercommunicationwithgovernmentorganisations orotherpolicymakingactors; •Numberofinstancesofincreasedpartnercollaborationwithotherorganisationsoutsideof government,specificallyforadvocacypurposes.Anincreaseinbothwillbeconsideredevidenceofincreasednetworkcapacitytoinfluencegovern-mentreform.

2 Inparticular,theOCPATisinfluencedbyamajorstudyoncapacitydevelopmentbytheEuropeanCentreforDevelopmentPolicyManagement(ECDPM)in2008.SeeBaser,H.andP.Morgan(2008),Capacity,ChangeandPerformanceStudyReport.(ECDPMDiscussionPaper59B).Maastricht:ECDPM.


WhilethisanalysiswillshowcapacitychangeamongpartnersandtheirnetworksitwillnotdirectlyassesstheextenttowhichMAMPUcontributedtosuchchanges.TofullyaddressKEQ1,MAMPUmustaddressasecondsubquestion:TowhatextentdidMAMPU’scontributetochangesincapacityandinwhatways?Therewillnotbeasingleanswertothisquestion.Rather,itislikelythatMAMPU’scontributionwillbemoreevidentinsomeaspectsofcapacitychangethaninothers.ItisalsolikelythatMAMPUwillmakeastrongercontributiontosomepartnersthanothers.Thiswillrequireanuancedapproach.

ThisanalysiswilluseinternallyhelddataonkeyfunctionsoftheMAMPUSecretariat:(i)grantfund-ing;(ii)technicalassistanceandadvice;(iii)conveningtheMAMPUnetwork;(iv)buildingandcom-municatingevidence;and(v)bridgingandlinkingpartners.ThesedatawillbebroughttogethertoassessthecasethatMAMPUmadeacontributiontochangesincapacityofpartnersandnetworks.Aqualitativescale(or‘rubric’)willbeusedtoenableclearjudgementsaboutMAMPU’scontributiontocapacitychange.Table2belowoutlinesadraftscale,whichwillneedtobefurtherrefinedbeforebeingapplied.

Table 2: Draft rubric to assess the strength of the case for MAMPU contri-bution to capacity change

12 Draft Monitoring and Evaluation Framework

3.3 Key Evaluation Question 2: How and to what extent have the partners and networks influ-enced government reform in relation to the needs and priorities of poor women?

WhileKEQ1addressescapacitychange,KEQ2focusesontheapplicationofthiscapacitytoinflu-encegovernmentreform.Therearetwointerrelatedfacetstothis.Firstly,partnersworkdirectlytoformnetworksandcoalitions (includingwithallies ingovernmentandprivatesector) to influencegovernmentdecision-making(formalandinformal)atmultiplelevels.Alongsidethis,partnersorgan-isewomenatlocallevel(villageanddistrict)andsupportthemtoexpresstheirviews(‘voice’)withtheexpectationthatthiswillinfluencechangethatbenefitspoorwomenandtheirfamilies.Bothofthese‘pathways’toachievinginfluencewillbeassessedunderKEQ2.

Subquestion1willfocusoninfluenceonformalgovernmentpolicies:How and what extent have MAMPU partners and networks influenced formal government policies?

DatatoassessthiswilldrawfromMAMPU’smonitoringsystem,particularlydata(qualitativeandquantitative)onengagementbetweenpartnersandpolicymakers,andpolicychanges.ThedataonMANISenablesMAMPUto:

•Tracktheprogressofdecision-makingonregulationsandformalpoliciesfromplanningto agendasetting,formaldebate,throughtoformalratification; •Monitorlevelsofengagementbetweenpartners,policymakers,themediaandother influentialstakeholdersincludingreligiousandcommunityleaders.

ThisenablesMAMPUtoidentifyinstancesofwhereMAMPUmaybeabletoclaimacontributiontodifferent typesofpolicychange, fromnational laws tovillage regulations.Minimumcriteria forinferringaclaimforcontributiontopolicyinfluenceare:

Gambar 5: Jumlah kebijakan di setiap tahap kemajuan, dashboard penelusuran kebijakan MANIS

50

01. In Prep 3. Under

Discussion4. Passed

47

22

37


(i) Therehasbeenpartnerengagementwiththepolicymakeroveratleasttwoquarterspriortothedecision,asevidencedinquantitativedataonengagement;and(ii) Priorengagementhasaddressedthesubstanceofthedecisiontakenbythepolicymakerasevidencedthroughnarrativematerialinpartnerreportingand/ormonitoringrecords.

Oncepotentialclaimsmeetthesecriteria,MAMPUfollowsuptocriticallyreviewthe‘influencesto-ry’,interviewingpartnerstoclarifydetailsandseekadditionalevidencewhererequired.UsingthisprocessMAMPUmaintainsalistofinstancesofpolicyinfluenceatmultiplelevels,backedbyverifi-ableevidence.

AsecondsetofsubquestionswilladdressthecriticalgrassrootsvoiceandinfluencethatMAMPUseekstodevelop: How and to what extent have partners contributed to the capacity of women at the village level to project ‘voice’? To what extent has this translated to influence on deci-sion-making at the household, village, and beyond?

Thesearecomplexquestions.ToaddressthemMAMPUwillsynthesizedatafromarangeofsourc-esagainstasetof‘evaluativecriteria’.Theseidentifythetypesofchangesthatwillbevaluedandtakentoindicatecapacity,readiness,voiceandinfluenceamongwomenatthevillagelevel.Table3containsadraftlist,developedwithpartners.

Table 3: Draft evaluative criteria to assess changes in capacity, readiness, voice and influence among women at the village level3

The data on these criteria will draw from several sources. Quantitative and qualitative data inQuarterlyProgressReportswillbecomplementedbyaqualitativemonitoring toolalready inuseby MAMPU– theMostSignificantChange (MSC) technique.Basedonfirst-personnarrativesaboutchanges(positiveornegative),MSCinvolvesthecollectionofstoriesfromwomenatthevil-lageacrossMAMPUwhicharethensystematicallyselectedbypartnersandMAMPU.Storiesare

3 ThesedraftcriteriawereidentifiedduringaworkshopfacilitatedbytheM&ESpecialistaspartoftheMAMPUPartner’sForuminJakartainJuly2017.


Table 4: Draft rubric scale to assess capacity, readiness, voice and influence at the village level

uploadedbypartnersintoacustom-designeddatabase(‘MAMPUStorybook’)whereMAMPUcon-ductssecondaryanalysisof their content.Anopen-endedmonitoring tool,MSC iswell suited tocapturingcomplexsocialchangesthatareoftenintangibleandhardtoobservedirectly4.MAMPU’sownfieldmonitoringrecordswillbeusedtoverifyandtriangulatethesedatawithdirectobservationofvillage-levelprocesses.Inaddition,MAMPUwillneedtodrawuponarangeofresearchstudiesforfurtherevidence.ThesewillincludetheMAMPUlongitudinalsurvey,whichexamineschangesinaccesstoservicesin1500women-andmale-headedhouseholdsin15villagesover3waves:2014(baseline),2017(midline),and2019(endline).Itwillalsoincludeafurtherstudyoflocallevelvoiceandinfluence,designedtobuildonthefindingsofthe2017Women’sCollectiveAction(WCA)Study.

Thesedatawillbebroughttogetheragainstarubricscale–draftedintable4below–thatdistin-guishesbetweendifferentlevelsofachievementagainsteachoftheevaluativecriterioncontainedintable3above.Itisproposedthatthisrubricisappliedonapartner-by-partnerbasis.However,itwillbeessentialthatthisisfirstrefinedwithpartnersandadaptedtosuittheirdiversecircumstances.Nevertheless,theuseofthescalewillenablesomeconsistencyinthesynthesisofevidence.

3.4 Key Evaluation Question 3: How and to what extent has MAMPU contributed to improved ac-cess to essential government services and programs?

Putsimply,twodistinctbutlinkedpiecesofanalysiswillbeneededtoassessMAMPU’scontributiontoimprovedaccesstoessentialservices.First,itmustbeclearifmorepoorwomenhaveaccesstoessentialservicesinMAMPUlocations.Secondly,theremustbeanassessmentoftheMAMPUcontributiontosuchincreases.Theseanalyticalpointswilladdressedthroughtwosub-questions.Thefirstsub-questionconcernstheextentandscaleofchange:Has access to services increased for women who are poor and if so, where and by how much?Thiswillrequirethecollectionandanalysisofquantitativedata.Table4proposesasetof8quantitativeindicatorsforthispurpose.

3 KloostermanJ.(2012),‘Measuringtheunmeasurable’:gendermainstreamingandculturalchange,inGenderandDevelopment,Vol.20,2012,Issue3


Table 5: Quantitative indicators to measure changes in access to services


Figure 6: Identifying quantitative indicators of access to services

Which theme ?

What governmentservice or program ?

What measures ofaccess are feasible ?(Direct & Indirect)

Indicatorsintable2havebeenidentifiedthroughacascadingprocessstartingwiththetheme,andthenconfirmingtheserviceorprogramaddressedbytherelevantpartners.MAMPUthenidentifiedindicatorsofaccesstotheserviceorprogram.TobefeasibleforMAMPU,these indicatorsneedtobesex-disaggregated,representativeatthedistrictormunicipality level,sufficiently linkedwithpartneractivity,andforwhichdataareavailableatbaseline(2015/2016)andendline(2019/2020).Assessingchangesintheseindicatorsacrossdifferenttargetareasshouldenableasufficientlynu-ancedpictureofchangesinaccesstoservicesacrossthediversityofMAMPUcontexts.

Quantitativeindicatorsintable2measuretwocategoriesofaccess:

Category1. DIRECTsupportbyMAMPU:wherewomenhavebeendirectlysupportedbypartnerstoaccessagovernmentserviceorprogram.Thisdatawillbecollectedandre-portedbypartnersthroughaspecificmoduleinthequarterlyprogressreportingtemplate.ThemagnitudeofincreasethatisachievableoverthelifeofMAMPUisexpectedtobeintheorderof10-15,000womenacrossallfivethemes.

Category2. INDIRECTbenefit throughpolicy implementation:Wherewomengain in-creasedaccesstoservicesthroughimplementationofgovernmentpolicydecisions.DatatomeasurethiswillcomefromselectedGovernmentofIndonesiadatasetsthatarerepre-sentativeatthedistrict/municipalitylevelandsex-disaggregated.TheseincludeSUSENAS(annual‘core’survey)aswellasadministrativedatapublishedroutinelybylineagenciesincluding theMinistryofLabourand theMinistryofWomen’sEmpowerment.Given thecoverageandscaleofMAMPU,thenumberofwomenwhocouldgainaccesstoservicesindirectlyisinthemillions.


Clearly,MAMPU’sinfluenceonpolicydevelopmentandimplementationiscriticalforwidescaleim-provementsinaccess(i.e.category2above).However,MAMPU’scontributionmustbeshownforthisoutcometobeclaimed.Thisisthefocusofthesecondsub-questionunderKEQ3:HowstrongisthecasethatMAMPUcontributedtoobservedincreasesinaccesstogovernmentservicesandprograms?

Data toanswer thiswill drawprimarily fromMAMPU’smonitoringsystem including thequarterlyprogressreportsandfieldmonitoring,supplementedwithcasestudiesandwhereavailable,otherqualitativematerialsuchasMostSignificantChangenarrativesandresearchstudies.ThisdatawillbedrawntogethertoassessthestrengthofthecaseforMAMPU’scontributionagainsta5-point‘rubric’(describedintable6).

Table 6: ‘Draft rubric to assess the strength of the case for MAMPU contribution to improved access to services

Combininganassessmentofcontribution(sub-question2)andquantitativeanalysisofchangesinaccess(sub-question1)willenableMAMPUtoanswerKEQ3clearly,butalsoinanuancedwaythatreflectsthediversityacrosstargetareas.UsingthisapproachMAMPUwilldistinguishbetweenfourpossibletypesofscenario,aslistedintable7.


Table 7: Combining sub-questions to assess EOPO achievement in target districts and municipalities

Itisimportanttodifferentiatebetweenthese.Targetareaswheretypes3or4haveoccurredwillberegardedashavingachievedtheEOPO.However,onlytype4willbecountedasanactualincreaseinthenumberofwomenwithaccesstoserviceswithintheMAMPUtime-frame.Furtheranalysisofthequantitativedatawillbedonewheresuchcasesareidentified.Thiswill includeattentiontotherelativedifferencesinaccesstoservicesexperiencedbywomenandmeninthetargetareatodrawtentativeconclusionsabouteffectsongenderequalitybeyondthelifeofMAMPU.

3.5 Judging success at the End-of-Program stage (Targets)

AlthoughMAMPUaimsforwidespreadimprovementsinaccess,thelinksbetweenpolicyinfluence–evenatlocallevels–andservicedeliveryareundeniablycomplex5.Toaccountforthis,MAMPUconsiderstargetsettingatthedistrictormunicipalityleveltobeanappro-priateapproach.Thealternative–expressingtargetsintermsofthenumberofwomenwithincreasedaccess to services–will inevitably understate the instanceswhereMAMPU’scontributionis‘strong’or‘verystrong’butnotyetreflectedattheservicelevel.MAMPUwilltrackquantitativechanges inwomen’saccess toservices,butultimatelysuccesswillbejudgedonthebasisofcontributiontogovernmentdecision-makingthathasalreadyledtoorislikelytoleadtoanincreaseinaccess.ThisisconsistentwiththeintentofKeyEvaluationQuestion 3.

5 ArecentevaluationofaDFAT-fundedprogramtoimproveservicedeliveryineasternIndonesiarecommendedthatdonorstakealong-termandmultilevelapproachthatrecognisesthiscomplexity.SeeL.KellyandSakriD.(2015),IndependentCompletionReport,Australia-IndonesiaPartnershipforDecentralisa-tion,Canberra:DFAT


Table5 sets targets for thenumberof districtsormunicipalitieswhereMAMPUexpectstheEOPOtobe‘largelyachieved’or‘fullyachieved’infourofthefivethemes.Tosetthesetargets,MAMPUconsultedwithpartners,reviewedtrendsinpartnerengagement,andcon-sideredrecentchangestoprogramcoverage.

Theme2,whichhasnotbeensetaspecifictargetinTable5,isaspecialcase.Becauseofthelowvisibilityofthehomeworkerissueamongpolicymakers,MAMPUdoesnotexpecttoachieve influence thatwill increaseaccess toservicesbefore2020. Instead,MAMPUexpects toachieveaquantitative targetofdirectly increasingaccess tohealth insurance(BPJSKetenagakerjaan).

Meetingthethematictargetsintable5willbeasoundbasisonwhichtojudgethatMAMPUhasachievedtheEOPOatascalethatreflectstheintentintheprogramdesign.

3.6 Types of evaluation

TwotypesofevaluationwillbeundertakenoverPhaseIIofMAMPU:(i)twointernalevalu-ationsbasedontheCollaborativeOutcomesReporting(COR)technique6;(ii)andaninde-pendentmid-termevaluationcommissionedbyDFAT.

Internal evaluations addressingallfourKEQswillbeundertakenattwopoints:duringthefirsthalfof2018;andinlate2019/early2020.FacilitatedbytheMAMPUM&ESpecialist,

Table 8: Targets for MAMPU achievement by 2020, by theme

6 http://www.betterevaluation.org/en/plan/approach/cort


thesewilldrawtogetherdatacollectedthroughtheM&EsystemtoassessthestrengthofMAMPU’scontribution toshort-,medium-termoutcomes,andexamineprogress towardstheEOPO.ThesecondinternalevaluationwillmakeasummativeassessmentofprogramachievementoftheEOPO,applyingthemethodologyoutlinedabove,alongwithdataavail-ablefromothersources.

WhiletheM&ESpecialistwillberesponsibleforleadingtheprocess,theCORtechniqueworks best as participatory exercise that involves different stakeholders in data collec-tion and analysis. Consistent with the COR methodology, all claims of contribution willtransparentlyreferencethesourceofevidencethatcanbeverifiedbyanindependentparty.MAMPUwillensureanindependentperspectivetoassessthestrengthofthecaseforcon-tributionforoutcomes.Thefindings,evidence,andrecommendationswillbedocumentedinashortreadablereportcalleda‘PerformanceStory’.

DFATinconsultationwithBappenaswillberesponsibleforcommissioninganindependent evaluationofprogressoverPhaseII.TheexactscopeandtimingofthisevaluativeexercisewillbedeterminedbyDFATbutithasbeententativelyscheduledtotakeplaceinthesecondhalfof2018.

4.0 How will the information reach decision-makers?

Thissectionbrieflydescribesthedifferentreportingandinformationproductsthatcontainfindingsandrecommendations fromtheM&Eprocessesdescribed in theprecedingsec-tions.Itexplainshowthesewillreachkeyaudiencestosupportdecision-makingabouttheprogram.

KeyaudiencesoftheinformationgatheredthroughtheM&EsystemaretheMAMPUSecre-tariat,partners,AustralianEmbassy,ThematicWorkingGroups(TWGs),TechnicalCommit-tee(TC),andSteeringCommittee(SC).ThesedifferentstakeholdersplaykeyroleswithinthegovernancestructureforMAMPUoutlinedintheSOPs.Therightinformationwillneedtoreachthesestakeholdersattherighttimeandintheappropriateformattosupportdecision-making.

Figure 7: Timeline of evaluations over Phase II


TheMAMPUSecretariatwillneedfrequentfeedbackonperformanceatthepartnerlevel,acrossathematic‘hub’,aswellasfortheprogramoverall.Ofparticularimportancewillberapidfeedback,earlywarningsigns,andrisksthatenabletheSecretariatstafftotakeac-tion.Thisneedstobealliedtomore‘strategic’andlong-termconsiderationsthatenabletheSecretariattoseehowimmediateconcernsaffectthelikelihoodofachievingtheEOPO.

Asthedonor,theAustralianEmbassyneedsinformationtoassessprogresstowardsshort-,medium-andlong-termoutcomes,aswellassummarizedinformationonimplementationoftheannualworkplanandbudgetaryperformance.ThisinformationshouldenableEmbassystafftomeetinternalaccountabilityandcompliancerequirementsbutalsomakedecisionsconcerningrisksandstrategicopportunitiesthatwouldbenefitfromofficialgovernment-to-governmentinvolvement.

The5ThematicWorkingGroups (TWGs)are venues for partner-government interactionandcoordination,notdecision-making.Tosupportthesefunctions,informationisneededonpoliciestargetedineachtheme,progressandobstacles,andhighlightsandlessonsfromwork‘ontheground’wherepartnersareattemptingtosolvepolicyissuesinpracticalways.TheTechnicalCommittee(TC)requiressummarizedinformationonprogramperformanceagainstoutcomes,particularlyprogresstowardstheEOPO.Theinformationshouldbesuf-ficientlydetailedtoenablememberstojudgetheadequacyofoverallperformance,considerrelativeprogressacrossthemes,andassesstheappropriatenessoftheAnnualWorkplanandbudgetproposedbytheMAMPUSecretariat.

TheSteeringCommittee (SC)needshigh levelsynthesized informationonperformance,sufficienttoapprovetheannualworkplanandbudget.AkeyconsiderationfortheSCwillbetheongoingrelevanceandalignmentofMAMPU’sportfoliowithstrategicprioritiesofbothgovernments.


Table 9: Key reporting and information products, audience, content and timing


Annex 1 Key terms and concepts

Analysis originates from a 2001 paper by John Mayne, formerly of

the Auditor-General of Canada. ( -

).

In it, Mayne argued that a narrow focus on outcomes to an . Instead, there should

be a focus on using evidence to show contributed. This , requires six steps: (i) acknowledge the

theory of change; (iii) show that the steps in the theory of change have occurred

the

Poor women, poverty locally relevant and suited to the range of contexts where they work. Overall, MAMPU is guided by the cent’,

Documents

Draft Monitoring and Evaluation Framework · 2018. 7. 26. · Draft Monitoring and Evaluation Framework 1 Introduction This document outlines a Monitoring and Evaluation (M&E) Framework