Findings from an external evaluation and peer review process of critical DEC member agency processess and proceedure.
<ul><li><p>The 2011/12 DECAccountabilityFramework Assessmentfindings from the externalevaluation and peer reviewprocess</p><p>July 2012</p><p>Alice Obrecht, Christina Laybourn,Michael Hammer & Stphanie Ray</p></li><li><p>Telephone: +44(0)20 7713 6790Email: firstname.lastname@example.orgVisit: www.oneworldtrust.orgCharity Commission No 1134438</p><p>About the One WorldTrustThe One World Trust is an independent charitableresearch organisation that conducts research,develops recommendations and advocates forreform to make policy and decision-makingprocesses in global governance more accountableto the people they affect now and in the future,and to ensure that international laws arestrengthened and applied equally to all.About the EvaluatorsThe 2011/12 DECAF assessment was conductedby Christina Laybourn and Alice Obrecht, bothSenior Researchers at the One World Trust, withsupport from the Executive Director MichaelHammer and Research Assistant Stphanie Ray.All members of the consultancy team, underAlices lead, contributed to the final report, forwhich The One World Trust takes responsibility asthe consultant.The One World Trust would like tothank the staff from all Member Agencies whocontributed to the validation process for theirparticipation, in particular the Leads tasked withorganising their Agencys submission. The DECSecretariat staff, through their openness toengagement and challenge, contributed invaluablyto their own accountability assessment as well asto the Member Agencies peer review workshops.The consultants are particularly grateful to CaitTurvey Roe for her active involvement andguidance throughout this process.</p><p> One WorldTrust 2012. This work is licensed under a Creative CommonsAttribution-NonCommercial-NoDerivs 2.0 UK: England & Wales License.</p><p>One WorldTrustSuite 301, 3rd Floor109-111 Farringdon RoadLondon EC1R 3BW</p><p>ISSN Series 2043-7943</p></li><li><p>DEC AccountabilityFramework: findings fromthe external evaluationand peer reviewAlice Obrecht, Christina Laybourn,Michael Hammer & Stphanie Ray</p><p>One WorldTrust 2012</p></li><li><p>Way of Working 1.4: Incidences of actual/suspectedfraud or loss are investigated and actioned.</p><p>Way of Working 3.3: Disaster affected populationsparticipate in programme assessment, design,implementation and evaluation.</p><p>Way of Working 3.4: In consultation with disaster affectedpopulations appropriately designed and describedcomplaints handling procedures are implemented.</p><p>Contents</p><p>Introduction.</p><p>Findings & Conclusion.</p><p>Way of Working 1.3: Programme design andprocurement processes maximise value formoneybalancing quality, cost and timeliness ateach phase of the response.</p><p>4</p><p>7</p><p>10</p><p>12</p><p>15</p><p>18</p><p>20</p><p>Way of Working 2.6: Programmes contribute to disasterrisk reduction and build the resilience of communities,partners and governments.</p><p>Introduction Way ofWorking 1.3 Way ofWorking1.4Way ofWorking2.6</p><p>Findings &ConclusionWay ofWorking3.3Way ofWorking3.4</p><p>Appendix 21</p></li><li><p>DEC Accountability Framework One World Trust</p><p>IntroductionEach year, members of the British public placetheir trust in the Disasters EmergencyCommittee to transform their donations into life-saving assistance for people affected by disaster.The Member Agencies of the DEC honour thistrust by committing themselves to processes thatsupport and challenge them to meet the higheststandards in humanitarian accountability andperformance. One of these important processesis the DEC Members annual self-assessmentsagainst their shared Accountability Framework.This Report summarises the findings from the2011/12 DEC Accountability Framework(DECAF) assessment process.The purpose of theDECAF assessment is to review the policies,practices and assurance systems through whichMembers meet best practice in their humanitarianwork using funds raised through the DEC. Inorder to lend focus to the assessment, Membersare asked to provide supporting evidence fortheir self-assessments from two countries inwhich they are currently using DEC funds toassist disaster-affected people. For the 2011/12assessment, these countries were selected fromthe Pakistan Floods Appeal, for which a total of71 million was raised from the British public, andthe East Africa Crisis Appeal, which reached atotal donation level of 79 million and has beenused to assist more than 1.3 million people.Through the DECAF assessment process,Member Agencies hold one another to accountfor mutually-recognised standards, providing aunique forum for peer challenge and horizontalaccountability. They are also held to account byexternal evaluators, who review the evidence-based self-assessments submitted by MemberAgencies, thereby supporting the externallegitimacy of the Accountability Framework.The Accountability Framework Assessmenttargets the underlying systems and policies thatare needed to support Members in their delivery</p><p>of effective and responsible aid. It does not assessthe quality of the DEC Members humanitarianresponse.As such, this Report is a complement tothe system of oversight of Members plans andprogramme reports as well as the Real-TimeEvaluations (RTEs) and other external evaluationsthat accompany each DEC appeal.Raising the bar: the DEC AccountabilityFramework (DECAF) IIThe 2011/12 DECAF process has been marked bya period of transition for the DisastersEmergency Committee.The DEC welcomed PlanUK as a new Member Agency this year, bringingits membership to 14 Agencies. DEC Membersalso revised significantly their best practicestandards, or, Ways of Working in humanitarianservice (Ways of Working). The new framework,DECAF II, was adopted in April 2011 andimplemented for the first time in this yearsDECAF assessment process. The framework is adynamic model based on four accountabilitypriorities that are considered most likely to driveimprovement:1) We use our resources efficiently and effectively.2) We achieve intended programme objectives inaccordance with agreed humanitarian standards,principles and behaviours.3) We are accountable to disaster affectedpopulations.4) We learn from our experiencetaking learningfrom one emergency to the next.These are enumerated in 21 specific Ways ofWorking which reflect best practice standards(see Appendix 1).The revised Ways of Working inDECAF-II offer a substantial step change from theprevious framework and, in several key areas,</p><p>IntroductionWay ofWorking 1.3</p><p>4</p><p>1</p><p>1. Evaluations funded by the DEC are independent and aremade publically available through their Appeals page:http://www.dec.org.uk/appeals2. DEC Accountability Framework,http://www.dec.org.uk/node/1951/</p><p>1</p><p>2</p><p>Way ofWorking1.4Way ofWorking2.6</p><p>Way ofWorking3.3Way ofWorking3.4</p><p>Findings &Conclusion</p></li><li><p>DEC Accountability Framework One World Trust5</p><p>IntroductionWay ofWorking 1.3</p><p>raise the bar for standards of accountability andperformance across the humanitarian sector. Forthe 2011/1 assessment, the Pakistan and EastAfrica Appeals presented unique opportunitiesand challenges for meeting the strongaccountability and performance standards set byDECAF-II. Members commitments to maintainingbeneficiary participation in all cycles ofprogramming were strong in countries covered bythe East Africa Appeal, while assurance processesfor value for money and detecting and respondingto suspected fraud were well-implemented inPakistan.The Assessment ProcessOn an annual basis, each Member Agency assessesitself against each Way of Working. Thisassessment targets three broad areas ofperformance:1) Policy or procedure2) Application of policy in a particularhumanitarian context (practice)3) The assurance systems that Members use toensure the quality and consistency of their workacross responses.As external evaluators, the One World Trust teamasked Agencies to submit evidence for policy,practice and assurance in support of their self-assessments for five selected Ways of Working:1.3 Programme design and procurementprocesses maximise value for money - balancingquality, cost and timeliness at each phase of theresponse1.4 Incidences of actual/suspected fraud or lossare investigated and actioned2.6 Programmes contribute to disaster riskreduction and build the resilience of communities,partners and governments3.3 Disaster affected populations participate inprogramme assessment, design, implementationand evaluation3.4 In consultation with disaster affected</p><p>populations appropriately designed and describedcomplaints handling procedures are implementedFocusing on these five standards offered a viewinto how Agencies are engaging with timely issuesin their donor environment and making progresson long-standing commitments, such asaccountability to beneficiaries, in new andchallenging contexts.After reviewing the submitted evidence, the OneWorld Trust facilitated a series of Peer Reviewworkshops, in which Members summarised theirself-assessments and received challenges andquestions from their peers, the One World Trustteam and representatives from the DECSecretariat. At the end of each workshop, eachMembers self-assessment was discussed andconfirmed by Peers and the One World Trustteam. In some cases, a Member Agency wasrequested to submit further supporting evidencefor an area of performance (policy, practiceand/or assurance).Findings & ConclusionBroadly speaking, three performance perspectiveson each Way of Working emerged throughout theassessment process: a conventional perspectivethat represents where most Agencies are at withrespect to a Way of Working and commonthemes or strategies; a perspective of innovationthat reflects the novel interpretations of a bestpractice and the new frontiers being explored bysome Members; and a perspective ofimprovement that embodies the challengesAgencies faced in meeting a Way of Working andthe strategies some Members are using to catchup with their peers or move to the front.Theseperspectives are represented in a diagram at thebeginning of each Way of Working summary andare discussed in greater detail in the text of eachsection.The One World Trust team found that, overall,performance across the five Ways of Working</p><p>Way ofWorking1.4Way ofWorking2.6</p><p>Way ofWorking3.3Way ofWorking3.4</p><p>Findings &Conclusion</p></li><li><p>DEC Accountability Framework One World Trust6</p><p>IntroductionWay ofWorking 1.3</p><p>examined for the assessment met many bestpractice standards, with strong policies andfrequent examples of humanitarian best practicebeing met in Pakistan and East Africa. However,performance on assurance was mixed across thefive Ways of Working: there were some examplesof strong quality assurance mechanisms but theprocess of strengthening or developing ways toassure quality and monitor the consistency oftheir performance continues. Innovation and theactive development of new systems to enhanceaccountability and performance are high across all14 Members.</p><p>Way ofWorking1.4Way ofWorking2.6</p><p>Way ofWorking3.3Way ofWorking3.4</p><p>Findings &Conclusion</p></li><li><p>DEC Accountability Framework One World Trust</p><p>Way ofWorking 1.3</p><p>7</p><p>Introduction Way ofWorking1.4Way ofWorking2.6</p><p>Way ofWorking3.3Way ofWorking3.4</p><p>Findings &Conclusion</p><p>Conventional perspective </p><p> Has position paper outlining organisational definition of value for money </p><p> Has strong assurance mechanisms for assessing costs, evaluating quality, and monitoring timeliness, however </p><p>there is no overarching mechanism that unifies all three </p><p>into a single decision-making or review process </p><p> However: confident that procurement and programme design on the ground are being guided by a holistic </p><p>balancing of costs, benefits and value added </p><p>Perspective of innovation </p><p> Embracing the opportunity to engage in and lead the discussion on how value for money should be </p><p>conceptualised for humanitarian aid </p><p> Specific attention paid to value for money considerations in programme design, not just procurement </p><p> Actively pursuing a holistic integration of VfM across all aspects of organisation </p><p>Evidence should demonstrate that we have a clear organisational position on </p><p>what value for money means and how this should be assessed. As a minimum, quality, </p><p>cost and timeliness should be analysed togetherrather than in complete isolation to </p><p>each otherduring the planning, implementation and evaluation phases. DEC </p><p>Accountability Framework Assessment Guidelines </p><p>Way of Working 1.3: Programme design and procurement </p><p>processes maximise value for moneybalancing quality, cost </p><p>and timeliness at each phase of the response </p><p>Continuous </p><p>Impro</p><p>vem</p><p>ent </p><p>Perspective of improvement </p><p> Organisational documents feature value for money as a term, but do not define it in detail. </p><p> Working groups established to develop an organisation-wide approach </p><p> Value for money will be an active area for improvement in 2012/13, with plans for developing tools, indicators, </p><p>and/or rolling out new performance systems </p></li><li><p>DEC Accountability Framework One World Trust</p><p>Background to this Way of WorkingValue for money has become a dominant conceptin the UK donor and aid environment in recentyears. The case for value for money as a bestpractice standard is relatively straight-forward:Given the limited amount of resources availableto them, aid organisations ought to accomplishthe most they can with these resources, bymaximising the value achieved by every unit ofcost. In practice, however, value for moneystandards raise important and complex questions,such as how to assess the value and financial costof intangible or longer-term outcomes, or how tocompare value for money across organisationsthat use different approaches to assessing qualityand to balancing quality against cost andtimeliness. The evidence submitted by the DECMembers reflected a shared commitment toachieving the very best with their limitedresources, showcasing a diverse set of approachesto how organisations are conceptualising andmeasuring value for money in their work.The Conventional Perspective:The Peer Review workshops addressed thequestion of whether a Member Agency needed tohave a defined approach to value for moneyoutlined within the organisation in order to beconsidered to meet minimum standards of bestpractice. Members felt that they all should have aclear position paper or approach to value formoney and many submitted papers that outlinedtheir approach to value for money in programmedesign.There is evidence that Agencies are usingconsiderations of cost, timeliness and quality toguide programming.At present, however, evidenceof value for money in practice focuses on theprocurement of goods and services: evidenceprimarily consisted of assessments of bids ordocumentation of vender selection,demonstrating how these procedures areregularly guided by a unified decision-makingprocess that looks at all three dimensions of cost,</p><p>timeliness and quality of product.The Perspective of Innovation:Measuring and demonstrating value for moneypresents a number of challenges, which Membersare overcoming through innovation. Value formoney is sometimes viewed as a top-downaccountability priority in which aid organisationsmust direct field staff to make decisions that canbe justified as cost-effective and efficient todonors.This challenges Members to find ways tobring intended beneficiary groups into the...</p></li></ul>