www.agulhas.co.uk

Budget Strengthening Initiative

External Evaluation 2010 - 2015

September 2015

Marcus Cox Ken Robson

Agulhas Applied Knowledge

i

Executive summary

This independent evaluation of the Overseas Development Institute’s Budget Strengthening Initiative (BSI) comes after five years of operations, at the end of the first core grant from DFID. It follows on from a mid-term evaluation by the same team in July 2013. Its purpose is to assess BSI’s achievements over the period 2010 to 2015 against its results framework, to test the validity of its theory of change and to draw lessons to inform the future development of the programme. BSI was designed as an innovative approach to supporting budgetary and public financial management (PFM) reforms in fragile states. It provides a combination of strategic advice, technical assistance and hands-on support for the implementation of reforms. Its funding arrangements provide greater latitude than most technical assistance programmes, enabling it to take a flexible, problem-solving approach. BSI has three main country programmes: South Sudan, which is by far the largest, Liberia and the Democratic Republic of Congo (DRC), with some additional activities in Uganda. In the past 18 months, it has faced a volatile operating context, owing to the outbreak of conflict in South Sudan and the Ebola crisis in Liberia. BSI also supports the g7+ group of fragile states and sets aside a proportion of its budget for research and dissemination. The evaluation consists of two main elements. First, it independently verifies BSI’s own reported results, including the quality of its advisory work and the extent to which it has delivered significant institutional change. Second, it tests key assumptions underlying the theory of change and BSI’s claim to offer an innovative and effective approach to supporting budget reforms in fragile states. Most of the evidence for the evaluation comes from feedback from the counterpart institutions and other stakeholders, collected during visits to all four countries, which was then used to triangulate BSI’s own reporting. Validation of results Inputs. BSI received a core grant of £7 million from DFID’s Africa Regional Department and has successfully raised an additional £8.56 million, including further funding from DFID South Sudan and a £2.2 million core grant from Sweden. While this is a good fundraising record, the uncertain and piecemeal nature of its funding has led to high transaction costs and hampered programme development. Funds have been utilised as planned, despite the need temporarily to withdraw BSI teams from South Sudan and Liberia during the crises. Feedback from counterparts on the calibre of BSI teams was universally positive. BSI staff are seen as providing an unusual mix of political insight, technical expertise and skills in communication and facilitation. The programme is able to offer good economy through modest fee rates and overheads, efficient use of part-time consultants and a limited infrastructure in its partner countries. We note that, despite a number of attempts in recent years, BSI has not been successfully at initiating engagements in new countries, suggesting that this aspect of the BSI model needs further thought. Effectiveness. Under its logframe, BSI’s main output is high-quality technical assistance. This is measured through internal narrative reports, independently verified by the evaluators. Over the two rounds of evaluation, we reviewed 25 annual workstream reviews and

ii

confirmed that 22 of them scored an ‘A’ or higher on DFID’s scale, showing that they met or exceeded expectations. This strong result was borne out by our assessment of BSI’s advisory outputs, which we found to be of consistently high quality. BSI also tracks the outcomes of its support through ‘stories of change’, which capture changes in policy, attitudes, behaviours and processes. We were able to verify a total of 14 significant institutional changes. Impact. As a relatively small programme working in volatile environments, BSI has had mixed success in delivering overall improvements in budget performance, as measured through Public Expenditure and Financial Accountability (PEFA) indicators. There have, however, been visible impacts at the country level. In South Sudan, BSI has been instrumental in designing and implementing many of the core budget processes, although the annual budget calendar has been regular disrupted by crises. It has been a strategic adviser to the ministry of finance, helping the government to respond to successive fiscal crises and to negotiate more effectively with the international community. It has achieved some key policy breakthroughs, such as building consensus on the need for an independent National Revenue Authority. Its most impressive achievement has been a system of intergovernmental transfers to fund local service delivery, to which donors are now beginning to align. This has helped to break the cycle of donors funding services through parallel mechanisms such as NGOs, in favour of supporting government service delivery. This highly strategic contribution to state-building in South Sudan is clearly attributable to BSI’s support. In Liberia, BSI has operated on a smaller scale, but has successfully supported a range of useful budgetary reforms, including a reclassified budget structure, a new chart of accounts, the introduction of Budget Policy Notes, improved Sector Working Groups and the introduction of Sector Expenditure Plans. Its work has facilitated greater political engagement in the budget process and increased alignment of the budget to national development priorities. BSI has also helped to build stronger capacity on tax policy within the ministry of finance, which will help the government to mobilise revenues to support its post-Ebola recovery plan. In DRC, BSI has been operating in a limited scale, but has helped the government to develop a prioritised action plan on PFM reform and has helped to build an institutional and regulatory framework to support the planned fiscal decentralisation. This is filling a gap identified by the donor-government PFM coordination group. In Uganda, BSI has supported some promising innovations around budget transparency, including a Ministry of Finance website where citizens can provide feedback on specific line items in national and local budgets. BSI’s support for the g7+ group of states affected by conflict and fragility has included capacity building support for its Secretariat in Dili, East Timor. BSI has helped to put in place basic administrative processes and capacities. The Secretariat is now able to manage the g7+ group’s participation in international events, with minimal external support. For the first time, an important group of fragile states now has a voice in international policy forums. Among other things, it has lobbied successfully for inclusion of the peacebuilding and statebuilding goals into the draft post-2015 Sustainable Development Goals. BSI was influential on the development of the New Deal principles on better international support

iii

for fragile states and has provided some support to the implementation of New Deal processes, such as fragility assessments, although this is not a major focus of its work. BSI’s research programme has produced a range of research and analysis on good aid practices in fragile states and on the challenges of strengthening budget processes in low capacity environments. It has also produced a range of dissemination products, including a well-utilised website. The impact of the research programme on international practice will have to be assessed in the coming years. Overall, we rate BSI’s level of achievement across its five years of operation at ‘A+’ (above expectations). If scored separately, its work on local service delivery in South Sudan would merit ‘A++’. Theory-based evaluation BSI’s theory of change posits that the programme will be able to identify political openings for reform, even in difficult political environments, and position itself as a trusted adviser to its counterparts to take advantage of those opportunities. It holds that reform can best be promoted through a flexible, problem-solving approach, combining strategic and technical advice with hands-on support for implementation. It suggests that, while some interventions will fail, others will deliver catalytic results that justify the overall investment. We find that these assumptions are broadly borne out by BSI’s experience, particularly in its larger engagement in South Sudan. BSI’s approach has proved to be well suited to the stop-go nature of reform in fragile contexts. When the political space for reform in a particular area closes, BSI has proved able to prevent past gains from slipping backwards and to identify other opportunities to move forward. By providing hands-on support to the implementation of reforms, BSI has proved to build constituencies for reform and overcome obstacles. Its work on local service delivery in South Sudan is a particularly good illustration of how a flexible, problem-solving approach can be used in the pursuit of long-term, strategic goals. It has produced catalytic results, in the form of influence on both government expenditure and donor programmes, that represent a strong return on the investment. We have identified a few areas where the BSI model and its delivery processes would benefit from refinement. There is scope for the programme to make more use of political economy analysis and risk-management tools such as scenario planning, in order to ensure it responds quickly to changes in the context. Its approach to individual capacity building is not well articulated or clearly understood by counterparts. Furthermore, the priority given in the theory of change to budget transparency is not fully reflected in BSI’s practice and more thought could be given as to what forms of transparency best support reforms in fragile states. Overall, however, the evaluation supports the proposition that BSI represents an innovative approach to technical assistance that addresses many of the shortcomings of conventional approaches and that has proved well suited to promoting budget reforms in fragile states.

iv

The Advisory Board asked the evaluation to consider BSI’s relevance and contribution to peacebuilding and state-building goals under the New Deal. We find that BSI has a potentially important contribution in a number of ways:

by strengthening ‘revenues and services’ – one of the peacebuilding and state-building goals;

by promoting the use of country systems for aid delivery, one of the TRUST principles;

by helping to facilitate donor-government communication on economic governance and to manage risks around the donor-government relationship;

through its support for the g7+ group’s international advocacy; and

by direct support to New Deal implementation activities, although this has not to date been a priority for the programme.

We note, however, that BSI faces an important strategic choice. If it chooses to focus on developing its model for supporting PFM reforms, this is potentially applicable in any low-capacity environment and need not to be restricted to fragile states. Alternatively, it could focus on the larger strategic questions of the role of economic governance in state-building processes and how to raise the quality of international engagement in fragile states. The latter challenge would provide more opportunity for developing synergies across the g7+ support, the research programme and the country engagements. In our view, the BSI approach is at its most distinctive when it moves beyond technical support for ministries of finance and engages a wider range of stakeholders on the larger questions of economic governance. There are few, if any, other actors who can play such a role effectively. Recommendations

1. In its country programmes, BSI should ensure as far as possible that its technical support to budget reforms makes a strategic contribution to wider economic governance issues (e.g., managing internal and external resource flows; financing service delivery; decentralisation). This may entail engaging with a wider range of counterparts and stakeholders, beyond ministries of finance.

2. BSI should refocus its g7+ work stream towards the larger strategic challenge of improving the quality of international support to fragile states, using its research and analytical capacity to support the international dialogue by helping to turn New Deal principles into practical proposals for improving aid management and delivery.

3. BSI should make more explicit use of political economy analysis to identify strategic risks and opportunities in its partner countries, with a focus on the long-term goal of reducing conflict and fragility.

4. BSI should explore the use of scenario planning and other risk management tools to help manage its engagement in volatile contexts. It should identify the objectives and activities appropriate to different scenarios, to help it to respond nimbly to changes in context. For high-risk work streams, it should identify and monitor the minimum conditions required for a successful engagement.

v

5. Based on its research programme and lessons learned from its country programmes, BSI should work towards a clearer articulation of the challenges of building capacity in low-capacity and volatile environments and develop explicit capacity building strategies for each of its country programmes. It should conclude agreements with its counterparts as to what forms of capacity building support it will provide, including training and mentoring for staff, as appropriate.

6. BSI should work towards a clearer understanding of the role of budget transparency in fragile contexts, including identifying what information needs to be communicated to which stakeholders in order to support budget reforms and better economic governance.

7. BSI and its Advisory Board should focus their attention on developing a sustainable funding model for the programme and how to establish a BSI presence in new countries.

8. The Advisory Committee should appoint a sub-committee or one of its members to look at its own functioning and develop proposals for how to strengthen BSI’s governance.

Contents

1. Introduction .............................................................................................................. 1

1.1 Purpose of the evaluation ............................................................................................................ 1

1.2 Programme overview ................................................................................................................... 1

1.3 Operating context ......................................................................................................................... 2

1.4 Evaluation methodology ............................................................................................................... 3

2. Validation of results................................................................................................... 6

2.1 Inputs ............................................................................................................................................ 6

2.2 Effective delivery of technical assistance ..................................................................................... 9

2.3 Achievement of outcomes – stories of change .......................................................................... 12

2.4 Results against logframe indicators ............................................................................................ 14

2.5 Impact at the country level ......................................................................................................... 16

2.6 Support for the g7+..................................................................................................................... 19

2.7 Research and dissemination ....................................................................................................... 22

2.9 Conflict sensitivity ....................................................................................................................... 24

2.10 Gender equality ........................................................................................................................ 25

3. Theory-based evaluation ......................................................................................... 25

3.1 Is BSI providing high quality and relevant advice? ..................................................................... 25

3.2 How well has the programme managed uncertainty and risk in fragile states? ........................ 26

3.3 How efficiently and cost-effectively has BSI provided its inputs? .............................................. 26

3.4 Are inputs provided by BSI contributing to identifiable institutional changes? ......................... 27

3.5 Is BSI having an effect on Government strategy and policy on PFM reforms? .......................... 28

3.6 How demand-led has BSI been and is the quality of its relationships with counterparts conducive to effective assistance? ................................................................................................... 29

3.7 Has BSI had a catalytic effect? .................................................................................................... 30

3.8 Is BSI having a sustainable impact on budget, policies, processes and systems capacity? ........ 31

3.9 Has BSI increased uptake of good practice and promoted innovation? .................................... 31

3.10 Has BSI increased partner Government capacity and capability sustainably? ......................... 32

3.11 In what ways is the support provided by BSI distinguishable from other providers of TA support? ............................................................................................................................................ 32

3.12 Does increased transparency in budget data help to drive reform? ........................................ 35

3.13 Does BSI contribute to peacebuilding and statebuilding goals? .............................................. 36

4. Conclusions and recommendations .......................................................................... 38

4.1 Conclusions ................................................................................................................................. 38

4.2 Recommendations ...................................................................................................................... 41

Annexes

Annex 1 Validation of AWRs........................................................................................ 45

Annex 2 Validation of stories of change ....................................................................... 54

Annex 3 Achievements against logframe indicators ..................................................... 62

Annex 4 BSI’s Theory of Change .................................................................................. 76

Annex 5 g7+ programme Theory of Change ................................................................. 85

Annex 6 Evaluation Terms of Reference ...................................................................... 92

Annex 7 List of key informant interviews ..................................................................... 97

Annex 8 Documents consulted .................................................................................... 99

Annex 9 Summary literature review .......................................................................... 101

1. Introduction

1.1 Purpose of the evaluation This independent evaluation of the Overseas Development Institute’s (ODI) Budget Strengthening Initiative (BSI) marks the end of its first five years of operation and its core grant from DFID. It is a final evaluation for DFID’s purposes, even though BSI has funding to continue to operate and continues to fundraise actively. It follows a mid-term evaluation by the same team in July 2013. The evaluation has distinct purposes. The first is to validate the results reported by the BSI programme. Under its logframe, BSI measures its outcomes by scoring each of its workstreams on their achievements and recording the number of significant institutional changes that are attributable to its efforts. These outcomes are captured through narrative reports by the programme teams, which are then validated by the external evaluators. The first section of this report provides that validation of results under the logframe. It emphasises accountability. The second purpose is to test the validity of BSI’s claim to provide a new and innovative approach to supporting economic governance and budget reforms in fragile states. BSI takes a flexible, problem-solving approach to promoting reform, known as problem-driven iterative adaptation (PDIA).1 The second part of the evaluation takes a theory-based approach, testing the assumptions underlying BSI’s theory of change in light of the programmes achievements. It is written to support learning, both by BSI’s management and Advisory Board and by other programmes interested in PDIA approaches. The primary audiences for the evaluation are ODI itself and BSI’s funders, both current and prospective. A secondary audience is the wider practitioner community interested in new approaches to reforming economic governance in fragile states. 1.2 Programme overview BSI was designed as an innovative approach to supporting budget and public financial management reforms in countries affected by conflict and fragility (hereafter ‘fragile states’). It is based on the premise that building economic governance capacity in low-capacity and unstable environments calls for a flexible, problem-solving approach, delivered differently than conventional technical assistance (the BSI theory of change is attached as Annex 4). It provides strategic and technical advice on a demand-driven basis – primarily to ministries of finance – focused on budgeting and PFM, the management of aid flows and transparency and accountability. Its arrangements with its funders allow it a higher level of flexibility in its activities and outputs than a typical technical assistance programme. BSI has three main country programmes – South Sudan, Liberia and the Democratic Republic of Congo (DRC) – with some additional activities in Uganda and, in the first two years,

1 Andrews, Matt, Lant Pritchett and Michael Woolcock (2012), Escaping Capability Traps through

Problem-Driven Iterative Adaptation (PDIA). Working Paper 299. Washington, DC: Center for Global Development.

2

Tajikistan. It also supports the g7+ Group of fragile states, which advocates for improvements in international assistance, and in particular its Secretariat, based in Dili, Timor-Leste. Fifteen percent of its core budget (non-earmarked, on average over the five years) is set aside for research and dissemination activities, which are intended to contribute to improved understanding of the challenges of strengthening economic governance in fragile states. 1.3 Operating context BSI is designed to operate in challenging national conditions. Its theory of change is premised on the ideas that fragile states are characterised by weak economic governance, which impedes state-building and the transition out of fragility. They also have low capacity to manage and absorb PFM reforms, giving rise to ‘capability traps’ and challenges of prioritisation and sequencing. Ministries of finance may receive multiple offers of international support, sometimes conflicting at the policy or technical level, and have low capacity to manage the assistance effectively. Institutional reforms may encounter political resistance and are subject to periodic setbacks. Political openings for reform may appear suddenly but also close again unexpectedly. South Sudan has faced repeated political and financial crises, including interruptions to its oil revenues that forced drastic reductions in expenditure and civil conflict since December 2013. South Sudan gained its independent in July 2011, facing a substantial state-building challenge alongside huge development needs linked to 50 years of conflict and under-investment. At the time of independence, government systems were rudimentary and most basic services were provided by donor-funded NGOs. The country has significant oil wealth but oil production have been extensively disrupted by conflict. In December 2013, there was a fresh outbreak of armed conflict, leading to period of withdrawal of many international programmes and a shift of assistance back to humanitarian support. BSI was also unable to visit the country for a number of months. The conflict caused major disruption to budget processes and a substantial breakdown in relations with international donors. This highly volatile context has forced BSI to adjust its activities and level of ambition, focusing at times on crisis management and preventing key budget processes from being abandoned. Some outcomes, including budget discipline and aid coordination, have deteriorated for reasons beyond the programme’s control. South Sudan is by far the largest of BSI’s country engagements. In 2014/15, BSI spend £1.3 million on South Sudan, compared to under £400,000 in Liberia, its next largest engagement. BSI plays the role of principal technical adviser to the Budget Department on the preparation of the annual budget, operating alongside a number of other technical assistance providers. There is a large USAID-funded programme focusing on budget execution and treasury functions, which has been suspended since the evaluation visit. BSI also interacts with a range of other programmes in the service delivery area. It advised the Ministry of Finance on the design of the World Bank’s Local Government Service Delivery Programme (LGSDP), which was the first programmes to attempt to build a funding mechanism for local services through government systems. It also collaborates with a DFID

3

programme providing capitation grants for girls’ education and an EU programme strengthening PFM capacity in local government. Liberia is a post-conflict country that has enjoyed a lengthy period of stability since its 2003 Peace Accord. It has made some important progress towards its Millennium Development Goals, but remains 175th on the Human Development Index.2 President Sirleaf has committed her government to improving transparency and accountability, but corruption remains a substantial concern. The government approved a PFM Reform Strategy in July 2011, which facilitated donors coming together around an Integrated PFM Reform Project, through a World Bank-led multi-donor trust fund (US$28.5 million in commitments) which works to improve budget coverage, fiscal policy management, financial control and oversight. BSI works in a gap-filling mode alongside this much larger programme, focusing on niche areas such as developing a medium-term expenditure framework (MTEF), budget sequencing and budget transparency. However, the 2015 Ebola epidemic triggered a major fiscal crisis in Liberia as a result of revenue shortfalls and unplanned expenditure,3 with significant macroeconomic impact. Alongside other development programmes, BSI was required to withdraw its presence for a period. Liberia is now working to develop a post-Ebola recovery programme. DRC is a challenging environment to promote PFM reform, with uncertain political commitment and institutional responsibilities divided across a number of ministries. As a legacy of long-running conflict, the public administration is highly fragmented, without a coherent management structure or salary.4 The government has a Strategic Plan for Public Finance Reform, agreed in 2010, overseen by a coordinating committee (COREF). Implementation is supported by a multi-donor trust fund on PFM reform, managed by the World Bank with contributions from Belgium and the UK. There has been progress on updating the legal framework for PFM, but relatively little headway on developing a coherent budget and sound budget execution processes. The government still makes excessive use of exceptional budget execution processes. Its Public Expenditure and Finance Accountability (PEFA) assessment places it at the lower end of the performance scale for sub-Saharan Africa. Many observers are pessimistic about the prospects of significant progress on governance reforms in the short term. DRC has made a constitutional commitment to fiscal decentralisation and is in the process of developing the laws and systems that this will require. 1.4 Evaluation methodology The evaluation methodology consists of two main elements (see the Terms of Reference in Annex 6). First, the independent evaluation is given the explicit role in the logframe of validating results reported by the BSI team. To provide it with the flexibility required for a problem-driven, iterative approach, the BSI logframe specifies its activities and outputs in an

2 http://hdr.undp.org/en/content/table-1-human-development-index-and-its-components. 3 “The Economic Impact of the 2014 Ebola Epidemic: Short and Medium Term Estimates for Guinea,

Liberia, and Sierra Leone”, World Bank, September 2014 4 Baudienville, Geraldine, ‘Public financial management reforms in fragile states: the case of

Democratic Republic of Congo’, ODI, October 2012, p. 6.

4

open-ended way. Its first output is ‘high-quality technical assistance to support improved budget policy, process capacity and financial management systems’, which is to be measured through two indicators:

1) the number of annual workstream assessments achieving a rating of ‘A’ or higher according to DFID’s annual review methodology; and

2) the number of significant changes from the catalytic effects of BSI, documented by BSI teams in the form of “stories of change”.

These annual workstream scores and stories of change are then verified by the evaluators, as an independent check on the quality and efficacy of the technical assistance. The verification process involved confirming the accuracy of BSI’s reports, based on written evidence and consultation with counterparts (government officials) and independent observers (including donors and other technical assistance providers working in the same institution). For the annual workstreams, we verified claimed outputs and outcomes and conducted our own scoring, by reference to the objectives and milestones set out in annual work plans and country-specific logframes. This involved our own expert assessment of the quality of outputs, as well as feedback from counterparts. For the stories of change, we verified that the change had taken place and that it counted as ‘significant’ in the operating context. Attribution was tested by whether stakeholders volunteered BSI’s support as a contributing factor to the change in question and whether any alternative causal hypotheses were available. Given that the changes claimed were close in time and institutional location to the BSI support, a simple ‘balance of evidence’ test was considered sufficient to validate the stories of change. We also collected evidence on BSI’s overall results against its logframe. The Reference Group for the evaluation5 stressed the importance of investigating BSI’s impact. This has been done both by exploring the impact indicators in the logframe and by investigating the causal links from BSI’s support towards wider changes in policies, systems and funding flows and their contribution to helping the partner countries overcome conflict and fragility. The evaluation was not, however, designed to test attribution at this level in a rigorous way. The second element of the evaluation methodology is a theory-based evaluation. We answer the eleven questions set out in the Terms of Reference (together with two other questions identified during the evaluation). These explore the causal linkages hypothesised in the BSI theory of change. All of the DAC evaluation criteria are addressed in the report, but owing to the specific requirements of the Terms of Reference for the evaluation, the report is not structured according to those criteria.

5 The reference group was made up of volunteer members of the BSI Advisory Board: Patricia Laverley

and Piper Hart from the AfDB, Jim Brumby from the World Bank, Elisabet Hedin from Sida and Tom Kelly from DfID.

5

The methodology included key informant interviews with BSI management and staff, Advisory Board members and funders (see the list of interviewees in Annex 7). We reviewed a wide selection of BSI documents and reports. The evaluation team visited South Sudan (4 days), Liberia and DRC (2 days each) and Uganda (1 day) in March 2015, carrying out interviews with the BSI team, counterparts (including ministers, senior management and working-level counterparts), donors, other technical assistance providers and a range of external observers. Because all of BSI’s current countries of engagement were visited and because activities are concentrated in ministries of finance, it was not necessary to select a formal sample of BSI activities to review. We validated the annual workstream reports and the stories of change completed in the last years, while also drawing on the findings of the mid-term evaluation in respect of earlier years. To test BSI’s reporting on its achievements, we carried out the following processes:

we carried out detailed interviews with BSI staff to obtain explanations of the objectives and approaches underlying each workstream and how they responded to implementation challenges and changes in the operating context;

we verified these accounts through written documentation, wherever available; and

we conducted interviews with officials from the counterpart institutions, at varying levels of seniority, with donor officials and staff from other development programmes, to assess whether their accounts of institutional changes were consistent with those provided by BSI (triangulation).

To test the proposition that BSI represents a novel approach to technical assistance on PFM, we conducted a brief review of the literature on fragile states to build up a picture of the shortcomings of conventional approaches. We synthesised learning on why donor-sponsored PFM and budget reforms often fail, looking at design and implementation weaknesses. We also looked at the challenges of PFM reform in fragile states. The main lessons from the literature review are extracted in Table 7 on pages 32-32 and a more detailed summary is attached as Annex 9. The literature review, supplemented by other consultations for the evaluation, provide a comparator for assessing the extent to which BSI is innovative and whether its innovations are relevant to the challenges of working in fragile states. There is a consistent enough critique of PFM reform emerging from the literature to make this a useful exercise. However, we were conscious of the risks of caricaturing ‘conventional assistance’, which in practice encompasses a variety of approaches and practices. There are a number of limitations to the methodology. It is heavily dependent on feedback from counterparts. As BSI’s support is a free good for the counterparts, this gives rise to a risk of positive bias. On the other hand, the nature of BSI’s support, working behind the scenes in a facilitative role, means that its influence is not always visible to counterparts. We sought to mitigate bias in feedback by triangulating across different groups of stakeholders (e.g., government officials and donors). There is also a risk that the process of validating results claimed by BSI leads to confirmation bias. In our interviews with counterparts, we tried to guard against that risk by avoiding leading questions.

6

The evaluation was guided by a Reference Group, made up of members of the BSI Advisory Board, who commented on the Terms of Reference and on a draft of the evaluation report. The report was fact-checked by the BSI management team and quality assured by DFID. In a departure from the Terms of Reference, we agreed with BSI management to dispense with preparing an inception report/evaluation plan, as the time available for conducting the evaluation was short and the evaluation methodology was sufficiently described in the Terms of Reference. In another departure from the Terms of Reference, we also decided not to conduct a counterpart survey. BSI had already conducted such a survey and provided us with the responses. DFID’s evaluation guidance suggests that evaluation address cross-cutting themes, such as equity, exclusion, anti-corruption and gender. BSI works primarily with ministries of finance around the development of budget and related systems. While its contribution to strengthening budget systems may make an indirect contribution to these cross-cutting policy goals, they are not an explicit part of the programme design or captured in the logframe. We have therefore not explored these areas individually. The evaluation team consisted of Dr. Marcus Cox, of Agulhas Applied Knowledge, and Ken Robson, an independent consultant specialising in PFM and other public sector reform.

2. Validation of results 2.1 Inputs Funding: The largest share of BSI’s has come from DFID. DFID’s Africa Regional Department provided core funding through an accountable grant of £7 million, initially for three years and then extended to five (with a final 6-month no-cost extension to September 2015). The DFID South Sudan office provided an additional £1.5 million for 2010/11 to 2012/13 and £2.4 million for 2013/14 and 2014/15, which has recently been extended with a further £2.3 million until September 2016. In addition, DFID’s Conflict, Humanitarian and Security Department (CHASE) provided a grant of £300,000 for g7+ support. BSI has recently received additional core funding from Sweden of around £2.2 million for 3 years, to 2016/17, and was recently informed its bid for funding of £833,000 from the African Development Bank’s Transition Support Facility for DRC was successful (though not yet signed). Other funders have included Australia (£563,000 for the g7+ and programme management) and Denmark (£212,000 for g7+), with small contributions from other sources for discrete activities or research products. In 2014/15, BSI spent £3.38 million, with South Sudan its largest programme (£1.32 million), followed by the g7+ (£436,000), Research and Dissemination (£414,000) and Liberia (£393,000).

7

BSI has used its core grant to establish operations in particular countries and then sought additional funding to scale up. The core funding has been key to BSI’s ability to operate in an innovative, flexible manner.6 BSI has raised £8.56 million in funding, above its core grant of £7 million, which is a good fundraising record and, from DFID’s perspective, demonstrates the value for money of core funding for a programme of this kind. However, the uncertain and piecemeal nature of the funding for Liberia and DRC has hampered planning and delivery. Overall, there have been no fewer than 22 separate negotiations with potential funders, which suggests very high transaction costs. DFID’s regional departments no longer fund multi-country activities of this type and the core grant has not been renewed. While SIDA has partially filled the resulting gap, security of funding remains a major risk for the programme going forward. BSI and its Advisory Board will need to consider alternative fundraising strategies. Funds have been spent in accordance with budgets reviewed by BSI’s Advisory Board. Civil conflict in South Sudan and the Ebola crisis in Liberia both required the withdrawal of BSI teams for a period. BSI limited the impact by providing remote support and inviting officials from South Sudan to attend events in Uganda. The South Sudan programme underspent by 9% (£113,000) in 2013-14, but is back on budget in 2014-15. Personnel: BSI draws on a mixture of ODI staff and external consultants to deliver its programmes. In addition to a programme management team of four in London, it has made use of around 40 individuals as short- and long-term advisors. In both this and the mid-term evaluation, feedback from counterparts and other stakeholders on the quality of BSI’s personnel has been highly positive. The box below contains a selection of quotes from counterparts and stakeholders collected by the evaluators. These closely correspond to feedback collected by BSI itself through beneficiary surveys. The consensus is that BSI advisers offer a combination of strong technical skills with the ability to win the confidence and trust of counterparts. Stakeholders particularly valued the advisors’ ability to broker agreements and facilitate coordination across government agencies and between governments and donors. BSI has done well in identifying advisors with existing high-level contacts in each of its countries, which has been key to enabling the teams to take a strategic advisory role. BSI also draws considerable strength from its location within ODI, giving it access to a broad range of expertise on economic governance. It makes effective use of alumni from the ODI Fellowship scheme, who bring excellent knowledge of the workings of ministries of finance. BSI has also made a point of sharing personnel between its research and advisory work, to maximise the sharing of lessons and experience. BSI has also been able to mobilise additional advisory support quickly and flexibly when the need arose. For example, it recruited a Natural Resources Tax Advisor for Liberia and a Revenue Advisor for South Sudan on short notice.

6 David Booth has categorised BSI as an ‘arms’-length’ programme, given the freedom by its funders to

operate in a flexible, problem-solving mode in pursuit of agreed goals. David Booth, “Facilitating development: an arm’s length approach to aid”, ODI Discussion Note, March 2013.

8

Table 1: Quotes from counterparts and stakeholders on BSI

“timely and extremely helpful”

“absolutely necessary and essential”

“highly relevant and instrumental”

“highly effective new form of TA”

“very strong at ensuring the engagement of stakeholders”

“Some of the best work that is going on here.”

“BSI’s budget work is strategic; it is used and well appreciated”

“The support provided by ODI’s Budget Support Initiative is outstanding. BSI enjoys the trust of the ministry.”

“BSI’s support has been incredibly important”

“These guys are serious and technically competent and highly relevant to what we’re trying to achieve.”

“Motivated, engaged, enthusiastic, smart”

“BSI helped to accelerate the process. Without BSI, we would not have succeeded.”

“at times aggressive” but “bring a great deal of drive”

“BSI can be preachy at times, pushing for things that are unrealistic”

“BSI’s early pressure on use of country systems was going too far, too early, but we understood the reasons for it”

“hard to communicate with them when they constantly come and go”

“we don’t like the short missions”

Many of its advisers are ‘peripatetic’ – that is, based outside the country and making repeated visits over the course of the year. There are pros and cons to this arrangement. It enables BSI to engage high-calibre advisors who would not be willingly to relocate permanently to the countries in question. The intermittent nature of the support also helps to mitigate the risk that BSI advisors will fall into a capacity substitution role by taking on routine responsibilities for the counterparts. It also enables BSI to minimise its overheads (see under Economy and Efficiency below). On the other hand, there was feedback to the effect that the quality of communication and continuity of effort suffer during the advisors’ periods of absence. On balance, BSI has made this pattern of engagement work well, with the advisors able to be flexible and responsive to client demands in planning their visits. During the Ebola crisis in Liberia and the period of insecurity in South Sudan, BSI managed to sustain some continuity of support. It provided remote advice, hosted a South Sudan Ministry of Finance retreat in Kampala and met with the Liberian Minister of Finance in Washington. In both cases, the remote support helped to preserve the integrity of the budget process through a very difficult period. While the budget timetables were truncated and some processes compressed or abandoned, the basic logic of the budget sequence was retained and, with BSI’s advice, conscious decisions were made as to what could be done in the time available.

9

Economy and efficiency. In the mid-term evaluation, we made a comparison of BSI’s fee rates with other technical assistance programmes, to assess economy. We found that BSI’s rates were similar to those paid under DFID Framework Agreements for mid-level and senior consultants, while BSI junior consultants were cheaper. However, that tells only part of the story. BSI’s use of part-time consultants entails considerable cost-savings, in terms of relocation and housing. It is able to hire vehicles as needed, rather than maintaining a permanent fleet. It operates out of government premises, rather than maintaining permanent country offices. Because the consultants are available to respond quickly and flexibly to client requests even during their periods away, BSI gets very good value from its personnel. BSI also sources consultants from within the region or makes use of retired public servants, which reduces costs. Its management overheads are forecast at 10% in 2014-15 (12% in earlier years). While overhead is difficult to compare directly between programmes, owing to differences in what is measured, in our experience this as at the low end for technical assistance programmes, which are usually at 15% or above. For these reasons, we assess BSI to be a lean and efficient operation. Governance arrangements. BSI has an Advisory Board, made up of its funders (DFID; SIDA), a number of other development partners (the World Bank; the International Monetary Fund), representatives of the partner countries and some independent experts. The Advisory Board meets via video conference every six months. It receives BSI’s activity and financial reports and provides guidance on strategic decisions. The Advisory Board is potentially a very useful resource, with very knowledgeable individuals. However, the format does not lend itself to a high level of input from them. There is too much documentation for members to absorb and too little time for substantive discussion. We also note that participation by country representatives in the overall Advisory Board does not lend itself to particularly robust oversight of the country programmes. For large country engagements like South Sudan, a country-level board or steering committee might be more effective. In addition, the Advisory Board has not been particularly effective at helping BSI with its fundraising or facilitating entry into new countries. 2.2 Effective delivery of technical assistance Under its logframe, BSI’s main output is high-quality technical assistance,7 measured by the number of annual workstreams achieving ratings of ‘A’ or above.8 In the mid-term

7 Although this is formulated as an activity, rather than an output, the AWRs are scored against the

delivery of expected outputs. 8 In its annual reviews, DFID scores projects on the following scale: A++ = Outputs substantially

exceeded expectation; A+ = Outputs moderately exceeded expectation; A = Outputs met expectation; B = Outputs moderately did not meet expectation; and C = Outputs substantially did not meet expectation. DFID, “How to note: Reviewing and Scoring Projects”, November 2011, p. 7,

10

evaluation, we validated ten annual workstream reviews (AWRs), of which nine received scores of ‘A’ or better. For this evaluation, we reviewed a further 15 AWRs for 2013-14 and 2014-15, including six for South Sudan, three for Liberia, two each for DRC and Uganda and two for the g7+ support. For each, we validated that the AWRs were a fair statements of the achievements of the workstream for the year, based on documentary evidence and counterparts interviews, and then assessed the level of achievement against the objectives set out in annual work plans, adjusted as necessary to respond to in-year events and opportunities. More detail on the scoring of each AWR is set out in Annex 1. Table 2: Scoring of Annual Work Stream Reviews

Annual Work Stream Reviews BSI score Our score

Mid-term evaluation

Liberia Support on budget process and policy reform (2011/12) A++ A+ Support on budget process and policy reform (2012/13) No score A

South Sudan Support on aid coordination (2011) A++ A++ Support on budget policy reform (2011) B B Support on PFM reform (2011) A++ A+ Support on the planning and budgeting process (2012) A A Support on aid coordination (2012) A++ A+ Support on peer learning (2012) A A Support on PFM reform (2012) B A Support to the Presidency (2012) A A

Final evaluation

Uganda Support to Budget Transparency and Accountability (2013-14) A+ A+ Support to Budget Transparency and Accountability (2014-15) A+ A++

South Sudan Support on the Planning and Budgeting process (2013-14) A B Support to Government Led Delivery of Local Services and Infrastructure (2013-14)

A++ A++

Support on PFM reform (2013-14) A B Planning, Budgeting and Aid (2014-15) A++ A+ Government-led Delivery of Local Services and Infrastructure (2014-15)

A++ A++

PFM Reform (2014-15) A++ A

Liberia Strengthening the budget process in basic education (2011-12) A Not scored Support on Citizen’s Guide to the Budget and Communications Strategy (2013-14)

A A

Support on budget policy and reform (2012-13) A Not scored Support on budget policy and process reform (2013-14) A A Support on budget policy and process reform (2014-15) A+ A

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67344/HTN-Reviewing-Scoring-Projects.pdf.

11

DRC Design and Agreement on new BSI Programme to support implementation of PRM reforms

A Not scored

Country Director for DRC Programme (2013-14) A A Advice and support to COREF (2014-15) A A

g7+ Support to the g7+ Secretariat 2013-14 A++ A++ Support to the g7+ Secretariat 2014-15 A++ A+

Of the 15 new AWRs reviewed for this evaluation, four were scored at A++, three at A+, six at A and two at B. The two South Sudan workstreams that scored B, for below expectations, were both affected by the conflict and resulting fiscal crisis, which forced the abandonment of planned activities. The logframe sets cumulative targets for the number of AWRs scoring ‘A’ or above, across four output areas (see Table 2). BSI has delivered at well above expectations in its two major areas of engagement – budget policies and PFM systems; and revenue management/service delivery. It has delivered to a high standard for its g7+ support and met expectations on transparency and accountability (we have some concerns about the relevance of this output area, which we come back to on pages 28-29). Table 3: Annual Workstream Reviews scoring ‘A’ or above

Logframe output areas

AWRs scoring ‘A’ or above

2015 target Results 1. Improved budget policy, process capacity and financial

management systems in Ministries of Finance 9

15, of which 12 validated

2. Improved management of domestic revenue and external resources by Ministry of Finance, and to support the establishment and implementation of processes and systems for coordination, financing, management and monitoring of government-led delivery of local services

3 6, all validated

3. Improved transparency and accountability in government budgets

3 3, all validated

4. More effective international engagement of fragile states being facilitated through the g7+ Group of Fragile States

3 3, of which 2

validated

The strong scoring of the AWRs is borne out by our overall assessment of the advisory outputs. BSI’s technical proficiency was highly appreciated by all counterparts. The written products we reviewed were of consistently high quality. They range from high-level strategy documents (e.g. the National Budget Plan and Budget Speech in South Sudan) to more technical products (e.g., budget calendars and budget call circulars in Liberia and South Sudan, a note on Presumptive Tax in Liberia and local government finance and human resource manuals in South Sudan). They are consistently succinct and well written, presenting technically complex issues in a straightforward way. Where appropriate, they

12

contain clear recommendations on the decisions and follow-up actions required. The majority of the documents are produced collaboratively, whether working jointly with counterparts on a one-to-one basis or through working groups made up of BSI advisors, other technical assistance providers and government officials. Most of the documents do not contain any BSI or ODI branding and outputs are often incorporated into official government documents as policy or guidelines. This willingness to work anonymously and behind the scene is an important element in BSI’s policy influence. 2.3 Achievement of outcomes – stories of change BSI also tracks its results through stories of change, which capture the outcomes of its assistance in the form of changes in policy, attitudes, behaviours, processes or content. To give the programme the freedom to work in a flexible, problem-driven way, these changes are not specified in advance in the logframe. Instead, BSI commits to achieving a certain number of significant changes in each of its output areas. Stories of change are produced by the programme teams and verified by the evaluators on a sample basis, both as to their factual accuracy and as to their significance in the context in which they occurred. In the mid-term evaluation, we examined 12 stories of change and verified nine of them as significant outcomes. For this evaluation, we examined a further 12 stories of change. One of them we were not able to assess, due to a lack of evidence. Of the remainder, six were assessed as significant changes (see Table 3). The changes include:

strengthening local service delivery in South Sudan by clarifying roles and responsibilities, creating systems for financial and human resource management at the local level and developing a system for inter-government transfers, including monitoring and oversight mechanisms;

strengthening the budget process in Liberia, including a revised budget calendar, establishing a Budget Working Group and strengthening budget hearings, resulting in more meaningful political engagement and stronger alignment of the budget with development priorities;

the introduction of a Citizen’s Guide to the Budget in Liberia, produced over three successive years, with an associated dissemination strategy;

support to the establishment of the Revenue and Tax Policy Division in the New Ministry of Finance and Development Planning and the amalgamation of the Liberian ministries of finance and development policy; and

the development of a new budget website in Uganda and other transparency measures.

13

Table 4: Results of validation of Stories of Change

Story of Change Significance

South Sudan Budget Planning and Aid Coordination (2013-14) Not assessed Government Led Delivery of Local Services and Infrastructure (2013-14)

Significant

Public Financial Management Reform (2013-14) Not significant Public Financial Management Reform (2014-15) Not significant Changing Behaviour in the Budget Directorate (2014-15) Not significant Rebuilding systems in a state of fragility (2014-15) Not significant More government funding of local services (Local Services Support) (2014-15)

Significant

Liberia Citizen’s Guide to the Budget and Communications Strategy (2013-14)

Significant

Developing strategic budgeting processes (2013-14) Significant Revenue (2014-15) Significant Improved budget preparation and execution during and after the Ebola crisis (2014-15)

Not significant

Uganda Making budget information accessible (2014-15) Significant

The logframe set targets of 17 significant changes achieved across the three output areas. BSI prepared a total of 26 stories of change, of which 14 were verified, 7 were assessed not to be significant and 5 were not assessed by the evaluators. This leaves BSI close to or just above its logframe target (depending on whether non-assessed stories of change are counted). Table 5: Significant changes against logframe targets

Logframe output areas

Significant changes

2015 target Verified 1. Improved budget policy, process capacity and financial

management systems in Ministries of Finance 9

6 verified 4 assessed as not significant 3 not assessed

2. Improved management of domestic revenue and external resources by Ministry of Finance, and to support the establishment and implementation of processes and systems for coordination, financing, management and monitoring of government-led delivery of local services

6 6 verified

1 not verified

3. Improved transparency and accountability in government budgets

2

2 verified 2 assessed as not significant 2 not assessed

14

We are not, however, unduly concerned by this result, for two reasons. First, as we noted in the Mid-Term Evaluation, the question as to what counts as a significant change is somewhat arbitrary. Where there has been gradual progress in a particular area over several years, it is often unclear whether this should be scored as a single, large change or a series of smaller changes. Second, the scope for sustainable institutional change was limited during the crises in South Sudan and Liberia over the past 18 months. Some of BSI’s planned activities had to be abandoned in favour of hands-on support to protect the integrity of the budget processes. Over this period, BSI has helped to deliver important preventative results – namely, ensuring that basic budget processes and behaviours were not abandoned under crisis conditions. Stakeholders in both countries confirmed the significance of this results, which should make it easier for both countries to recover from the crises. DFID South Sudan, in particular, regards BSI as an important investment in risk management for its country engagement as a whole. Though not captured in the logframe, these preventative results are fully consistent with BSI’s theory of change. 2.4 Results against logframe indicators In this section, we assess BSI’s results against the outcome and impact indicators in its logframes. BSI has an overall logframe, plus logframes for individual country and thematic programmes. In our mid-term evaluation, we noted the difficulty that BSI faces in finding objective indicators to measure its impact on budget processes. The original logframe had used international indices such as the World Bank CPIA score,9 the Mo Ibrahim Index and the Open Budget Index. These are not updated with the frequency required for programme monitoring and in any case were unlikely to change significantly within the timeframe of the programme. At outcome level, the old logframe made use of individual indicators from the PEFA Public Financial Management Performance Measurement Framework,10 but again faced challenges with the regularity of assessments. Furthermore, the indicators were at too high a level to be useful outcome measures for what are often quite small, focused interventions by BSI, operating alongside other actors. Since the mid-term evaluation, BSI has revised its logframes, reducing its use of international indices and selecting as outcome indicators individual PEFA measures that are more likely to be responsive to BSI’s interventions. Table 6 summarises the results for the programme as a whole at impact and outcome levels (a complete set of logframe results are included as Annex 3). At impact level, there has been some slight improvement in fiscal discipline but no aggregate increase in the proportion of budgets being spent on development priorities. At output level, the overall quality of the budget processes remains static, there has been a slight improvement in budget

9 Country Policy and Institutional Assessment: http://data.worldbank.org/data-catalog/CPIA. 10 PEFA Secretariat, “Public Financial Management Performance Measurement Framework”, January

2011 edition, http://www.pefa.org/sites/pefa.org/files/attachments/PMFEng-finalSZreprint04-12_1.pdf.

15

transparency but the quality of expenditure data provided by donors has gone backwards, as a result of a deterioration in donor-government relations in South Sudan. Table 6: BSI performance against logframe impact and outcome indicators

Goal Indicator Target Result

Impact

Improved economic governance in partner fragile states

Greater allocative efficiency, in the form of an increased share of public resources going to social spending priorities

Improvement in 3 countries

Not achieved. Not measured in Liberia. Slight regression in DRC. Regression in South Sudan.

Improved fiscal discipline, in the form of reduced variation between approved budgets and actual expenditure (PEFA Indicator PI-2)

Improvement in 2 countries and no deterioration in a third

Achieved. Liberia has progressed from D to D+, South Sudan has remained on D+ but with some improvement in component measures, while DRC has remained unchanged at D.

Outcome

More effective, transparent and accountable budget policies, processes and systems in the poorest and most fragile states

An orderly budget process with political involvement (PEFA indicator PI-11

Maintain a positive trajectory in Liberia and South Sudan

Not achieved. Both countries have remained at the same level.

Transparent and accountable budgets, measured by Open Budget Index (maximum=100)

Maintain a positive trajectory in Liberia and South Sudan

Partially achieved. Liberia improved from 40 to 49 (BSI estimate); South Sudan: no data.

Quality of financial information provided by donors for budgeting and reporting on project and programme aid, measured by PEFA performance indicator D-2

Maintain a positive trajectory in Liberia and South Sudan

Not achieved: Liberia: no improvement from D+ (2012); South Sudan: deterioration from C in 2012 to D.

Number of countries implementing New Deal activities

At least 3 fragile states introduce new deal approaches

Achieved: 6 fragility assessments, 3 compacts and a range

16

of other processes underway

In the individual programme logframes, where the indicators are more directly relevant to activities in each country, the picture is more positive:

i) South Sudan: Overall fiscal discipline has not improved significantly but there has been an increase in the funds transferred to subnational government for service delivery. The budget process is more transparent and BSI has successfully encouraged the first major donor programme to deliver through country systems.

ii) Liberia: Fiscal discipline, the use of country systems by aid programmes and the quality of expenditure data provided by donors all remain static. There have been improvements to aspects of the budget process (the budget calendar and the setting of budget ceilings), while budget transparency has improved.

iii) DRC: There has been no improvement in allocative efficiency, budget reliability data

is unavailable and as much as 40% of expenditure remains outside the budget process. The annual budget is now published. BSI has helped to prepare a Priority Action Plan for PFM reform, contributing to the preparation of a number of important PFM laws and regulations and helped to put in place necessary preparatory actions for fiscal decentralisation.

Despite the revisions to the logframe, we continue to have a number of concerns about the appropriateness of some of the indicators and, more broadly, about the usefulness of the logframe format for capturing the results of this type of programme. There are objective reasons, beyond the influence of the programme, why budget processes are slow to improve and at times deteriorate. Piecemeal progress punctuated by regular setbacks is to be expected in fragile states, and indeed is anticipated in BSI’s theory of change. The programme has nonetheless been successful at finding opportunities to work around obstacles and identify areas where progress is possible. Where necessary, it has shifted its focus to preventing past gains from being lost. These kinds of results are not easily captured through pre-defined indicators. 2.5 Impact at the country level This section summarises what we see as the most important results from BSI’s work across its three main countries of engagement. South Sudan is by far the largest of the programmes and has achieved commensurately greater results. The four years of support has, however, been punctuated by a succession of fiscal and political crises that have resulted in setbacks and required frequent course corrections.

Budget process: BSI has been instrumental in designing and establishing many of the core processes for the annual budget cycle. It developed budget and planning guidelines that have been broadly followed, despite repeated fiscal crises. It has

17

helped to prepare national budget strategies, showing how to translate government policies into the budget. It has provided mentoring and some formal training to budget officers, who can now manage some routine budget processes without external assistance. It has helped to protect the regular budget process through successive fiscal and political crises. The former Minister of Finance describes BSI’s support as “absolutely necessary and essential” to putting in place a credible annual budget and planning process. However, the results need to be qualified in a number of respects. The budget is yet to become a tool for strategic resource allocation. Political conditions have prevented the introduction of the necessary austerity measures. Furthermore, budget discipline is weak and has deteriorated as a result of conflict. The structure of the Ministry of Finance is unreformed and many staff lack basic qualifications. In the absence of a normal budget cycle, it is difficult to determine at this point what level of capacity has emerged. During the absence of the BSI team, Ministry staff produced a supplementary budget with only remote assistance, but according to the then Finance Minister this was “at a terribly reduced pace and quality”.

Local service delivery: BSI first embarked on the development of a new aid instrument, by which donors could fund local service delivery through country systems. Following the breakdown in government and donor relations after the 2013 outbreak of conflict, it shifted focus to developing the government’s own systems for funding local service delivery. It helped to clarify responsibilities, develop PFM and human resource management guidelines for local government and introduce a system of intergovernmental transfers. It achieved the inclusion these transfers in the budget, with funding so far released as planned. Without BSI’s strong brokering and technical skills, sustained over several years, these advances would not have occurred. This result is of considerable strategic significant in the South Sudan context, creating for the first time a credible mechanism for donors to support local service delivery through country systems. One World Bank programme is now delivered through country systems, while DFID has ‘shadow aligned’11 its education support to the government’s system for capitation grants to schools. While it is too early to look for improvements in services, we heard from the Ministry of Education that, with regular capitation payments, schools were no longer levying as many ad hoc fees on pupils, which had had an immediate effect in reducing drop-out rates. Many stakeholders confirmed that this is the one area of economic governance in South Sudan that has made significant progress despite the crises.

Strategic advice: BSI has provided strategic advice to successive ministers of finance and other senior officials on public financial management reform, budget composition, the management of austerity, domestic revenues, borrowing and the handling of donor relations (including relations with the IMF). This is a difficult area

11 I.e., the support uses the same principles and procedures for allocation to schools, even though the

funding is channelled outside government.

18

in which to assess results. It appears, however, that BSI has helped to make the Government of South Sudan better placed to formulate its needs and priorities and to negotiate with the international community. It has helped to advance the PFM reform programme in a variety of areas. It has recently helped to build consensus on the need for a National Revenue Authority. Development partners in Juba clearly value the role played by BSI in facilitating better communication with government, which they see as an important risk-mitigation measure in a difficult political climate.

Aid management: In the early years of the programme, BSI made some important progress on capturing aid expenditure data on the budget. It supported a range of donor-government dialogue and coordination processes, including the preparation of a draft New Deal ‘compact’. Most of the progress, however, lapsed as a result of the outbreak of conflict in 2013. Attempts are now underway to revitalise the initiative.

The Liberia programme is smaller, with a narrower remit. The country’s PFM reform process is more defined and is supported by a multi-donor trust fund, leaving BSI to work in a gap-filling mode. There have nonetheless been some useful results.

Budget process: BSI’s initial remit was to help develop an MTEF. While the MTEF is only partially effective, the work has provided an entry point for other useful results, including a reclassified budget structure, a new chart of accounts, the introduction of Budget Policy Notes, improved Sector Working Groups and the introduction of Sector Expenditure Plans. BSI has supported the introduction of a budget options paper for submission to Cabinet and promoted a stronger challenge function in budget hearings, which has strengthened political engagement. BSI facilitated the establishment of a Budget Working Group to help align the budget to national development priorities. These contributions have helped to make the budget more strategic in orientation. The Ebola crisis forced the evacuation of BSI staff between August and December 2014. BSI was able to provide some technical support remotely, to enable the budget process to continue with a truncated timetable. BSI provided advisory support throughout the crisis and, although it is difficult to attribute any specific results to this, the feedback was positive.

Revenues: BSI supported the amalgamation of the former ministries of finance and development planning. In particular, it supported the establishment of the Revenue and Tax Policy Division in the new Ministry of Finance and Development Planning, to develop strong analytical capacity to work on tax policy. Increased revenue generation will be an important part of the government’s post-Ebola stabilisation and recovery plan.

Transparency: BSI has supported the introduction of Citizen’s Guide to the Budgets, which present the budget in an accessible form to support transparency and accountability. The first two editions were produced by BSI staff while a third has now been produced by ministry staff, with an associated dissemination strategy.

19

Aid management: In earlier phases of the support, BSI developed a new structure and staffing plan for the Ministry of Finance’s Aid Management Unit and helped to align aid data to the new budget structure.

The DRC programme has been operating on a very limited scale, with two embedded advisors and occasional visits from the country director and other external experts, pending more substantial funding. It provides support to COREF, the coordinating committee for PFM reform. It has provided capacity building to COREF staff, helping to make it a more capable and credible organisation. It has helped COREF to update and elaborate a Priority Action Plan for PFM reform. It has provided high quality technical advice on the legal and institutional framework needed for the government’s planned decentralisation programme. 2.6 Support for the g7+ Though not part of the original design of the programme, BSI has provided a substantial amount of support over the past four years to the g7+, a self-selected block of states affected by conflict and fragility. First convened in 2010, the g7+ engages in mutual support and joint lobbying for improved international support. The g7+ support has cost an average of £400,000 per year over the past three years, with grants from AusAID (now DFAT), Danida, the World Bank and DFID. BSI has provided capacity building support for the g7+ Secretariat, together with research and peer learning, international engagement and support for New Deal implementation activities in particular countries. Capacity building: Recognising the very limited capacity within the g7+ Secretariat in Dili, East Timor, BSI initiated a programme of capacity-building support. It seconded a full-time project officer to Dili, who had the language skills to support communication with francophone and lusophone members and also provided administrative support. Other BSI staff and consultants provided ‘surge’ support, travelling regularly to Dili and helping to organise international meetings and events. ODI also facilitated the hiring of two g7+ nationals, in part using funds from Danida earmarked for this purpose, to boost the Secretariat’s capacity in policy development and administration. ODI helped to put in place basic administrative systems, creating a website, establishing regular communication among g7+ members and introducing procedures for managing meetings (e.g. the preparation of agendas, room documents and presentations). The Secretariat now has a permanent staff of 10, the remaining 8 of which are funded from a grant from the Government of Timor-Leste. As capacity in the Secretariat has developed, ODI has gradually pulled back from direct support to these core administrative and logistical functions. Feedback from the g7+ on BSI’s support has been consistently very positive. The g7+ General Secretary described it to the evaluators as “incredibly important” to the development of the g7+. He confirmed that the Secretariat has now run major events, including two side events at the UN General Assembly, without logistical support from BSI. Furthermore, the communication systems and practices introduced by BSI means that the g7+ is now able to formulate common positions on areas of mutual interest and

20

communicate them to international partners. The General Secretary stated that, while he would like BSI support to continue, it can now focus on policy and research inputs, rather than capacity building. Research support: BSI supports the preparation of various research outputs, which are published by the g7+ Secretariat. They have included support for the preparation of a ‘New Deal Progress Report’ (published by the International Dialogue on Peacebuilding and Statebuilding), a handbook on natural resource management across g7+ members, some learning notes on New Deal implementation activities and a set of ‘New Deal Innovation papers’ (still under preparation) showcasing innovative donor behaviour. While the research outputs have been useful and relevant, there is as yet no evidence of them leading to an increase in peer learning across g7+ members, although this may evolve over time. International engagement: BSI support for g7+ international engagement has been extensive, including informing the g7+ about key international dialogue processes, brokering relationships between the g7+ and international organisations and helping the g7+ members to formulate common positions. The dialogue has been marked by some tension, as the g7+ has pushed for major shifts in aid practices (such as a move to budget support) that donors have been reluctant to accept. We had some feedback from external observers to the effect that ODI may have been at times too visible and assertive in promoting this agenda. However, BSI documents show a recognition of the importance of stepping back and allowing the g7+ to develop its own voice. Beginning from the preparations for Busan and the formulation of the New Deal itself, stakeholders affirm that BSI played a major role in establishing the g7+ in international dialogue. It helped to secure the inclusion of peacebuilding and statebuilding indicators in the post-2015 sustainable development framework (“Goal 16”). It also alerted the g7+ to proposed changes in the World Bank’s IDA allocation formula and helped it to formulate a common position at the IDA17 replenishment negotiations. At the negotiations, the decision was taken to increase the resources available to fragile states through a change in the allocation formula and the introduction of a dedicated funding window for transition countries. These changes were reportedly under preparation for some time and would likely have taken place without the g7+ advocacy. It was, however, important for the development of the g7+ itself that it has a voice in the process. New Deal implementation: BSI has also provided research and advisory support for New Deal implementation activities in a number of countries. It advised Sierra Leone on piloting a New Deal fragility assessment, Liberia with various New Deal processes, Afghanistan on an Aid Management Policy and Mutual Accountability Framework and South Sudan on a draft New Deal compact (abandoned following the outbreak of conflict). BSI has chosen not to link this work stream to its country programmes. It took the view that ODI promotion of the New Deal would undermine country ownership. While this is a defensible position, it means that the g7+ workstream still feels somewhat disjointed from the rest of BSI. Stakeholders also observed that neither the g7+ nor BSI had been particularly strong in advocating for New Deal implementation, which appears to have stalled somewhat in the face of limited support from donors.

21

Overview of results: Overall, the g7+ has been successful at the output level. The support has been flexible and tailored to the needs of the counterpart. Sustaining capacity building support to the Secretariat over several years, while gradually withdrawing from direct administrative and logistical support, has proved a successful strategy. The Secretariat now has significantly greater capacity at the organisational and individual levels, making use of embedded advisers from g7+ countries, enabling BSI to move towards an advisory role. The first output in the logframe is therefore achieved. At the outcome level, the g7+ has, in a relatively short space of time, given a voice to a self-selected group of fragile states willing to engage with the international community on the challenges of conflict and fragility. Given the history of many fragile states refusing to discuss peacebuilding and statebuilding, in order to protect their national sovereignty, the existence of the g7+ and accompanying dialogue is an important step forward. The g7+ has become a recognised grouping that is regularly invited to participate in key international forums and is able to articulate common positions and advocate on shared interests. The achievement of the second output (“2 significant changes in international policy due to g7+ engagement”) is substantially achieved through the adoption of the New Deal itself and the inclusion of peacebuilding and statebuilding goals in the draft Sustainable Development Goals. Wider impact on the quality of international support to fragile states is, however, still uncertain. The New Deal provides a potentially useful normative framework, and a range of implementation activities have been undertaken in Afghanistan, Comoros, DRC, Liberia, Sierra Leone, Somalia, South Sudan and Timor-Leste. (In its logframe for BSI’s g7+ work, DFID set a target of at least 3 fragile states implementing New Deal activities, which has been exceeded.) The record on implementation is mixed, however. The participating countries have struggled to integrate peacebuilding and statebuilding goals into their national planning frameworks, and progress on increased use of country systems to deliver aid and coordinated capacity-building support from donors has been limited.12 Most donors have not engaged actively with the New Deal and there are signs of a trend back to political conditionality or disengagement when faced with setbacks. While this is due to many factors beyond BSI influence, it is clear that the New Deal is yet to establish its credibility as a new paradigm for assistance to fragile states. However, the problems the New Deal set out to address remain live and the New Deal remains the only viable normative framework available. Modest funding to ODI to provide ongoing policy and advisory support therefore still looks like a strategic investment. With the g7+ Secretariat now substantially established, ODI has an opportunity to refocus its efforts towards the wider problem of raising the quality of international support to fragile states, using a more PDIA-type approach. We come back to this in the final chapter.

12 International Dialogue on Peacebuilding and Statebuilding, “New Deal Monitoring Report 2014”,

November 2014.

22

2.7 Research and dissemination BSI reserves 15% of its core budget (on average, over the life of the programme) for research on the challenges of strengthening economic governance in fragile states, and for associated dissemination. The research explores three themes:

i) How does budgeting work in fragile states? ii) How do reforms happen, and how can they be made to work better?

iii) How can external actors enable and support positive change?13 The research explores the theme that conventional approaches to PFM reform, based on importing international best-practice models, are not effective in fragile, low-capacity environments. It expands on work done by Professor Matt Andrews, of the Harvard Kennedy School, and others on how to use problem-driven approaches to overcome the poor record of traditional technical assistance programmes.14 As an independent research institute that is not beholden to an existing body of practice, ODI is well placed to take on a research agenda that challenges conventional donor programming. It also benefits from the substantial expertise available in ODI’s Centre for Aid and Public Expenditure, with its team of around 30 researchers. The research is intended as a contribution to the global public good of knowledge and understanding of this area, and is therefore addressed to the wider practitioner community. While relevant to BSI’s in-country assistance, it is not directly responsive to requests from country programmes or counterparts. As major research projects have long lead times, they are not suited to answering policy questions arising from BSI country programmes in real time. The research programme does, however, include a range of secondary products, such as syntheses of available knowledge, briefing papers and practice notes on particular issues or initiatives, which have immediate practical value. There is cross-over of team members between research and in-country advisory roles, which helps to ensure mutual learning. In addition, one final research output will be a summary of lessons learned from BSI implementation. The research and dissemination work started relatively slowly in the first two years. A dedicated head of the research programme was then appointed in 2012-13, a research and dissemination strategy adopted and multi-year research programmes were initiated. The workstream is now on track to use all its resources, having been given a no-cost extension to September 2015. BSI set itself the goal of producing 12 substantive research papers by September 2015. (The original logframe target was much lower, and this was revised upward following the mid-term evaluation). So far, it has produced the following:

13 BSI Research Strategy, November 2012. 14 Andrews, Matt, Lant Pritchett and Michael Woolcock (2012) Escaping Capability Traps through

Problem-Driven Iterative Adaptation (PDIA). Working Paper 299. Washington, DC: Center for Global Development.

23

The Road to the New Deal (2010): a set of working papers written for the International Dialogue on Peacebuilding and Statebuilding to facilitate preparations for the Busan High Level Forum;

Innovative aid instruments and flexible financing: providing better support to fragile states (2012);

Measuring good pooled funds in fragile states (2012);

Managing natural resource revenues: the Timor-Leste Petroleum Fund (2013);

Special fiscal institutions for resource-rich developing countries (2013);

Linking public financial management dimensions to development priorities (2013);

Unblocking results: using aid to address governance constraints in public service delivery (2013); and

Change in Challenging Contexts: how does it happen? – a summary of lessons learned from BSI’s implementation.15

In addition, the following products are in the pipeline.

Basic enough budgets (available in draft form);

From capacity substitution to capacity growth (available in draft form);

Escaping capability traps – a conceptual framework for understanding organisational capacity (available in draft form); and

Capabilities of finance ministries (draft launched in South Africa in collaboration with CABRI in March 2015).

The programme is therefore substantially on track to meet its targets. In addition, there have been a range of other synthesis or knowledge dissemination products, including a literature guide on PFM reforms (so far the most popular output, in terms of downloads), a major international conference, teaching at Harvard Kennedy School, a series of country learning notes and the regular ‘Beyond Budgets’ blog (although there have been no entries since August 2014). All its outputs are published on a dedicated BSI website,16 as well as on the main ODI website. The major research outputs are published by ODI and go through ODI’s regular quality assurance process. All products are reviewed by one internal and one external peer reviewer, prior to publication. In our view, the research products are of very good quality. They are also highly relevant to the wider aims of BSI in promoting new approaches to building economic governance in fragile states. The earlier research products helped to provide the conceptual underpinnings for the New Deal and challenged development partners to think through how to adapt their instruments and practices to fragile contexts. The work on managing natural resource revenues is relevant to many of Africa’s fragile states and the challenge of escaping the ‘resource curse’. The paper Unblocking results: using aid to address governance constraints

15 The synthesis paper is entitled “Change in Challenging Contexts” and is was launched on 14

September 2015: http://www.odi.org/publications/9877-change-challenging-contexts-does-happen. 16 http://www.budgetstrengthening.org/.

24

in public service delivery contains an extremely useful conceptual framework for diagnosing blockages on service delivery,17 which has helped to inform BSI’s work in South Sudan. The forthcoming research will help to unpick the component elements of capacity constraints in fragile states and diagnose why they are difficult to overcome. Together, the research offers a substantial challenge to conventional ways of thinking about aid and capacity building in low-capacity environments and helps to build an evidence base for alternative approaches. BSI tracks page views and content downloads on its website. It set a target of 25,000 downloads, which has been far exceeded at 60,000. Its Unblocking Results report was downloaded nearly 2,000 times, which for a technical paper is a strong result. Page views also appear to have exceeded the targets, although changes to ODI’s systems prevent them from generating an exact figure. Beyond website utilisation, BSI has not developed a system for tracking the usage and influence of its research. While there are techniques available for doing so, it would require a substantial investment of time and resources, which is probably not warranted for a research programme of this size. 2.9 Conflict sensitivity The Reference Group asked the evaluation to assess whether BSI was conflict sensitive in its approach. Conflict sensitivity means being aware of how the programme interacts with conflict dynamics, and taking active steps to maximise positive and minimise potentially negative impact on the conflict. Conflict sensitivity is not part of BSI’s management lexicon. This is perhaps not surprising. Being located within ministries of finance, BSI does not directly interact with communities in conflict. The risks of unintended harms are therefore relatively low. We also note that, in South Sudan, BSI’s work on local service delivery is likely to make a positive contribution to conflict reduction over time. Nonetheless, the resumption of conflict in South Sudan does raise certain risks for the programme. The government’s budget is being used to fund the conflict and BSI’s counterparts in the ministry of finance are presumably signing off on irregular budget releases to the military. While there is no suggestion that BSI has in any way facilitated this, its continued presence in the budget department does pose certain reputational risks. In addition, the ongoing conflict means that the rationale for BSI’s presence – to strengthen the budget process – has weakened, as the space for introducing useful reforms has narrowed (although BSI’s presence reduces the risk of the budget process breaking down under the pressure of the crisis). Given these conditions, it would be appropriate for BSI to set out explicitly what the minimum conditions are to justify its continued support for the budget department, and

17 It uses a three-way typology: i) policy and institutional incoherence; ii) a lack of top-down

performance discipline and bottom-up accountability; and iii) limited scope for collective action and problem solving.

25

what ‘red lines’ it should avoid crossing in order to avoid inadvertently supporting the financing of the conflict. Beyond that, we see potential for BSI to do more analytical work on the links between the political economy of South Sudan and conflict risk, and to use that analysis to inform its advisory work. For example, we understand that there is a high level of rent seeking around the dual exchange rate. IMF pressure on the government to end the dual exchange rate could therefore heighten conflict risk. BSI might be able to help the international community develop a risk management strategy around that reform. Furthermore, there might be links between conflict and the budget, if particular groups or communities are being denied a fair share of resources. BSI analysis could help the government to understand those linkages. We understand that BSI is now undertaking a context analysis in South Sudan, on the request of one of its funders. 2.10 Gender equality The Reference Group also asked whether BSI’s programme supports gender equality. There is no explicit discussion of gender issues in BSI’s programming documents and no activities that specifically address gender equality.

3. Theory-based evaluation In this section, we assess BSI’s theory of change. First, we address each of the eleven evaluation questions. Second, we test each of five key assumptions underlying BSI’s theory of change. Finally, we assess BSI’s claim to offer a novel and effective means of supporting PFM reform in fragile states, drawing on the literature’s critique of conventional approaches as a comparator, and examine its contribution to peacebuilding and state-building goals. 3.1 Is BSI providing high quality and relevant advice? Under the BSI logframe, the delivery of high-quality technical assistance is measured by the number of annual workstreams achieving ratings of ‘A’ or above. Over the course of the mid-term and final evaluations, we reviewed 25 annual workstream reviews, including by reviewing the quality of written outputs and obtaining from counterparts. Twenty-two of them met or exceeded an ‘A’ rating, demonstrating consistently high quality support. Feedback from counterparts and peer organisations was strongly supportive of the quality of BSI’s technical and advisory work. BSI advisors combine strong technical skills with a deep understanding of the institutional contexts in which they work. This understanding, combined with the inherent flexibility of the programme, makes BSI effective at promoting reforms that are suitable for the context, building on existing capacities in a stepwise fashion. The BSI team also offers strong skills in communication and facilitation, which are essential to working in a flexible, problem-solving mode. They are adept at identifying opportunities to advance useful reforms, even in difficult and unstable political contexts, and at building and sustaining coalitions of interest in support of change.

26

A distinctive feature of the BSI approach is that much of the support focuses on implementation. This differs from programmes such as the IMF’s Afritac, which offer strategic and technical advice without implementation support. In fragile states, budget reforms do not pose particularly complex technical challenges. In fact, most of the technical solutions proposed by BSI are similar to those that other technical assistance providers might suggest. The main challenge is in implementation – building constituencies for change, working around vested interests, making up for the lack of change-management capacity and managing a volatile delivery environment. In fact, with a PDIA approach, the distinction between the design and implementation of reforms becomes blurred – reform processes call for continuous design, in the form of solutions to implementation challenges. 3.2 How well has the programme managed uncertainty and risk in fragile states? BSI operates in very volatile environments. In the past two years, it has faced both the Ebola epidemic in Liberia and the renewed outbreak of conflict in South Sudan. BSI has proved able to adapt fairly quickly and flexibly to these changing conditions. When required to withdraw personnel from South Sudan and Liberia, it managed to continue some support remotely, helping the respective ministries of finance to meet the challenges raised by the crises. When new opportunities are identified, BSI is able to change the balance of its support quickly. A recent example was the provision of an experienced advisor to South Sudan to take advantage of a political window of opportunity to promote an independent review authority. As with any technical assistance programme, there are limits to the speed with which BSI can change its advisory staff. BSI’s greater use of consultants based abroad, however, means that the balance of inputs from the existing team can be changed quickly and easily. We nonetheless take the view that the programme could be more active in its risk management stance, to increase the efficiency of its response to changing circumstances. BSI does not make much use of risk management tools such as scenario planning. Nor, when undertaking high-risk interventions, is it explicit in identifying the conditions under which the intervention is justified or the triggers that would make a withdrawal necessary. In South Sudan, where the budget is being used to fund the conflict through irregular channels, BSI’s continuing presence in the budget department poses reputational risks for the programme and its funders. It also raises ‘do no harm’ considerations. While at the time of our visit, we were satisfied that BSI’s activities were justifiable, we took the view that the circumstances called for more explicit attention to risk management. 3.3 How efficiently and cost-effectively has BSI provided its inputs? Efficiency and cost-effectiveness are assessed in section 2.1 above. We conclude that BSI is a very efficient programme in a number of respects. Its use of peripatetic advisers keeps its overheads low while enabling it to respond flexibly in dynamic environments. Its daily fee rates are comparable or somewhat below those charged by comparable technical assistance programmes, but when its lower ‘footprint’ is taken into account, in terms of permanent offices, vehicles and accommodation, it gets better value for money for money for its inputs than similar programmes. BSI’s forecast management overheads of 10% in 2014-15 look to be at the low end for technical assistance programmes.

27

3.4 Are inputs provided by BSI contributing to identifiable institutional changes? This is assessed in detail in section 2.3 above, through the validation of BSI’s stories of change. Over the course of the two evaluations, we have verified 14 identifiable institutional changes that were directly attributable to BSI’s support and advice. There are another 5 institutional changes reported by BSI that we have not attempted to validate. Some of the institutional changes in South Sudan have been highly significant, including core budget processes and a system for intergovernmental transfers for local service delivery. In Liberia and DRC, they have been of a smaller scale, commensurate with a much smaller presence, but in both cases BSI has made some clearly identifiable contributions to a wider process of institutional development. BSI’s usual mode of institutional development is to support the design of new systems or processes and then to provide hands-on support for their implementation, including on-the-job mentoring for counterparts. This combination of systems design and hands-on implementation support is key to BSI’s success. BSI advisers help to make up for the lack of change management capacity in the counterparts. Along the way, they are able to build support for change and identify where obstacles are likely to occur. It is not clear, however, whether BSI has helped to raise the underlying level of capacity in its counterpart institutions. Ministries of finance in fragile states often have very limited capacity at the organisational and individual levels. In South Sudan, we were told that the Ministry of Finance still has the same basic structure as the old provincial department of finance from the colonial era, lacking many of the functional departments required for a ministry of finance of a sovereign state. In addition, a legacy of the independence war was that many poorly educated individuals were appointed to the public administration, who not only lack the requisite qualifications but are also difficult to train. Post-conflict institutions can therefore combine low capacity with low absorption capacity for capacity building assistance, calling for a patient, long-term approach. BSI’s capacity building support is limited mainly to on-the-job training. It has not played any major role in restructuring the organisations it advises. It is not equipped to deliver training programmes on any scale, although BSI advisers from time to time conduct training sessions. Its programme documents make some reference to coaching and mentoring, but it has not attempted to do this in a structured way. It does not carry out individual capacity assessments, tailor mentoring to individual needs, reach agreement with the individual on what a mentoring relationship involves or monitor progress on skills acquisition. This is the only area of BSI’s operations where the feedback from counterparts was mixed. On-the-job training was seen as both necessary and useful. However, some counterparts felt that BSI had not lived up to its commitments in this area. Others thought that on-the-job training was not sufficient to address skills gaps in a systematic way. In the Mid-Term Evaluation, we pointed out that BSI’s approach to capacity building was not well articulated and that its efforts were unstructured. While there was some evidence (in South Sudan) of an increase in the amount of training provided, the basic finding remains the same. Clearly, there is a link between the level of individual skills and capacities

28

available within a ministry of finance and the kinds of budget reform that are likely to be viable. Many reform programmes rest on the implicit assumption that individual skills can be built up quite rapidly, yet this may be unrealistic in post-conflict environments where turnover of staff is high and appointments are not always merit-based. BSI needs a clearer doctrine on how to identify the range of feasible reforms in a low-capacity environment. BSI’s forthcoming research focuses on this ‘capability trap’ and how it can be overcome. Hopefully, this will assist the country programmes to think in a more structured way about the different determinants of organisational capacity (systems, structures and skills) and come up with more explicit capacity-building strategies. 3.5 Is BSI having an effect on Government strategy and policy on PFM reforms? A key assumption in BSI’s approach is that there exist political openings for constructive engagement on PFM reform, even in difficult and unstable political environments. This has proved to be valid. Through successive crises in South Sudan, political openings for reform have appeared and disappeared in quick succession. BSI has proved able to position itself as a strategic adviser to senior figures and present its support as solutions to the challenges of the day. For example, during the oil crisis, there was political space to discuss austerity measures and BSI helped to provide the analysis and budget systems to support that. Since the conflict, there has been no appetite for austerity, but BSI has helped to mobilise support for improved non-oil revenue generation. At the time of our visit, it was close to securing Cabinet approval for an independent revenue authority. BSI has also been influential in carving out political space to develop a system of transfers to subnational government. Even in the midst of crises, BSI has been able to identify reforms that are useful to the political leadership. The experience supports the proposition that the political space for reform in fragile states may be greater than it appears from the outside. In Liberia, the government’s approach to PFM reform was already clearly articulated before BSI’s arrival. BSI has therefore had less influence at the level of strategy and policy, although it has nonetheless made a useful contribution to the reform process. In DRC, BSI has helped to formulate a prioritised action plan on PFM reform, although stakeholders appear unsure as to the level of government commitment to the reform process. Arguably, official PFM policies and strategies are of less importance when following a PDIA-type approach to institutional development. PFM strategies do not serve as a blueprint for the delivery of support, which must remain flexible and to some extent opportunistic. However, the PFM strategy provides an overarching authorisation for BSI to engage in PFM reform in a problem-solving way. BSI’s experience bears out the stop-go nature of reform in fragile states. Political windows close as suddenly as they open. For example, a recent changeover in Minister of Finance in South Sudan has required BSI to rebuild political support for budget reforms. Loss of support or political attention creates risks that past reforms will be reversed or left to lapse. BSI has had some success in preserving budget processes in both South Sudan and Liberia through crisis periods. It puts significant effort into ensuring that routine budget processes continue

29

to happen, so that the capacity is not lost. This is an aspect of BSI’s support that could be more explicit in its theory of change – the ability not just to exploit openings, but to defend past gains against the risk of reversal. While BSI has been fairly successful at identifying openings for reform, it is not systematic in the way it uses political analysis. It relies on the judgment of its consultants, brought together for occasional brainstorming sessions. It does not use formal political economy analysis or stakeholder mapping. It does not plot out possible scenarios and how it might respond to them. As a result, though the political instincts of its advisers appear to be very good, there is a danger that BSI remains reactive to events. At the time of our visit, BSI consultants in South Sudan had been working on an analysis of the vested interests involved in the dual currency regime – a key point of contention between the Government and the IMF. This is an example of an area where BSI could use its knowledge of the political economy to construct possible reform strategies, including exploring ways of influencing actors who are not part of the regular policy debate. More use of political analysis would also add to the programme’s conflict sensitivity (see section 2.9). 3.6 How demand-led has BSI been and is the quality of its relationships with counterparts conducive to effective assistance? When it comes to PDIA-type approaches, the concept of ‘demand led’ needs to be unpacked. All technical assistance must to some extent be demand-led, as the presence of foreign advisers within an institution requires the permission of ministers or senior officials. Most programmes are formally aligned to a reform strategy, in that they pursue objectives to which the counterpart institutions has officially committed itself. However, it is clear from the literature that a formal PFM strategy or request for assistance does not necessarily indicate a meaningful level of political support when it comes to tackling complex reform. The premise of a PDIA-approach is that formal statements of intention cannot be taken at face value. Instead, the challenge is to identify and mobilise potential constituencies for change and to work around political obstacles. This type of engagement, however, requires a significant level of trust from the counterpart agency, who must be convinced that this engagement in internal institutional dynamics will be used in its interest. Thus, the concept of demand-led applies not to the specific content of the reforms being pursued, but to the permission given to the programme to pursue them in a flexible manner. BSI has done very well at positioning itself as a trusted partner and strategic adviser in South Sudan and, to a slightly lesser extent, in Liberia. In DRC, it is not yet in such a position but may be able to achieve it in the coming year, as its activities scale up. In South Sudan, BSI has a high degree of license to engage in a flexible manner. It engages well at both the political and official levels and, as mentioned, has often been able to present its support as a solution to the political challenges of the day. Both government and donors recognise that BSI plays a useful intermediary role, helping to prevent successive crises from leading to a complete breakdown in communication. In all three countries, BSI has managed to engage consultants who bring strong personal networks, which facilitates access.

30

One unresolved question is whether the BSI programmes needs stronger governance arrangements at national level. Officials from all three countries participate in BSI’s central Advisory Board, giving them some voice in the direction of the programme. However, our observation is that substantive input from the three countries is difficult to obtain through the format of a 6-monthly videoconference. BSI might be better advised to put its effort into creating country-level governance structures for each major engagement. While BSI has done well at maintaining good relationships with its established counterparts, it is not clear how it initiates relationships in new countries or institutions. Exploratory work in Burma, Somaliland and other potential partner countries have yet to lead to new engagements. DFID’s core grant gave BSI the resources to undertake exploratory activities in new countries, but this model of expansion has not yet proved effective. A key issue for the programme and its Advisory Board in the coming period will be how to initiate engagements in new countries. BSI argues that its arm’s-length relationship with its funders enables it to establish stronger relationships with its counterparts and be trusted with confidential information. We were therefore somewhat concerned that DFID South Sudan, which funds BSI’s country engagement, now expects to receive regular briefings from BSI as part of the funding relationship. While both BSI and DFID state that these briefings only relate to matters in the public domain, there is a risk that this practice gives the wrong impression to government and undermines BSI’s effectiveness. BSI will need to manage this risk carefully, if necessary by increasing its distance from the DFID country office. 3.7 Has BSI had a catalytic effect? A key premise for the BSI theory of change is that, while some initiatives will inevitably fail, a few will produce catalytic results that justify the overall investment. There are several ways in which the BSI approach may generate catalytic results. One is to add value to other donor programmes operating in the same space. In South Sudan, the BSI team has contributed to the design and implementation of other programmes, often by helping the government to articulate what it wants and needs. BSI has also helped to create systems and processes that other donors can make use of. It has helped to develop the system of budgetary transfers to local governments and service providers, to which a number of other donor programmes are now aligned or ‘shadow’ aligned. In 2013, preparations were underway for an EU budget support operation, although this was derailed by the conflict. Changing the way in which donors support services in a post-conflict state certainly has the potential for catalytic impact. It is notable that these results are more likely when BSI goes beyond internal budget systems in the ministry of finance and engages with a wider range of stakeholders. Second, BSI plays a risk management role in fragile states. It provides strategic advice to ministers on how to manage external shocks to the budget process and helps to communicate fiscal issues to the political class. It facilitates communication between government and donors, to help guard against a breakdown in relations – a persistent risk in post-conflict countries. Its presence makes it less likely that donors will revert to unhelpful

31

political conditionality or interrupt their aid flows. In Liberia and South Sudan, it helped to prevent core budget processes from being abandoned in times of crisis, keeping the systems operating at a basic level so that credibility and capacity was not lost. It was clear in South Sudan that this risk management role was valued by donors. Potentially, BSI can also help to promote broader improvements in aid practices in fragile states. This happened to some degree in South Sudan prior to the conflict, where BSI supported the development of a New Deal Compact. BSI has not, however, done much explicit promotion of New Deal activities in its partner countries. According to BSI, this is because it remains a demand-led programme, working according to the needs and priorities of its ministry of finance counterparts. While this is legitimate, we take the view that the combination of its research programme and its support to the g7+ should put BSI in a position to be a more active advocate on improving aid practices in fragile states, if it chooses to prioritise this set of issues. Overall, therefore, the BSI approach clearly does have the potential for catalytic impact. Such impact was less evident, however, in Liberia or DRC. This suggests that BSI may need a higher level of engagement in a particular country before it can hope to achieve breakthroughs of the kind seen in South Sudan. BSI is also more likely to achieve catalytic impact where it goes beyond internal ministry of finance processes and engages with a wider group of stakeholders. 3.8 Is BSI having a sustainable impact on budget, policies, processes and systems capacity? Sustainability is a difficult criterion to apply to the PDIA approach in fluid environments. BSI’s work is punctuated almost continuously by setbacks and reversals, but this seems to be an inevitable aspect of working in volatile contexts. Part of the art of the PDIA approach is the willingness to abandon or defer initiatives that cease to be viable due to political changes and to identify new areas where progress can be made. Furthermore, the institutional changes it promotes are interim steps in long and complex reform processes whose destinations are uncertain. If BSI is successful, other reform processes will take over and continue after it disengages. Therefore, we take the view that the direct sustainability of BSI-sponsored initiatives is not necessarily a useful measure of success. 3.9 Has BSI increased uptake of good practice and promoted innovation? Across the two evaluations, we have noted a number of examples of innovation. BSI helped to develop innovative systems for recording aid on the budget, in both South Sudan and Liberia. Its support for budget transparency in Uganda is highly innovative, backed up by an impact evaluation to assess what difference transparency initiatives make at the local level. Its work on local service delivery in South Sudan, moving from the attempted development of a new aid instrument to develop a system of intergovernmental transfers, is innovative and potentially transformative. Innovation is not, however, always the objective. The reform needs in fragile states are often rather basic. The innovation required is not at the technical level, but in the design of

32

reform processes that are viable in the local context. BSI has been innovative in overcoming or working around vested interests and building momentum in favour of change. 3.10 Has BSI increased partner Government capacity and capability sustainably? One of the challenges facing BSI is to ensure that the specific institutional reforms it supports – usually, the introduction of new systems – are sustainable given the underlying level of capacity of the counterpart. There is inevitably a significant level of capacity substitution during the introductory phase of any new system. There have been relatively few opportunities to test whether those systems can be sustained without direct BSI support. The temporary withdrawal of BSI staff after the 2013 crisis in South Sudan provided one such opportunity. The budget department managed to produce a revised budget with only remote support. While BSI asserts this as evidence of sustainability, national stakeholders were less convinced. One senior counterpart informed us that the work was done “at terribly reduced pace and quality”. As noted above, the lack of clear BSI approach to capacity building makes it difficult to conclude that it is maximising its potential for sustainable capacity development. 3.11 In what ways is the support provided by BSI distinguishable from other providers of TA support? Overall, the evaluation findings support the proposition that BSI has identified a form of support for budget reforms that is distinctive from traditional forms of technical assistance on PFM reform. The BSI model a well-informed response to a cogent critique of conventional approaches emerging from the literature. This critique is summarised in Table 7. It includes over-reliance on international best practice templates, overly technical and overly complex approaches to reform that pay too little attention to political context, over-ambitious budget reforms that are out of sync with the development of wider public administration systems, and rigid contracting modalities that create unhelpful incentives for technical assistance providers.

Table 7: Key message emerging from the PFM literature

The challenges of externally sponsored budget reforms

There is broad agreement in the literature that externally sponsored budget and PFM reforms, backed by technical assistance programmes, have a disappointing record of achievement. Key explanations include:

the PFM field is dominated by a few actors espousing a tight normative framework, codified in the PEFA assessment tool;

reform strategies and programmes based on international best practice give insufficient attention to context, particularly the political and institutional constraints on change;

reform programmes are overly dependent on a single or a few ‘change champions’ to overcome vested interests;

33

budget reforms that are out of sync with broader public administration development, including the state of performance management across government;

overly complex, technical or ambitious reform programmes, loaded with hundreds of actions, supported by donors as a package through programmatic assistance, overload the capacity of counterparts to manage and absorb change;

timelines for change are frequently unrealistic;

a lack of project management capacity on the part of counterparts means that reforms programmes are poorly delivered; and

conventional contracting modalities lead to rigid approaches to delivery of TA.

The literature suggests an emerging consensus that the problem is not the content of reforms (the principles of good budgeting being fairly constant across different country contexts), but the question of how change happens and what contributions outsiders can usefully make.

Budget reforms in situations of conflict and fragility

The concerns with conventional approaches to reform are heightened in fragile contexts:

while principles of good budgeting remain broadly the same, the more difficult context calls for simplicity and programmes that are adapted to whatever limited capacity already exists;

PFM reform may need to take account of depleted financial resources, scarce information and widespread institutional incoherence;

when the centre of government is not coherent enough to operate as a single actor, externally sponsored reforms risk taking over and ‘ventriloquizing’ the limited capacity that is available;

capacity is constrained by the limited pool of skilled people and low wages for civil servants, particularly at mid and senior levels; there is no quick solution to this;

donor programmes face hard choices around the risks of capacity substitution and diversion and between quick fixes and long-term approaches; and

TA programmes that are accountable for concrete results do not tend to prioritise capacity development in low-capacity environments.

BSI is not necessarily different from traditional technical assistance providers in the types of advisory support it offers. Other programmes also claim to respect the principles of flexibility, tailor-made solutions, independence, trust and confidentiality. However, BSI’s funding arrangements give it far greater autonomy, enabling it to pursue a more strategic set of objectives in a flexible manner. Its theory of change posits that the most appropriate means of supporting reform in fragile states is through a flexible, problem-solving approach (known sometimes as problem-driven iterative adaptation, or PDIA, using Matt Andrews’ terminology18). The two core elements of the approach are flexibility in the choice of which issues to engage with, based on the political openings, and the use of problem-solving

18 Andrews, Matt, Lant Pritchett and Michael Woolcock (2012) Escaping Capability Traps through

Problem-Driven Iterative Adaptation (PDIA). Working Paper 299. Washington, DC: Center for Global Development.

34

techniques to address barriers to institutional change. A PDIA approach begins from the current institutional setup and looks for opportunities for incremental improvement, rather than comparing current institutions to an internationally accepted template and developing a sequence of comprehensive reforms to address the deficits. The PFM literature confirms that the PDIA approach responds to a cogent criticism of conventional approaches to technical assistance. Many reform efforts are said to have failed because they were over-ambitious and overly technical in nature and poorly suited to the local context. This is matched by a common critique of standard technical assistance projects, where activities and outputs are written into logframes and contracts, leading to inflexibility and creating incentives for implementers to press ahead with delivering formal outputs even in the absence of the national ownership required to make them meaningful (see Table 8 for some representative quotes from the literature).

Table 8: An emerging consensus on the need for flexible approaches to reform

“More successful improvement strategies have a relatively short-term horizon, focus deliberately on a quite narrow and specific set of objectives (for example, how to correct a fiscal imbalance or a problem of arrears, or to improve budgetary reporting), and involve a large element of trial and error, learning from mistakes, and ‘fumbling around in the dark.’ This requires deliberate selectivity in the choice of topics on which to focus the reform effort: selection rather than sequencing is the keyword.”

Allen, R., 2009

“Taking time to properly identify problems, underlying causes and the appropriate change agents at each of the relevant locations required and finding contextually appropriate solutions probably takes more time and requires more (apparent) risk than a typical development partner organisation (and individuals therein) is prepared to take.”

Johnson, M., 2013

BSI’s work on local service delivery in South Sudan is a good example of the PDIA approach at work. The problem that this work stream addresses is the difficulty of moving from donor funding of services through parallel systems to funding through government, when the national systems are not in place to do so. Initially, BSI worked on developing a new aid instrument that would facilitate a shift to use of country systems. When that initiative was derailed by the outbreak of conflict, the tactic changed to building a national system of budgetary transfers to local government, so as to build a system to which donors could eventually align. The example shows that PDIA does not simply mean short-term, opportunistic interventions. It means using a flexible approach to the pursuit of longer-term, strategic goals. It is not just about the design of reforms, but working through the politics and practical challenges of implementing them. BSI’s experience suggest that this approach is particularly well suited to the fragile states context. It is difficult to imagine a conventional technical assistance programme being able to take on a problem as broad as this one. The accountabilities that are built into typical donor

35

contract management processes work against that level of flexibility and create powerful disincentives for the pursuit of long-term, high-risk but high-payoff objectives. In Liberia and DRC, where BSI works at a small scale alongside larger actors, it is harder to see the distinctive nature of the PDIA approach, but there is some evidence of it being effective. In Liberia, BSI’s work on the MTEF was probably overambitious for the context, but it did provide the opportunity for BSI to work on a range of other issues, such as improving engagement by cabinet and line ministries with the budget process. In DRC, we were concerned that BSI may become locked into the delivery of a traditional, overambitious reform programme. Its positioning within COREF leaves it one step removed from the ministries it needs to influence, which makes it difficult to play a problem-solving, facilitative role. However, there are some positive signs that BSI has been able to use this platform to engage flexibly in the government’s planned decentralisation programme. 3.12 Increased transparency in budget data will help to drive reform While this was not an evaluation question in the TORs, it emerged as an issue during the course of the evaluation. BSI’s theory of change makes the assumption that increasing the transparency of budget data will help to drive reform. The inclusion of budget transparency looks more like a statement of conventional wisdom rather than a necessary part of the BSI approach. There are two main transparency initiatives within the current set of activities. In Liberia, BSI has helped to introduce a Citizen’s Guide to the Budget. The document explains in simple terms the government’s main policy goals and spending priorities and how the budget process works. It has accessible infographics on revenues and expenditure, simple explanations of some key spending programmes and a breakdown of county budgets. The first two iterations were produced primarily by BSI but a third has now been produced independently by the Ministry of Finance. There is no information available as to what use is made by whom of the information. In Uganda, which is one of the leading African countries on budget transparency, BSI has helped to develop a new budget website, where an increasing number of budget documents are published. The site contains disaggregated budget data for both national and subnational governments. It receives around 50 unique visitors a day. BSI is also engaging with intermediaries (e.g., parliamentarians, journalists and civil society organisations) who can interpret the information on the site for the public. Working in partnership with a local NGO, BSI has also helped to develop a ‘hotline’ for queries on the budget from members of the public. Citizens can raise concerns related to specific budget lines, which are passed to the responsible department. BSI is working to introduce a feedback process. The initiative is still young; there is no evidence yet available as to whether the mechanism helps to drive reform. BSI in Uganda is also engaging in a randomised control trial of information initiatives in local government. Since the introduction of an output budgeting tool some years ago, a considerable amount of information is available on the performance of local governments but little use is made of it. Across 28 districts, BSI is testing whether providing information

36

and training to intermediaries (local councillors and other local opinion leaders) has an effect on their level of engagement in local decision making. The experiment is being conducted in partnership with Innovations for Poverty Action, an NGO which specialises in randomised control trials.19 Results are expected in late 2015. Given the paucity of evidence behind the budget transparency movement, these initiatives are a potentially useful contribution. They are not, however, integral to BSI’s way of working or clearly linked to the PDIA approach. A PDIA approach should start with the problem, rather than the solution. While transparency might emerge as a solution, one would expect to see more explicit analysis as to what information needs to be transparent to whom, and what other processes are required to maximise the benefits. Under present conditions in South Sudan, for example, the general public or civil society may not be the best focus for transparency initiatives. It may be more useful to improve understanding across government (including state governments) and parliament about the budget process and the fiscal constraints facing the country. On the whole, it may be more appropriate for transparency to be treated as a guiding principle for BSI’s work, rather than a predetermined output in its logframe. 3.13 Does BSI contribute to peacebuilding and statebuilding goals? The Reference Group requested that the evaluation consider BSI’s relevance and contribution to peacebuilding and state-building goals (PSGs). While this objective is not explicit in BSI’s results framework, it is clearly important to BSI’s funders. It is also a pertinent question, reflecting a strategic choice facing the programme for its future development. Should it focus on developing its alternative model for supporting PFM reforms? If so, its approach is likely to be suitable for any low-capacity environment, not just fragile states. The programme could potentially over a wider geographic area, as well as piloting approaches for other TA programmes to take to scale. The alternative is to focus on the larger strategic question of the role that budget reforms play in state-building processes and how a PDIA approach to economic governance can help to raise the quality of international engagement in fragile states. The programme has a potentially important contribution to make to the second challenge.

1. Revenues and services: Under the New Deal, one of the PSGs is ‘Revenues and Services – Manage revenue and build capacity for accountable and fair service delivery’. In South Sudan, BSI has already made a significant contribution to this goal in a number of ways, including strengthening basic budget processes, developing a system of fiscal transfers for service delivery, building government support for increasing non-oil revenues, and helping with the management of successful fiscal crises. Even at a time when government-donor relations have been poor, BSI has made important progress towards the long-term goal of creating of mechanisms for funding service delivery through the state, rather than through parallel mechanisms. In the South Sudan context, this is one of the few statebuilding goals that continues

19 See http://www.poverty-action.org/project/1063 for a description of the trial.

37

to make appreciable progress. There is a strong argument to suggest that, in major post-conflict interventions, having a BSI-style programme in the ministry of finance is key to getting revenues and expenditure back on an even keel.

2. Use of country systems: The New Deal also contains a firm commitment to using and strengthening country systems, including strengthening PFM systems from the ground up and building fiduciary and administrative capacity at the national level. In South Sudan, BSI has supported this goal very directly through its work on aid instruments and the financing of local service delivery. It has helped to develop the management systems required at local government level to manage funds and to organise donors to provide capacity building assistance. It had made good progress on getting aid recorded on the budget, to increase the predictability of aid and facilitate more efficient use of resources, although this process was set back by the conflict.

3. De-risking the donor/government relationship: BSI in South Sudan has also played the role of an intermediary between donors and the government on economic governance issues. It has helped to reduce (although not eliminate) the risk that miscommunication leads to a breakdown in the relationship and interruptions to aid flows. It has helped to facilitate South Sudan’s communications with the IMF and other international actors. DFID South Sudan was clear that it saw its support for BSI as an investment in risk management for the country programme as a whole.

4. Support for the g7+: BSI has helped to develop the capacity of the g7+ group to formulate common positions and advocate in their interest in international policy forums. It was instrumental in the development of the New Deal itself. The question as to whether, since then, the g7+ has been effective at promoting better quality international engagement in fragile states remains an open one. Clearly, however, BSI now has the potential to move on from building basic capacity in the g7+ Secretariat towards playing a more strategic advisory role, to help the New Deal process regain its momentum.

5. New Deal implementation activities: BSI has also provided research and advisory

support for New Deal implementation activities in a number of countries, including the Sierra Leone fragility assessment, Afghanistan’s Aid Management Policy and Mutual Accountability and Framework and various New Deal processes in Liberia. In South Sudan, it was instrumental in the development of a draft New Deal Compact, which was abandoned following the output of conflict. However, BSI has not sought out opportunities to engage with the New Deal process in its partner countries, believing that this is not compatible with its demand-led approach.

Given this range of contributions, there is no question that BSI has made a useful contribution to the PSGs. It is not clear, however, that this is always central to the way ODI runs the programme. BSI makes its most strategic contribution when it goes beyond purely demand-driven, technical support for PFM reform and moves into the wider economic governance space, engaging with more government stakeholders and playing an intermediary role between government and donors. There are few, if any, other actors who

38

can play such an intermediary role effectively. Conversely, when it focuses purely on budget processes that are internal to ministries of finance, the support is less obviously about the PSGs.

4. Conclusions and recommendations 4.1 Conclusions Overall, we find that BSI has continued to grow and develop since the Mid-Term Evaluation, achieving important results despite a more difficult operating environment. Its work continues to be of consistently high quality. We heard universally positive feedback on the BSI country teams, who combined strong technical knowledge with outstanding communication and facilitation skills. Our validation of BSI’s results shows that the programme continues to deliver at well above expectations in the majority of its work streams. Despite setbacks caused by the crises in South Sudan and Liberia, it has comfortably exceeded its targets for effective delivery of assistance. At the outcome level, we have been able to verify 14 significant institutional changes that have resulted from BSI’s work across the evaluation period. That number is slightly short of the logframe target, mainly because of the difficulties in achieving sustainable institutional change in South Sudan and Liberia over the past 18 months. It is also a slightly arbitrary number, given the difficulties involved in counting institutional change. Nonetheless, we are satisfied that it represents a significant contribution to the partner countries. While it is difficult to capture the impact of BSI’s work through high-level indicators of budgetary effectiveness, there have been a range of important results from BSI’s work. In South Sudan, basic budget processes have been strengthened, transparency has improved, the funds available for local service delivery have increased (with anecdotal evidence of resulting improvements in primary school enrolment) and the first donor programmes have begun to deliver through country systems. These results have been achieved despite a very difficult environment in which many reform initiatives have slipped backwards. BSI’s work on local service delivery stands out among these results as an example of the kind of catalytic impact that the programme hopes to achieve. By creating a system of intergovernmental transfers to which donors can align, BSI has achieved impressive progress towards the key state-building goal of country-led service delivery. Without BSI’s support, there would almost certainly have been regression away from that goal. In Liberia, while overall fiscal discipline has yet to improve, BSI has helped to introduce improvements in the budget process that have strengthened the engagement of cabinet and line ministries and improved transparency. The DRC engagement is less mature, but BSI has helped to strengthen the government’s plans, laws and institutional arrangements for PFM reform, and has made useful progress in putting in place preparatory actions for fiscal decentralisation.

39

In its g7+ programme, BSI has provided effective support to the development of the Secretariat, helping it put in place basic administrative systems and capacities. The Secretariat is now capable of organising international events, identifying common positions among its member countries and representing those in international policy dialogue. The g7+ is now a recognised grouping that is regularly invited to international events, giving for the first time a voice to an important group of fragile states. In terms of international policy, BSI had a direct influence on the formulation of the New Deal. Since then, the g7+ helped to ensure that the interests of its member countries were taken into account in the IDA17 negotiations and in the development of the draft Sustainable Development Goals. BSI has not prioritised implementation of the New Deal at the country level and the contribution of the initiative to improving the quality of international engagement is not yet clearly identifiable. BSI’s research programme is on track to deliver its planned research products, with seven pieces of substantive research already produced and another five in the pipeline. It has also produced a range of dissemination products, including a website that is well utilised. The influence of the research on international practice will take time to emerge. However, it looks likely that BSI will succeed in developing a body of work that challenges conventional thinking on external assistance in fragile contexts and builds an evidence base for alternative approaches. Overall, we rate BSI’s level of achievement across its five years of operation at ‘A+’ (above expectations). If scored separately, its work on local service delivery in South Sudan would merit ‘A++’. We find that the core assumptions in BSI’s theory of change have been borne out by the experience of implementation. The programme has demonstrated that a flexible, problem-solving approach can support improvements in economic governance even in very difficult environments. Because it is not locked into predetermined activities or outputs, BSI is better placed than most programmes to respond to the stop-go nature of reforms in fragile contexts. It has shown an ability to identify useful reforms even in difficult political environments and, where necessary, to defend past gains against the risk of slipping back. It combines strategic and technical advice with hands-on support to implementation, enabling it to build constituencies for change and overcome implementation bottlenecks. These attributes of the BSI model have been carefully designed to address many of the shortcomings of traditional technical assistance, as identified in the literature. These include over-reliance on institutional templates, technical approaches to reform that disregard the political context and rigid contracting modalities that create unhelpful incentives for technical assistance providers. We conclude that BSI offers a powerful alternative model to the conventional approach to supporting PFM reform. There are also areas where the model and its delivery need to be refined. We find that BSI’s approach to individual capacity building is not fully articulated. It has not yet clearly identified what forms of capacity building support work best in low capacity environments

40

and its capacity building offer to its counterparts is not always or understood. BSI is also yet to demonstrate how budget transparency fits into a PDIA approach to budget reforms. Maintaining a flexible approach to delivery in a volatile environment poses difficult management challenges. The effectiveness and efficiency of the assistance depends in large part on BSI’s ability to respond quickly as opportunities open up or close down. The BSI model is inherently flexible and the programme has shown an ability to mobilise high-quality support at short notice. However, BSI makes relatively little use of political economy analysis or scenario planning to guide its management decisions. It does not explicitly identify and monitor the preconditions for effective support in particular areas. As a result, we found it difficult to be certain that BSI resources in South Sudan were being used to maximum effect, given the impact of conflict on the budget process. There is therefore scope for BSI to improve its risk management. We have found that BSI has a potentially important contribution to make to the larger strategic objectives of improving the quality of international engagement in fragile states and implementing the New Deal Peacebuilding and Statebuilding Goals. As it plans the next phase of its development, the programme faces an important strategic choice as to whether to focus on the delivery of PFM reforms, as traditionally defined, or to tackle wider economic governance issues, including the quality of international support. Both are legitimate and important goals. In our view, BSI’s experience in South Sudan suggests that the programme is at its strongest when it moves beyond internal ministry of finance processes and engages a wider range of stakeholders on economic governance issues, such as how to achieve country-led delivery of public services. This larger problem set would also seem to offer more opportunity for BSI to leverage its unique combination of country support, research and international advocacy. Finally, we note that, while BSI has had some success at fundraising, its recent attempts to initiate operations in new countries have not been successful. This raises important questions about how the programme should develop in the future. BSI and its Advisory Board need to give some careful thought to how the programme will both obtain funding and establish relationships of trust in new countries. Is BSI a model that could be replicated in other fragile states by different funders and implementers? It is notable BSI’s successes have been greatest in South Sudan, where it has the largest and longest running programme. Its engagements in Liberia and DRC have been successful but less distinctive, and BSI has struggled to establish itself in other countries. It therefore remains an open question whether the major successes that BSI has achieved in South Sudan are a product of particular local circumstances or opportunities, and whether they can be replicated elsewhere. There are also particularities of ODI as an implementer. As a research institute, it is able to draw on a wide pool of expertise and an active research programme. It also makes effective use of the alumni network from the ODI Fellows programme, who bring a particular set of skills and knowledge.

41

On the other hand, BSI has made a significant effort to articulate its theory of change as clearly as possible and to identify the elements required for successful implementation of the PDIA approach. It has documented its lessons extensively. It has identified the kinds of skills that are required and has demonstrated that such people can be recruited. There seems no reason in principle why the core elements of the BSI model could not be replicated by others. In fact, many of the distinctive features of the BSI programme are linked to its different funding model. It is certainly plausible that the difference between BSI and other technical assistance programmes is primarily down to differences in project design and contract management. These patterns can be certainly be changed by funders. The BSI experience stands as a useful example to donors interested in exploring the possibilities for more flexible and adaptive programming. 4.2 Recommendations

1. In its country programmes, BSI should ensure as far as possible that its technical support to budget reforms makes a strategic contribution to wider economic governance issues (e.g., managing internal and external resource flows; financing service delivery; decentralisation). This may entail engaging with a wider range of counterparts and stakeholders, beyond ministries of finance.

The distinctive contribution of BSI is greatest where it tackles larger strategic questions around economic management and the quality of the development partnership. It is least evident where BSI focuses on delivering narrow or technical PFM reforms. While BSI has demonstrated that its PDIA-type approach can be useful applied to many areas of PFM reform, we take the view that BSI should focus its efforts at the strategic level. This means focusing on areas of economic governance that are important to wider peacebuilding and statebuilding processes and that can contribute to a more productive development partnership. This in turn may mean broadening BSI’s circle of engagement beyond ministries of finance.

2. BSI should refocus its g7+ work stream towards the larger strategic challenge of improving the quality of international support to fragile states, using its research and analytical capacity to support the international dialogue by helping to turn New Deal principles into practical proposals for improving aid management and delivery.

BSI has successfully delivered technical assistance to the g7 Secretariat over a number of years. The Secretariat is now in a position to manage its routine work programme without BSI inputs. BSI should therefore move from capacity building towards intellectual support. In keeping with BSI’s PDIA approach, the objective should be raising the quality of donor support to fragile states, rather than strengthening the g7+ per se – although the latter goal is a legitimate means of pursuing the former. While many observers are concerned that the New Deal initiative has lost direction or momentum, the problems that it was set up to address are still very much in evidence. International debates on how to operate effectively

42

in fragile environments are going to continue. On the back of the research programme and five years’ of country programming, ODI and BSI are well placed to contribute to policy development in this area, particularly on economic governance and improving aid practices. BSI should use its relationship with the g7+ to promote new thinking on development assistance in fragile states, including innovations in aid instruments and practices. The implication of the first two recommendations are that, as it plans its future development, BSI should look for opportunities to build synergy across the g7+ engagement, the country engagements and the research programme.

3. BSI should make more explicit use of conflict and fragility analysis to identify strategic risks and opportunities in its partner countries, with a focus on the long-term goal of reducing conflict and fragility.

While the evaluation has found that BSI support is based on a good understanding of the political context, it could make more use of conflict and fragility to identify priority areas of engagement. As part of its planning processes for its country engagements, BSI should identify more explicitly which economic governance reforms are most likely to contribute to peacebuilding and statebuilding, and which risks or threats are most likely to trigger conflict or instability. The recommendation is primarily applicable to South Sudan, where BSI has greater resources and more flexibility to engage outside the ministry of finance in wider state-building processes.

4. BSI should explore the use of scenario planning and other risk management tools

to help manage its engagement in volatile contexts. It should identify the objectives and activities appropriate to different scenarios, to help it to respond nimbly to changes in context. For high-risk work streams, it should identify and monitor the minimum conditions required for a successful engagement.

BSI is engaged in high-risk interventions in volatile contexts. A significant determinant of its efficiency and effectiveness is the speed with which it identifies and responds to changes in the political environment and the level of risk associated with its activities. High-risk programmes may fail to respond appropriately to gradually deteriorating contexts, if they do not identify the risks explicitly and put in place active monitoring arrangements. In South Sudan, conditions have been deteriorating for some time and some of BSI’s peer programmes have already been suspended. There is a heightened risk that the engagement loses its focus or even causes inadvertent harm. As part of its routine planning processes, BSI should map different scenarios in the political context and the types of engagement that are likely to be appropriate to each, including the types of reforms that might be viable and where the programme should focus on action to prevent past gains from being lost. When undertaking high-risk interventions, BSI should identify the minimum conditions required for a successful engagement and then monitor whether those conditions remain in place. This should help to determine when activities should be abandoned or when risks become excessive.

43

5. Based on its research programme and lessons learned from its country

programmes, BSI should work towards a clearer articulation of the challenges of building capacity in low-capacity and volatile environments and develop explicit capacity building strategies for each of its country programmes. It should conclude agreements with its counterparts as to what forms of capacity building support it will provide, including training and mentoring for staff, as appropriate.

The evaluation found that BSI’s capacity building approach remains under-articulated. Its research programme, however, is generating a rich set of findings on the core capacities required in finance ministries and how to match institutional reforms to the underlying level of capacity. BSI should soon be able to apply these findings and develop more explicit capacity building strategies for each of its country programmes. By developing capacity building plans jointly with its counterparts, BSI will be better placed to manage expectations and also ensure a regular tempo of capacity development activities through the budget calendar. If it chooses to offer coaching or mentoring for counterpart officials, this should be formalised into individually tailored plans with associated activities and monitoring arrangements.

6. BSI should work towards a clearer understanding of the role of budget transparency in fragile contexts, including identifying what information needs to be communicated to which stakeholders in order to support budget reforms and better economic governance.

BSI’s theory of change makes reference to budget transparency as a driver of reform, but without being very clear how that fits within the wider PDIA approach. The work in Uganda on transparency is innovative and should generate useful evidence on how and under what circumstances transparency contributes effective pressure for reform. BSI should apply a PDIA lens to the transparency agenda and begin to develop a more explicit set of theories as to what information needs to be transparent to which stakeholders in what form, in order to drive reform. This might be a useful topic for the research programme to pick up.

7. BSI and its Advisory Board should focus their attention on developing a sustainable

funding model for the programme and how to establish a BSI presence in new countries.

The greatest threat to the continuing success of BSI at present is the uncertainty of its funding model. BSI has not yet leveraged its current success at the country level into engagements in new countries. Without core funding from DFID to enable exploratory activities, it needs a more considered strategy for any future expansion. This is an area where BSI’s funders and Advisory Board members could be more active. There is good evidence from South Sudan, in particular, that a BSI presence in the ministry of finance can deliver wider benefits to other development programmes, by making the ministry of finance a more informed consumer of technical assistance. It can also help to manage the development partnership in environments where it is prone to breaking down.

44

If the Advisory Board members agree with this analysis, they could do more to promote the idea to their country programmes.

8. The Advisory Committee should appoint a sub-committee or one of its members to look at its own functioning and develop proposals for how to strengthen BSI’s governance.

The Advisory Board is very important for the legitimacy of BSI, giving its flexible funding arrangements, and a very useful source of knowledge and experience. However, it is not being used as effectively as it could be. The six-monthly meetings have overcrowded agendas and the format does not give enough scope for substantive discussion. The Advisory Board itself may be best placed to suggest different ways of structuring itself and its work. There might be a case for sub-committees on the research programme, the g7+ and on each country engagement, to provide more granular oversight. Individuals on the Advisory Board could take a watching brief over particular elements of the programme, to provide some continuity in between full Board meetings. The Advisory Board could be more active in facilitating entry for BSI into new countries. The representatives of the bilateral and multilateral institutions could also do more to raise BSI’s profile within their own institutions, and to advocate for more flexible funding models more generally within the sector.

45

Annex 1 Validation of AWRs Introduction BSI’s monitoring system includes the production of internally generated reviews of the work completed each year on those work streams where the annual investment has exceeded £100,00020. These annual work stream reviews (AWRs) are prepared by the Country Team Leaders following a standard format:

Background, including title, period under review, spend in the year and AWR preparation date

Outputs, covering responses to these questions: what was the situation at the outset of the project i.e. the baseline? and what progress has been made against the expected results i.e. actual achievements against milestones and indicators?

Additional comments, to provide more detail on the context and any wider achievements

Score.

The final section of the AWR, relating to the score, is completed by the BSI senior management in London. The structure of the scoring adopts DFID’s standard ranking21. The scoring of these AWRs is used as the measurement of the achievement of the Output indicators in the overall programme logical framework22 which refers to the “relevance, effectiveness, efficiency and sustainability of TA support”. Approach to the validation of the scoring of the AWRs The following AWRs were validated and scored:

Uganda: Support to Budget Transparency and Accountability (2013-14, 2014-15)

South Sudan: Support on the Planning and Budgeting process (2013-14); Support to Government Led Delivery of Local Services and Infrastructure (2013-14); Support on PFM reform (2013-14); Planning, Budgeting and Aid (2014-15); Government-led Delivery of Local Services and Infrastructure (2014-15); and Public Financial Management Reform (2014-15)

Liberia: Support on Citizen’s Guide to the Budget and Communications Strategy (2013-14); Support on budget policy and process reform (2013-14, 2014-15)

20 For Uganda, the work stream expenditure was less than £100,000 in 2013-14 21 Reviewing and Scoring Projects How to Note, DFID, November 2011 provides the following ranking: A++ =

Outputs substantially exceeded expectation, A+ = Outputs moderately exceeded expectation, A = Outputs met expectation, B = Outputs moderately did not meet expectation and C = Outputs substantially did not meet expectation 22 Specifically, Output indicators 1.3 (in relation to budget policy and process and financial management), 2.2 (domestic revenue and external resources), 3.3 (budget oversight by Parliament and civil society), and 4.3 (international engagement in fragile states)

46

DRC: Country Director for DRC Programme (2013-14); Country Director, External Experts and Resident Experts for DRC Programme (2014-15)’

For the purpose of this assessment, the objective has to been to rate progress by reviewing the achievements made in each year within each work stream. To do this, the following approach was adopted:

from the annual programme summaries for each country presented to the Advisory Board in May, identify the expected results (and relevant indicators if provided) for each work stream

review the actual performance in delivering the expected results as reported by each Country Team Leader in the respective country Annual Report

review the narrative provided in the relevant AWR to ensure consistency with the Annual Report

confirm that the results described are a fair assessment (through cross reference to the validation of the Stories of Change, where appropriate)

verify that the results align with the indicators and milestones reported in the country logical frameworks23

derive a score for each AWR using the DFID rankings. Summary of the scoring A comparison of the AWR scoring by ODI and by the Evaluation Team is shown in the table below.

Country and AWRs BSI score Evaluation

score

Uganda - Support to Budget Transparency and Accountability (2013-14)

A+ A+

Uganda - Support to Budget Transparency and Accountability (2014-15)

A+ A++

South Sudan - Support on the Planning and Budgeting process (2013-14)

A B

South Sudan - Support to Government Led Delivery of Local Services and Infrastructure (2013-14)

A++ A++

South Sudan - Support on PFM reform (2013-14) A B

South Sudan - Planning, Budgeting and Aid (2014-15) A++ A+

South Sudan - Government-led Delivery of Local Services and Infrastructure (2014-15)

A++ A++

South Sudan - Public Financial Management Reform (2014-15)

A++ A

Liberia - Support on Citizen’s Guide to the Budget and Communications Strategy (2013-14)

A A

23 Uganda does not have a logical framework

47

Liberia - Support on budget policy and process reform (2013-14)

A A

Liberia - Support on budget policy and process reform (2014-15)

A+ A

DRC - Country Director for DRC Programme (2013-14) A A

DRC - Country Director, External Experts and Resident Experts for DRC Programme (2014-15)

A A

Support to the g7+ Secretariat 2013-14 A++ A++

Support to the g7+ Secretariat 2014-15 A++ A+

Results of the validation and scoring Uganda - Support to Budget Transparency and Accountability (2013-14) During this year, BSI continued its work started in 2012-13 to support the implementation of the Government’s budget transparency initiative (BTI) in partnership with 2 NGOs: Advocates Coalition for Development and Environment (ACODE) who are establishing a network of Budget Champions across the country, and Innovations for Poverty Action (IPA) who will conduct a randomised impact assessment of the access to and use of the budget data now available on the website. The new budget website (www.budget.go.ug) provides a comprehensive library of budget documents and innovative tools for the public to access location-specific budgetary information and provide feedback. Considerable interest, in-country and internationally, has been established in BTI. Ministry of Finance, Planning and Economic Development’s (MoFPED) commitment to budget transparency increased after the corruption scandal uncovered in the Prime Minister’s Office at the end of 2012. Although there were some delays arising from the development of the technology, the results exceeded expectations. Score: A+, Output moderately exceeded expectation Uganda - Support to Budget Transparency and Accountability (2014-15) This has been a year of significant progress as evidenced by several achievements including: BSI attracting additional non-core funding to help sustain its engagement in the BTI; further enhancements to the budget website; stronger working relations with ACODE and IPA and progress with their initiatives, the introduction of a budget ‘hotline’/call centre and the ‘mainstreaming’ of the ongoing support for the budget website within MoFPED. Score: A++, Output substantially exceeded expectation South Sudan - Support on the Planning and Budgeting process (2013-14)

48

Achievements have been made including: support to the preparation of the national budget strategy which articulates the President’s spending priorities on health, education and water service delivery, agriculture and infrastructure provision; the 2013/14 national budget set out details of increased and new transfers to counties and facilities in line with the framework for financing local service delivery that had been developed in 2012 (by March 2014 the first of these service delivery transfers were released); and BSI supported the development of tools and processes to prepare a Public Investment Plan, the first of which was published as a part of the 2013/14 budget. However, the main objective for 2013-14 was to help deliver the following result “planning, budgeting and reporting processes are better integrated, with more strategic political engagement, greater transparency and reduced reliance on TA for their management and coordination”. 24 BSI planned “to continue to support a stronger, better integrated and more transparent national policy making, planning, budgeting and reporting process at the national and sector level.”25 In practice, this result failed to materialise in its entirety, largely due to the continued fiscal crisis and to the civil unrest from December 2013 which resulted in a 3 months’ absence of BSI from Juba through to March 2014. Score: B, Output moderately did not meet expectation South Sudan - Support to Government Led Delivery of Local Services and Infrastructure (2013-14) For 2013-14, this work stream set out to deliver progress in three main areas, namely:

“Sectoral responsibilities and coordination mechanisms for local service delivery are clarified and institutional capacities and systems for local services delivery are strengthened

Intergovernmental transfers that are supportive of local service delivery developed and implemented alongside the county PFM system

Local service delivery aid programmes that are managed by government institutions and use government systems for delivering local services designed and approved.”26

Coordination mechanisms became stronger with increased participation and leadership by Government staff. The service delivery frameworks (SDFs) were finalised, setting out clearer sector responsibilities for different levels of Government for local service delivery. A Joint Plan of Action was endorsed by the LSS Task Force and approved by the Undersecretaries to provide the framework for improving service delivery.

Progress with getting aid funding on Government systems for local service delivery stalled with the hiatus created over the non-compliance with the exchange rate equalisation condition set by the IMF, followed by the crisis in December 2013. The then current proposals for donors to use Government systems, such as the EU’s proposed budget

24 BSI South Sudan Annual Programme Summary presented to the Advisory Board, May 2013 25 Ibid 26 Ibid

49

support for state building, were halted. Currently, only the World Bank’s LSS programme (LGSDP) retains funding through Government systems. If progress was to be scored on the progress made in delivering these 2 objectives, then the rating at best would be a ‘B, Output moderately failed to meet expectation.’ However, despite the fiscal crisis and the political turmoil, exceeding expectations, new transfers for health, education and water were designed, and allocations made in the 2013/14 budget were released early in 2014. State Transfer Monitoring Committees (STMC) have been supported to operate and the states have been supported to establish Country Transfer Monitoring Committees (CTMCs) to ensure proper use and reporting of the transfers. These transfers have continued to be paid on a regular basis. The significance of these continued payments justifies a significant improvement in the work stream score, especially since it provides a very strong signal of Government’s continued support to LSS, despite the major fiscal and political challenges. Performance against the logframe indicators and milestones presents a ‘below expected’ performance. However, the reality is that this work stream has delivered impressive results. On this basis, the assessment is shown below. Score: A++, Output substantially exceeded expectation South Sudan - Support on PFM reform (2013-14) BSI’s planning for the year recognised that there was little appetite in Government to drive forward with the Ministry of Finance and Action Plan for reform given the focus on trying to tackle the fiscal crisis. The plan for 2013-14 was articulated as follows “At a minimum, BSI will help facilitate an annual external review of PFM and senior management discussions on the findings. BSI will also be able to source and deliver expertise to address key gaps and bottlenecks in the reform process. One area where BSI is mobilising support is on capital investment management.”27 Given the pressures elsewhere these limited interventions did not materialise.

Score B: Output moderately did not meet expectation South Sudan - Planning, Budgeting and Aid (2014-15) At the beginning of the year BSI’s priorities were as follows:

Continued support to the planning, budgeting and reporting processes to help ensure that have more strategic political engagement, greater participation, transparency and reduced reliance on TA for their management and coordination.

Support to enhance the link between allocations of budget and aid resources to infrastructure and service delivery in spending agency budgets and budget reports in the health, education and infrastructure sectors.

27 Ibid

50

Support to consolidate aid management processes and to provide guidance for implementing the aid strategy, with a framework for monitoring aid strategy benchmarks.

Given the extreme fiscal crisis and the continued DPs’ cautious approach to re-engaging, and waiting to see what happens about a peace accord, it is perhaps not surprising that the achievements failed to meet the ambitious objectives. Much good work was done but this was more in terms of delivering specific products and targeted interventions rather than helping to build systemic and sustainable change across the broad PFM reform agenda. However, in terms of the logframe indicators, and delivering the targets, actual performance was slightly above expectation. Score: A+, Output moderately exceeded expectation South Sudan - Government-led Delivery of Local Services and Infrastructure (2014-15) BSI’s work continued the work of earlier years in building systems and increasing confidence in decentralised service delivery through local governments. At the outset the objectives were to support the following:

Re-configuration and continued implementation of a government framework and mechanisms for coordinating local service delivery

Implementation of mechanisms for budgeting, financing and management of funds for

local service delivery

Implementation of aid programmes which use and strengthen government systems All of these objectives were achieved. Government continued to process the transfers to local governments despite the fiscal crisis. Nearly all of the logframe targets were met or exceeded. Score: A++, Output substantially exceeded expectation South Sudan - Public Financial Management Reform (2014-15) The key challenge that BSI identified at the start of 2014 was how to support MoFEP to ensure system continuation and maintain a trajectory towards PFM reform. BSI provided high quality advice on a range of PFM issues including increased transparency relating to the Government’s debt and arrears position. BSI’s facilitation of the process in guiding Government towards establishing a National Revenue Authority is particularly noteworthy and has the potential to make a significant contribution in increasing domestic revenue mobilisation. The logframe indicators, and related March 2015 targets, have been changed since the Mid Term Evaluation. Although performance in delivering the targets either met or exceeded expectation, it seems that the focus of the measurement has moved away from helping to deliver systemic change in the PFM processes to delivering individual PFM related ‘products’. If correct, this is a change in the original objective set back in 2011/12. But it may

51

be an appropriate, pragmatic adjustment to reflect the uncertainty around delivering a broad based PFM reform agenda in a period of fiscal crisis. Score: A, Output met expectation Liberia - Support on Citizen’s Guide to the Budget and Communications Strategy (2013-14)28 This work built on the initial success in 2012-13 when the first Citizen’s Guide to the Budget was produced. The majority of the intended results were achieved, most notably in the following areas: the preparation of a communications strategy that can be used in future years to help publicise the Guide; and the creation of a reusable structure, content, and design of the Guide together with clarity of the roles and responsibilities of Ministry staff in its preparation and production. Score: A, Output met expectation Liberia - Support on budget policy and process reform (2013-14)

BSI’s work in this year continued its support to the PFM Reform Strategy by providing technical support and capacity building to embed the MTEF process with a focus on establishing a credible and comprehensive budget. In this regard, progress was made across a number of fronts, including: an added strategic dimension to the budget process with the production of a Budget Options Paper for consideration by the Cabinet prior to the drafting of the Budget Call Circular; and the creation of a Budget Working Group to help improve coordination and discussion of budget options and proposals especially in helping to better align spending proposals with the priorities in the national development plan – the Agenda for Transformation. This work was underpinned by the delivery of job related training to the staff in the Budget Department of the Ministry of Finance.

Score: A, Output met expectation Liberia - Support on budget policy and process reform (2014-15) In May 2014, the original plan for BSI’s support set out its proposed continued support to help establish a credible and comprehensive budget process with continued strengthening of the key elements of the MTEF approach. The objectives and priorities were framed in the following terms “the aim is to continue contributing to improvements in performance indicators in the monitoring framework for the PFM reform strategy, and ultimately relevant scores in the PEFA assessment. More specifically, BSI will provide support to implementing elements of the new budget calendar for FY2015/16, based on the ODI-BSI developed training plan, integrating M&E into the budget cycle; and budget execution by supporting expenditure management and policy for improved service delivery”.29

28 Total spend on this work stream was £20,000 29 BSI Liberia Annual Programme Summary reported to the Advisory Board, May 2014

52

However, the outbreak of Ebola in August 2014 disrupted these plans with the evacuation of the BSI staff and the Ministry’s preoccupation with tackling the crisis. During this period BSI was able to continue providing technical support remotely. Since returning to Liberia in December 2014, BSI’s support has been in helping to preserve the integrity of the budget process albeit within a much compressed timetable for the 2015-16 budget preparation. Necessarily the advisory work has been responsive to needs as they arise and as such it is difficult to gauge actual performance against expectation. Certainly the feedback from beneficiaries has been very positive. Score: A, Output met expectation DRC - Country Director for DRC Programme (2013-14) The expectation during the year was for the more substantial TA support from BSI to COREF to start (to be funded through the AfDB’s Fragile States Facility). The bidding/selection process was delayed. As a result, BSI had to resort to using its core funding (supplemented by additional funding from GIZ and World Bank to cover the cost of 2 visits by the Country Team Leader and two other external experts) to implement a limited range of activities in support of COREF as a ‘bridging’ exercise in anticipation of the AfDB funding, now expected in the summer of 2015. BSI completed successfully its work in assisting COREF with the updating and elaboration of the Priority Action Plan for PFM reform and in advising on a range of legal and institutional issues regarding PFM reform and financial decentralisation, including suggested revisions to laws, ordinances and draft procedures manuals. The milestones for both Outputs in the country logframe were met. Score: A, Output met expectation. DRC - Country Director, External Experts and Resident Experts for DRC Programme (2014-15) The AfDB funding failed to materialise during 2014-15. Consequently, BSI’s support continued along the lines of the previous year with inputs to improve the Priority Action Plan, further advisory support to the decentralisation process and proposals for the revision of laws and ordinances. A particular feature of BSI’s work has been the support provided to the 6 technical assistants working in COREF regarding the work of the five COREF sub-committees in monitoring PFM reform implementation. Score: A, Output met expectation. Support to the g7+ Secretariat 2013-14

BSI provided support to the Secretariat from a number of staff and contractors, including a programme manager, a Dili-based programme officer and inputs from a range of others. It also facilitated the contracting of a g7+ national as Senior Policy Officer, giving the Secretariat a capacity to undertake policy work. There has been progress on developing a

53

permanent institutional structure for the g7+, with a BSI governance options paper setting out options for structure, decision making and financing. The Secretariat began to function increasingly well, taking on more core functions such as regular communication with members and coordinating common positions for international evens. It managed two major events without ODI assistance, including a meeting at the ILO and a side-event at the 2013 UN General Assembly. In the international engagement workstream, the g7+ engaged successfully with the IDA allocation process, adding its voice to proposed change in the allocation formula that resulted in a significant increase in resource allocations to fragile states. The g7+ also began to engage with the Global Partnership for Effectiveness Development Coordination, with support from BSI in the form of articles and blogs in the media and preparation of a state on implementation of the New Deal. The team produce a number of research and peer learning outputs, including a Note on the Fragility Spectrum and an updated guidance note on conducting fragility assessments. Score A++, Output exceeded expectation Support to the g7+ Secretariat 2014-15 Over 2014/15, BSI’s support for the Secretariat continued to move away from administrative and logistical support by ODI staff, towards remote support and occasional visits. The BSI full-time contractor was replaced by a g7+ national, whose contract is facilitated by BSI but who reports directly to the Secretariat’s General Secretary, to enhance ownership. The g7+ has demonstrated the ability to organise major international events and coordinate among its members. It supported a 2014 UN General Assembly side event, with only minimal support from BSI. While there was a slight loss in the quality of preparation, the trajectory is positive as the Secretariat gains experience. BSI organised a Secretariat staff retreat and training event in December 2014, covering skills such as project management, communication, stakeholder management and personnel effectiveness. Feedback from the g7+ Secretariat on BSI’s support remains extremely positive. BSI is now more focused on the provision of substantive support, including bespoke research and policy papers. It contributed to the production of the first ‘New Deal Progress Report’, undertook a major research project into Natural Resource Management and is producing a set of ‘New Deal Innovation papers’, which are in draft. Score A+, Output moderately exceeded expectation

54

Annex 2 Validation of stories of change Introduction An important component of BSI’s monitoring strategy is to capture significant institutional change through Stories of Change. The intention is to capture institutional changes that have occurred as a result of BSI’s interventions that were not individually anticipated in the programme design and therefore not reflected directly in outcome or impact indicators. The logframe captures the number of these significant institutional changes that are produced, as an output indicator. The Stories of Change also complement the indicators in the logframe by providing a more detailed and contextualised assessment in narrative form of the programme’s achievements. The output indicator is as follows: “Significant changes (policy, institutional, financial or services) from catalytic effects of BSI.” These are to be generated by BSI staff, informed by recipient assessment and verified on a sample basis by this external evaluation. The following Stories of Change have been prepared by the BSI team: South Sudan 1. 2013-14 – Budget Planning and Aid Coordination 2. 2013-14 - Government Led Delivery of Local Services and Infrastructure 3. 2013-14 – Public Financial Management Reform 4. 2014-15 - Public Financial Management Reform 5. 2014-15 - Changing Behaviour in the Budget Directorate 6. 2014-15 - Rebuilding systems in a state of fragility 7. 2014-15 - More government funding of local services (Local Services Support) Liberia 1. 2013-14 – Citizen’s Guide to the Budget and Communications Strategy 2. 2013-14 - Developing strategic budgeting processes 3. 2014-15 – Revenue 4. 2014-15-Improved budget preparation and execution during and after the Ebola crisis Uganda 1. 2014-15 – Making budget information accessible During our field visits to Liberia, South Sudan and Uganda30, we examined a sample of activities in detail and others more briefly. Consequently, we are able to verify some of the Stories of Change in full, while for others we did not undertake an independent assessment of results claimed or their attribution to BSI activities.

30 For DRC, no Stories of Change have been documented

55

For each Story of Change, we have assessed: 1. whether the Story of Change is a fair assessment of the results that are achieved and BSI’s

contribution to them (verified as significant/ assessed as not significant/not assessed); and

2. whether the change in question merits being treated as ‘significant’ in terms of its actual or potential contribution to the programme’s overall goal of improving economic governance in the partner countries (significant/not significant/not assessed).

Summary of the validation The table below shows the results of the validation exercise for all 12 Stories of Change.

Story of Change Accuracy Significance

South Sudan - Budget Planning and Aid Coordination (2013-14)

Not assessed Not assessed

South Sudan - Government Led Delivery of Local Services and Infrastructure (2013-14)

Verified Significant

South Sudan - Public Financial Management Reform (2013-14)

Verified Not significant

South Sudan - Public Financial Management Reform (2014-15)

Verified Not significant

South Sudan - Changing Behaviour in the Budget Directorate (2014-15)

Verified Not significant

South Sudan - Rebuilding systems in a state of fragility (2014-15)

Verified Not significant

South Sudan - More government funding of local services (Local Services Support) (2014-15)

Verified Significant

Liberia - Citizen’s Guide to the Budget and Communications Strategy (2013-14)

Verified Significant

Liberia - Developing strategic budgeting processes (2013-14)

Verified Significant

Liberia – Revenue (2014-15) Verified Significant

Liberia - Improved budget preparation and execution during and after the Ebola crisis (2014-15)

Verified Not significant

Uganda - Making budget information accessible (2014-15)

Verified Significant

Results of the validation South Sudan - Budget Planning and Aid Coordination (2013-14)

56

Not assessed. Much of this Story of Change recounts the ability of the Budget staff to maintain the budget process during the absence of the BSI advisors during the period from December 2013 through to March 2014. It was not possible to form a view on the nature and extent to which the competencies Budget staff developed during this time. South Sudan - Government Led Delivery of Local Services and Infrastructure (2013-14) Verified. Building on the progress made in 2012/13, BSI continued its support in 3 main areas: better define sector responsibilities for service delivery, and strengthen institutional capacities and systems for service delivery; develop inter-governmental transfers supportive of local service delivery and improve Local Government’s PFM system; and develop aid programmes supporting local service delivery that are managed by government organisations and use government systems. A Joint Plan of Action was agreed which set out the Government’s agenda for reform. BSI assisted with facilitating technical work on financial and human resource management. BSI worked with the Directorate of Budget to propose the inclusion of new county transfers in the 2013/14 budget ceilings. This paved the way for supporting the Finance Ministry and relevant line ministries to design the transfers for basic healthcare, primary education, rural water and local infrastructure, which were then included in the approved 2013/14 budget. In the education sector, BSI helped to ensure that the secondary school capitation grant to be provided by the DFID GESS project ‘shadow-aligned’ to the government-funded primary school capitation grant systems. BSI’s continued to support the Ministry’s original objective of supporting the design of aid programmes supporting local service delivery that are managed by government institutions and use government systems. BSI supported the government in the design of the Local Governance and Service Delivery Project (LGSDP). Significant. Essentially this continued support in facilitating Government to adopt a greater focus on local service delivery has been a major achievement. The progress has been very significant for several reasons, namely: the needs and the constraints are now clearly articulated and understood; there is a consensus within government on what needs to be done and how to do it; the initial barrier and negative perceptions around fiduciary risk have been moderated to the extent that DPs not only started to commit funds through this mechanism but in the case of the World Bank actually negotiated a local government support project using government systems. South Sudan - Public Financial Management Reform (2013-14) Verified. At the start of 2013-14 the challenge facing BSI was how to revitalise the Finance Ministry’s commitment to, and ownership of, the PFM reform agenda, in a context of continuing fiscal uncertainty. The early part of the financial year brought significant progress with commitment of DPs to provide budget support. A number of multilaterals indicated their commitment to provide budget support within the context of an IMF programme, while bilateral agencies expressed interest in establishing the South Sudan Partnership Fund (SSPF), in order to strengthen and make greater use of country systems. After extensive stakeholder consultations, the design process for the SSPF commenced in August 2013. In addition, an EU State Building Contract (SBC) to support education and

57

health salaries was being discussed. All of these endeavours, plus any chance of re-invigorating the PFM reform programme came to a halt in December 2013. Not significant. It is not evident from the content of this Story as to what is the claimed change that occurred during 2013-14, other than the major disruption and severe funding consequences arising from the recommencement of the civil unrest in December 2013. South Sudan - Public Financial Management Reform (2014-15) Verified. The past year has proved a challenging environment in which to introduce PFM reform. The civil unrest, leading to increased security spending, combined with the decline in oil revenues, all led to a very difficult context in which to push ahead with the Finance Ministry’s reform programme. The ‘withdrawal’ of funding commitments from the Development Partners added to the funding constraints. However, despite these difficulties, there have been opportunities, supported by BSI, to improve elements of the wider PFM system. The most notable features include: greater awareness of the implications of the unfunded arrears; increased compliance of the 2014/15 Budget with the legal requirements relating, inter alia, to debt and borrowings; producing and publishing quarterly macro-fiscal reports, and accelerating the consideration of setting up of a National Revenue Authority (NRA). Not significant. The achievements have been important. But they are largely, stand alone events delivered outside the mainstream of the Ministry’s PFM reform agenda. Their significance is difficult to discern. However, BSI’s flexible response in mobilising an advisor in relation to the setting up of an NRA has the potential to be a very significant, should the Government decide to proceed as a matter of urgency. South Sudan - Changing Behaviour in the Budget Directorate (2014-15) Verified. It was possible to confirm that the training of the Budget staff took place through interviews and a review of the training programme together with the follow up report prepared by BSI. However it was not possible to validate the changes in individual competencies resulting from individuals’ participation in the training. The claimed change in behaviour could not be assessed. A selection of the participants did confirm that the training had been beneficial. Not significant. As a ‘one off’ event, the importance of the training is diminished by the lack of sustainability. Given the acknowledged need to build skills, it would have been significant if this event had started a routine, structured approach to training delivery by building on the success claimed. South Sudan - Rebuilding systems in a state of fragility (2014-15) Verified. Strengthening the aid management system in South Sudan had been a significant achievement of BSI during its first few years of engagement. This success was acknowledged in the Mid-Term Evaluation. However since Mid 2013 the system was not maintained due in

58

part to the deterioration in the relationship between Government and the Development Partners. Between October 2014 and March 2015, BSI has helped the Aid Coordination Department , working alongside the UNDP Advisor, to restart aid data collection and to start supplying data to other government and donor processes. Not significant. BSI’s inputs have been directed towards modifying the design and resurrecting the use of the database for the Aid Information Management System. As such, the recovery of a failed system does not have the significance of the original implementation which already has been assessed as a positive change in the period 2011-12. Furthermore, the value of this work has yet to be seen as the Government/DPs relationships have yet to be restored to their status prior to the December 2013 crisis. South Sudan - More government funding of local services (Local Services Support) (2014-15) Verified. The initial work by BSI in 2011-12 to identify a new aid instrument for donor funding has evolved in the last two years to the design and implementation of a new fiscal transfer mechanism to help fund service delivery at the local government level. BSI supported government in progressing the Local Services Support agenda and BSI has facilitated greater engagement between the Ministry of Finance and sector ministries on release of funding and on monitoring and accountability for use of funds. In addition, it has supported the implementation of the oversight and monitoring arrangements through the establishment of the State Transfer monitoring Committees and the State Mentoring Teams. Significant. This is the most impressive achievement of BSI’s work in all four countries. BSI has played a pivotal role in assisting Government introduce a system of fiscal transfers which enables funding to flow to local governments. BSI’s role has been both strategic (in terms of helping to set up the planning processes through the sector development frameworks and joint plan of action) and tactical (in relation to helping design the detailed operational procedures such as the finance and human resource management manuals for use by local governments). The significance of this major reform is evidenced by the Government’s continued payment of transfers to local governments despite the severe budget constraints nationally. As of February 2015, SSP 70,240,311 had been transferred to cater for county block, education operating and capitation grant, health departments and hospitals operating as well as water operating costs. Anecdotally, it was reported that this funding is now resulting in improved attendance rates in primary schools. Liberia - Citizen’s Guide to the Budget and Communications Strategy (2013-14) Verified. In 2013, Liberia’s MoF requested BSI to assist the Open Budget Initiative (OBI) for the second year running to devise a strategy for communicating and disseminating information on the 2013/14 National Budget to the wider public. BSI’s communications officer worked closely with the OBI to devise a strategy centred around publishing and disseminating a “Citizen’s Guide to the Budget” for 2013/14. This second round of producing the Guide resulted in a ‘step change’ in the quality of the published document resulting in a high quality product. Furthermore, the output was positioned in a wider communications strategy with the specific objective of enabling the OBI to repeat the exercise for 2014/15.

59

Significant. In the Mid Term Evaluation the production of the first guide was assessed as ‘not significant’, as the product was not at that stage anchored in a wider institutional process. Since then, the team has produced a second iteration while MoF staff were able to produce a third edition (despite delays caused by the Ebola crisis) with minimal BSI support. As this now seems to be an established process, it can be verified as a significant institutional change. Liberia - Developing strategic budgeting processes (2013-14) Verified. BSI’s initial engagement with the Ministry of Finance was focused on helping with the introduction of an MTEF and its related processes. In 2012/13 the credibility of the MTEF process was reduced amongst the ministries and agencies due in part to the volatility in the actual budget releases caused by over estimates of revenues. The ensuing 2013/14 budget process was flawed with many of the stages and logic of the MTEF process were either hurried or even abandoned. As a consequence the IMF observed that “These weaknesses have undermined the credibility of the budget process and the consequent budget estimates and appropriations.” For the 2014/15 budget exercise BSI was successful in helping the Finance Ministry adopt a more structured and coherent approach through a number of measures, including: reformatting the Budget Calendar, establishing a Budget Working Group, introducing Budget Options Paper and strengthening the challenge function in the Budget Hearings. As a result, there was increased and more meaningful political engagement in the process, budgets were more closely aligned to development priorities, and overall the budget was perceived as being more credible than in the previous two years. Significant. The achievements made in the 2014/15 budget process demonstrated that is possible to continue with budget reforms despite the difficulties and constraints that arise in the absence of strong political leadership, unrealistic and unaffordable spending levels, and the limited technical capacity within the Ministry of Finance. The fact that the budget has not been implemented as planned due to the Ebola crisis should not be seen as a weakness. Indeed, the approach to the 2015/16 budget process has been informed by the results achieved for the 2014/15 budget. Liberia – Revenue (2014-15) Verified. In the context of domestic revenue mobilisation, two important events have taken place. Firstly, the setting up of the semi-autonomous Liberia Revenue Authority (LRA) which has taken over the responsibility from the Ministry of Finance and Development Planning (MoFDP) for tax administration. Secondly, the amalgamation of the former Ministry of Finance and Ministry of Development Planning into the single MoFDP. Both events brought into the focus the question of the respective roles of LRA and MoFDP especially with regard to review and formulation of tax policy. BSI provided detailed guidance to MoFDP on the proposed organisational structure of the new tax policy division which was adopted as part of the reorganisation process. Indeed the structure of the recommendations was taken up by the Ministry as a template for guiding the restructuring of other functions. In addition, BSI provided a synopsis, informed by

60

international experience, of the division of roles and responsibilities between MoFDP and LRA. Significant. There are examples in sub Saharan Africa where the setting up of a revenue authority proceeds without sufficient attention being paid to developing the analytical skills to review and recommend improvements to tax policy. BSI’s contribution to this area is especially timely given the fiscal crisis arising from Ebola. The Government’s economic recovery programme (Economic Stabilisation and Recovery Plan) is reliant, in large part, to increasing domestic revenue mobilisation. LRA will have a role in achieving this objective through more efficient and effective tax administration. Tax policy reforms will complement these efforts. Liberia - Improved budget preparation and execution during and after the Ebola crisis (2014-15) Verified. The Ebola crisis severely disrupted the adoption of the orderly budget cycle which BSI had worked to support in previous years. BSI had to withdraw for the period from August to December 2014 but however retained contact with the key staff in the Ministry through e mails and conference calls. As a result of BSI’s engagement, it is evident that the budget timetable has been planned, and re-planned, in the best way to try and retain the integrity of the budget process. But these efforts have not resulted in any overall improvement in budget preparation and execution. The budget timetable, necessarily, has been ‘squeezed’ into the remaining time available to meet the deadline for submission to Parliament. So many of the desirable features (underpinning the MTEF approach), such as aligning the budget submissions to the priorities in the Agenda for Transformation, have been adopted. In terms of budget execution, progress has been made, with BSI’s inputs, to design a cash management system; it is too early to assess its impact. Not significant. BSI’s work in support of the budget process has been, and continues to be, important for the Ministry. Clearly the role and the contributions are highly regarded; indeed the Ministry has asked for a second Budget Advisor. But the contribution during this period is not assessed as ‘significant’ in terms of the changes that have occurred. The contribution has been mainly in a ‘fire fighting’ role helping the Ministry preserve the integrity of the budget process. Uganda - Making budget information accessible (2014-15) Verified. BSI has been working with the Ministry of Finance, Planning and Economic Development (MoFPED) since 2012 in helping to implement the Government’s Budget Transparency Initiative. In 2013, a new budget website was launched following BSI’s support to its development. A range of disaggregated budget data is available for both national and subnational governments. In the same year, a local NGO called ACODE, which had been developing a network of Budget Champions, was introduced to the partnership with BSI and MoFPED. In 2014 the budget website reached a more sustainable future with financial support from several international agencies as well as the financial commitment from MoFPED. A budget

61

call centre ‘hotline’ was started in September 2014 which responds to queries from members of the public. Plans are in place to further enhance the functionality of the website through new features such as a mobile website, a smart phone app and SMS subscriptions and feedback. Significant. The Government has a longstanding interest in and commitment to budget transparency. The implementation of this budget website represents a major step forward not only in publicising budget data but also in terms of increasing accountability through the feedback mechanism that is built into the design of the system. There is the potential for the website to help support the Government’s move towards reforming fiscal transfers and funding of service delivery.

62

Annex 3 Achievements against logframe indicators

Progress in achieving overall programme results

Narrative Indicator Target end March 2015

Achievement

Impact

Improved economic governance in partner fragile states

Allocative efficiency: Percentage government spend on social spending priorities in countries where BSI has operated continuously

Improvements in all 3 countries. Indicative targets: Liberia 30% South Sudan 6% DRC 6.5%

Not achieved: Liberia, not measured South Sudan, regression. DRC, slight regression.

Fiscal discipline: Variances between the approved budget and executed expenditure in countries where BSI has operated continuously, measured by PEFA indicator 1-231

Improvement in 2 countries where BSI has operated continuously and no deterioration in third country

Achieved: Liberia 2014: D+ South Sudan (March 2015 estimate): D+ in each year since 2010 but the PI-2 composition variance is improving year on year since 2011/12 DRC: PI-2 stayed the same at D, while PI-1 improved from D to D+

Outcome

More effective, transparent and accountable budget policies, processes and systems in the poorest and most fragile states

A clear budget process with political involvement is in operation, measured by PEFA indicator PI-11

Maintain a positive trajectory in Liberia and South Sudan

Not achieved: Both countries maintained their performance level from the start of the programme, namely B

Transparent and accountable budgets, measured by Open Budget Index (maximum=100)

Maintain a positive trajectory in Liberia and South Sudan

Partially achieved: Liberia – 4932, improved from 40 in 2010 and 43 in 2013 South Sudan: no data available

Quality of financial information provided by donors for budgeting and reporting on project and programme aid, measured

Maintain a positive trajectory in Liberia and South Sudan

Not achieved: Liberia: no improvement from D+ (2012). South Sudan: deterioration from C in 2012 to D

31 Estimated PEFA self-assessment by the MoF. 32 This is an unverified score, calculated by BSI staff using data from 2014 only. This score is in lieu of the

OBI, which has not yet been released

63

by PEFA performance indicator D-2

Number of countries implementing New Deal activities and average number of processes underway (Fragility Assessment, Compact, Aid Strategy aligned to New Delivery of high-quality Technical Assistance (TA) to support improved budget policy, process capacity and financial management systems in Ministries of Finance Deal etc)

At least 3 fragile states introduce new deal approaches

Achieved: 6 fragility assessments, 3 compacts and a range of other processes underway

Outputs

1. Delivery of high-quality Technical Assistance (TA) to support improved budget policy, process capacity and financial management systems in Ministries of Finance

1.1 Number of annual work stream assessments achieving rating of A or higher (cumulative).

9 completed assignments scored at A or above

Achieved: 9, including an Aid component for South Sudan

1.2 Number of significant changes (policy, institutional , financial or services) from catalytic effects of BSI (cumulative)

9 significant changes in the partner countries

Not achieved: 6

2. Delivery of high-quality Technical Assistance (TA) to support improved management of domestic revenue and external resources by Ministry of Finance, and to support the establishment & implementation of processes and systems for coordination, financing, management and monitoring of

2.1 Number of annual work stream assessments achieving rating of A or higher (cumulative).

3 completed assignments scored at A or above

Achieved: 6

64

government-led delivery of local services.

2.3 Number of significant changes (policy, institutional , financial or services) from catalytic effects of BSI (cumulative)

6 significant changes in the partner countries

Achieved: 6

3. Delivery of high-quality Technical Assistance (TA) to support improved transparency and accountability in government budgets

3.1. Number of significant changes in the transparency, availability, accessibility, or oversight of government budget data (cumulative)

2 significant changes in partner countries.

Achieved: 2

3.2 Number of annual work stream assessments achieving rating of A or higher (cumulative)

3 completed assignments scored at A or above

Achieved: 3 independently evaluated

4. More effective international engagement of fragile states being facilitated through the g7+ Group of Fragile States

4.1 Number of annual work stream assessments achieving a rating of A or higher (cumulative).

3 assignments completed scored at A or above

Achieved 3 2 independently verified and 1 BSI assessed

4.2 Implementation capacity of g7+ Secretariat enhanced and progress against plans in line with expectations

Includes March and September 2015 targets: 1. Fully staffed and operational Secretariat no longer requires in-country external ODI administrative support. (March 2015) 2 significant changes in international policy due to g7+ engagement. (March 2015) 3. Embedded advisors in Dili demonstrably increase on-site capacity of the

Achieved

65

secretariat, with evidence of decreasing reliance on off-site BSI advisors. (Sept 2015) 4. g7+ countries' peer-learning supported through the provision of bespoke research in at least one new thematic area. (Sept 2015)

5. High quality research and dissemination of papers on the challenges of budget strengthening in fragile states

5.1 Number of research papers produced (cumulative)

12 substantive research papers which are disseminated through web site and an international event. BSI produces some lessons-learned products and disseminates them (including within DFID).

Partially achieved 13 substantial research papers have been produced, but some are awaiting publication & dissemination on the website. One of these was the lesson-learning piece which was disseminated in September 2015

5.2 Number of (i) visits to the ODI website, across all BSI content, (ii) downloads of BSI publications, and (iii) visitors to the beyond budgets blog site. (Cumulative since 2010)

(i) ODI website BSI Page views: 25,000 (ii) BSI Content Downloads: 25,000 (iii) Beyond Budgets Blog Page views: 12,000

Achieved [but one part unverifiable in final year] Page view data unavailable due to a change in ODI systems Cumulative downloads of BSI content: 59,634 Beyond Budgets blog page views: 21,201

66

Progress in achieving programme results in South Sudan

Narrative Indicator Target end March 2015

Achievement

Impact

Improved public financial management in South Sudan

Extent of the variance in expenditure composition during the last complete FY year as measured in the first dimension of PEFA indicators, PI-2

25% Partly achieved: 24.9%

Actual amount and % of RSS budget outturns transferred for sub national service and infrastructure delivery in health, education, water and infrastructure development (including on-budget aid).

SSP 260m /3% Achieved: SSP 315m/3%

Outcome

More effective, transparent and accountable budget and aid management policies, processes and systems in South Sudan

Public Access to key fiscal information as measured by PEFA performance indicator PI-10

B Achieved: B, evidenced by: (i) Budget Docs available on the RSS website; (ii) Budget execution reports available on website; (iii) Audit Reports available on audit chamber website)

Transparency of Inter governmental Fiscal Relations, as measured by PEFA performance indicator PI-8

C+ Not achieved: Awaiting data, likely to be C

Number of aid programmes supporting local service delivery & infrastructure that are aligned with GRSS policies, strengthen GRSS systems and managed by GRSS institutions designed, approved & implemented

1 Achieved: LGSDP

Outputs

Output 1: More transparent and accountable policy, planning and budgeting processes, with greater sector

1.1 Number of policy documents prepared (PNBP, NBP, budget speech, donor book, budget book, citizens budget) in line with stages

4 Achieved: 5, comprising: PNBP, the National Budget Plan, the Budget Speech, the budget book including the Financial

67

focus on improved delivery of services and infrastructure

in budget calendar and PFMA Act

Bill and budget at a glance

1.2 No of quarterly budget performance reports prepared and made public by Ministry of Finance and Economic Planning (MoFEP) in line with the PFMA Act

4 Partially achieved: 3

1.3 Number of structured training programme/courses for budget officers that takes place

3 training levels/sessions -1, 2 & 3 take place benefitting at least 12 Budget Officers of which 30% are women

Achieved: Level 1 9 total / 5 female Level 2-1 12 total/7 female Level 2-2 9 total/0 female Level 3 22 total (8 female) MS Access training 7 total/2 female BPS training - 7 total/2 female

1.4 Result of annual BSI assessments of delivery of relevant work stream outputs (yearly - not cumulative)

One assessment scores A or above

Achieved: 1. Planning, Budgeting and Aid, A+

Output 2: Aid management processes are established that deliver progress towards the benchmarks for aid delivery set out in the GoSS Aid Strategy, which includes the design and implementation of aid programmes which use government institutions systems for service and infrastructure delivery .

2.1 Number of high level aid coordination events that take place in accordance with the aid strategy and aid management calendar.

1 Partially achieved: 3 ad hoc briefings given but not aligned to the established aid management guidelines

2.2 No. of Aid Management Guidelines published on the MoFEP website and % where associated policies/processes are being implemented

10, 40% (The target is to publish 10 guidelines on the website and ensure that at least 4 (40%) of them have associated policies being implemented

Partially achieved: 7 published

2.3 No. and sector of donor projects being

4 Achieved: 4 reported to Parliament

68

implemented by Government reflected in the budget, and the respective/related loans being tabled to Parliament at the time of budget presentation

2.4 Result of annual BSI assessments of delivery of relevant outputs (annual not cumulative).

1 Achieved: As one part of the Government-led Delivery of Local Services and Infrastructure, A++

Output 3 The Government establishes processes which provide strategic direction to economic and financial management reforms at the national and local levels that absorb good practice from the experience of other countries, international organisations and research

3.1 BSI facilitates MoFEP Senior Management meetingss to identify and review the implementation of MoFEP PFM reform priorities

Discuss implementation of a limited set of budget preparation and execution reform priorities e.g. (arrears, debt management, non-oil revenue, budget execution including service delivery transfers)

Achieved

3.2 BSI supports the organisation and participates in external/joint review of support to and implementation of Core MoFEP PFM reforms/functions

Quarterly Joint GRSS-DP &TA review meetings to discuss PFM reform priorities and review implementation progress

Not achieved: Minister held 3 informal fiscal briefings with the international community in October & December 2014 and March 2015

3.3 BSI supports MoFEP to identify and develop guidelines for implementation of reforms to improve budget execution (arrears and contingency fund) being operated and managed by MoFEP

2 reforms Achieved: (NB possible ‘double counting’ of the work on arrears with indicator 3.1 above)

3.4 Result of annual BSI assessments of delivery of relevant work stream outputs (assessed on an annual basis)

One assessment scores A or above

Achieved: Public Financial Management Reform, A

69

Output 4 The Government establishes processes which provide strategic direction to economic and financial management reforms at the national and local levels that absorb good practice from the experience of other countries, international organisations and research

4.1 Number of service delivery frameworks developed, disseminated and implemented as part of the Local Services Support Joint Plan of Action

4 (the target is to synthensise and disseminate the 4 SDFs in education, water, health and local infrastructure - CDG)

Achieved

4.2 Number of sectoral and inter-sectoral structures for coordination and management of local service delivery functional - meeting and performing stipulated

5 Achieved: All the 8 TWGs have been meeting

4.3 Local Government PFM and HRM system designed, approved and under implementation

1 All LG PFM reforms and training follow the LG PFM manual 2 LG HRM manual published

Achieved

4.4 National and State level institutions and systems for oversight, supervision, inspection and top-down accountability are designed and operational

1. STMC meeting at least 6 times, 2. CTMCs established in at least 5 states and meet quarterly to review budget and performance reports, 3. LG Audit and Performance Assessment System designed.

Achieved

4.5 Social Accountability Framework developed

1. Development of the

Not achieved

70

framework outline, 2. Consultation workshop with state and non-state stakeholders, 3Validation meeting

4.6 Result of annual BSI assessments of delivery of relevant work stream outputs (assessed on annual basis)

One assessment scores A or above

Achieved: Government-led Delivery of Local Services and Infrastructure, A++

Progress in achieving programme results in Liberia

Narrative Indicator Target end March 2015

Achievement

Impact

Improved economic governance in partner fragile states

Allocative efficiency: Increased government spend on social spending priorities in countries where BSI has operated continuously, measured by % expenditure on budget priorities as defined by partner country, or on social spending as defined by IMF if not available

30% Not assessed Figures expected later in the year

Fiscal discipline: 1. Decrease in variances between the budget and executed expenditure 2. Extent of the variance in expenditure composition during the last complete FY year in Partner Countries as measured in the first dimension of PEFA indicators, PI-2

Improvement or no deterioration

Achieved: Overall, no deterioration at D+ which was the assessment in 2012/13 Overall assessment derived from: Indicator 1 = D: The variance in expenditure composition has exceeded 15% in two of the last three years. Indicator 2 = A: The average level of expenditures charged directly to the Contingency Reserve Fund Vote, has been

71

substantially less than 3% (0.5%).

Increased government capacity to deliver services: Increased use of government procedures for aid monies. Proportion of aid using government procedures, measured using definition in PEFA indicator D-3.

Improvement or no deterioration

Achieved: No deterioration at D. Less than 50% of aid funds to CG are managed through national procedures.

Outcome

More effective, transparent and accountable budget policies, processes and systems in the poorest and most fragile states

A clear budget process with political involvement is in operation as measured by PEFA performance indicator 1-1 or other indicator if more appropriate

Improvement or no deterioration

Achieved: No deterioration at B overall Components: Sub indicator 1 = B: The budget calendar is included in the Budget Call Circular. It is broadly respected, and leaves M&As over 4 weeks to complete their submissions. Sub indicator 2 = A: A comprehensive and clear Budget Call Circular is issued to M&As, which reflected Budget ceilings pre-approved by Cabinet prior to the circular’s distribution to M&As.

More transparent and accountable budgets, measured by Open Budget Index (maximum score 100) or by other indicator if not available

Improvement or no deterioration

Achieved: Improved to 49 from 43 in 2012. (this is an unverified score, calculated by BSI staff using data from 2014 only. This score is in lieu of the OBI, which has not yet been released).

Quality of financial information provided by donors for budgeting and reporting on project and program aid as measured by PEFA performance indicator D-2

Improvement or no deterioration

Achieved: No deterioration from 2012 at D+ Components: Sub indicator 1 = C: Most donors provide budget estimates of anticipated

72

disbursements of project aid at least 3 months in advance of the government’s fiscal year but not in a format consistent with the Government’s budget classification. Sub indicator 2 = D: Most donors do not provide quarterly reports on the disbursements made within two months of end of quarter.

Outputs

Output 1: Stronger budget policy and process capacity in the Liberian Ministry of Finance

1.1 A more coordinated budget preparation process, measured by: - budget working group meeting on a regular basis - budget working group meetings involve key stakeholders - budget working group coordinated by local staff - better adherence to the budget calendar than in previous years

Budget Working Group meets during the strategic budget phase and is embedded into new Ministry Finance and Development Planning structure

Partially achieved: Budget Working Group was unable to meet in the second half of 2014 in order to prepare for Budget FY2015/16 due to the pressures and challenges of the Ebola Crisis. The BWG has met regularly since December 2014 to coordinate the strategic phase of the budget, with attendees from the Budget and Planning divisions within the Department, as well as Economic Management (macro, aid and debt). However, temporary reduced staffing in the budget coordination unit has slowed the coordination process, e.g. fewer minutes taken, delayed Budget Call Circular. Attendance was also initially weak due to staff settling into their responsibilities in the new structure.

73

1.2 Budget options paper produced for the first time, in order to inform Ministers early on of the direction of the budget. Quality indicators: - correct issues addressed - process involved the right people - timely (before the President's speech)

Budget Options Paper produced and used by Ministers to make allocative decisions to inform the Budget Call Circular. Revised Budget Options Paper presented to the President/Cabinet and legislature

Not achieved: Delays in budget preparation due to the Ebola crisis meant that no Budget Options Paper or BCC was drafted in 2014 for FY15/16 planning. Budget Options Paper is currently being drafted.

1.3 Continued capacity development of national civil servants involved in the budget process

Budget analysts (trained in 2013/14) working in Ministries and Agencies with the Budget Committees

Not achieved: Budget analysts have not been rolled out to M&As, partly due to the Ebola crisis. Devolved budget analysts are on hold until an implementation plan is developed: there is uncertainty about whose payroll they should be on and how they can be effectively integrated, avoiding natural distrust from M&As and sidelining.

Output 2: Increased transparency and accountability around the budget in Liberia.

2.1 Citizen's Guide to the Budget produced. Quality indicators: - Improvements in presentation of data (more accessible to the general public) - Smaller quantity of mistakes

Citizen's Guide to the 2014/15 budget produced, with quality improvements.

Achieved: Citizen's Guide to the FY2014/15 budget delayed until early 2015, due to the recasting of the budget in response to the Ebola crisis (budget passed in December 2014).

2.2 Budget data available online

Basic budget website set up.

Not achieved: Change in personnel in Department of Budget and Development Planning, as well as new MFDP website, restarted the basic website discussions.

2.3 Ministry of Finance more strategic in their

A communications

Not achieved.

74

communications around the budget data.

strategy is developed, covering CSO engagement, training to journalists and more.

Although a strategy has been developed, no engagement on training and outreach due to the Ebola crisis and other competing work streams.

Output 3: Stronger tax policy function in MFDP to support new LRA and provide enabling legal framework for revenue mobilisation and delivery of fair, equitable, efficient tax system

3.1 Appropriate structure for Revenue and Tax Policy Division (RTPD) in new MFDP in place, with clear separation of functions between LRA and RTPD and templates for roles and responsibilities for RTPD key staff.

RTPD structure to be agreed; with terms of reference for staff with clear functions/ responsibilities

Achieved

Progress in achieving programme results in DRC

Narrative Indicator Target end March 2015

Achievement

Impact

Improved economic governance in DRC

Allocative efficiency: % government expenditure on social sector spending

6.5% Not achieved: 5.8%

Budget execution: 1. Variance between budgeted and actual expenditure. - PEFA Indicator PI-1 2. Total actual expenditure relative to the initially approved budget - PEFA indicator PI-2:

PI-1: 31% PI-2: 44%

Achieved: PI-1: 32.5% PI-2 48.1%

Increased fiscal transparency: 1. Publication of annual central government budget 2. Decline in percentage of transactions outside the expenditure chain.

1. Budget on MoB website 2. Spending outside chain =0%

Partially achieved: 1. 2014 Budget published on internet. 2. Spending outside the expenditure chain continued & was as much as 40%.

Outcome(s)

1. COREF is able to effectively oversee and coordinate PFM reform implementation.

Preparation/revision of a three-year rolling, Priority Action Plan (PAP) for PFM reform and serves as centrepiece by government and donors to monitor

Revised PAP 2015-17

Achieved

75

implementation of PFM reform

2. Good progress in implementing financial decentralisation in the 11 provinces.

Preparation of a core set of preparatory actions (Platforme Minimale or "PM") to be taken by each of the 11 provinces regarding the financial decentralisation of delivery of public services.

Revision of PM Achieved

Outputs

Output 1: Improvement in COREF coordination of PFM reforms

1.1 Number of monthly meetings of each of the five COREF technical sub-committees to review status of implementation of PAPs regarding each pillar of PFM reform

At least one meeting per month

Partially achieved: Hiatus in meetings from February until September 2014, but since Sept 2014, now regular sub-committee meetings.

1.2 Government and donors meet regularly to discuss status of PFM reform implementation and donor coordination

At least one meeting per quarter of the "Cadre Permanante de Concertation"

Achieved

1.3 COREF raises and resolves key implementation issues with senior ministerial officials as needed.

Monthly meeting between COREF Coordinator and Minister of Finance and Minister of Budget.

Partially achieved Throughout 2014, meetings were only with Minister of Finance, for political reasons. This was resolved with a Cabinet reshuffle in January 2015 and COREF is now able to raise implementation issues as required.

Output 2: Provinces are prepared for financial decentralisation

2.1 Preparation and implementation of road maps for implementation of financial decentralisation at the individual provincial level

Implementation of Platforme Minimale in at least half of the 11 provinces

Achieved: Platforme Minimales have been completed in all 11 provinces and are being implemented.

2.2 Development and monitoring of quantifiable indicators of financial decentralisation at the provincial level

Implementation of indicators in 3 initial provinces.

Achieved: Indicators have been identified in all 11 provinces

2.3 Preparation and initial implementation of priority action plans for financial decentralisation at the provincial level

Preparation of draft PAPs in 3 initial provinces

Not achieved: Draft PAPs have yet to be prepared in any of the 11 provinces

76

Annex 4 BSI’s Theory of Change 1. Introduction

1.1 The Budget Strengthening Initiative is hosted by ODI and was set up in 2010, as a means of providing high quality, independent advice to governments of fragile or transitional states, in the area of economic governance.33 The BSI programme has evolved in its first three years of operation and experience to date – in particular the external mid-term evaluation - has generated new insights about the way in which it is possible to promote change. The programme has developed effective programmes at an international and country partner level which necessitates a further refinement of the earlier theory of change.

1.2 This note sets out the various elements of the theory of change (TOC) including the cause and effect relationships between BSI inputs and expected outputs. We intend to use it as an adjunct to the log frame and to draw on both as a basis for the monitoring and evaluation of the programme.

1.3 The workstream supporting the g7+ secretariat has been removed from the Theory of Change, as recommended by the mid-term evaluation. This removes the international fragile states policy dimension.

2. Problem Statement

2.1 Weak economic governance exists in fragile states and has significant implications on prospects for development. It results in inefficient allocation of resources (with negative effects on service delivery), increased opportunities for corruption and slow development of government capacity. All of these factors delay or impede a transition out of fragility and statebuilding.

2.2 Governments of fragile states by definition have low capacity to implement improvements to their public finances, in particular in a challenging environment where management is weak, priorities are many, and uncoordinated demands are made by a variety of actors.

2.3 This need for better budgeting and better accountability in the poorest and most fragile states has been recognised by the international community. However, technical assistance in the area of public finance management has often been inappropriate to the needs of fragile state governments34: either through being unsuitable (not possible to implement in their context, or one-size-fits-all), lacking political awareness, insufficient (not easily available especially at short notice), unsustained (short-term advice with little implementation assistance) or implemented as window-dressing (as a superficial response to donor requirements).

2.4 What is needed is high-quality advice and implementation assistance to governments of fragile states as a means of strengthening their capacity and decreasing their fragility.

33 BSI works more specifically in the area of public financial governance. 34 C.f. Matt Andrews, The Limits of Institutional Reform in Development, 2013

77

Advice should be suitable to the needs, possess political awareness, be available at short notice, be sustained and not respond to donor requirements but actual demand. The Budget Strengthening Initiative seeks to provide this kind of advice and implementation assistance, with the characteristics of: (i) being independent and trusted by government; (ii) being mindful of implementation; (iii) being problem driven and iterative; (iv) providing sustained support, from advice to implementation; (v) being research-based; (vi) having capacity development as a key aim.

3. Key Hypotheses and Assumptions

3.1 The Key Hypothesis behind this theory of change is that:

Problem-driven support …

… which takes account of context, constraints, capacity and political incentives …

… will enable Ministries of Finance to sustainably spearhead improvements in public financial governance (resource allocation, management and reporting).

3.2 Key assumptions behind this theory of change are:

a. The BSI project selects partner countries to work with on the basis of need but also

on the basis of available political space for taking action to improve public financial

institutions. As a pre requisite there should be an emergent strategy or potential to

develop a reform programme into which TA inputs can address well defined

bottlenecks or obstacles to progress. However interventions can also be

opportunistic so that the level of commitment can be tested. Country interventions

are scaled and paced in line with developments and this “politically intelligent”

approach makes it more likely that BSI managers can anticipate risks of political

instability or of lack of sufficient support for reforms. They can also ensure technical

interventions take into account the political (and other) incentives of stakeholders.

b. It is accepted that not all the interventions will deliver results but it is expected that

there will be certain investments which generate a disproportionate return with very

high benefits.

c. Capacity development is challenging in fragile state contexts, yet extremely

important. BSI-type support in a particular area is likely to commence as direct

support for implementation and progressively evolve towards an increased focus on

capacity development, skills and knowledge transfer. It should be noted that while

BSI aims to build capacity, it does not seek to be the provider of large scale training,

but may as part of its catalytic/problem-solving approach facilitate the provision of

training by other providers.

d. Increased government financial transparency in terms of availability and

dissemination of budget and expenditure data will be able to be used by civil society

and other stakeholders to increase accountability. BSI, following its Mid-Term

Evaluation, will work mainly on the former “supply” side, and will look for

78

opportunities to work on the “demand” side where this is appropriate (and not

constrained by work on the “supply” side).

e. There are many other actors and stakeholders working on improving public financial

governance in the countries where BSI works; BSI input is small compared to that of

others, but it can hope to be a catalyst for larger changes.

Further assumptions based on each step are detailed in the table in section 4.3.

4. Interventions to Results: Processes and Causality 4.1 The BSI programme log frame and the country programme nested log frames set out the broad parameters of the expected results. By adding information on the processes and the causal mechanisms we can show the links between the planned results and the underlying TOC more explicitly.

4.2 BSI provides four distinctive types of support (as identified in the mid-term evaluation):

I. help the partner government to articulate an overall PFM reform strategy, providing strategic advice to ministers and senior managers

on the prioritisation, sequencing and design of reforms.

II. help with the design and implementation of core budget and aid management systems. In doing so, it works alongside other TA

providers, often in gap-filling mode. The BSI team offers particular skills on system design.

III. help identify and resolve problems or blockages in the delivery of budget reforms and in the management of the development

partnership. BSI is an embodiment of the approach to TA known as ‘problem-driven iterative adaptation’. Its distinctive features are

that it begins from recognised problems, rather than international best practices, it pursues institutional change as an iterative, step-

by-step process, and it emphasises building consensus among stakeholders and across institutional boundaries. David Booth describes

it in terms of brokering solutions to collective action problems.

IV. help to fill in a missing management layer in its counterpart institutions. While the Ministries of Finance in both South Sudan and

Liberia have policy-competent individuals in senior positions, they lack a middle management capable of seeing reform initiatives

through to a successful conclusion. In South Sudan, in particular, where BSI has a larger team, BSI’s presence boosts the capacity of the

counterpart institution to manage and absorb change. In both countries, it supports the capacity of the Ministries of Finance to

communicate and support line ministries, strengthening their engagement with the budget process.

In addition, BSI has:

V. A specific research capacity, focused on primary issues of public financial governance in fragile states. The research–implementation

relationship is intended to be a virtuous circle.

80

VI. An explicit aim of capacity development in the countries (mainly Ministries of Finance) where it works, by peer learning or other

mechanisms.

4.3 BSI aims to work mainly in the following three areas, considered critical to good public financial governance in fragile states:

i. Budget policy, process capacity and financial management systems in Ministries of Finance

ii. Management of resources (external resources and domestic revenue) by Ministries of Finance

iii. Transparency and accountability in government budgets.

4.4 Other intervening factors and other actors: This Theory of Change cannot appropriately give reference to all the other intervening factors on the ground that BSI outputs interact with, be it our government counterparts or the many other actors involved in public financial governance. This is diagrammatically described by the purple box in the TOC diagram at the end of this document.

4.5 There are several processes whereby interventions are expected to lead to results within the BSI. The tables below summarise how these processes are expected to lead to desired results and the likely causal mechanism. It should be read in conjunction with Diagram 2, which is a simplified rendition of this table. The range of cause and effects can vary but are likely to include:

81

Type of support Process Example Change -Outputs

Assumptions/Assumed Causal Mechanism

Longer Term Outcome Impact / Rationale

III.

Pro

ble

m-s

olv

ing

I . Strategic Advice

Strategic analysis for partner countries by BSI, covering policy and programme options. Confidential and independent TA & papers in partner countries for MoF or other Government organisations.

Appropriate PFM reform design and approaches aiming to achieve higher impact and sustainability Reforms promulgated in law or administrative directives.

Consultancy visits and reports are (even partly) absorbed and acted upon by decision makers. They are followed up by (IV) implementation support. Advice on PFM reforms, prioritisation, sequencing and approach persuades officials and influences Ministers to accept new policy and directions.

PFM reforms and implementation are realistic, focussed on priorities and can deliver improved budget policy and process capacity of finance Ministries.

PEFA or similar indicators will improve over time. Strategic influencing of national PFM policy through high quality expertise offering relevant, appropriate, research-based and independent advice.

More strategic choices made by partner governments in the management of their finances, leading to more resource to allocate, and better resource allocation for development outcomes. Measurement indicator (Impact indicator 1): Increased government spend on social spending priorities.

82

III.

Pro

ble

m-s

olv

ing

II. Systems Design (core budget and aid management)

Specific new procedures and practices proposed by BSI. Also designed and implemented through facilitation by TA working with national staff. Most BSI advisors have experience of working within and with Ministries of Finance and have a strong problem solving orientation. This is combined with direct experience of working on implementation.

More systematic & comprehensive approaches to budget preparation institutionalised New system (e.g. for budget execution or aid information management) introduced & supported by motivated and capable national staff. A method is devised to enable donors to work more effectively through government systems. For example, a budget support-type instrument for local service delivery is set up.

Government wants improvements in basic mechanisms of expenditure planning, management and financial control introduced quickly. BSI can respond to urgent needs. Addressing a specific problem makes TA inputs more immediately relevant.

Planned public spending is realistic, comprehensive and determined centrally. Actual spending is controlled, on target and diversion minimised. Government has better data on aid available to it. PEFA or similar indicators will improve over time. Improved performance or the specific development of a defined solution. For example ensuring aid is captured on budget or financing instruments are developed that can be used to channel donor funding through government systems.

Partner governments are more capable of delivering on what they promise in terms of the budget, to ensure that services are actually delivered to populations. Measurement indicator for fiscal discipline (Impact indicator 2): decrease in variances between the budget and executed expenditure.

83

III.

P

rob

lem

-so

lvin

g

IV. Facilitating implementation

Opportunistic or demand led interventions supporting specific reforms and institutions: long-term BSI advisors assist in the detailed implementation of particular reforms.

A functional system aimed at solving a direct problem is in place or developed. BSI staff work alongside partners to ensure they are motivated and capable to both sustain and continue to improve the system.

Consultancy inputs can assist reformers to demonstrate the effects of new approaches in specific areas (e.g. budget planning or execution across Government) or in selected sectors (e.g. Health or Education) or at different levels of Government (e.g. central/Provincial)

Effective planning and budgeting systems leading to better decisions on allocation and more effective use of domestic and external finance.

A capable workforce, able to put policy recommendations into practice and set up systems that are appropriate to and functional in the context. Measurement indicator as above (Impact indicator 2).

V. Research

BSI conducts research into PFM experience

Improved knowledge base and ideas, with scope for new approach to PFM by developing countries and practitioners.

New knowledge identifies high impact and most practical innovations. Partners show interest in taking these up.

Innovations in approach being tried and tested with positive results.

More and better quality research available on the challenges of budget strengthening in fragile states, leading to more efficient interventions by donors and partner governments.

VI. Capacity Development interventions

Key practitioners from budget, finance and revenue functions are given hands on mentoring by BSI staff.

A more confident and capable workforce.

Key staff want skills and capability to manage and sustain new systems. BSI can provide relevant, practical on the job training and mentoring.

Higher and sustained levels of performance by budget, finance and revenue personnel

A more capable government, able to deliver services accountably to their citizens. Measurement indicator (Impact indicator 3): increased use of government procedures for aid monies (a key part of statebuilding and emerging from humanitarian situations where most aid monies flow through non-government actors).

84

BSI PROBLEM SOLVING APPROACH (PDIA)

SYSTEMS DESIGN

FACILITATING IMPLEMENTATION

TARGETED RESEARCH

(Examples) Selected civil servants mentored and coached. BSI staff help to fill in a missing management layer in its counterpart institutions, setting up basic processes & systems. BSI staff help to identify a key problem/ blockage and asked to work on it.

Institutional processes

(Examples) A budget support –type instrument for local service delivery is set up (South Sudan). A new system for aid information management is introduced & supported by motivated and capable national staff (Liberia). Better trained budget staff follow established procedures for the budget process (S. Sudan).

(Examples) Local services receive planned budget allocations Government systems are strengthened Increased government revenue Budget is freed up for priority expenditures Budget credibility increases

STRATEGIC ADVICE

CAPACITY DEVELOPMENT

RESEARCH Informing all stakeholders: new ideas and good practice

Outputs: High Quality

TA

BSI Activities Outcome: More effective, transparent, and accountable

policies and systems

Impact: Improved Economic

Governance resulting from

partner Government action with

donor support.

Other Actors Government agencies, Donors, IFIs, civil society, etc. have major role in policy dialogue, provision of advice and TA on the ground, offering direct support of their own. These inputs cannot be individually detailed here as they are numerous and vary from country to country – but clearly they are essential to overall progress and are taken into account by BSI.

SUPER IMPACT: The promotion of peace and state-building through the development of capable, effective, transparent and accountable public financial management institutions which ensure macroeconomic stability, increased domestic resource mobilisation and the delivery of services at a

BSI ToC Diagram - Summary of Results Chain and Change Processes (for more details including assumptions see table in section 4.5)

Annex 5 g7+ programme Theory of Change 1. Long term change being sought

1.1 The g7+ programme within BSI aims to support the more effective international engagement of fragile states through the g7+ Group of Fragile States.

1.2 This Theory of Change (TOC) has been separated from the overall BSI TOC upon the recommendation of the mid-term evaluation, which was finalised in July 2013.

2. Context for intervention

2.1 Context analysis and problem statement

2.1 Fragile states are an increasingly important focus of development efforts worldwide. This is because of a growing recognition that poverty reduction efforts have had a limited impact in countries affected by conflict, and that development and peace are mutually reinforcing, whereas the absence of one undermines the other. Research shows that the majority of extreme poverty is now in fragile states, and that by 2025 the vast majority of the extreme poor will live in fragile settings.35 As the world looks towards the next phase of global development efforts post-2015, including ambitions to eradicate extreme poverty by 2030, fragile states have become the new frontline of poverty.

2.2 Meanwhile, there is a greater expectation, partly emanating from aid effectiveness concerns, that recipient countries take more of a lead in development cooperation. This presents a problem for development partners, as Fragile States are by their nature both fragmented and lacking in capacity.

2.3 The g7+ has emerged as a timely solution to this dilemma. Acting as a country-led mechanism for fragile states to share and learn from each other, and also as a platform for collective action and advocacy, the g7+ represents an opportunity to improve the understanding and engagement of development actors working in fragile states.

2.4 The g7+ achieved its first major triumph at the Busan High Level Forum on Aid Effectiveness in 2011 – an agreement on a ‘New Deal for Engagement in Fragile States’. This success was due to the vision and commitment of the g7+, chaired by the Minister of Finance of Timor-Leste, Emilia Pires and the support of the ‘International Dialogue for Peacebuilding and Statebuilding’ Partners. Over 2011 ODI’s Budget Strengthening Initiative (BSI) provided key research for the New Deal, while its technical and administrative support helped to raise the profile of the g7+ and substantively increased the effectiveness of the g7+ Secretariat in mobilising the support of its members.

2.5 ODI’s work was appreciated by g7+ colleagues, and in early 2012 we were asked to continue and increase our support to the g7+ in a variety of ways. Four streams of work now comprise the g7+ programme. These are:

35 See Kharas and Rogerson 2012, Horizon 2025.

86

A. Strengthening the secretariat: The low capacity of the g7+ Secretariat has been an

on-going issue, particularly as the g7+ has embarked on an ambitious programme of

work since the New Deal was agreed. BSI assistance provides additional staffing

capacity, as well as support to develop more streamlined systems and processes, and

strategic advice on the future development of the organisation.

B. Implementation of the New Deal at the country level: Seven New Deal pilot

countries were announced at Busan; Afghanistan, Central African Republic,

Democratic Republic of the Congo, Liberia, Sierra Leone, South Sudan and Timor-

Leste. Somalia also expressed an interest in piloting the New Deal. BSI has provided

support to these countries on a demand-led basis, as they seek to take forward

specific aspects of the New Deal framework.

C. Support the g7+’s engagement with international partners: The g7+, as part of the

International Dialogue, have been working hard since Busan to develop tools to

implement the different parts of the New Deal, including the fragility spectrum,

Peacebuiling and Statebuilding Goal (PSG) indicators and New Deal Compact

guidelines. They have also been moving forward with efforts to get peacebuilding

and statebuilding prioritised in international fora such as the United Nations General

Assembly, particularly in the post-2015 development agenda. Finally, the g7+

engages regularly with multilateral organisations including the World Bank and the

IMF, to advocate for policy and operations changes in fragile states. This process

continues to be supported by BSI, primarily through research, briefing notes and

advisory support at high-level meetings.

D. Peer-learning and research: BSI provides ad hoc research support on a demand-led

basis, and also supports g7+ countries to capture the lessons from their experiences,

to share with fellow members.

3. The process of change, assumptions and causality

3.1 The g7+ programme log frame sets out the broad parameters of the expected results of the intervention. By adding information on the processes and the causal mechanisms we can show the links between the planned results and the underlying TOC more explicitly.

3.2 There are several processes whereby interventions are expected to lead to results within the BSI approach. The below tables summarise how these processes are expected to work and the likely causal mechanism, organised around the four workstreams from the programme logframe.

3.2.1 Strengthening the secretariat:

The range of cause and effects can vary but are likely to include:

87

Process Example Change -Outputs

Assumptions/causal mechanism

Longer Term Result

New Systems and processes proposed by BSI. Also designed and implemented through facilitation by TA, working with national staff & peer learning.

More systematic & comprehensive approach to g7+ organisation institutionalised

New systems and processes maintained and supported by motivated and capable national staff.

g7+ chair and Head of Secretariat wants improvements in basic mechanisms of secretariat management and g7+ group operations.

BSI can respond flexibly to needs.

Secretariat staff are responsive and open to BSI suggestions.

g7+ secretariat is able to create a stronger, more consultative, effective and influential g7+ organisation, in turn increasing the influence of the g7+ externally.

Advice given in timely manner on challenges, issues and opportunities facing the secretariat, and the overall strategic direction of the group, and practical solutions proposed.

Challenges are identified and addressed more quickly by secretariat staff

Opportunities identified and exploited by secretariat staff.

BSI has necessary objectivity to be able to assess issues effectively.

BSI has necessary connections to identify opportunities effectively.

g7+ Secretariat leaders are responsive and open to BSI suggestions.

g7+ secretariat is able to create a stronger, more consultative, effective and influential g7+ organisation, in turn increasing the influence of the g7+ externally.

Support given to improve the secretariat’s engagement of g7+ members, through regular communication , translation and

Translated documents and emails.

Ad hoc interpretation

BSI staff have language abilities that are not otherwise present in secretariat (including French).

g7+ secretariat is able to create a stronger, more consultative, effective and influential g7+ organisation, in turn increasing the

88

interpretation services, and the organisation of meetings and VCs.

during VCs and meetings.

Greater number of meetings an VCs between g7+ members.

BSI provides additional capacity for this kind of activity.

BSI is able to use its country network to improve participation in VCs and meetings.

influence of the g7+ externally.

Key practitioners from g7+ secretariat team are given hands on coaching and mentoring by BSI staff.

A more confident and capable workforce.

Key staff want to gain skills and capability to manage and sustain new systems.

BSI can provide relevant, practical on the job training and mentoring.

Higher and sustained levels of performance by budget, finance and revenue personnel

3.2.2. Implementation of the New Deal at the country level:

The range of cause and effects can vary but are likely to include:

Process Example Change -Outputs

Assumptions/causal mechanism

Longer Term Result

BSI provides ad hoc, flexible and demand-led support to countries wishing to pilot aspects of the New Deal.

New Deal elements are piloted by g7+ countries, such as fragility assessments and compacts.

BSI support has catalytic impact, inspiring other countries to pilot the New Deal.

BSI uses its network in fragile states to identify opportunities in pilot countries.

BSI is a trusted provider of advice and support on the New Deal.

BSI can respond quickly and flexibly to

New Deal is implemented more quickly, more effectively and across a greater number of countries.

89

needs and opportunities.

3.2.3 Support the g7+’s engagement with international partners:

Process Example Change -Outputs

Assumptions/causal mechanism

Longer Term Result

Strategic influencing of international policy on fragile states through facilitation of international agreements.

The New Deal for Engagement in Fragile States

BSI provides research, negotiation and implementation capacity to g7+.

Support has a “catalytic” effect, accelerating reforms and changing the approach of international partners.

Support to g7+ interaction with multilateral institutions, bilateral donors, and international fora including the UN and International Dialogue.

More effective, productive and informed relationships between the g7+ and its interlocutors.

Greater influence of g7+ group in international discussions relevant to fragile states.

Secretariat staff are responsive and open to BSI suggestions.

BSI has connections and experience that enables it to understand relevant international processes and stakeholders.

Multilateral institutions, bilateral donors, and international fora include and respond to the voices of fragile states actors more frequently.

3.2.4 Peer-learning and research:

Process Example Change - Outputs

Assumptions/causal mechanism

Result

g7+ focal points and Ministers regularly liaise with each other, via VC or in meetings, to get structured exposure

Knowledge and skills of partner country enhanced and new approaches are quickly absorbed

Good practice captured more systematically, and disseminated more quickly, and take up is facilitated by TA.

Faster adoption of appropriate, tried and tested policies in the areas of aid effectiveness, donor

90

to relevant fragile states experience.

through South-South information & experience transfer.

relations and national planning.

Relevant existing ODI, BSI and other research is made available and used by g7+ secretariat and member countries.

Knowledge and skills of partner countries enhanced, and policy decisions and advocacy positions are more informed.

g7+ actors are responsive and open to BSI suggestions.

Research is fed in a timely way, and is cognisant of absorptive capacity constraints.

Faster adoption of appropriate, tried and tested policies in the areas of aid effectiveness, donor relations and national planning.

BSI conducts new research into aid effectiveness in fragile states, and into planning for peacebuilding and statebuilding

Improved knowledge base and ideas, with scope for new approaches to aid effectiveness and to peacebuilding and statebuilding by g7+ member countries and practitioners.

New knowledge identifies high impact and most practical innovations.

Partners show interest in taking these up.

Innovations in approach being tried and tested with positive results.

BSI PROBLEM SOLVING APPROACH (PDIA)

SYSTEMS DESIGN AND SUPPPORT

FACILITATING GLOBAL ENGAGEMENT

TARGETED RESEARCH

Secretariat staff trained, mentored, and coached. BSI staff set up basic processes & systems. BSI staff help to identify a key problem/ blockage and are asked to work on it.

Institutional process (ex.)

More capable Senior Policy Specialist in Secretariat. Systems for planning work and events in place. g7+ consulted on and feedback on proposed changes to IDA17 Bespoke research on natural resource management.

g7+ able to engage across a wider range of policy areas. g7+ secretariat’s organisational effectiveness improved. Fragile states consulted in key IFI reforms. g7+ able to advocate in more informed way on NRM.

More effective international

engagement of fragile states

through the g7+ Group of Fragile

States.

STRATEGIC ADVICE

CAPACITY DEVELOPMENT

RESEARCH Informing all stakeholders: new ideas and good practice

Intermediate Outcome

(ex.)

g7+ programme Activities

Outcome (examples)

Longer term outcome

g7+ Theory of Change

Other Actors Governments, Donors, IFIs, civil society, etc. have major role in the nature of g7+ engagement, and other international Technical Assistance provide advice to individual g7+ members and the Secretariat/Government of Timor Leste on the ground.

Annex 6 Evaluation Terms of Reference Introduction

The Budget Strengthening Initiative36 (BSI) provides advice and support to partner Governments in fragile states on a demand led basis to assist with the development of financial and budgetary systems, and to promote more effective international engagement in these areas through the g7+ process. It also aims to support the management of domestic revenue and external aid and to promote budget transparency.

The aim of BSI is to be responsive and flexible to meet urgent needs and to overcome bottlenecks to development. It is anticipated that inputs provided by BSI will support existing efforts by Governments and donors and will help to promote changes in attitudes, behaviour, policy or the systems used by organisations.

Objectives

The final evaluation will examine the achievements of the BSI programme between 2010 and March 2015. It will review the overall impact of the programme in fragile states both through its work at the country level also through its engagement with the g7+. The evaluation will assess progress against the objectives set out in the agreed log frame and independently validate the internal assessments carried out so far (Stories of Change and Annual Workstream Reports) which were not covered during the mid-term evaluationThe results of the evaluation and the review of the BSI model will add to the growing literature on PFM reform and the design of initiatives to support them as well as the design of future phases of the BSI programme.

The evaluator will therefore examine the inputs, outputs and outcomes of the programme to date.

The revised theory of change (TOC) underlying the programme will also be tested to determine whether the assumptions about cause and effect underlying the project actually hold in practice.

(S)he will recommend revisions to the programme design and implementation in future phases of the programme; and will make a judgement on the value of the implementation model even as it continues to adapt to the demands and requirements in fragile states.

Scope

The evaluator will be expected to review the progress of the programme as a whole and in relation to specific programmes of support to individual countries and the g7+ group of fragile states, and the research programme.

36 See www.budgetstrengthening.org for more details

93

The final evaluation is designed to assess the efficiency and effectiveness of the BSI programme including whether inputs were provided cost effectively and outputs were delivered.

The evaluation should test the BSI theory of change in relation to other approaches by other donors and programmes to public financial management reform and capacity building. This should include a synthesis of evidence from recent evaluations and reviews (e.g. from the Joint Evaluation of PFM led by the AfDB) and interviews with donors operating in the countries where BSI currently works. The comparison will enable the evaluator to assess the level of innovation in the BSI approach and whether these innovations help to overcome shortcomings in traditional donor programming.

Key Aspects of the final Evaluation

The evaluation should focus firstly on validating the evidence of achievement against the outputs and indicators set out in the individual partner country log frames and the overall log frame for BSI as a whole, in the period from programme start in 2010 until March 2015.

Secondly, it will extend the analysis of programme performance using the latest evidence from BSI monitoring and reporting systems through in-depth interviews at the country and g7+ programme level. Thirdly, on the basis of the evidence produced, the evaluation will consider the implications for the theory of change (TOC) underlying the revised BSI programme. This is attached as Annex 1. Any lessons and areas for improvement in the approach or the TOC itself will also be identified.

As part of the third component the evaluator will focus on the following key questions derived from the OECD DAC evaluation criteria (see Annex 2) and discuss with the BSI management team how best to gather the evidence to make an assessment. It is proposed that an inception report will set out how the data will be obtained for each of the research questions below.

1. Is BSI providing high quality, relevant and cost effective advice?

2. How well has the programme managed uncertainty and risk in fragile states?

3. How efficiently and cost-effectively has BSI provided its inputs?

4. Are inputs provided by BSI contributing to identifiable institutional changes?

5. Is BSI having an effect on Government strategy and policy on PFM reforms?

6. How demand-led has BSI been and is the quality of its relationships with counterparts conductive to effective assistance?

7. Has BSI had a catalytic effect?

8. Is BSI having a sustainable impact on budget, policies, processes and systems capacity?

94

The evaluator will provide:

an overall assessment of BSI, including its overall strategy and approach, management

arrangements and funds utilisation;

a review of the South Sudan, DRC and Liberia country programmes, including their

achievements against their logframes, testing the main linkages in the BSI theory of

change;

a review of the g7+ programme against of logframe and theory of change and an

assessment of its contribution to strengthening the g7+ secretariat and capacity for

engaging with international partners and to strengthening the implementation of the

New Deal at country level;

a review the Research & dissemination component, including its relevance, quality and

uptake.

The main requirement is to assess BSI’s overall effectiveness and value for money in achieving its planned results. It is also expected that any general lessons will be drawn out and specific recommendations for improvement in programme design or delivery will be made.

Finally, the evaluation should put forward recommendations for future phases of BSI.

Method

The evaluator will initially examine project reports, Annual reviews and PCRs, stories of change and self-assessments by project staff as well as feedback from partner Governments. This will be done through a desk-based review of BSI programme documentation held in London and meetings with BSI staff in London.

As part of this initial process, the evaluator will examine the annual reviews conducted by DFID thus far and any trends in the data collected against milestone indicators in the BSI log frame. The range and relevance of the indicators for assessing programme achievements and impact and the time frames for data collection will also be considered.

The evaluator will also select examples of specific interventions by BSI for further investigation which will cover the programmes in South Sudan, Liberia, DRC and g7+. The aim should be to conduct a deeper examination of these cases and to explore the key evaluation questions with individual contacts identified with the help of BSI programme managers. Methods for this will include:

9. Has BSI increased uptake of good practice and promoted innovation?

10. Has BSI increased partner Government capacity and capability sustainably?

11. In what ways is the support provided by BSI distinguishable from other providers of TA support?

95

Telephone interviews

E mail exchanges

Questionnaire surveys

Arrangements will be made for the evaluator to visit South Sudan, DRC and Liberia to enable face to face structured interviews with partner Governments, other donors and stakeholders. The review of the g7 + work and other activities will be based around desk reviews and remote interviews.

In undertaking the country programme analysis the evaluator should conduct interviews with the other donors involved in budget strengthening and PFM programmes and develop a good understanding of the alternative approaches being used in that country context. In South Sudan this is likely to include USAID, DFID and UNDP, in Liberia the World Bank and the IMF and in the DRC the donor group supporting PFM reform. BSI staff will be able to support the setting up of meetings. The evaluator will consider whether these interviews need to be complemented by a questionnaire sent to a wider group of interested stakeholders.

The overall evaluation will be undertaken in accordance with the OECD-DAC evaluation criteria and meet the associated quality standards.

Inputs and Timing

Overall the evaluation is expected to require up to 65 person days including 15 days in the UK, 25 days overseas (allows for 3 country programme visits) and 10 days report writing.

The evaluation will take place in the January - March 2015 and will be expected to take place over a period of six weeks of elapsed time. The draft report must be available by end March 2015 and the final report must be available by end April 2015 to inform the 2015 annual review.

Reporting

The evaluation will be commissioned by the BSI Advisory Board who will oversee and review the evaluation with additional expertise secured from DFID’s external evaluation QA panel. Oversight of the evaluation will be undertaken by an Evaluation reference group which will be set up by the Advisory Board with co-opted expertise as required. Day to day support will be provided by the BSI Programme Manager. The final report will be submitted and presented to the Advisory Board. It is anticipated that the evaluation will be of interest to a wider PFM community involving partner Governments, PFM practitioners and others. The final report will be published internally (within DFID) and externally, along withy a formal DFID Management response. Further dissemination of the report will be considered by ODI-CAPE

Deliverables

The evaluator will provide a report of 30 pages with an executive summary and supported by annexes as required. This will be presented in an agreed format and in draft to the

96

Evaluation steering group and then completed after comments are received. The report will be formally submitted to the Advisory Board and will be published online by DFID.

A list of recommendations for BSI and for its Advisory Board (including DFID) will be proposed to increase efficiency, effectiveness and impact of the programme amongst other things. This list will also be provided separately to the main report for the Advisory Board to consider. The evaluators may also be required to make a presentation or to provide a concise statement about the evaluation for communication purposes internally and externally.

Job description for evaluator

Experience of working with/evaluating other innovative TA approaches

Experience in qualitative evaluation methods and of taking forward comparable evaluations.

Experience of working in/with Ministries of Finance in sub-Saharan Africa

Experience of working on broad public financial management issues including budgeting and aid management issues in sub-Saharan Africa

Experience of working on fragile state issues

The evaluator should be independent from BSI, ODI and DFID.

97

Annex 7 List of key informant interviews

UK/telephone

SIDA Programme Manager

DFID Adviser Programme Manager

OECD Representative Representative, International Dialogue on Peacebuilding and Statebuilding

UNDP Representative

ODI/BSI Current and former staff (8)

Danida Representative, Conflict and Fragility Team

G7+ General Secretary Senior Policy Specialist

Uganda

BSI Uganda BSI staff (2)

ACODE Director

Ministry of Finance, Planning and Economic Development

Director of Budget Acting Assistant Commissioner, Budget Policy Budget team: 4 staff

Innovations for Poverty Action

Evaluation Manager

South Sudan

BSI South Sudan BSI staff (6)

ODI Fellows Two Fellows

Ministry of Finance and Economic Planning

Former Minister of Finance and Economic Planning

Director of Aid Director of Budget Senior Inspector, intergovernmental relations Inspector, intergovernmental relations Assistant inspector, intergovernment relations Inspector of Budget

Ministry of Infrastructure Director, Rural Water

Institutional Development and Capacity Building Project

Two officials

USAID CORE project Head of CORE II PFM team

DFID South Sudan Senior Governance Advisor Head, Basic Services Team Programme Manager

Charlie Goldsmith Two staff

World Bank LGSDP project Three staff

World Bank Two staff

European Union Head of Delegation

98

CIDA Head of Mission

EU-TAPP project Two staff

UNDP Aid Advisor

DRC

COREF Coordinator, COREF Deputy Ministry Coordinator Technical Assistants (6)

Ministry of Education CAT/EPSP – two staff

Ministry of Finance Director Budget

Development partners DFID EU Belgium World Bank GIZ (two staff) Resident Representative, IMF

BSI DRC Three staff

Liberia

Ministry of Finance Deputy Minister, Budget Asst Minister for Expenditure Asst Minister, Tax Director, Indirect Tax Director, Direct Tax

Development partners USAID GIZ SIDA

BSI Liberia Five staff

99

Annex 8 Documents consulted ODI internal documents

Arm’s length organisations (ALO) and Monitoring & Evaluation, undated

Capacity Development Round Table Meeting, 30 April 2013

Mid-Term Evaluation of BSI Management Response, 9 August 2013

Outcome Mapping, An Overview, September 2014

BSI programme documents

Logframes (current and historic), for BSI, each country engagement and g7+

BSI: A Refined Theory of Change, undated

BSI: A refined Theory of Change for the g7+ programme

DFID Accountable Grant letter, 6 July 2010 and subsequent amendments

Advisory Board papers, including Director’s Report, Budget Report, Risk Report,

country programme reports and minutes, for each 6-monthly meeting from

November 2010 to May 2015

Minutes of Annual Learning Retreats 2012, 2013 and 2014

DFID Annual Reviews, 2011 to 2015

DFID BSI Programme Memorandum, July 2011

Beneficiary surveys (various dates)

Cross-ODI research utilisation figures

Capacity Development Roundtable Meetings, April 2013

Lessons Learned Documents

o Growing Momentum for Budget Transparency – Uganda, June 2015

o Change in Challenging Contexts, draft, January 2015

o Insider Accounts of Reform – g7+ Secretariat Support

o Insider Accounts – Liberia, draft, undated

o Insider Accounts – South Sudan Budged, draft, undated

o Insider Accounts – Local Service Delivery in South Sudan, draft, undated

South Sudan

AWRs and Stories of Change

Proposal to DFID South Sudan to extend BSI funding, February 2015

Local Services Support JPA 2013/14 Review and Priorities, October 2014

Coordination Mechanisms for Government Led Local Service Delivery, October 2014

2014/15 Budget Speech

Education Sector Planning, Budgeting and Reporting Guidelines 2014/15

Report on Training, undated

Interim Local Government Human Resource Management Manual, December 2014

100

National Budget Plan 2013/14

Approved Budget Tables 2014/15

State Planning and Budgeting Guidelines 2014/15

South Sudan Contingency Planning Meeting/Retreat, January 2014

South Sudan Post March 2015 Planning Retreat Report, draft, September 2014

BSI South Sudan – September Retreat 2014 – Main personal reflections

Uganda

Growing Momentum for Budget Transparency – Uganda, undated

Change in Challenging Contexts – Uganda, draft, undated

DRC

AWRs and Stories of Change

Exercice d’auto-évaluation de la Gestion des Finances Publiques de la République Démocratique du Congo, undated

Overview and outstanding questions on DRC programme, undated

PEFA Report, 2015

Liberia

Workstream Outcome Log, Liberia Revenue Support, 2015

Liberia, Workstream Planning Form

New Deal/g7+ documents

Proposed World Bank Reforms to IDA Allocations for Fragile States, presentation,

undated

Briefing Note: Th World Bank’s ‘Billion Dollar Minerals Map’, AKA the African Minerals

Geoscience Initiative (AMGI), undated

G7+ Ministers Meeting with President of the World Bank, Annotated Agenda, 9 April

2014

International Dialogue on Peacebuilding and Statebuilding, New Deal Monitoring

Report 2014, November 2014

Governance Options Paper, November 2013

Note on the Fragility Spectrum, 2013

UNDP, UNDP New Deal Support Facility: Implementation Overview, 2014

UNDP Implementation Support Facility, December 2013

Natural Resources in g7+ Countries, 2014

101

Annex 9 Summary literature review

To inform the theory-based evaluation, we carried out a brief review of literature on conventional approaches to external assistance for public financial reform, to help us assess BSI’s claim to (i) offer a distinctive model of engagement to conventional technical assistance programmes; and (ii) offer a way of working that is better suited to low-capacity or fragile environments. The results of the literature review are summarised here, in the form of key quotes and findings from the literature. A. Why do PFM/budget reforms go wrong? A1. Design weaknesses A1.1 Reform strategies, programmes/projects and action plans are too technical with insufficient attention to/reflection of the context: political, social/cultural, economic, incentives, skills

“The fundamental constraint on improvement is not the absence of technical understanding or knowledge, or the lack of sufficient skills or capacity in the public service or efficient IT systems—though these issues certainly need to be addressed during the reform process—but the absence of an enabling institutional environment that makes it possible to move forward in the first instance.” (Allen, R. 2009) “......have unrealistic or inappropriate log-frame design, both in terms of nonsensical periods of time over which fully-formed reforms are expected to be designed and delivered and in terms of the contextual ‘constraints’ that are simply overlooked or assumed away. By way of contrast, taking time to properly identify problems, underlying causes and the appropriate change agents at each of the relevant locations required and finding contextually appropriate solutions probably takes more time and requires more (apparent) risk than a typical development partner organisation (and individuals therein) is prepared to take.” (Johnson, M. 2013) “The overly complex, rigid, and technocratic budget improvement programs that have bedeviled would-be reformers in many developing countries need to be avoided. Budget reform is an art, not a science.” (Allen, R. 2009) “What is clearly needed in such circumstances is for the authors of the technical report to have a much greater understanding of the political drivers of decision-making in the country concerned, and the powerful institutional constraints on implementing change.” (Allen, R. 2010) “Wildavsky emphasized these less visible social processes, referring to underlying “politics of the budgetary process” as deeply influential on how systems work, but lamented that they are not always visible, and difficult for reformers to grasp and influence. He distinguished between formal and “practical” budgeting, for example, saying that the latter takes place “in a twilight zone between politics and efficiency” (Wildavsky 1992, 598). He saw the lack of knowledge about these processes as a key limit on reform

102

prospects: “Until we know something about the ‘existential situation’ in which the participants find themselves… proposals for major reform must be based on woefully inadequate understanding.” (Andrews M, 2009) “Bank performance might have achieved greater success with deeper institutional and governance analysis, greater attention to addressing basic systems before moving to advanced PFM tools”.(IEG, 2008) “First, the design of PSR projects and resource allocation to them needs to reflect the fact that they face more complex political and sequencing issues than in most traditional areas of the Bank’s activity. This implies, therefore, the need to (i) set project objectives with realistic recognition of the time it takes to get significant results, (ii) understand the political context, identifying prerequisites to achieve the objectives, and (iii) focus first on the basic reforms that a country needs in its initial situation and that generate political support for the process.” (IEG. 2008)

A1.2 Reform programmes are too complex and overambitious: timescales too short, erroneously striving for international best practice, with the scope of reform too broad

“I have little doubt that an analysis of development partner support portfolios will tend to suggest that the typical menu of best practice reforms have tended not to achieve the desired results.” (Johnson, M. 2013) “Unfortunately, in Africa the vision of PFM reform is one of attempting the summits of international best practice, rather than consolidating the basics of a firm financial plateau. The success of the Ethiopian reform was in large measure due to an appropriate strategy of reform focused on recognition and improvement of existing systems and judicious change.” (Peterson, S. 2011) “In recent years, financial summits in Africa have included several techniques that governments and their technical advisors have slipped on—Medium Term Expenditure Frameworks, performance/program budgeting, Integrated Financial Management Information Systems, accrual accounting, and business process reengineering to name just a few.” (Peterson, S. 2011) “The challenge for the Bank and the Fund is that so much of their technical assistance and advisory work on PFM is based, explicitly or implicitly, on a model (implicit or explicit) of best practice. As Dani Rodrick has explained, this reflects a more general bias towards best practice models in the economic work of the Bank and Fund, which allows for cross-national comparisons and benchmarking institutional performance, but takes scant account of local constraints and opportunities.” (Allen, R. 2010) “Reformers can have tunnel vision and see the worth of what they want to do as self-evident, when to others it is not. Better PFM needs to gain political support by solving real problems that matter to decision makers and the public - even if this does not look tidy through the lens of a model of best practice sequencing of reform.” (Scott, G. 2014)

103

“Telling every government they need to mimic best practices is a very poor strategy by donors.” (Krause, P. 2013) “The similarities in reform content belie dissimilarities between countries. The different contexts imply different PFM challenges, opportunities and constraints, with some countries having enjoyed long periods of credible annual budgets and others experiencing periods with no budget at all, for example. Even with these dissimilarities, however, one finds similar PFM reforms.” (Andrews, M. 2011)

“In context of PFM reform initiatives in Africa, where governments are dependent on donors, face great goal ambiguity (or ambiguity about the means to achieve goals like fiscal discipline and allocative and technical efficiency, or the MDGs6) and are thus uncertain as to what their PFM structures should look like (which leads organizations to resort to best practices as a ‘safe’ reform option). Similarly, the PFM field in development is small, dominated by one or two donors that fund most activities, and espouses a fairly tight normative model (now manifest, perhaps, in PEFA). (Andrews, M. 2011)

“Politically vulnerable, resource poor governments are highly dependent on donor organizations, which offer support on condition of specific types of PFM reform. The reform designs are often influenced by “good international practice” that countries appear willing to replicate because of a dearth of local solutions to the problems they face. PFM is also an area dominated by professionals, from accountants to auditors and other groups bent on standardization. These professionals, mostly working in donor organizations, have become central to the growing PFM field in development. They have contributed to the emergence of the PEFA assessment tool, which is an important standardizing device that establishes the normative legitimacy of a set of practices and characteristics, inferring the need for governments to adopt such. “ (Andrews, M. 2011))

A1.4 Unrealistic expectations/requirements of development partners

“In many cases, donors have focused their efforts on “big ticket” initiatives—such as the introduction of a medium-term expenditure framework (MTEF) and integrated financial management information systems (IFMIS)—that are dependent upon large injections of external funding and technical assistance. The presumed idea behind such an approach is that the programs concerned may act as a centrifugal force around which other reforms would gravitate. However, a recent survey of MTEF and IFMIS projects in Ghana, Tanzania, and Uganda suggests that the approach has been a mixed success (Wynne, 2005). Another recent study by Schiavo-Campo (2009) notes that there have been some positive lessons from a decade of MTEF experiments in Africa―greater awareness of the need to look beyond the annual budget horizon, and to focus on the results of government spending―but also “costly failures” arising from premature implementation of MTEFs, little or no local ownership, damaging distraction from basic PFM improvements, heavy stress on limited budget capacity, and little improvement in macroeconomic balances, financial control and predictability, or spending efficiency.” (R Allen, R. 2009)

104

“In addition to overoptimistic time horizons, many plans are overloaded with hundreds of activities and actions. The IFIs and donors—and their advisors—are in many cases to blame, for this enables them to package together measures into a “public sector reform” grant or loan that gains approval from the management board, and gives the donor increased leverage in the country concerned. However, these packages often prove unmanageable and ineffective, raise unrealistic expectations about what can be achieved, and fall into disarray within only a small fraction of the measures having been implemented.” (Allen, R. 2009)

A1.5 DP over emphasis on sequencing reforms and their attraction to adopting reform ‘models’

“Technocratic solutions to the design of reform strategies in developing countries―such as the platform approach—have a certain logic, but their sheer weight, restricted timelines, and complexity vitiate against them as a practical tool for use in developing countries. They are also prone to capture by the agencies who fund and provide the technical assistance. The highly structured reform agenda, with platforms, sequenced steps, coordination, and participation of the authorities, donors, and other stakeholders may to some extent give a false impression of order in a fundamentally chaotic process. By contrast, the more successful improvement strategies have a relatively short-term horizon, focus deliberately on a quite narrow and specific set of objectives (for example, how to correct a fiscal imbalance or a problem of arrears, or to improve budgetary reporting), and involve a large element of trial and error, learning from mistakes, and “fumbling around in the dark.” This requires deliberate selectivity in the choice of topics on which to focus the reform effort: selection rather than sequencing is the keyword.” (Allen, R. 2009)

A1.6 Weaknesses in the available PFM diagnostic tools, primarily PEFA

“This instrument has been widely used in developing countries and emerging markets—about 80 countries have undertaken some form of PEFA assessment in the past four years, some on several occasions. However, in practice, the tool was not designed to take account of the weak institutions and governance that are critical to the improvement of budgetary systems. As a result, the diagnostic information deriving from a PEFA assessment does not provide a robust basis for preparing an action plan to be used by national authorities in strengthening their budgetary institutions.” (Allen, R. 2010)

A2. Implementation weaknesses A2.1 The challenges are underestimated; budget reform challenges vested interests and takes a long time

“The budget, however, is both a central institution of the state and a key mechanism for determining the distribution of resources and economic rents to the elites that dominate natural states, and to the wider groups that influence the development of open-access societies. Because of this, the budget is, almost by definition, very hard to reform except in the unlikely circumstance that such improvements enhance or facilitate rent-seeking

105

behavior. Evidence for the extremely gradual evolution of budgetary institutions can be found in the history of the three countries—France, the United Kingdom, and the United States.” (Allen, R. 2009) “It is interesting to speculate on the areas of the budget system that are especially prone to rent-seeking behavior, which may frustrate improvement programs. These areas cover a substantial part of the budget process, such as control of the allocation of budgetary appropriations by ministry or sector; planning and management of government investment projects, including procurement operations; control of decisions relating to the planning and management of external aid; management and control of government bank accounts (and resistance to consolidating these into a treasury single account) and the management of cash disbursed through the budget; and management of the external audit process. Many of these areas have indeed proved difficult to reform in developing countries. However, all areas of the budget process are likely to be affected to a varying degree.” (Allen, R. 2009)

A2.2 Lack of leadership, authority and ownership

“The financial reform succeeded because it was embedded in a government led political and administrative reform—decentralization. The reforms were driven by a domestic political imperative not a foreign technical agenda, and rapid results were needed in public financial management to keep up with the accelerating pace of decentralization. Rapid results required improving the existing financial system rather than changing it with advanced financial techniques of ‘international best practice.” (Peterson, S. 2011) “Budget reforms are also difficult because they require political willingness to make hard choices.” (Allen, R. 2009) “Successful PFM reforms are always largely indigenous even though PFM systems at different stages of development have general similarities. The narrative that eventually emerges, over the years that any substantial reform takes, is about local responses to local needs, within a local political and administrative environment and culture. Pressures from development partners, international standard setting bodies or whoever for best practice solutions can be a feature of the story but is never the core of the narrative over the long haul.” (Scott, G. 2014)

A2.3 Resistance to change

“PFM reforms can be quite confronting to the power structures in a ministry of finance. In the heart of the ministry of finance, the people in the high positions sometimes do not understand what is required by way of people and associated resources to get the reform done.” (Scott, G. 2014)

A2.4 Overdependence on single change champions, and the limited number of effective change agents and competent technicians

106

“It is common for a PFM reform to be critically dependent on the leadership and charisma of one or a few people who keep the reform rolling, through the sometimes tedious processes of government and give others the confidence to go along, or at least wait and see, rather than resist. For capacity poor countries, my view is that the reform has to be tailored to the capacity that can reasonably be expected to be available and if necessary the reform aspirations cut back to fit within the capacity constraints. It is too common for low capacity countries to embark on reforms they do not have the capacity to implement. Turning book keepers into accountants and budget analysts is challenging and takes time and resources. Inevitably some new staff with high levels of skill are essential for operating a reformed PFM system that uses modern concepts and methods. Much of the skill in PFM reform is what economists call ‘tacit knowledge’ – the knowledge embodied in a person that is not easily passed on to others. The way to accumulate tacit knowledge is to learn it on the job by trying, sometimes failing and sometimes succeeding on the path to your vision of what great PFM looks like in your circumstances.” (Scott, G. 2014)

A2.5 Limited project management capability

“I have seen ministries of finance setting out a PFM reform program that is beyond their capability to plan, organise and deliver. Project design and management is a discipline that is not part of the normal professional training of most finance ministries. PMUs are only a partial answer and can have difficulty in their integration with the host ministry. A great looking program can crash because of poor project management. Resistance to the reform gathers and focuses on the shortcomings in implementation, political will for the reform fades as expected milestones and results are not achieved. A simple incremental reform done in manageable steps that does not overstretch project management capability can be a wise approach. Even though it does not promise a lot in the early stages, a lot can be learned about how to do reform, which can sustain more ambitious further reforms, as the reformers get better at project management.” (Scott, G. 2014)

A2.6 Process change out of synchronisation with institutional change (this applies especially to Ministries of Finance) – not integrated with/supported by essential institutional changes across the civil service

“Modernization of the budget needs to be linked to reforms in public administration more generally, and to the establishment of a professional, merit-based civil service” (Allen, R. 2010)

A2.7 The focus on core PFM agencies and the difficulty of extending reforms into sectors

“Securing the support of line ministers and their ministries to take better PFM practices across their ministries is always a challenge. They have to see what is in it for them or they will not own the PFM system and use it, because it is seen as a finance ministry operation.” (Scott, G. 2014)

107

A2.8 Not sufficiently transparent and accountable – work proceeds ‘behind closed doors’

“The people and institutions that may change the balance of power will need to know what government is doing with public resources before they can change anything about it.” (van Zyl, A. 2009)

A2.9 Inflated expectations of what PFM can do to promote better government performance which when not realised can undermine the credibility of the achievements made

“While the PFM system can usefully support a system of performance management it cannot substitute for one. PFM tools will not work well in isolation from a wider system of performance management that integrates planning, institutional reform, capability development, governance and accountability systems and incentives of some kind - not necessarily financial. Adorning the budget with performance indicators will not achieve much on its own.” (Scott.G. 2014)

A2.10 Role of DPs; over ambitious, lack of coordination, with a major focus on results

“A problem with PFM reforms in Africa is having too many cooks in the kitchen. In many countries, separate foreign aid initiatives compete, overlap and undermine existing PFM operations and ongoing reforms.” (Peterson, S. 2011) “In many countries, donors and their associated international consultants have exerted a strong (and not necessarily benign or beneficial) influence on the reform process (Allen, 2008).Donors frequently provide a substantial proportion of the funds required to finance such measures, in addition to supplying technical assistance. This may lead to a dependency on donors and their consultants that inhibits the development of local capacity and channels finance into elaborate projects such as computerized information systems that may not be appropriate or timely for the country concerned. By extracting large fees-for-services without direct accountability for results, consultants can be viewed as complicit in the rent-seeking process.” (Allen, R. 2009)

B. What are the shortcomings of conventional TA in PFM/budget reforms? B.1 Delivered outside government systems

“In the three case study countries, a significant proportion of donor funding was directed towards technical assistance activities, which were outside of the Government’s PFM reform priorities and for which there was no political support and commitment.” (Lawson, A. 2012)

B.2 External ‘interference’ and lack of communication

108

“These experiences suggest that the problem is not one of country-level coordination mechanisms but an issue of the design and conduct of policies on technical assistance at the DP headquarters level. Significant difficulties also arose in the case study countries due to the confusion of the different objectives of TA activities. Government representatives commented that it was often unclear which TA activities were supposed to be part of the PFM reform programme and which were being done on behalf of the Development Agency.” (Lawson, A. 2012)

B.3 Absence of active learning and adaptation

“...suggests an important weakness in the management mechanisms for PFM reform: namely, the absence of an active learning and adaptation process. In all three countries there were PFM reform ideas and models, which continued to be pursued over long periods, even though, when viewed retrospectively, they were obviously inappropriate. How could management mechanisms have been designed to avoid this? Learning and adaptation processes need to be introduced both into Government mechanisms for coordination of PFM reforms and into the supervision and peer review processes of the Development Agencies.” (Lawson, A. 2012)

C. What are the characteristic challenges of PFM reform in fragile states? C.1 Very limited capability; start with what you have and progress incrementally

“Although the principles of good budgeting are the same, the core requirements of budgeting in a post conflict situation are simplicity and adaptation to whatever limited capacity exists in the new transitional government. Post conflict budgeting must, in the first place, be fully cognizant of the realities of depleted resources, scarce information, and weak administrative capacity. Budgeting must be deliberately selective, tailored to the basic needs of the economy, and oriented in part toward the quick wins that are necessary to re-establish government credibility and to restore hope.” (Schiavo-Campo, S. 2007)

C.2 Mistrust, suspicion of conspiracies, lack of information and poor communication.

“In a post conflict setting, transparency and participation are especially important. The climate of reciprocal suspicion generated by the conflict means that every shadow is seen as a threat and every closed door as a conspiracy. In such a setting, no other public management practice can dispel those shadows and suspicions as effectively as a wide-open budgeting process—as consultative and participatory as possible.” (Schiavo-Campo, S. 2007)

C.3 Increased tendency/inclination/likelihood to mimic the donors’ expected responses rather than to adopt substantive changes

“Institutional ventriloquism....occurs when fragmentation and informality in government are so high that purposeful action is very difficult: in effect, there is a formal state, but it does not have the capability to behave as a single actor. When the centre of government

109

does not have enough autonomy and capability to learn and adapt, externally sponsored formal reforms become very dangerous. This is not isomorphic mimicry, but institutional ventriloquism. It happens when best-practice reforms are articulated, planned and implemented following external prompting and via externally funded advisers and consultants. While the first two kinds of mimicry assume government agency, ventriloquism is the absence of whole-of-government intention. In the context of fragmented institutions, informality and fragility (where state agency is lacking) ventriloquising what donors consider to be best practice as a substitute for real reform is actively harmful. Domestic government units are further fragmented as they align themselves to available external funding and manpower, to tap into external capacity for new strategies and projects. This institutional ventriloquism worsens capability traps because it keeps states from developing the autonomous capability to adapt and change. It is the phenomenon Pritchett, et al. call ‘looking like a state’.” (Krause, P.)

C.4 Fragmentation of budgetary processes

“Integrating the management of recurrent and capital budgets has been a challenge in most of the eight cases. A standard premise of good PFM is the development of a unified and comprehensive budget that captures all forms of revenue and expenditure (Webber 2007). Evidence from the cases suggests this principle is complicated by the tendency for fragmented budget management arrangements centrally (between ministries of finance and planning) as well as within line ministries and sectors”. (Fritz, V et al. 2012)

C.5 Strong role of the ‘centre’

“Establishing effective transparency and accountability mechanisms appears to be particularly challenging in authoritarian and potentially also in hybrid regimes that result in (overly) dominant executives.” (Fritz, V et al. 2012)

D. What is distinctive about building capacity when the starting point is low? D.1 Very limited human resource capacity

“In fragile states and post conflict countries the capacity constraints can be so severe that it makes sense to build capability just to run the present system before thinking about how it might be reformed” (Scott, G. 2014) “There is too little local capacity to enhance in the short-term. In post-conflict situations, ministries of finance may be so depleted in terms of human resources that there is little local capacity to work with. Where a cadre of permanent civil servants needs to be recruited and/or redeployed into key positions, capacity substitution may be the only way to fill key gaps, at least in the short to medium term. In fragile situations, particularly post-conflict, external support may be necessary simply to ‘get things done’ to an acceptable level on an accelerated timescale, before longer-term capacity enhancement can be addressed.” (Tavakoli, H and Saneja, S. 2012).

110

D.2 Capacity building dilemmas, including choices around (a) substitution and supplementation and (b) ‘quick fixes’ and long term approaches

“Capacity development is constrained by the limited pool of skilled people in-country and in the civil service and by the low wages paid to civil servants, particularly middle- and senior-level officials. Low salaries for key PFM positions result in high staff turnover (as experienced in Afghanistan and Tajikistan), or in an inability to move away from technical assistance arrangements and to bring donor-funded staff onto the public pay roll (Sierra Leone and Liberia). At the same time, donors offer substantial salaries to their locally hired staff, further undermining the ability of the public sector to attract the most skilled people.” (Fritz, V et al. 2012) “In fragile situations the risk of doing harm to existing national capacity is considerable. Current approaches to capacity development have not been very effective in most fragile and post-conflict states and have even been detrimental in some cases. Part of the reason for this is that fragile and post-conflict states are rife with dilemmas where no solution will accommodate all concerns and solutions may be far from ideal. Careful analysis of the context such as through political economy and dilemma analyses is important to understand the entry points for action and the implications of various choices. Most TAs in fragile situations are under pressure from both country partners and donors to achieve specific goals, usually formulated in terms of tangible results. Many do not have a clear idea of what they are expected to do in terms of capacity development. They have usually been recruited for their technical knowledge rather than for their process skills and TORs and their annual assessments tend to focus on task accomplishment. Under these conditions, it is little wonder that capacity development is a second priority for most TAs. Topping up of the salaries of national public servants is only appropriate when there is broad agreement among the main donors working in the country on how it should be done. In fragile situations, it is especially important to have a code of conduct specifying how to avoid undermining the limited capacity of the public service.” (Baser, H. 2011)

References: Johnson, Martin ‘Some thoughts on Matthew Andrew's 'Limits to PFM Reform in Development' http:///pfmboard.com. 2013 Allen, Richard. The Challenge of Reforming Budgetary Institutions in Developing Countries. IMF Working Paper WP/09/96. 2009 Allen, Richard. Two Very Modest Proposals for Making Technical Assistance More Effective! 2010 Andrews, Matt. Isomorphism and the Limits to African Public Financial Management Reform. Faculty Research Working Paper Series, Harvard Kennedy School. 2009

111

Baser, Heather. Special Report: Perspectives on Capacity Development in Fragile Situations. OECD/DAC. 2011 Fritz, Verena; Ana Paula Fialho Lopes, with Hedger, Edward. Public Financial Management Reforms in Post Conflict Countries, Synthesis Report. World Bank. 2012 Independent Evaluation Group (IEG). Public Sector Reform: What Works and Why? World Bank. 2008 Krause, Philipp. Of institutions and butterflies: is isomorphism in developing countries necessarily a bad thing? ODI Background Note. 2013 Lawson, Andrew. Evaluation of Public Financial Management Reform, Burkina Faso, Ghana, Malawi, 2001-2010 Final Synthesis Report. SIDA. 2012 Peterson, Stephen B. Plateaus not Summits: Reforming Public Financial Management in Africa. Public Administration and Development, 31, 205-213. Wiley Online Library. 2011 Schiavo-Campo, Salvatore. Budgeting in Post Conflict Countries. Budgeting and Budgetary Institutions. World Bank. 2007 Scott, Graham. PFM Reform: Promises and Tears - Lessons Learnt from Reforms Gone Wrong. Fourth Asian Regional Conference on Public Financial Management IMF Fiscal Affairs Dept. 2014 Tavakoli, Heidi and Saneja, Shkumbin. Public financial management in fragile states - The case of Kosovo. ODI. 2012 van Zyl, Albert. How to drive public financial reforms in poor countries. http://internationalbudget.org/blog/2009/08/14/how-to-drive-public-finance-management-reforms-in-poor-countries/. 2009 Welham, Bryn; Krause, Philipp and Hedger, Edward. Linking PFM dimensions to development priorities. ODI Working Paper 380. 2013 Baser, Heather, Special Report: Perspectives on Capacity Development in Fragile Situations, OECD/DAC, 2011