Knowledge Generation and Sharing by Multilateral ... · Knowledge Generation and Sharing by MDBs: Evidence from Evaluation 7 knowledge capabilities for local audiences at country

Knowledge Generation and Sharing by MDBs: Evidence from Evaluation

1

Knowledge Generation and Sharing by Multilateral Development Banks: Evidence from Evaluation A Synthesis Paper for the Evaluation Cooperation Group (ECG) Thierry Senechal 20 October 2013

Draft Report For Review


2

Knowledge Generation and Sharing by Multilateral Development Banks:

Evidence from Evaluation

This publication was led by the Independent Evaluation Department of the Asian Development Bank. The report was prepared by Thierry Senechal (Consultant) for the Evaluation Cooperation Group (ECG), under the direction of Mala Hettige (Director, Independent Evaluation Department, Division 2) and Walter Kolkma (Director Independent Evaluation Department, Division 1). Published as an Evaluation Cooperation Group (ECG) paper, it does not reflect the ECG policy, nor the views of ECG members. Responsibility for the contents of this report rests with the author.


3

Contents Abbreviations ............................................................................................................................. 4

Preface ....................................................................................................................................... 5

Executive Summary .................................................................................................................. 6

1. INTRODUCTION ....................................................................................................................... 9 1.1 Purpose and scope of the report ....................................................................................................... 9 1.2 Approach and evaluation reports to be reviewed ....................................................................... 10 1.3 Limitations of the analysis ................................................................................................................... 11

2. Areas of progress and strength .......................................................................................... 12 2.1 Knowledge management gained wide support in late 1990s .................................................... 12 2.2 Knowledge management evolved as a strategic asset .............................................................. 13 2.3 Significant funding allocated to knowledge management ........................................................ 14 2.4 Knowledge strategies were useful to streamline knowledge ................................................... 1516 2.5 Organizational arrangements strengthened knowledge generation ........................................ 17 2.6 Providing knowledge to various audiences, both internal and external ................................... 18 2.7 The power to be “knowledge broker” is another key strength .................................................... 19 2.8 Contribution of TA and AS to knowledge generation ............................................................... 2021 2.9 International recognition of MDS’ Knowledge management ................................................. 2122

3. CRITICAL KNOWLEDGE MANAGEMENT GAPS ................................................................... 23 3.1 The “knowledge bank” concept remains largely unachieved ................................................... 23 3.2 An overall lack of conceptual clarity and definitions ............................................................... 2425 3.3 Knowledge is increasingly dispersed, unplanned and poorly coordinated .............................. 25 3.4 Just-in-time, operationally-relevant knowledge is lacking ............................................................ 27 3.5 Managing for quality and incentives to do better ........................................................................ 29 3.6 IT systems are not set up to support full knowledge dissemination ............................................. 30 3.7 Tacit knowledge is largely not captured and well utilized ....................................................... 3132 3.8 Capturing knowledge on “frontier issues” is difficult ...................................................................... 32 3.9 Measuring knowledge impacts remains a challenge ............................................................... 3233

4. Conclusions and areas for improvement ......................................................................... 34

Appendixes .............................................................................................................................. 35

Appendix 1. Glossary of key terms ....................................................................................... 36

Appendix 2. Conceptual framework of knowledge management .................................. 38

Appendix 3. Key findings from the 14 selected knowledge-related evaluations ........... 39

Appendix 4. Bibliography ...................................................................................................5859

Appendix 5. List of interviews ................................................................................................. 62

Appendix 6. Consultant’s background and expertise ........................................................ 63


4

Abbreviations AAA Analytical and Advisory Activities AfDB African Development Bank AsDB Asian Development Bank AsDBI Asian Development Bank Institute AS Advisory Services BRIC Fast-growing developing economies of Brazil, Russia, India, and China CSO Civil Society Organization DMC Developing Member Country EBRD European Bank for Reconstruction and Development ECG Evaluation Cooperation Group EIB European Investment Bank ESW Economic and Sector Work FX Foreign Exchange GDP Gross Domestic Product HQ Headquarters IDB Inter-American Development Bank IBRD International Bank for Reconstruction and Development IDA International Development Association IFAD International Fund for Agricultural Development IFC International Finance Corporation IFI International Financial Institution IMF International Monetary Fund IT Information Technology KM Knowledge Management KPS Knowledge Products and Services MDB Multilateral Development Bank M&E Monitoring and Evaluation MIC Middle-Income Countries MOPAN Multilateral Organisation Performance Assessment Network NGO Non-Governmental Organization OECD Organisation for Economic Co-operation and Development QA Quality Assurance TA Technical Assistance UNDP United Nations Development Programme USD United States Dollar WB The World Bank Group WBI World Bank Institute


5

Preface To be provided in final draft


6

Executive Summary

Purpose and scope This paper synthesized the findings of fourteen evaluation reports from six development banks 1 on knowledge management. None of these reports deal with the subject in exactly the same way, depth, or time2, and they often represent complementary points of view on the use of knowledge, technical assistance and research within the development field. However, the paper sought to present findings on the most common points raised in these reports and to focus on areas of progress and critical gaps in knowledge management. The analysis opens a very wide range of issues for consideration of development financial institutions. In this summary only a few are highlighted.

Areas of progress The survey of evaluations of the diverse knowledge management practices reveals development banks have made progress in establishing the architecture to support their knowledge initiatives. The World Bank, with great vision, pioneered the “knowledge bank” concept in the late 1990s, although it is still a concept largely unachieved today. Subsequently, many other development banks followed. The spread of networked computers in the 2000s has made it possible for those organizations to codify, store, and share certain kinds of knowledge more easily and cheaply than ever before. Following the World Bank initiative, most development banks acknowledged the importance of knowledge as a strategic asset. They started to pay special attention to their knowledge management functions at the corporate level. The number of international organizations that have developed specific knowledge strategies or frameworks has steadily grown in the early 2000s, a sign of their desire to structure and plan their knowledge products and services more systematically. Funding for knowledge products and

1 AsDB, AfDB, IAsDB, IFAD, IMF, World Bank. 2 The period covered by the evaluation period is 2006-2013.

services significantly increased in past years. Administrative budget was increased in most banks for core knowledge work in order to better meet clients’ expectations and needs. Funding for knowledge management was also confirmed to be in line with such budget allocated by the private sector. For instance, the global consultancy McKinsey spends about 10% of its revenues on knowledge management, a level also reached by many development banks. Still, some sources indicate that resource allocations are not always commensurate with broad and expanding demands for customized knowledge solutions. Many reports, including the 2011 World Bank Knowledge For Development report, recognized that development banks have transformed the way they conduct knowledge work, opening data and knowledge to clients, researchers, policy makers, and civil society throughout the world. Drawing on their mandates as a producer, customizer, broker and connector of knowledge to respond to client needs, these institutions have achieved some tangible results. In particular, the total number of knowledge products and advisory services has significantly increased and diversified, with some internationally-praised flagship publications and trainings being proposed as public goods. In addition, organizational arrangements have been made to strengthen knowledge management, some of them with great success. Such arrangements may differ from one organization to the other but evaluation reports concluded that significant efforts have been made to institutionalize knowledge management within each development bank. MDBs have been compelled to service many different types of audiences, either internal or external, with some gaps to fill. A major achievement has been the ability of MDBs to differentiate among the various audiences serviced by knowledge solutions. A consensus has emerged to focus the knowledge generation and sharing on three categories of audiences: knowledge for internal use, knowledge for external clients and partners, and knowledge as a global public good without a designated client. Still, many development banks are grappling with the question of how best to leverage


7

knowledge capabilities for local audiences at country level. Furthermore, the emergence of knowledge as a public good, a feature of development support that gained prominence in recent years, is causing some concerns. Determining the needs and characteristics of an audience for public goods is difficult. Then, once the knowledge products are produced, it is challenging to ensure that the benefits of such public goods can be equally split up and distributed to the target audience(s). The question of funding has been raised with those providing such freestanding knowledge products; development banks usually not capturing all of the benefits but bearing all the costs. Another key strength is linked to the role of MDBs as “knowledge connector” or “knowledge broker”. Good practices in knowledge brokering across boundaries tends to happen in particular locations—e.g. in Communities of Practice (CoPs) or thematic networks. Let us mention the increasing importance and influence of open data and knowledge partnerships for development banks to perform more effectively as knowledge brokers. Due to their openness, the thematic groups and CoPs, when sufficiently resourced, are able to connect the various producers and clients of knowledge products and services. Under recent initiatives, high volume datasets are also made available to all, thus allowing a multitude of analysis being performed by the development partners. TAs and advisory services have significantly contributed to knowledge generation. Many evaluations revealed that a significant modality for financing and facilitating knowledge generation activities is through various forms of Technical Assistance (TA) and Advisory Services (AS). In past years, the volume of TA and AS has significantly increased, with knowledge-related TA and AS growing faster than lending-related ones. For instance, the World Bank TA supporting client efforts to implement reforms and strengthen development banks has increased from 782 in 2005 to 1,424 in 2010. Similarly, AsDB approved a total of USD249 million on knowledge generation-related TA over the 2002–2011 decade for advisory services, research, studies, capacity development, and other knowledge related activities. At IFC, AS have grown rapidly since 2001, with expenditures increasing tenfold from

USD24 million in 2001 to US245 million in 2008.

Critical Knowledge management gaps Yet, the strategic intent of including knowledge products and services as a major new offering of MDBs, has been only partly realized. The case of the World Bank is notable as the institution is now entering the sixteenth year of the 1996 “knowledge bank” initiative. Still, in 2012, the evaluation of knowledge management points out many difficulties in creating knowledge, embedding it in the core operational processes and providing more knowledge capacity enhancement for clients in the field. Despite the growing importance of knowledge for development, evaluation reports have exposed a significant number of problems with the way in which the development banks manage their knowledge agendas, research work and the production of studies. First, the lack of standard definition(s), guiding principles and metrics has also been identified as a major pitfall to implementing a robust knowledge management culture in a few organizations, leaving staff with a poor understanding of what constitutes knowledge management. Without a clear knowledge management definition at all levels and within a variety of interventions (national, sector, program, project, and organization), development banks staff are tempted to use their own knowledge tactics and tools at various phases of national and development programming. A number of the evaluation papers stressed that realizing the potential of MDBs to become more efficient in knowledge management will depend on the strategies devoted to it. Only a few development banks drafted a corporate-wide strategy on knowledge management, while several others allowed decision-making and priority-setting for knowledge to remain largely at lower levels. This fact also underscores the critical role of the leadership and senior management of development banks in shaping more focused and well defined knowledge strategies. That simple observation adds an important dimension to the topic of knowledge management and may explain why many


8

development banks are developing a multitude of independent strategies at sub-corporate level. Several evaluation studies acknowledged that knowledge is often created through a supply-driven approach under which the institution tends to concentrate knowledge products and services at headquarters, leaving country offices with too little opportunities and resources to carry out specific knowledge generation and sharing, thus not helping operations. A few evaluations indicated that too many knowledge management approaches prevail, thus preventing the coherent use of the knowledge products and services. As a result, the question of control over the quantity and quality of knowledge products and services is a continuous problem. With time, such ambiguity has hindered the development of a cohesive system-wide knowledge approach to promoting knowledge in a coherent fashion, thus making it difficult for staff to integrate knowledge into day-to-day work. Several evaluation studies have also pointed out the need to better capture tacit knowledge and use it beyond the circumstances in which it was gained. True, to some extent, development banks have been transferring tacit knowledge among staff and partners. Yet, existing corporate systems put in place for capturing knowledge do not allow harnessing the value of tacit experience. In addition, no matter how well they are designed, such systems, databases or registries, cannot fully capture tacit knowledge, a great part of it being socially and informally constructed. Moreover, staff do not find reasons to explain away such advantage unless getting a serious incentive to do it, either in time or in terms of recognition or bonus. Finally, some evaluations indicated that getting the knowledge from beneficiaries

beyond the immediate clients can be a challenge. Many MDBs have made no particular efforts to strengthen the capacity of their field offices to bring their knowledge back to the corporate level. As a result, the knowledge agenda of many development organizations is indeed weak in tackling country specific issues of extreme poverty and social exclusion; there is also limited cooperation with civil society organizations (CSOs) in the production and sharing of knowledge and yet such support is important to help understand the surrounding and evolving environment. Finally, no particular efforts have been made to define knowledge and develop metrics to measure its impact.

Conclusion Taking this synthesis paper as whole, and the evaluation reports upon which it is based, six sets of implications emerge for development banks: a. Customize according to specific client

needs b. Create more knowledge that is

operationally relevant

c. Improve the overall quality and coordination of knowledge products and services

d. Assign clear indicators to monitor

knowledge management effectiveness e. Use the new technologies to open data

according to the needs of the various audiences

f. Better capture frontier issues that needs

special attention


9

Knowledge is like light. Weightless and intangible, it can easily travel the world, enlightening the lives of people everywhere. Yet billions of people still live in the darkness of poverty—unnecessarily. World Development Report 1998/99, World Bank

1. INTRODUCTION

1.1 Purpose and scope of the report 2. This report presents the findings of an examination of recent evaluative work undertaken by selected multilateral development banks (MDBs). It examines fourteen thematic knowledge evaluations to share findings about what works well in supporting knowledge generation and what does not. Development banks have invested a significant amount of time and resources in knowledge creation in past years. However, the efficient management and dissemination of the created knowledge remains a fairly young practice with no ready-to-use road maps to serve as guides for those banks willing to improve and progress to a higher level. Because development banks are likely to continue supporting the creation of knowledge products and services on a sustained basis, their evaluation units forming the Evaluation Cooperation Group (ECG)3 decided to review a selection of their knowledge-related evaluation reports in order to identify areas of progress and critical gaps in knowledge creation and management in the 2000s. 3. Most of the ECG members now have many years of experience in using knowledge generation approaches and agendas to contribute to the learning of a wide variety of audiences, either internal or external. Building on this evaluative work, this synthesis paper seeks to promote the learning and good practices among development practitioners and policy makers. In particular, the paper focuses on the way the MDBs have built their organizational capability allowing to turn knowledge and organizational learning into sources of competitive advantage. The paper also reviews the challenge of developing, embedding, and leveraging intellectual capital within a vast network of operations. In that sense, the reports reviewed in this synthesis paper provide a unique background to the most pressing issues facing development banks operating in the current information-intensive global environment. 4. The purpose of the assessment is three-fold:

I. To provide insight into the way in which development banks have developed knowledge

management agendas in past years;

II. To review areas of progress and outcomes achieved from generating, sharing and using knowledge products and advisory services; and

III. To identify critical gaps and derive lessons on how to better support effectiveness of knowledge operations in the future across development banks.

3 African Development Bank (Operations Evaluation Department), Asian Development Bank (Independent Evaluation Department), Black Sea Trade and Development Bank (Evaluation Office), Council of Europe Development Bank (Evaluation Department), European Bank for Reconstruction and Development (Evaluation Department), European Investment Bank Inspectorate General), Inter-American Development Bank (Office of Evaluation and Oversight), International Fund for Agricultural Development (Independent Evaluation Office), International Monetary Fund (Independent Evaluation Office), Islamic Development Bank (Group Operations Evaluation Department), The World Bank Group (Independent Evaluation Group), and Global Environment Facility (Evaluation Office).


10

1.2 Approach and evaluation reports to be reviewed

5. The approach undertaken has been somewhat distinctive. First the paper reviews fourteen evaluations provided by six development banks. Then, it seeks to identify commonalities in the reports and the key issues encountered. The paper also selectively reviews some additional material on knowledge management, including good practice from the private sector. Finally, the paper summarizes key findings, highlights good practices and outlines critical gaps in knowledge management.4 It is not the intention of the synthesis paper to compare evaluations or evaluation approaches but to extract a set of common issues on how development knowledge is created and used. Table 1: List of the 14 evaluations reviewed

Institution Year Name of evaluation (and evaluation unit)AfDB 2013 Review of the African Development Bank’s Economic and Sector Work

(2005−2010). Operations Evaluation Department (OPEV)

AsDB 2012 Knowledge Products and Services: Building a Stronger Knowledge Institution. Special Evaluation Study, Independent Evaluation Department

AsDB 2011 Performance of the AsDBI: Research, Capacity Building and Training, and Outreach and Knowledge Management, Independent Evaluation Department

ECG 2012 Evaluating Technical Assistance: Taking Stock of the Practices of International Financial Institutions

IDB 2006 Evaluation of the IDB's Studies (RE-323). Office of Evaluation and Oversight

IFAD 2013 IFAD’s institutional efficiency and efficiency of IFAD-funded operations Corporate-level evaluation, Independent Office of Evaluation of IFAD

IMF 2011 Research at the IMF: Relevance and Utilization. Independent Evaluation Office (IEO)

World Bank 2013 Knowledge-Based Country Programs, An Evaluation of the World Bank Group Experience. Independent Evaluation Group (IEG)

World Bank 2012 The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG),

World Bank 2011 The State of World Bank Knowledge Services: Knowledge for Development

World Bank Institute

2011 Using Knowledge Exchange for Capacity Development: What Works in Global Practice? Three case studies in assessment of knowledge exchange programs using a results-focused methodology (with the Korea Development Institute)

World Bank 2009 Evaluation of knowledge for Private Sector Development. Independent Evaluation Group (IEG)

World Bank 2008 Using Knowledge to Improve Development Effectiveness, An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006. Independent Evaluation Group (IEG)

World Bank 2003 Sharing Knowledge: Innovations and Remaining Challenges. Operations

4 The present paper is based upon evaluation material provided to the writer. Appendix 3 provides a summary of each evaluation report. Evaluation reports were obtained from the evaluation units of the following organizations: AfDB, AsDB, Asian Development Bank Institute (AsDBI), IDB, IFAD, IMF, and The World Bank. No independent fieldwork on knowledge management was conducted for this synthesis paper.


11

Evaluation Department

1.3 Limitations of the analysis 6. The current methodological approach has some limitations. The analysis focused on a limited number of development banks and selected knowledge-related evaluation papers. Owing to the large variety of MDBs and the scope of their programs dealing with knowledge, it was not feasible to complete a comprehensive in-depth review of knowledge management in its entirety, neither conduct a sufficient number of interviews with staff working in the field of knowledge management. In addition, the evaluation reports reviewed in this study adopted different perspectives on knowledge generation, some reports taking a holistic view of knowledge generation and other studies focusing the analysis to some subsets of knowledge management, e.g. research, technical assistance, or country experiences. The fact that the evaluation reports covered a wide range of periods covering that last decade has been another constraint. In next draft of this report, it is proposed to remedy some weaknesses of the current analysis by conducting several interviews in selected MDBs. Further information from other banks on areas of strength and weakness in knowledge generation may be needed as well.


12

2. Areas of progress and strength

2.1 Knowledge management gained wide support in late 1990s 7. The quest for knowledge is nothing new but we have to wait for the 1990s to see the concept of “knowledge management” emerging in most development banks. An influential OECD report issued in 1996 gave new importance to knowledge management as a tool for low income countries to help their development. “The Knowledge-Based Economy”5 outlined the needs to better generate, mobilize, and adapt knowledge to improve the human condition. It recognized that knowledge was increasingly being codified and transmitted through computer and communications networks in the emerging “information society”. At the same time, the spread of networked computers made it possible to codify, store, and share certain kinds of knowledge more easily and cheaply than ever before. Thus, a panorama had opened up in the late 1990s that painted the advances of knowledge in both encouraging and disturbing tones. It was encouraging because of the potential for creating and sharing new forms of knowledge. It was also disturbing, as there was a potential for the development of a knowledge divide between the developed and the developing countries. So, the report called on development banks to bridge this knowledge gap. 8. In advancing the concept of a “knowledge bank”, the World Bank made another major step forward for promoting development knowledge at around the same time. Unlike traditional capital assets like plant and machinery which is bolted to the floor, the World Bank recognized that an institution’s intellectual capital could walk out the office door each evening in the heads and briefcases of employees unless the organization found a way to capture it in the form of operating procedures, shared practices, publications, data repositories, etc. In 1996, James Wolfensohn, the World Bank’s then new President, unveiled his vision of the knowledge bank: “We have been in the business of researching and disseminating the lessons of development for a long time. But the revolution in information technology increases the potential value of these efforts by vastly extending their reach. To capture this potential, we need to invest in the necessary systems, in Washington and worldwide, that will enhance our ability to gather development information and experience, and share it with our clients. We need to become, in effect, the Knowledge Bank”. 6 9. This World Bank knowledge initiative considerably raised the profile of the institution and led it to embark on a series of institutional changes inside and outside to leverage knowledge for development more effectively. This new direction was to be a cornerstone in the World Bank new vision for knowledge management. The “Knowledge Bank” concept was greeted by a combination of high expectations within and outside of the institution. It was seen as promising because more attention to knowledge could enhance the quality and hence effectiveness of investment projects lending. It could offer an alternative stream of services to developing countries and a unique value proposition combining knowledge and finance. As a result, knowledge became visible on international agendas, with the World Bank turning “Knowledge for Development” into the theme of its World Development Report 1998-1999.7 Its main argument was that the development of poorer countries necessitated assigning the highest priority to building “knowledge-based economies”, like the OECD report had done earlier. 10. Development institutions were seen as having three roles in reducing knowledge gaps: (1) provide international public goods; (2) act as intermediaries in the transfer of knowledge; and (3) manage the rapidly growing body of knowledge about development. Although international research was expected to produce relevant knowledge, the report made it clear that knowledge for development

5 OECD. The Knowledge-Based Economy. OCDE/GD(96)102, 1996. 6 The World Bank (2012). The Matrix System at Work: An Evaluation of the World Bank's Organizational Effectiveness. See Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG). This chapter includes an extract of Wolfensohn’s Speech at the 1996 Annual Meeting. 7 World Bank, World Development Report: Knowledge for Development. World Bank, Oxford University Press, 1999.


13

would need to come also from developing countries themselves. “Each change in policy in one country produces knowledge that may help another. Every project, successful or not, yields information about what works and what does not. Amassing this knowledge, assessing it, and making it available to others is a task beyond the capacity (and self-interest) of any single country”. 8 So such task fell to international institutions.

2.2 Knowledge management evolved as a strategic asset 11. Following the World Bank initiative, most other development banks acknowledged the importance of knowledge as a strategic asset and started to pay special attention to their knowledge management functions at the corporate level. All evaluation reports reviewed for this paper acknowledge that the early and mid-2000s saw a surge in the knowledge management agendas of most development banks and other multilateral organizations (e.g., OECD, the World Trade Organization, the United Nations System, and many others). Many MDBs identified knowledge management as one of their core management tools or formed dedicated knowledge departments. Several development banks went so far as to adopt specific knowledge policies and strategies, incurring considerable investments in human resources, connectivity tools, and specific institutional arrangements to create dedicated knowledge applications ranging from publications and databases to media solutions and decentralized training. In part this was for internal purposes, but there were also purposes of making more information available externally. 12. In the 2000s, MDBs developed relevant knowledge agendas and provided a wide range of knowledge services that addressed the needs of different audiences. Not only capacity development was provided in the form of various knowledge products and services (e.g. training, networking, or access to shared information) but public knowledge goods were created and typically consisted of reports available to a broad audience primarily for disseminating analyses of developments or particular sectors or issues. Around that time, a much-needed distinction was made between information and knowledge, the former consisting of explicit formatted data which could not on its own generate new knowledge. Indeed, over the past years, development banks have learned to transform information into a wide variety of knowledge-generating applications allowing the replication of experiments through the mobilization of practical, common sense and experience knowledge. Emerging from the desire to exchange knowledge by making its transmission more efficient, sophisticated management information and knowledge sharing platforms started to emerge within the development field. 13. Recognizing that knowledge management was a core activity, most development banks strengthened their capacity to create and share knowledge by creating specific units and organizations. For instance, the Asian Development Bank Institute (AsDBI) was established in 1997 in Tokyo to help build capacity, skills and knowledge related to poverty reduction and other areas supporting long-term growth in developing economies of the Asia-Pacific region. In particular, AsDBI‘s strategic vision was to become a “leader in creating and sharing knowledge on economic development in the Asia and Pacific region”.9 In 2004, ASDB approved a Knowledge Management Framework. The World Bank Institute (WBI), formerly named the Economic Development Institute when it was founded in 1955 was renamed in 2000 and reorganized to better leverage the power of knowledge, innovation, and coalitions for change. 14. The spread of knowledge institutions and strategies within regional and specialized development banks also signaled the overall increasing importance attached to knowledge management in the MDB community. The IDB also made a significant commitment to knowledge in

8 See World Bank Calls for a Narrowing of the "Knowledge Gap" Between Rich and Poor, Press Release No:99/1955, WASHINGTON, October 2, 1998. 9 ASDB. November 2012. Special Evaluation Study, Performance of the Asian Development Bank Institute: Research, Capacity Building and Training, and Outreach and Knowledge Management, Independent Evaluation Department, Manila, Philippines. See also ASDBI. Year in Review 2009. Tokyo, page 4.


14

the 2000s, with an emphasis on research and knowledge dissemination to complement lending operations. As indicated in a 2006 evaluation report,10 IDB had fully recognized the importance of knowledge in the development process by mid-2000s. “Loans are the end-point of a broad process involving both analytical work and dialogue among stakeholders in borrowing member countries, and the Bank performs a wide array of non-financial activities in support of these areas”.11 IFAD similarly increasingly recognized that knowledge management positively impacted on development effectiveness. An evaluation report indicated that “results on KM show a steady improvement from around 10 per cent of country programmes being evaluated as moderately satisfactory or better in 2006-2008, to around 70 per cent in 2009-2011.”12 15. The evaluation studies reported that progress is being made in MDBs in applying generating and disseminating knowledge, and applying it in their own operations. The 2011 World Bank knowledge report notes that “the Bank’s ability to provide customized advice to clients rests on a broad ecosystem of complementary and interrelated knowledge activities. What clients usually experience is the tip of a complex interaction of many knowledge products and services, which includes research, country and sector reports, project knowledge, and inputs from outside the Bank. Client services inform—and are informed by—the Bank’s global knowledge products AsDB 2012 knowledge management evaluation concluded that AsDB’s knowledge management efforts were responsive in putting in place the necessary enabling environment needed to support the implementation of the knowledge agenda.13

2.3 Significant funding allocated to knowledge management 16. MDBs have increased funding for knowledge products and services in past years. Administrative budget was increased in most development banks for core knowledge work in order to better meet clients’ expectations and needs. Funding for knowledge management was also confirmed to be in line with such budget allocated by the private sector. For instance, the global consultancy McKinsey spends more than 10% of its revenues on knowledge management, a level also reached by many development banks. While AsDB knowledge management accounted for 11% of its operational budget in 2011, IMF’s research activities accounted for about 10% of its gross administrative budget and about 8% of staff time in recent years. 17. Total budget dedicated to knowledge management can be significant. The World Bank has steadily allocated a larger share of its administrative budget to “core” knowledge work, which includes technical assistance, economic and social work, impact evaluations, external client training, research, global monitoring, and new product development. In 2011 these core knowledge services amounted to 31 percent of the World Bank’s budget, compared with 24 percent in 2002. In addition, the amount spent on technical assistance nearly quadrupled between 2002 and 2010.14 18. It is difficult to compare the budget allocated to knowledge generation across MDBs because of differences in the ways to record budget information for knowledge products and services. However, many evaluation reports have disclosed some estimates or ranges of figures, which can be summarized as follows: At AsDB, the three core knowledge departments were allocated USD41.5 million in 2011 (without AsDBI, which operates on a separate budget). In addition, various operations and other departments can draw on budget for technical assistance estimated at around

10 Evaluation of the IDB’s Studies. Office of Evaluation and Oversight. Inter-American Development Bank, November 2006. 11 Inter-American Development Bank (IDB). Evaluation of the IDB's Studies (RE-323). Office of Evaluation and Oversight, 2006. 12 International Fund for Agricultural Development (IFAD). IFAD’s institutional efficiency and efficiency of IFAD-funded operations Corporate-level evaluation, Independent Office of Evaluation of IFAD, July 2013, page 17. 13 Asian Development Bank (ASDB). Knowledge Products and Services: Building a Stronger Knowledge Institution. Special Evaluation Study, Independent Evaluation Department, 2012, page 58. 14 Evaluation Cooperation Group. Evaluating Technical Assistance: Taking Stock of the Practices of International Financial Institutions, December 17, 2012, page 10.


15

USD250-300 million a year.15 At the World Bank, funding for “core knowledge” increased from 2.4% in 2002 for a total of USD302 million to USD606 million in 2010 for country clients (USD444 million), for global public goods (USD62 million), and for internal use (USD99 million). The knowledge spending embedded in World Bank lending was estimated at about USD2.5 billion for 2010, so about four times the amount spent on core knowledge products and services. 16 Taken together, a broad estimate of the total spending on explicit knowledge services in 2010—funded by the institution itself, by partnerships, and from client loan proceeds—comes to about USD4 billion. At IFC, advisory services have grown rapidly since 2001 with expenditures increasing tenfold, from USD24 million in 2001 to USD245 million in 2008. 19. Still, some sources indicate that resource allocations are not always commensurate with broad and expanding demands for customized knowledge solutions. Examining reported lessons on knowledge budgets in evaluation reports, the study finds that while the evaluations indicate that the development banks have made some important financial contributions to knowledge generation, there remains an overall difficulty in budgeting for increased demands. Several development banks have now included knowledge management into their regular programming, with a few pledging additional resources or adopting measures to earmark funding to knowledge. AsDB adopted specific operational and institutional measures to deliver its Strategy 2020 commitments in 2010. Resources were devoted to developing knowledge in the AsDB’s core areas of operations (e.g. sustainable transport, education, etc.) but the various departments have progressively been given the opportunity to establish knowledge-results-management units.17 20. However, providers of development support face challenges as they are now expected to play a combination of roles for providing financing to the poorest regions and creating and disseminating various types of knowledge under new demands. While many of these institutions remain, by and large, development financing institutions, their clients and beneficiaries increasingly demand more than just a simple transfer of financial resources; they ask for customized technical and knowledge support to climb out of the middle-income trap, move forward on the sustainable development path, grow more inclusively, and reap the rewards of regional cooperation. While filling financing gaps remains important, the identification, generation, sharing and use of knowledge is now just as important, and requires substantial budgets and resource allocation, thus raising the question of who will support all the costs of knowledge generation, in particular when a large part of the current knowledge products and services are considered a “global public good” . 21. Some evaluations indicated that the achievement of knowledge results and outcomes is not necessarily correlated with financing arrangements for knowledge services. Factors related to the relevance of design, quality, timeliness, client participation, and use of local expertise have been said to be more closely associated with achievement of results than volume and source of financing.18 As such, responding to needs for funding of knowledge products and services remains a complex issue, with some organizations struggling to leverage the needed resources for knowledge generation and little consistent and reliable evidence to support a call for more substantial funding.

2.4 Knowledge strategies were useful to streamline knowledge 22. The number of international organizations that have developed specific knowledge strategies or frameworks has steadily grown in the early 2000s, a sign of their desire to structure and plan their knowledge products and services more systematically. Most of these strategies have rightly sought to blend knowledge with financing. Overall, knowledge strategies were

15 Lending products may occasionally also include the generation of knowledge products and knowledge services – but the amount is difficult to assess in the aggregate. 16 The World Bank. The State of World Bank Knowledge Services: Knowledge for Development, 2011. 17 Asian Development Bank (ASDB). Knowledge Products and Services: Building a Stronger Knowledge Institution. Special Evaluation Study, Independent Evaluation Department, 2012. 18 See for instance the World Bank evaluation of Knowledge-Based Country Programs, Independent Evaluation Group (IEG), July 2013.


16

useful to spell out how the organization could create value for its constituencies, how that value supported an economic model, and how the staff delivered on the value and the economics. IFAD, for instance, decided to formalize a specific strategy in 2007 to advance its knowledge agenda faster than before. It realized that knowledge was mainly held by individual members of IFAD staff and that little had been done to document and codify it.19 The IFAD knowledge strategy was a logical step so that the organization would become a more relevant knowledge institution. Among the objectives of IFAD’s knowledge strategy were: (i) generating and monitoring a knowledge management strategy; (ii) installing an organizational infrastructure with specific knowledge management roles and structures; (iii) implementing processes and tools for staff to collect, store and share knowledge; and (iv) implementing information technology support.20

IFAD’S EFFORTS TO INSTITUTIONALIZE KNOWLEDGE THROUGH ITS 2007 KNOWLEDGE STRATEGY

Before establishing its 2007 strategy on knowledge management, IFAD took stock of a number of critical lessons emerging from the examination of IFAD’s efforts to streamline knowledge management. Some of the key points outlined by IFAD were as follows: Knowledge management initiatives did not succeed when they were merely “bolted to” established

activities: careful attention and institutional leadership were required to ensure that knowledge management initiatives are embedded in the organization’s work processes and its main delivery instruments (i.e. loans and grants).

IFAD’s knowledge management initiatives before the adoption of the knowledge strategy in 2007 tended to involve isolated activities or to be treated as “add on” activities without a clear unity of purpose. Unsurprisingly, before the strategy, efforts were seen as fragmented and poorly integrated, resources thinly distributed and responsibilities diluted. Knowledge management activities were not planned or implemented with a clear strategic focus nor within a coherent sourcing, planning, reporting and results framework.

A carefully constructed and valued inventory of knowledge assets was essential for improving an institution’s capabilities and performance in knowledge management, and this must be the starting point for any knowledge management strategy.

An institutional culture of learning and sharing knowledge required appropriate human resource policies and practices, including incentives. Inadequate attention was paid to this factor in previous efforts.

It was also noted that the roles, responsibilities, competencies and incentives to perform the knowledge management processes and practices needed clear, careful and consistent attention and institutional support. IFAD’s past efforts attached insufficient importance to these requirements.

Source: IFAD Knowledge Strategy 2007 23. Similarly, AfDB and the World Bank benefited from their knowledge strategy. AfDB’s knowledge policies provided broad guidance on knowledge work, with a first strategy adopted by the AfDB Board in 2005 aiming at strengthening AfDB’s role as a knowledge bank; this strategy was followed by a more comprehensive Knowledge Management and Development Strategy 2008–2012 in 2008 spelling out the knowledge vision and objectives to become the “premier knowledge Bank for Africa”.21 In 2010 the World Bank also adopted a knowledge strategy, called “Transforming the Bank’s Knowledge Agenda: A Framework for Action”. In the same year, the World Bank established its Knowledge and Learning Council to manage knowledge initiatives, including this first Knowledge Report issued in 2011.22 The strategy focused on three key pillars: developing global practices to promote mobility of staff and knowledge; managing the Bank’s knowledge products as a portfolio to ensure greater impact; and strengthening the Bank's role as a global connector of knowledge. Evaluation reports indicated that the World Bank strategy yielded demonstrable results in last few years. For instance, the

19 International Fund for Agricultural Development (IFAD). IFAD’s institutional efficiency and efficiency of IFAD-funded operations Corporate-level evaluation, Independent Office of Evaluation of IFAD, July 2013. 20 IFAD Knowledge Strategy 2007. 21 The African Development Bank (AfDB). Review of the African Development Bank’s Economic and Sector Work (2005−2010). Operations Evaluation Department (OPEV), 2013, page 12. 22 The World Bank. The State of World Bank Knowledge Services: Knowledge for Development, 2011, page 7.


17

newly established Knowledge Platforms have been useful for sharing knowledge, supporting global collaboration and fostering knowledge connectivity with external knowledge centers.

2.5 Organizational arrangements strengthened knowledge generation 24. Many organizational arrangements have been made to strengthen knowledge management, some of them with great success. Institutional arrangements for implementation of knowledge agendas are different among development banks, but evaluation reports concluded that significant efforts have been made to institutionalize knowledge management within each organization. Of course, the development banks have gone through different stages in institutionalizing knowledge management but a model has emerged and it consisted in a more centralized arrangement approach, with knowledge management being first organized at the headquarters level. 25. At the corporate level, a few achievements are notable. At IDB, institutional strengthening took place through the 2007 strategic realignment leading to the creation of the Knowledge and Learning Sector (KNL) under the Vice-Presidency for Sectors and Knowledge, and a Knowledge Management Division. The IFAD knowledge management strategy provided a basis for institutional strengthening of various information technology infrastructure and knowledge management tools, notably web portal and content management tools, virtual workspace platforms with collaborative authoring and editing applications, and better communications solutions. Reorganization in 2011 created the IFAD Strategy and Knowledge Management Department (SKM), which includes an Interdepartmental Knowledge Management and Innovation (KMI) Coordination Group. In May 2010 the World Bank formed the already mentioned Knowledge and Learning Council, comprising members of the World Bank’s Senior Management, to catalyze new approaches to the way it managed and disseminated knowledge work. Similarly, as a result of its institutional strengthening, AfDB spent considerable time to define the purposes and audience of different types of knowledge products and significantly improving its corporate programming of the economic and sector work and other non-lending activities.23 26. AsDB, on the other hand, has not adopted a specific knowledge management strategy but is increasingly perceived to be a pioneer knowledge organization in Asia-Pacific, as also argued by MAKE. 24 AsDB has been developing a number of business processes to streamline knowledge management within the organization. For instance, AsDB’s 2004 Knowledge Framework was perceived as an important effort to create a learning organization at AsDB. In the same year, the Knowledge Management Center was established. The center played a critical role in introducing new approaches, monitoring progress, and reporting to ASDB Management. Strategy 2020 subsequently identified knowledge solutions as one of its five drivers of change for stimulating growth and synergizing broader development assistance in its core and other areas of operations (next to governance and capacity development, private sector development, partnerships, and gender). AsDB’s Knowledge Action Plan 2009-2011 envisaged enhancing the generation and sharing knowledge solutions through specific activities, including Communities of Practice (CoPs), external knowledge partnerships and staff learning and skills development. Of the 37 action points listed under specific activity indicators of the AsDB Action Plan, 70% of had been fully adopted, 8% largely adopted, and 16% partially adopted by July 2011.25

23 The African Development Bank (AfDB). Review of the African Development Bank’s Economic and Sector Work (2005−2010). Operations Evaluation Department (OPEV), 2013, page 18. 24 The Most Admired Knowledge Enterprises (MAKE) program was established in 1998 to recognize leading organizations for their ability to transform corporate knowledge into enterprise intellectual capital and shareholder wealth or societal wealth (for public sector and non-profit organizations). Regional MAKE studies are also conducted for Asia, Europe and North America, as well as the Global MAKE study. 25 Progress report dated April 2011 (Enhancing Knowledge Management Under Strategy 2020: Plan of Action for 2009-2011).


18

27. In conclusion, the new knowledge management capabilities are being used well. In crafting knowledge strategies and deploying new institutional arrangements, many development banks were able to refine their approaches to knowledge management and develop a wider set of capabilities to innovate and respond to changing needs. These new knowledge capabilities which emerged in the 2000s—the knowledge management departments and centers, the knowledge platforms and networks, the CoPs, the knowledge frameworks and strategies, etc.—are the result of sound investments in organizational strengthening. Such investments took the form of extra resources, new action plans, additional staffing, and special training in knowledge products and services and communication.

2.6 Providing knowledge to various audiences, both internal and external 28. MDBs have been compelled to service many different types of audiences, either internal or external, with some gaps to fill. Development banks have largely recognized the need to generate knowledge and share it to all stakeholders, e.g. clients in member countries, partners, internal staff, senior management, public at large, etc. Many development banks developed considerable efforts to understand and categorize the needs of their potential “knowledge audience” and differentiate among the knowledge solutions to be provided, either for their internal or external purposes. For instance, in 2009, the World Bank Knowledge Strategy Group, formed to oversee the preparation of a knowledge strategy, identifies nine product lines as the core of the institution’s knowledge business, for three audiences: (i) knowledge for clients, (ii) knowledge as a public good, and (iii) knowledge for internal use. 29. Other development banks (AsDB, IDN, AfDB) have achieved spent significant time and resources to evaluate and understand the nature of their knowledge products and services. The AfDB, for instance, recognized that there was considerable confusion about how economic and social work, research, and internal reports such as portfolio analyses were positioned for the various audiences.26 In 2013, based upon the findings of the evaluation of economic and social work, AfDB adopted a audience classification similar to the World Bank: Group 1: Knowledge for external clients, including economic and social work, for specific countries

and at the regional or continental levels;

Group 2: Knowledge as a public good without a designated client that advances knowledge in specific areas of development and stimulate debates, for example, products from the Chief Economist Complex (African Development Report, research);

Group 3: Knowledge for internal use to inform Bank staff about the latest approaches or findings, instruments or client products (sector reports, policy papers and portfolio examinations) and tools developed for Bank staff.

30. Similar categorization of knowledge products and services per types of audiences (external, internal and public goods) can be found in other development banks. However, many are grappling with the question of how best to leverage knowledge capabilities for local audiences at country level. The fact that knowledge for development should be shared at country level is obviously not new but many evaluation reports found that the transfer of knowledge is inherently difficult at local level, especially in countries having unequal means to disseminate the knowledge products and services

26 The African Development Bank (AfDB). Review of the African Development Bank’s Economic and Sector Work (2005−2010). Operations Evaluation Department (OPEV), 2013, page XVI.


19

(e.g. lack of knowledge partners, lack of time of local staff, lack of knowledge strategy or framework at country level, etc.). 31. Furthermore, the emergence of knowledge as a public good, a feature of development assistance that gained prominence in recent years, is causing some concerns. A new category of audience has emerged in past years with global public goods becoming regular products of on the shelves of the development banks. Knowledge as a public good denotes work that has no pre-specified external or internal clients, but that ultimately benefits the cause of development worldwide. At the World Bank, these public good services accounted for about USD62 million (or 10 percent) of core knowledge spending in fiscal 2010. However, determining the needs and characteristics of an audience for public goods is difficult. Once they are produced, it is also challenging to ensure that the benefits of such public goods can be equally split up and distributed to the target audience(s). The question of funding of these public goods has been raised with those providing such goods not capturing all of the benefits but bearing all the costs.

2.7 The power to be “knowledge broker” is another key strength 32. Another key strength of development banks is linked to their role as “knowledge connector” or “knowledge broker”. Recent evaluation reports tend to conclude that the main strengths of development banks in knowledge are not only intimately linked to the roles as “knowledge producer” but also as “knowledge broker.” Development banks as knowledge brokers have acted in different manners: as knowledge managers, linkage agents (between producers and users of knowledge), or capacity builders (through enhancing access to knowledge. The evaluations find that development banks spend a lot of time and resources to facilitate the sharing and use of knowledge, with some evidence of success. For instance, a World Bank evaluation revealed that the World Bank has been successful in its ability of connecting the many stakeholders involved in knowledge generation. “The Bank’s convening power was often used to mobilize top international experts for brainstorming sessions and seminars with high-level government officials, or for TA and working sessions with government agencies”.27 33. Knowledge brokering tends to happen in particular locations—e.g. in CoPs or thematic networks that privilege the brokering of knowledge across boundaries, of which the following are a few examples. CoPs are a good example of an appropriate organizational instrument supporting knowledge brokering activities. According to several evaluations, CoPs, thematic networks, informal communication and learning practices have been rightly put in place to share knowledge to the wider communities, many of them showcasing powerful knowledge databases. While CoPs are not new within the field of development, they have evolved in a positive way in some of the MDBs. Although often under-resourced and under-staffed, CoPs are a place where knowledge can be constantly preserved, adapted, and enhanced to allow it to operate effectively in the much larger and more dispersed global operations that development banks have built. The case of AsDB can serve as a good practice to show how an institution can be a “knowledge broker” through the use of CoPs. Some 15 CoPs were constituted, each consisting of 20-150 staff members in areas of AsDB’s expertise like energy, agriculture, water, social development and poverty, and education. The CoPs gather periodically to conduct peer reviews of projects and exchanging confidential information on the business pipelines. Knowing who to contact, and getting good information can become more valuable than any structured and codified knowledge product and service. 34. Other MDBs have also used various formal or informal networks and group to create good knowledge brokering spaces. At the World Bank, thematic groups have become mechanisms for staff to share knowledge and experience and are the primary vehicles through which sharing of tacit and explicit knowledge among regional staff occurs. World Bank CoPs such as those on Financial

27 The World Bank. Knowledge-Based Country Programs, An Evaluation of the World Bank Group Experience. Independent Evaluation Group (IEG), July 2013.


20

Management Information Systems and Political Economy are similar to thematic groups and their inclusive nature and willingness to share knowledge is heavily valued by staff.28 Like thematic groups, membership is open to all interested staff who may access and contribute knowledge. Due to their openness, the thematic groups and communities of practice, when sufficiently resourced, are able to connect the various producers and clients of knowledge products and services.

The ASDB Energy Community of Practice The AsDB Energy CoP is seen as a good practice for providing strategic directions on ASDB's energy sector operations and advised on staff competencies, skills mix distribution, and innovative initiatives. In addition, an informal Energy CoP hosts brown bag seminars and other information exchanges on various energy sector issues. The CoP assumes that a staff in the field of Energy, after years of experience, develops a wealth of expertise at his/her fingertips, e.g. in the form of strategic outlook, extensive networks, privileged contacts in the field, tactical information on project implementation, and so on. With members from 5 regional departments, the Energy CoP is thus a relevant and efficient platform to capture and disseminate such tacit knowledge throughout the network.

Source: AsDB 35. Let us mention the increasing importance and influence of open data and knowledge partnerships for MDBs to perform more effectively as knowledge brokers. Efforts have been made to make knowledge available to all through open-data initiatives. For instance, through its Open Development initiative, the World Bank has been a leader in making information and data freely available and searchable, encouraging feedback, information-sharing, and accountability. Under the new “Access to Information” policy, any World Bank document not on a small exclusionary list will be soon available to the public.29 According to evaluations, most MDBs have now opened their data, knowledge and research to foster innovation and increase transparency on all development projects and aid flows. Institutions such as the World Bank, ADB, or IMF have developed a policy for continually expanding the amount of data available for download. 36. In addition, good practices are reported on how partnerships have led MDBs to become better “knowledge connector”. Evaluations indicated that development banks have significantly increased the scope of informal and external knowledge partnerships over the last years. Indeed, knowledge generation requires a wide range of stakeholders working together toward common goals and, rightly, development banks have created knowledge partnerships to facilitate the interchange of knowledge among partners. Many evaluations (IFAD, AsDB, IFC, World Bank) found that most development have now implemented structured and long-lasting partnerships for the delivery of the knowledge-based programs, although in some cases there was no results framework to monitor impact of these programs. Partnerships at the local level are also said to provide customized development solutions, allowing MDBs to be more responsive to client demands and needs and thus to remain closely connected to those producing and applying development knowledge. Connecting has involved drawing from the data, research, and expertise of client countries and development partners.

2.8 Contribution of TA and AS to knowledge generation 37. TA and AS have significantly contributed to knowledge generation. Many evaluations revealed that a significant modality for financing and facilitating knowledge generation activities is through various forms of Technical Assistance (TA) and Advisory Services (AS). With varying degrees of emphasis and purpose, development banks have provided TA and AS to either improve the design,

28 The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012, page 47. 29 The World Bank. The State of World Bank Knowledge Services: Knowledge for Development, 2011.


21

execution and impact of projects or build institutional capacity, skills, and knowledge in client countries. At IDB, TA aims to facilitate the transfer of technical know�how and qualified experience with the purpose of complementing, strengthening, and building capacity both at the national and regional levels, thus presenting a strong knowledge generation goal. At AsDB, TA has been extensively used to improve the knowledge about development issues in the Asia and Pacific Region. EBRD has extensively used knowledge generation-related TA to promote the development of free market economies, by providing comprehensive training and advisory services to address all aspects of economic and social transformation. With the MDB community, the wide variety of knowledge-related technical assistance typically include: project preparation TAs; policy and advisory services; capacity development; research and development, sector studies, trainings; etc. 38. Knowledge generation-related TA and AS has steadily grown over the past years. Several evaluation reports acknowledged that the volume of TA and AS has increased in past years, with knowledge-related TA and AS growing faster than lending-related ones.30 Evidence was found that World Bank TA supporting client efforts to implement reforms and strengthen institutions has increased from 782 in 2005 to 1,424 in 2010.31 Similarly, AsDB approved a total of USD249.1 million on knowledge generation-related TA over the 2002–2011 decade for advisory services, research, studies, capacity development, and other knowledge related activities. Over the period, AsDB approved more than 2,700 TA operations worth nearly USD2.5 billion. At IFC, AS have grown rapidly since 2001, with expenditures increasing tenfold from USD24 million in 2001 to US245 million in 2008. Based on published data, IFC’s share of MDB advisory services to the private sector is about a quarter. 32 In addition, in 2008, the cost for employing consultants delivering AS (some 1,332) was USD72.3 million, only slightly less than IFC staff costs (1,262 staff at USD82.7 million). 39. Although TA and AS are well perceived by clients, efforts are still needed to better capture knowledge from technical and advisory services. Many reports pointed to the general satisfaction of clients with TA and AS provided by development banks. In particular, clients of AsDB, World Bank, IDB and IFC placed a high value on these TA and AS because they provide “just-in-time” and accessible knowledge products and services that require their engagement and collaboration. A recently released World Bank evaluation of knowledge-based country programs 33 revealed that intended outcomes were fully achieved or likely to be achieved in 47 percent of the knowledge activities reviewed and partly achieved in another 37 percent. Similarly, outcomes of IFC’s AS were achieved in about 38 percent of the reviewed projects. However, several reports indicated that findings and lessons identified in the various TA were not systematically shared, thus point to a weakness in the knowledge management of TA products and processes.34 35

2.9 International recognition of MDBs’ Knowledge management 40. Most development banks have taken various measures to develop intellectual capital as a strategic asset and this is well recognized. While there is variations in MDBs’ strategies to implement a knowledge management agenda, the review of several evaluations revealed that that common objectives, tools, and language related to knowledge generation have emerged in recent years. Overall, development banks as a group have made significant progress in institutionalizing knowledge products and advisory services within their organization.

30 As indicated in the ECG 2012 report “Evaluating Technical Assistance: Taking Stock of the Practices of International Financial Institutions”, the estimated annual amount of TA provided by ECG members in 2011 ranged from USD44 million for IDB to USD634 million for the World Bank. Information gathered from ECG consultant’s report on TA shows that the share of the administrative budget spent by four institutions on TA ranges from 0.5 percent to 3 percent. 31 The World Bank. The State of World Bank Knowledge Services: Knowledge for Development, 2011. 32 The World Bank. Knowledge for Private Sector Development. Independent Evaluation Group (IEG), 2009, page 31. 33 The World Bank. Knowledge-Based Country Programs, An Evaluation of the World Bank Group Experience. Independent Evaluation Group (IEG), July 2013. 34 Independent Evaluation Department. 2007. Special Evaluation Study: Performance of Technical Assistance. Manila: ADB. 35 The World Bank. Knowledge for Private Sector Development. Independent Evaluation Group (IEG), 2009, page 51.


22

41. Importantly, evaluations have evidenced substantial progress in mainstreaming knowledge management agendas over the last years. For instance, the MOPAN assessments are quite revealing. The MOPAN reports assess each year a certain number of institutions by focusing on strategic, operational, relationship and knowledge management. By benchmarking MOPAN on knowledge management indicators for IFAD, AsDB, AfDB, IDB and the World Bank, high consistency in the ratings of knowledge management are achieved, with a range of value fluctuating between 4.2 and 4.5 (out of 6 as the strongest score).36 Moreover, the ability of several institutions to strengthen knowledge management capacity was noted by outside organizations. Some multilateral organizations have been able to benchmark their knowledge agenda against leading public and private sector organization in the Most Admired Knowledge Enterprises (MAKE) initiative.

Recognition of AsDB knowledge management efforts in the MAKE survey

The MAKE program was established in 1998 to recognize leading organizations for their ability to transform corporate knowledge into enterprise intellectual capital and shareholder wealth or societal wealth (for public sector and non-profit organizations). Regional MAKE studies are conducted for Asia, Europe and North America, as well as the Global MAKE study. For instance, the World Bank Group has been named as one of the world's top 20 Most Admired Knowledge Enterprises over several years. In 2011, AsDB’s efforts were also rewarded with global recognition with a citation for collaborative, enterprise-wide knowledge sharing in Asian MAKE award. AsDB’s first-ever win in this award placed the organization among Asian 20 top entities (and one of only four from the public sector) picked by a panel of Fortune 500 executives and knowledge management experts. As a result, AsDB became an internationally recognized knowledge organization fully aligned to best knowledge practices of leading private sector firms such as Accenture, Apple, Google, or Microsoft. To conclude, several development banks have made significant progress to conform their knowledge management with international best practice and reaching recognition for doing so.

36 International Fund for Agricultural Development (IFAD). IFAD’s institutional efficiency and efficiency of IFAD-funded operations Corporate-level evaluation, Independent Office of Evaluation of IFAD, July 2013, page 106.


23

3. CRITICAL KNOWLEDGE MANAGEMENT GAPS 42. The last section provided an opportunity to reflect on some of the progress made in knowledge management. Several development banks have achieved progress in terms of their commitment to learning and knowledge, development of a knowledge culture, recognition of the value of knowledge partnerships and network solutions, and increased resources earmarked for knowledge products and services and flagship publications. Nevertheless, it will be useful to a briefly review some critical gaps in development knowledge.

3.1 The “knowledge bank” concept remains largely unachieved 43. The new and promising “knowledge bank” concept of late 1990s to which gradually more MDBs have wanted to subscribe has not yet materialized. The recent evaluation of the World Bank Matrix system is unambiguous. The matrix system has been highly inefficient and modestly effective in achieving this outcome. Rather than functioning as a global organization, the evidence from the flow of virtual knowledge and technical expertise indicates that the World Bank is “at risk of evolving into a group of regional banks interconnected by fraying ties among the Regions and between the Regions and networks”. Knowledge that is generated by the World Bank or captured from operations implemented by clients is rarely used in future operations or in other contexts. “Decentralization has created further impediments to the flows of knowledge and expertise, and until the underlying incentives and constraints inhibiting effective knowledge flows are addressed, further decentralization will inevitably aggravate these problems”.37 44. A similar conclusion is reached in the 2012 AsDB evaluation of knowledge products and services. “Faced with the ASDB President’s vision of transforming the institution into a “knowledge bank,” AsDB did not respond predictably. Instead of drafting a corporate wide policy or strategy on KM, AsDB has allowed decision making and priority setting for knowledge to remain largely within each department, whether designated knowledge departments or operations departments. With time, this ambiguity has hindered the development of a cohesive system-wide approach to promoting knowledge in a coherent fashion, thus making it difficult for staff to integrate knowledge into the day-to-day work in a logical fashion.38

The World Bank: Efficiency and Effectiveness of the Knowledge Bank

A few years later after announcing the vision for a “Knowledge Bank”, an evaluation39 noted that only limited progress had been achieved. The study concluded that the World Bank had made more progress in establishing the architecture to support its knowledge initiative than in creating the governance arrangements and work processes for carrying it out. As a result, the strategic intent of making knowledge sharing a way of doing business had been only partly realized. The World Bank needed to move deliberately to embed knowledge sharing in its core operational processes by providing more direct support to task teams and more knowledge capacity enhancement for clients, and it needed to manage its knowledge services for results. In other words, despite the spread of new technologies and the emergence of the internet as a public network, still the creation and dissemination of knowledge remained a challenge in the early 2000s. The spread and democratization of knowledge (e.g. through the new technologies) represented a significant shift of focus in the approach to the generation and sharing of knowledge. Put in the simplest terms, it had become easier to make available knowledge products and advisory services to all stakeholders of development assistance. Still many

37 The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012. 38 Asian Development Bank (ASDB). Knowledge Products and Services: Building a Stronger Knowledge Institution. Special Evaluation Study, Independent Evaluation Department, 2012, page 49. 39 World Bank, Operations Evaluation Department, Sharing Knowledge: Innovations and Remaining Challenges. World Bank, 2003. See also Appendix 1 of this report for a review of this evaluation study.


24

gaps do exist.

In its 2012 evaluation of the Matrix at Work,40 the World Bank stressed that the institution’s role in development knowledge was equally vital to the quality of its lending portfolio. In addition to knowledge services that have a direct bearing on policy advice and institutional development, there had always been a complementarity between knowledge and lending. However, the evaluation report noted some weaknesses in the knowledge approach:

The three forms of knowledge—explicit, tacit, and embedded—do not draw on each other efficiently, which reduces the World Bank’s development effectiveness.

The World Bank generates a large volume of knowledge embedded in its operations but lacks the ability to efficiently capture and share that knowledge.

The availability of trust funds has led to an increase in core knowledge production and sharing by 50 percent, but the World Bank’s budget for knowledge barely increased. The growing reliance on trust funds generates inefficiencies and risks.

Tacit knowledge is largely untracked, making it difficult to assess its efficiency and effectiveness.

Sector silos persist, but World Bank region and network silos are even stronger. Overall, the flow of tacit knowledge through cross support has been rather limited, is declining in volume, and is constrained by structural boundaries.

The communication tools for the World Bank to act as a connector of knowledge are available, but not systematically used.

Innovation in the field is not well captured across the institution, and country operations do not draw efficiently on global knowledge to meet client needs.

Based on the World Bank’s performance since the introduction of the matrix, both in efficiency and effectiveness, the intermediate outcome of the World Bank’s knowledge work—capturing innovation across the institution, and drawing on global knowledge to meet client needs—has not been achieved.

Source: The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012. 45. Thus, lessons from the experiences of the World Bank and other development banks underscore the complexities, difficulties and pitfalls of transforming development finance institutions into “knowledge banks”. Knowledge was wrongly viewed as an operational issue only, not a strategic one, and senior management often failed to articulate why knowledge products and services would be produced and what value they would bring to the institution. Knowledge was seen as a commodity produced by all departments of the organization and without the need for specific strategic alignment or coherence. In other words, many institutions failed to leverage knowledge products and services to drive strategy, generate new investment opportunities, solve problems, develop employees' professional skills, and retain knowledge talent. 46. Weak strategic and operational guidance has been identified as one of the key constraints for mainstreaming knowledge management agendas. Senior management of a few development banks has now placed knowledge management at the top of their agendas. Others have not given it the same attention as they have let knowledge be managed—if at all—by a wide variety of functional departments. Without prioritization of knowledge products and services by senior management, development banks will continue to follow ‘multi-silo’ knowledge approach implemented at departmental level—based upon departmental strategies all different in format and content.

3.2 An overall lack of conceptual clarity and definitions 47. The MDBs’ ability to give guidance and promote a cohesive and integrated response in

40 The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012, page 60.


25

knowledge management is hampered by the lack of conceptual clarity. The lack of standard definition(s), guiding principles, and metrics is a major impediment to inculcating a knowledge culture in most organizations. Staff requires a clear definition of what constitutes knowledge, knowledge products, knowledge services, and what the priority is for knowledge generation, dissemination and management vis-a-vis other pursuits. Without a strict application of a knowledge definition at all operational levels and within a variety of interventions (national, sector, program, project, and organization), staff are tempted to use their own concepts of knowledge management at various phases of national and development programming. Knowledge management has enjoyed increasing popularity in recent years, but as a term it often means different things to different people. 48. MDBs have proposed different typologies of knowledge products and services. Some are based on the source of the knowledge and its format. Traditionally a distinction is made between implicit knowledge (e.g. daily life or common sense knowledge, experience knowledge, local or indigenous knowledge, action knowledge) and explicit knowledge (practical, theoretical or creative knowledge). However, other typologies emphasized the context in which knowledge is used, as defined by the knowledge itself (normative and descriptive knowledge, strategic and operative knowledge, scientific and empirical knowledge, past- and future-oriented knowledge). Finally, certain organizations have focused more on the modes of inscription of knowledge, and thus distinguish between: ‘embrained’ knowledge (based on certain conceptual and cognitive skills), embodied knowledge, ‘encultured’ knowledge (built up in the processes of socialisation that lead to shared forms of understanding), embedded knowledge (in systemic routines), and encoded knowledge (which can be considered as equivalent to information). 49. Not surprisingly, many surveys or studies conducted in recent years revealed that staff is overly concerned by the lack of a common understanding and vision on knowledge management. For instance, at AsDB, it was noted that the lack of standard definition(s), guiding principles and metrics was a major pitfall to implementing a knowledge management culture. The 2012 evaluation of knowledge products and services considered the operational implications of the knowledge management definition(s), principles and metrics and confirmed that most AsDB informants indicated that they require a clear definition of what constitutes ‘knowledge’. Without a strict application of a knowledge management definition at all corporate levels of the organization and within a variety of interventions (national, sector, program, project, and organization), staff are tempted to use their own knowledge tactics and tools at various phases of national and development programming. That simple observation noted above adds an important dimension to the topic of knowledge management and may explain why many institutions have been developing a multitude of knowledge action plans and policies at sub-corporate levels, thus adding more confusion on how to use knowledge management.

3.3 Knowledge is increasingly dispersed, unplanned and poorly coordinated 50. MDBs have increasingly served as generators and repositories of knowledge, with many approaches and systems in place. Evaluations revealed that development banks have allowed decision-making and priority-setting for knowledge to remain largely dispersed, whether it is core-knowledge departments or operations ones. With time, this ambiguity has hindered the development of a well-defined and cohesive system-wide knowledge approach for many institutions, thus making it difficult for staff to integrate knowledge into the day-to-day work in a logical fashion. Indeed, most institutions did not respond predictably with their knowledge management and, as a result, the production of knowledge has spread in all directions. 51. This lack of direction has resulted in a wide variety of knowledge products and services, many of them bringing no clear benefits to stakeholders. More worrisome, the volume of knowledge products and services seems to be difficult. The 2012 AsDB evaluation revealed that the stocktaking of knowledge products for 2004–2011 for AsDB (including AsDBI) resulted in at least


26

5,138 products.41 Over the same period, knowledge generation and sharing increased about five-fold with a substantial increase every year of the number of books, reports, journals, briefs, working papers, training and instructive materials as well as a general and multi-media materials posted on the ASDB website. Similarly, a recent AfDB evaluation report 42 found that knowledge products for economic and sector work has increased substantially in recent years, with the 2010 production up 27% over 2009 and more than five times higher than 2005. Only 11 economic and social work products were released in 2005, compared with 45 in 2009 and 62 in 2010. During 1999–2008, the IMF issued an average of 650 research pieces annually, representing a vast body of research at an annual cost of about 10 percent of the IMF budget.43 Hence, development banks are today providers of an increasingly sprawling range of knowledge products. 52. The diverse nature of knowledge products and advisory services poses challenges to the effectiveness of knowledge coordination. Among the many forces driving the scattering of knowledge, an important problem occurring progressively over the past decade, has become an increasing source of concern in the development community. Various World Bank reports mention the concern over coordination of knowledge. The 2012 Matrix System at Work reports that a frequent concern among staff was that the regional sector units seemed disconnected from each other, a fact that is complicated by the far reaching decentralization of sector staff in recent years. As one World Bank survey respondent said, “with some technical staff decentralized, and little resources devoted to knowledge sharing among sector staff, it is difficult for operational staff to reflect experiences from other Regions or from outside the Bank to enhance [the] quality of the operations.”44 53. The 2009 IMF evaluation of research also concluded that there was limited coordination or prioritization across departments despite the widespread view among staff that priorities for research should be set across departments, if not IMF-wide. Attempts have been made in the IMF to introduce a coordinating mechanism, for instance through the Committee on Research Priorities, but none of these has endured. 45 Similarly, at AsDB, there is limited success in implementing a successful coordination mechanism for knowledge activities. AsDB’s approach to knowledge management is multidimensional as it focuses on cooperation from various departments, including the core-knowledge departments and AsDBI. Although there is growing cooperation on knowledge generation and sharing at all levels of AsDB, still many of the agreed-upon modalities for better coordination are yet to be implemented. For instance, the Knowledge Management Center established in 2004 has not been serving as the intended focal point to oversee the coherence of these knowledge efforts. The 2013 AfDB evaluation report provided a similar conclusion and pointed out that the dissemination of reports is ad hoc with no formal guidelines and coordination mechanisms.46 54. A large share of knowledge products and services is not planned, with costs not always anticipated well. Evaluations also confirmed that knowledge is not only produced in a non-organized fashion, but also the great share of products and services is not planned at all. The IAsDB evaluation47 reported that a large share of IAsDB studies appearing on web pages are not officially recorded in the database tracking studies, showing that a very large fraction of studies is simply unplanned. As a result development banks are often confronted with the reality of not being able to monitor knowledge

41 Asian Development Bank (ASDB). Knowledge Products and Services: Building a Stronger Knowledge Institution. Special Evaluation Study, Independent Evaluation Department, 2012, page 8. 42 The African Development Bank (AfDB). Review of the African Development Bank’s Economic and Sector Work (2005−2010). Operations Evaluation Department (OPEV), 2013, page 15. 43 International Monetary Fund (IMF). Research at the IMF: Relevance and Utilization. Independent Evaluation Office (IEO), 2011, page 7. 44 The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012, page 48. 45 International Monetary Fund (IMF). Research at the IMF: Relevance and Utilization. Independent Evaluation Office (IEO), 2011, page 21. 46 The African Development Bank (AfDB). Review of the African Development Bank’s Economic and Sector Work (2005−2010). Operations Evaluation Department (OPEV), 2013, page 23. 47 Inter-American Development Bank (IDB). Evaluation of the IDB's Studies (RE-323). Office of Evaluation and Oversight, 2006, page 12.


27

products and services produced each year and therefore in a difficult position to orchestrate an efficient programming. Although many development banks have sought to nurture a knowledge culture across the organization in past years, e.g. drafting a specific framework and action plan, they have been lacking a major impulse to anticipate the needs for knowledge products and services and thus plan the production in a control manner. Finally, given the explosion in the sheer amount of knowledge that exists in development banks, it is seen as fundamental to develop a mechanism to prioritize needs and ensure that only the most relevant knowledge products and services are retained for production and dissemination.

3.4 Just-in-time, operationally-relevant knowledge is lacking 55. In the early 2000s, evaluation reports already warned of the risks of developing knowledge products and services not related to operations. A 2003 evaluation report indicated that knowledge management in the World Bank primarily consisted of knowledge aggregation and sharing—processes that by themselves did not guarantee that the shared knowledge would be adopted, adapted, and applied. “For that to happen, knowledge sharing has to be embedded in work processes. But, for the most part, the new activities have not been tightly linked to the Bank’s core lending and nonlending tasks. As a result, staff and clients do not view the new knowledge-sharing programs and activities as sufficiently relevant to their operational work.”48 The 2003 report concluded that these weaknesses in reach and operational linkage led the World Bank’s knowledge-sharing activities and programs to have “limited impact” on the client countries. 56. Unfortunately, a 2012 evaluation found similar problems some ten years later. Lending volume are the most important metric of success at the World Bank and the report concluded that the lack of incentive for staff and mangers to give priority and funding to knowledge activities was still a major hurdle. Not only the dissemination of operationally useful knowledge was perceived as weak at the regional levels. Also the impact of knowledge products was limited, due to “weaker dialogue on the analytical findings and inadequate client ownership and involvement in the analytical activities”. The 2012 evaluation report summarized the issue as follows: “At the institutional level, knowledge management has not worked well. In the Regions, we are under too much pressure to deliver lending. We have no time to think or document our experience, although this was originally envisaged as part of the matrix reform. We have no resources to document and disseminate experience systematically. The main constraint is pressure of lending delivery”. A 2013 World Bank evaluation reiterated that knowledge products and services used in the design of lending operations were more likely to succeed than freestanding knowledge services.49 57. Many evaluations concluded that knowledge management is often supply-driven, not always operationally-relevant and not responding to clients’ needs. Knowledge products and services tend to be too often produced by headquarters, with few inputs from experience of governments and authorities at the local level. IFAD noted that “knowledge management in country programs has not yet been sufficiently focused on scaling up IFAD’s impact on the ground”, with the institution not focusing enough on mining the rich knowledge embedded within the operations it funds.50

58. Similarly, the 2011 IMF evaluation of research concluded that the relevance of research was often hampered by lack of early consultation with country authorities on research themes and by lack of sufficient country and institutional context. Also, authorities indicated that some important issues, such as macro-financial linkages and aspects of monetary policy, were not adequately covered. The

48 The World Bank. Sharing Knowledge: Innovations and Remaining Challenges. Operations Evaluation Department, 2003, page XIV. 49 The World Bank. Knowledge-Based Country Programs, An Evaluation of the World Bank Group Experience. Independent Evaluation Group (IEG), July 2013. 50 International Fund for Agricultural Development (IFAD). IFAD’s institutional efficiency and efficiency of IFAD-funded operations Corporate-level evaluation, Independent Office of Evaluation of IFAD, July 2013, page 47.


28

IMF report indicated that national authorities often considered IMF research to be “quite insular—authors tended to cite mostly other IMF research and often failed to cite work written by local researchers, thereby missing important institutional dimensions.” 51 World Bank evaluations indicated that advisory services often take too long and is outdated by the time it is finalized. Clients do not always see the value of such services if it is not demand-driven and does not provide options and solutions to problems. 52 A consensus seems to emerge to say that, to be relevant, knowledge products and services must respond to clearly established demand; it should address important topics and be adequately informed about country context and institutional setup and be provided in a timely fashion. 59. A purely supply-driven approach is not advisable. The supply-driven approach now in place in many development banks is preventing management and staff from recognizing specific knowledge needs, thus undermining operations. This argument is to be taken seriously. Any lack of access to timely and operationally relevant information at country level would undercut the organization’s ability to create a real competitive advantage from knowledge products and services. The AsDB evaluation notes that “any lack of access to timely and operationally relevant information at the country level would undermine the resident mission’s ability to create a real competitive advantage from knowledge products and services. Not only do effective operational strategies at the country level require adequate knowledge to engage government officials and partners, but also resident missions cannot operate in a competitive business environment in equivalent detail and timeliness if such services and products are less relevant”.53 Finally, a supply-driven approach would often fail to recognize changing patterns and needs in developing countries. The clients of most development banks are diversified and include graduated countries, middle-income countries, low-income countries, and fragile states. 60. Knowledge products and services should have a greater emphasis on country needs. Evaluations highlighted the need for the knowledge organization to avoid concentrating knowledge products and services at the corporate level, thus leaving regional and country offices with too little opportunities and resources to carry out specific knowledge tasks at the field level. Knowledge generation is undermined by the lack of use of knowledge products and services in influencing decision making, in particular in the countries. For instance, an IMF evaluation noted that “many country authorities expressed a dislike of cross-country panel regressions, saying the policy advice that emerged from this methodology was not very informative”. 54 A 2013 AfDB study concluded that there was “little attention to country priority needs with a tendency to follow management or the Board’s topics or themes (governance, gender, private sector, infrastructure, regional integration)” and a sudden drop in the interest in some topics once trust funds are no longer available, and thus raising the “question of the ESW relevance for the Bank”.55 With limited resources, some people from field offices argue that knowledge products and services need to be focused in the first instance on scaling up successful operations in countries, “rather than pursued as ends in themselves”.56 This last point is matter to a debate as many freestanding knowledge products and services are said to be relevant to their audience.

51 International Monetary Fund (IMF). Research at the IMF: Relevance and Utilization. Independent Evaluation Office (IEO), 2011, page 12. 52 The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012. 53 Asian Development Bank (ASDB). Knowledge Products and Services: Building a Stronger Knowledge Institution. Special Evaluation Study, Independent Evaluation Department, 2012, page 45. 54 International Monetary Fund (IMF). Research at the IMF: Relevance and Utilization. Independent Evaluation Office (IEO), 2011, page 13. 55 The African Development Bank (AfDB). Review of the African Development Bank’s Economic and Sector Work (2005−2010). Operations Evaluation Department (OPEV), 2013, pages 18 and 24. 56 International Fund for Agricultural Development (IFAD). IFAD’s institutional efficiency and efficiency of IFAD-funded operations Corporate-level evaluation, Independent Office of Evaluation of IFAD, July 2013, page 17.


29

3.5 Managing for quality and incentives to do better 61. While there is unanimous, strong appreciation for generation of knowledge, a fact undisputed in MDBs, the overall knowledge output is best described as of “mixed” quality. Evaluations found that many flagship studies have been a particularly successful knowledge product. They achieved many of their intended outcomes, including robust analytics to policymaking and better international debate on economic and development issues. For instance, the IMF World Economic Outlook is one of the two IMF flagship publications widely utilized and, overall, one of the most influential publications in terms of the reach of analysis, findings, and messages. Almost all country authorities reported that they paid regular attention to the World Economic Outlook and said that it was widely read. While the almost-universal use of this publication was driven in part by the extensive use of the analytics, the usefulness of messages to inform decision-making was also cited as very important.57

62. At AsDB, such flagship publications included the Asian Development Outlook flagship series, some of which had contributed to global and regional policy discussions on many development issues and in particular led the debate on Asia's post-global crisis growth strategy. ASDB’s Economics of Climate Change in Southeast Asia: A Regional Review, informed climate change policy making and created a momentum for more in-depth country-level analyses. These policy reports are noted as successful because they presented policy options to national authorities and decision makers on country-specific responses to some challenges.58 Many other knowledge products and services were mentioned as important, all of them with the characteristics of informing decision makers with timely information and data. Relevant were also many statistical reports and economic guides and monitors. 63. Evaluations are all unanimous to conclude that quality management problems undermined the production of knowledge products and services. A 2013 AfDB evaluation report noted that harmonized procedures for quality control of ESW did not exist. “Internal and external reviews are conducted for some ESW products, particularly those with a high profile, but only on an ad-hoc basis. The Operations Manual does not provide guidance. The section devoted to ESW was planned but never written, so there are still no formal guidelines or procedures for ESW.59 Similarly, the evaluation of IDB studies concluded that formal methods of quality control were not systematically applied to the production of most studies. Although some IDB departments use quality control procedures for some types of studies, “for the majority of studies in the Bank, there is no indication of any sort of quality review”.60

64. A recent evaluation of IMF research also indicated that the analytical chapters of flagship publications such as the World Economic Outlook or the Global Financial Stability Reports were of high quality, while the quality of other product lines was mixed, with great variability within products and across themes.61 Similar to the findings of many evaluation, the IMF study put into evidence a lack of systematic IMF-wide process to review research products. “The review process varied across departments and research products. The World Economic Outlook and Global Financial Stability Reports were subjected to a formal and structured interdepartmental review process. Regional Economic Outlook reports were reviewed through a structured process in each area department, but these processes varied across departments. There was no systematic and uniform process for

57 International Monetary Fund (IMF). Research at the IMF: Relevance and Utilization. Independent Evaluation Office (IEO), 2011, page 11. 58 Asian Development Bank (ASDB). Knowledge Products and Services: Building a Stronger Knowledge Institution. Special Evaluation Study, Independent Evaluation Department, 2012, page 38. 59 The African Development Bank (AfDB). Review of the African Development Bank’s Economic and Sector Work (2005−2010). Operations Evaluation Department (OPEV), 2013, page 21. 60 Inter-American Development Bank (IDB). Evaluation of the IDB's Studies (RE-323). Office of Evaluation and Oversight, 2006, page 17. 61 International Monetary Fund (IMF). Research at the IMF: Relevance and Utilization. Independent Evaluation Office (IEO), 2011, page 14.


30

reviewing paper”.62 Departmental seminars were used in some, but not all, departments as part of the review process and to gather comments. Indeed, the lack of rigorous quality management procedures for knowledge products and advisory services hamper the ways development banks can function coherently under their development agenda. 65. In summary, staff needs more incentives to produce better quality knowledge products and services. Any instances of poor knowledge can adversely affect the reputation of development intuitions and their ability to respond appropriately. Evaluations suggested that development banks encourage staff to write down what they know and to get those documents into the public space. However, incentives to share knowledge should be not forgotten. It is often easier to create knowledge than to disseminate it. Consulting firms such as McKinsey or Bain & Co annually review the degree of high-quality person-to-person dialogue a staff member has had with clients, an approach which can account for as much as one-quarter of a staff annual compensation.

3.6 IT systems are not set up to support full knowledge dissemination 66. Because knowledge must be de-coded and re-coded, sophisticated IT-driven knowledge management tools have been deployed to avoid calcifying knowledge into static, inert information. Development banks have rightly invested into technology to better streamline content management and improve tools for archiving, indexing, searchability, and retrieval. The World Bank governance and anticorruption team seems to be in the forefront of using technology to reach out to regional staff. 63 Nowadays IT is as much about creating direct connections among staff and development partners through such applications as electronic mail, webinars, video-conferencing, open learning platforms, social media and other types of groupware, as about storing high volume, high velocity data into large databases and other types of repositories. Thus, linking together different systems into a more integrated and seamless infrastructure is a necessary step and many MDBs recognized investments in IT are unavoidable in order to scale up knowledge management projects. 67. Still, some MDBs do not yet have good information technology (IT) supporting knowledge management. Evaluations found that several development banks are operating with fragmented, often incompatible systems. Most MDBs are using a multitude of sophisticated electronic repository systems with proprietary search engines but do not have a logically-developed hierarchy of databases, e.g. smaller database for exclusive use of selected staff/partners and larger “holding tanks” containing publicly-available data and knowledge. The case is serious at AsDB. Despite significant investment into the physical and digital infrastructure that permits the flow of informational contents (e.g. C-Cube, the system allowing to share knowledge in various networks and CoPs), AsDB systems are not conducive to staff members devoting time to knowledge sharing and thus ensuring a proper dissemination of knowledge products and services. A major gap was noted in the 2012 knowledge evaluation report in respect to the lack of an integrated information-sharing platform, e.g. a dedicated knowledge extranet to provide search and retrieval capabilities across all AsDB document repositories of knowledge. 68. Disparate knowledge management visions, models, and needs are influencing the choice of IT systems, creating some serious concerns for properly disseminating knowledge. Most evaluations noted that comprehensive IT support is critical to the success of knowledge management. However, each institution seems to have its own problems with the IT systems. At IDB, a fairly decentralized and content MIS resulted in inefficiency, constraining access, visibility, and dissemination of knowledge products. The processes by which the development banks shares information with the field, and how information from various partners is shared throughout the entire

62 IMF, 2011. 63 The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012.


31

ecosystem, is “flawed”, with a direct negative impact of the quality of knowledge and analytical products eventually provided in support of operations. Over time, the dissemination and proper storing for public usage has become a concern.64 69. Similarly, the World Bank evaluation of the matrix system revealed some flaws in the management information systems. “The communication tools for the Bank to act as a connector of knowledge are available, but not systematically used”.65 The 2011 IMF evaluation also revealed a stunning problem. During the evaluation period, the IMF changed its IT strategy to move towards Internet-based dissemination in an effort to reduce the production and distribution of hard copies of knowledge products. The shift to electronic dissemination facilitated access in most member countries and by staff. But, authorities in 40% of countries eligible to extended credit facilities indicated that it had diminished their usage of IMF publications, partly because of connectivity problems.66 AfDB admitted that one of the fundamental issues facing the institution in undertaking knowledge activities was the lack of a management information system with a comprehensive database that documents economic and social work results and outcomes.67 70. Finally, any IT deployment will not achieve much, if it is not accompanied by a global cultural change toward knowledge values. Today, many development banks are building a corporate memory in a totally unstructured way. They maintain all documents and record all practices of the institution in IT systems, amassing a lot of irrelevant information that is not filtered, and put it on the web and other open datasets as “public goods”. Over the past years, the motto has been to involve an intensive IT and knowledge engineering effort leading to the construction of a wide variety of corporate knowledge databases, trusted networks and expert systems. In doing so, some development banks overrated the possibilities of knowledge management to the point where some people believed that computer-managed knowledge would be sufficient for the institution. However, having most knowledge products and services online does not mean that knowledge if properly disseminated. Therefore, IT should not be seen as a process through which knowledge is easily and automatically shared at large. Knowledge management is not only about tools. Implementing a good IT strategy for knowledge management needs to involve many complementary changes that can predate and accompany the system installation. More importantly, development banks have to deal with the issue of rapid change in knowledge and outdated information; they also need to be asked what kind of knowledge exists in the organization and what kind of knowledge is needed to improve performance. Such approach will require a cultural change on the value of knowledge in the organization.

3.7 Tacit knowledge is largely not captured and well utilized 71. Tacit knowledge, a complex topic and by nature difficult to quantify, remains largely unmonitored. Tacit knowledge exists in the form of operational experience of staff, clients, and partners. While the magnitude of codified knowledge is known—at least in terms of the number of knowledge products delivered and their cost—most development banks have not been able to put a value on tacit knowledge.68 Some development banks have made some progress in better utilizing tacit knowledge. AsDB’s CoPs have been a conduit for transferring tacit knowledge among AsDB staff. CoP discussion of projects and business plans is a good illustration on how tacit knowledge can be socially constructed, through an informal process allowing staff to gain a broader perspective and understanding of various projects and good practices. But, overall, each institution’s tacit knowledge

64 Inter-American Development Bank (IDB). Evaluation of the IDB's Studies (RE-323). Office of Evaluation and Oversight, 2006, page 22. 65 The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012, page 59. 66 International Monetary Fund (IMF). Research at the IMF: Relevance and Utilization. Independent Evaluation Office (IEO), 2011, page 23. 67 The African Development Bank (AfDB). Review of the African Development Bank’s Economic and Sector Work (2005−2010). Operations Evaluation Department (OPEV), 2013. 68 The World Bank. The State of World Bank Knowledge Services: Knowledge for Development, 2011.


32

remains largely untapped beyond the more immediate context within which it is employed. A World Bank evaluation stated that “the only portion of tacit knowledge that can be tracked and analyzed at present is the movement of staff and cross support, and even that is constrained by incomplete and inconsistent data across the Regions… Overall, the flow of tacit knowledge through cross support has been rather limited, is declining in volume, and is constrained by structural boundaries.”69 72. The retention of managerial knowledge, a sub-category of tacit knowledge, is most problematic. Such knowledge centers on the realities of organizational life, organizational processes, and the working of organizational units and staff. Unlike technical knowledge, managerial knowledge comprises a wide range of experiences difficult to harness and store. The World Bank evaluation of the matrix system noted that the movement of tacit knowledge within the World Bank is not easy to capture and mostly due to the periodic rotation of staff between departments and regions. A natural time to capture tacit knowledge would be when a long-serving staff member moves to a new region or departments (or retires), but the World Bank lacks any mechanism and resources to debrief staff upon leaving a unit or to prepare staff moving to a new region. This concern is shared by most development banks and one of the goals of any knowledge management strategy or action plan should be to capture tacit knowledge from departing staff.

3.8 Capturing knowledge on “frontier issues” is difficult 73. Capturing “frontier issues” is difficult as well and not many development banks have been capable of developing truly forward-looking knowledge products and services. The emergence of global challenges, e.g. climate change or increased rate of urbanization, has upset existing development models and paradigms, requiring a more comprehensive analysis and openness to a plurality of views, rather than reliance on established economic models. The 2012 ASDB evaluation70 pointed to the forward-looking Asia 2050 publication which drew on experience of many stakeholders in the Asia and Pacific region to use scenario planning as a tool for decision-making. However, good forward-looking knowledge products and services are the exception. Another gap is persisting as insufficient attention is paid to the voices of the poor themselves and associated knowledge practitioners. This was pointed out in the AsDB evaluation. The experience shows that there is only limited direct cooperation with civil society and beneficiaries to create knowledge products and services. Yet such cooperation is indispensable to understand the surrounding and evolving environment.

3.9 Measuring knowledge impacts remains a challenge 74. A major difficulty has been to demonstrate the ultimate impact of knowledge work. Despite the growth of knowledge management and some positive progress made in past years, there is surprisingly little firm evidence on its impact on development. Most of the indicators used in many performance systems are unable to clearly demonstrate that their knowledge management strategies have brought significant changes to their core business processes. Reviews of knowledge management work in many development banks have pointed to the lack of impact indicators. The case of the World Bank is well documented. In the first few years of knowledge management implementation, the strategy seemed to have been relevant, responding to the World Bank's mission and the needs of its clients, while taking advantage of its vast resources of development expertise and knowledge assets. Subsequent reviews, including the Matrix evaluation of 2012, have pointed to the lack of impact metrics, thus making it impossible to measure how the institution was capturing the

69 The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012, page 58. 70 ASDB. Special Evaluation Study, Knowledge Products and Services: Building a Stronger Knowledge Institution, November 2012.


33

effects of knowledge of knowledge products and services on core business processes.71 75. Other evaluations have spotted similar problems. The 2009 evaluation of IFC development results indicated that indicators for advisory services were very limited, which “reflected the relative immaturity of the project M&E system, but also the absence of established M&E indicators for IFC’s impact at a programmatic level.” 72 The targets that were included for advisory services indeed pertained to the number of public-private partnership advisory mandates and the level of overall AS expenditures, neither of which captured IFC’s development impact. The case of AsDB stands out as slightly different. Since 2008, knowledge management has been measured with perception indicators; knowledge management significantly improved its rating in AsDB’s scorecard, achieving an overall rating of “good” for managing knowledge better in 2011 in AsDB DEfR. Still, AsDB evaluation concluded that more specific knowledge indicators would need to be created and incorporated into the Results Framework, e.g. indicators on country partnership strategies referring to knowledge products or indicators on the use of knowledge by different audiences. The example of IDB's Development Effectiveness Overview reports is quite revealing as well. Despite compiling significant data on knowledge activities, the report still provides little metrics and insight on the actual application of this knowledge. 76. The AsDB, World Bank and IDB experiences suggest that a relatively good amount of quantitative metrics on knowledge products is probably available but not fully used yet. Such metrics could include publication output; publication downloads and purchases; web traffic indicators; citations (especially for journal articles); media mentions/references; internal-external perception survey results, etc. The challenge is to develop and track more indicators to better assess progress at each stage of the knowledge cycle, e.g. Knowledge activities-outputs-outcomes-use/impacts. Furthermore, those cases also reveal the limits of such metrics, in the absence of complementary qualitative analysis of the impact of knowledge activities on core business processes. 77. In summary, too often, a knowledge result is thought of in terms of outputs (e.g., number of people trained), deliverables (e.g., policy recommendations delivered), or milestones (e.g., learning module completed). These serve an important monitoring function, along with results information, for assessing whether a project is being implemented as planned and adjustments are needed to achieve results. However, these indicators are not enough and the main challenge ahead is to account for the ultimate impact of its knowledge work.

71 The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012. 72 The World Bank. Knowledge for Private Sector Development. Independent Evaluation Group (IEG), 2009, page 46.


34

4. Conclusions and areas for improvement 78. This synthesis study revealed important implications for effectively developing knowledge generation programs that are applicable across organizations. Its findings indicate support for knowledge management approaches that emphasize the roles of a just-in-time, operational and quality knowledge. Above everything else, knowledge management needs top level support, and well-defined organizational and work process changes. Emerging economic and social issues play an increasingly important part in the direction of focus for most development banks, in particular when food, demographics and the environment are at stake. Such emerging challenges will also require a set of knowledge, a redefinition of standards, new systems to provide relevant knowledge products and advisory services and an overall strengthening of development banks to disseminate to the various audiences. 79. Taking this synthesis paper as whole, and the evaluation reports upon which it is based, six sets of implications emerge for MDBs: a. Customize according to specific client needs: To develop an efficient set of knowledge

products and advisory services, development banks must be able to identify the needs of clients and describe their demands. In this process, it is crucial to match demand and supply of knowledge. Knowledge products and advisory services requested by the client and designed specifically to achieve client objectives are more likely to achieve the desired outcomes than knowledge services of a more generic character.

b. Create more knowledge that is operationally-relevant: There is a fundamental question about what role knowledge will play with respect to operations. Many evaluations noted that knowledge products and advisory services used in the design of lending operations were more likely to succeed than freestanding knowledge not responding to a clear demand.

c. Improve the overall quality and coordination of knowledge products and services: Weaknesses in relevance of design, quality, timeliness of delivery are major constraint. To do so, it may be necessary to improve the quality control systems as well as the incentive structures to better reward staff doing and sharing of knowledge and collaboration across sectors.

d. Assign clear indicators to monitor knowledge management effectiveness: It is important to

better monitor and measure knowledge to track progress, document practical lessons in implementation, and make timely adjustments to the knowledge agendas when necessitated.

e. Use the new technologies to open data according to the needs of the various audiences:

There is scope to view knowledge management in a different way. To do so, it will be important to improve enabling technologies, particularly for knowledge storage retrieval and sharing.

f. Better capture frontier issues that needs special attention. Whichever way the need for such

knowledge is satisfied, knowledge on frontier issues where the major threats and opportunities are likely to arise is becoming increasingly urgent. As a result, there is a need to develop more forward-looking knowledge products and services.


35

Appendixes


36

Appendix 1. Glossary of key terms (From ADB evaluation of knowledge Products and Services, 2012. ADB and François Fortier) Communities of Practice (CoPs): A group of people linked across units, organizations or sectors, having common interests in a given type of activities, and about which they exchange knowledge through discussions and learning, and may collaborate to apply and further develop such knowledge. CoPs can be linked physically, for example through meetings, casual encounters at lunch or formal workshops, or virtually through mailing lists discussions and workspaces. Knowledge: “Knowledge is the understanding of relations and causalities, and is therefore essential in making operations effective, building business process, or predicting the outcomes of business models.”73 Similarly, knowledge can be seen as: “Understanding the why, what, how, who, when, and where relative to taking some action. Knowledge is the product of organization and reasoning applied to raw data.” 74 These views complement a now classic hierarchical projection of knowledge management: data – information – knowledge – wisdom, whereby data is compiled and sorted into information, whose relations are analyzed into knowledge, which builds wisdom through experiential context. Sub-forms of knowledge include:

Explicit Knowledge: “That component that can be codified and transmitted in systematic and formal languages, for example, documents, databases, Web sites and emails.”75

Implicit Knowledge: “Implicit knowledge helps individuals know what is socially and culturally appropriate in a given circumstance; it is knowledge of shared beliefs, values and expectations (e.g. knowing that it is inappropriate to undermine colleagues in public, understanding management attitudes within a given organisation).”76

Tacit Knowledge: “Personal, context specific knowledge that is difficult to formalize, record, or articulate: it is stored in the heads of people. It is mainly developed through a process of interaction, debate, and trial and error encountered in practice.”77

Knowledge Application (or Use): When knowledge is actually used to develop or change practices and behavior. Changes can occur through policies or programs guidelines revisions, and with business processes transformation (or reengineering). In other words: “What is done with knowledge gained from an information product or service. It is the way in which information products or services are absorbed and applied to institute or implement changes.”78 Knowledge Bank: A financial organization, such as the ADB or the World Bank, that systematically applies knowledge management processes to develop, share and apply knowledge-based solutions to improve the effectiveness and efficiency of loans and finance-related technical assistance for development.79 Knowledge Development: Includes activities creating new knowledge through research and reflections, but also of capturing (documenting of tacit knowledge), codifying (adding meta-data), or transforming knowledge to new formats and for new media (including translation, formatting and re-interpretation).

73 Talisayon, Serafin D. (2009, “Monitoring and Evaluation in Knowledge Management for Development”, IKM Working Paper No. 3, IKM Emergent Research Programme, Dutch Ministry of Foreign Affairs. p. 9, citing McKinsey & Company, 2001. 74 ADB (2004), Knowledge Management in ADB, p. 4. 75 Idem. 76 Ramalingam, Ben (2005), “Implementing Knowledge Strategies: Lessons from international development agencies”. Working Paper. London, Overseas Development Institute. p. 4. 77 ADB, op. cit., p. 4. 78 Sullivan, Tara M., et al. (2007), Guide to Monitoring and Evaluation of Health Information Products and Services. Baltimore, Washington D.C., Cambridge MA, Center for Communication Programs, Johns Hopkins Bloomberg, School of Public Health; Constella Futures; Management Sciences for Health, USAID. Citing NCDDR, 1996 and Malchup, 1993. p. 3. 79 Building on ADB President statement, 2009.


37

Knowledge Management: A combination of management processes, institutional arrangements, and technologies with which knowledge is developed, shared and applied for improving the efficiency and effectiveness of activities.80 For Snowden, “Knowledge management is the identification, optimization, and active management of intellectual assets, either in the form of explicit knowledge held in artefacts or as tacit knowledge possessed by individuals, teams, organizations or communities. Knowledge management is about managing professionals, and creating the right culture, structure, processes and systems that allow knowledge workers to professionalize, be effective and innovate.”81 Knowledge Products and Services (KPS): KPS are tangible outputs (products) and activities (services) of development, sharing, or application of information and knowledge contents. Strictly speaking, and in accordance to the definition of knowledge provided here, KPS should be those where the analytical value addition is significant, increasing the understanding of relations and causality between elements in a body of evidence. In turn, information products and services (IPS) are those that offer an organized but not analyzed compilation and classification of data through artifacts or activities. Information products include bibliographies and staff directories, expertise locators, catalogues, archives, databases, standard operational manuals, mission statements and legal documents. In turn, knowledge products include annotated bibliographies, relational or mapped directories, analytical narratives such as arguments shared as speeches, evaluation reports or working papers, disseminated as podcasts, or published as journal articles, wikis or monographs. Knowledge services include analytical research taking various forms of advisory services, facilitation and capacity development, including problem-solving engagements, the moderation of CoPs, and learning programs. Knowledge Sharing: An act of exchanging knowledge, between two or more parties. If the flow of knowledge is unidirectional, the “sharing” is limited to one side disseminating to another, or broadcasting to others. When flows are multidirectional, exchange occurs, and many sides can benefit by learning from others. Sharing can occur face-to-face, be mediated by diverse communication practices and technologies, synchronous or asynchronous, such as conversation, in-class teaching, distance learning, publishing and broadcasting, peer mentoring, mailing list discussions, micro-blogging, etc. Knowledge Reach: “The breadth and saturation of product dissemination. It describes the extent to which information is distributed, redistributed, and referred to by organizations and individual users.”82 Knowledge Solution: In generic terms, a knowledge solution refers to improving or resolving a problem through the identification and use of a solution based on knowledge development and sharing among concerned stakeholders. In the ADB context, “Knowledge Solutions” also refer to concise guidelines on knowledge processes – such as nurturing internal knowledge markets or conducting a premortem analysis. The Knowledge Solution series “aims to build competencies in the areas of strategy development, management techniques, collaboration mechanisms, knowledge sharing and learning, and knowledge capture and storage.”83

80 See notably Ferguson, Julie E., et al. (2008), “Management of Knowledge for Development: Meta-review and scoping study.” IKM Working Paper, IKM Emergent Research Prog., Dutch Ministry of Foreign Affairs. p. 8. 81 Cited in Hulsebosch, Joitske, et al. (2009), “Monitoring and Evaluating Knowledge Management Strategies.” IKM Background Paper, IKM Emergent Research Programme, Dutch Ministry of Foreign Affairs. p. 11. 82 Sullivan et al., op. cit., p. 3. 83 ADB, http://www.adb.org/site/knowledge-management/knowledge-solutions.


38

Appendix 2. Conceptual framework of knowledge management


39

Appendix 3. Key findings from the 14 selected knowledge-related evaluations

The African Development Bank (2013). Review of the African Development Bank’s Economic and Sector Work (2005−2010). Operations Evaluation Department (OPEV) Objective and scope

This report is a review of the performance of the AfDB’s Economic and Sector Work (ESW) over the period 2005-2010 and constitutes the first phase of a comprehensive ESW evaluation. The report focuses mainly on the processes and procedures for the preparation of ESW, including on issues related to knowledge management. It also assesses the performance of ESW activities of the Bank based on a portfolio review, literature review and five country case studies. It was noted that AfDB’s Management is currently preparing a new policy on Knowledge Management (including the creation of a Bank-wide Knowledge Management Committee), and it is expected that this review will also inform this policy.

Methodology This report was produced by AfDB’s Operations Evaluation Department (OPEV) in 2013. The purpose of the review was to assess AfDB’s ESW over the period 2005-2010 and to draw lessons and formulate recommendations to improve the effectiveness of future ESW. The study should be seen as a formative evaluation as AfDB is putting in place an internal process for ESW. To investigate ESW activity along the entire process, this review addressed four questions: (1) How consistent are Bank policies and strategies for ESW?; (2) What are the features of ESW produced by the Bank?; (3) How efficient are the Bank’s processes and management of ESW?; and (4) To what extent is the Bank’s ESW useable and useful?

Main findings and recommendations

The report noted that knowledge management as a new field of study has drawn attention to the significant and important role of ESW. The role of knowledge as a resource for development is of crucial importance. In response to the growing demand for cutting-edge knowledge, development organizations publish studies and reports, both to enhance quality of lending and as a business line for policy and program advice to clients. A subset of these knowledge products is known as ESW. As noted, the volume of ESW prepared has increased substantially since 2008, with increased focus on the Bank’s strategic areas, reflecting AfDB’s desire to become a “knowledge bank”. The first AfDB strategy on knowledge management aimed at strengthening the Bank’s role as a knowledge bank was approved by the Board in March 2005. It covered the period to the end of 2007. Since then the Bank’s knowledge activities have been guided by the Knowledge Management and Development Strategy 2008–2012, approved by the Board in 2008. After a major independent evaluation of the ADF VII, VIII, and IX in 2004 found that knowledge and ESW at the Bank were inadequate, AfDB responded by issuing knowledge strategies in 2005 and 2008. The current strategy states that the Bank’s knowledge vision is to be the premier knowledge Bank for Africa. Its two key objectives are to establish and establish a knowledge culture within the Bank and to enhance the operational effectiveness of Bank initiatives. The reported noted that trust fund expenditures on knowledge and ESW have significantly increased in nominal terms since 2009. The following issues were noted in the evaluation report: AfDB’s knowledge policies provide broad guidance on knowledge work but do not

explicitly refer to ESW. AfDB’s 2008–2012 knowledge strategy provided limited guidance on ESW. It has

three main limitations: o The strategy focuses mostly on the complex of the Chief Economist

providing scant policy guidance for knowledge across the AfDB. o The strategy makes little distinction between knowledge products, so ESW

is not singled out.


40

o The strategy has a more inward-looking approach, with little consideration on the strategic positioning of the Bank in relation to regional member countries and other donors in the area of knowledge. For example, whether the Bank should invest more in “what to do” (ESW) or “how to do it” (technical assistance) is not addressed.

There is no mechanism in place for managing ESW as a portfolio. In addition, harmonized procedures for quality control of ESW do not exist. As a result, the evaluation report recommended to create a business process and increase corporate oversight to support quality assurance, dissemination and management of ESW as a coherent portfolio.

Dissemination of reports is ad hoc with no formal guidelines. Dissemination varies across Bank units, from posting studies on the web (internally or externally) to sending them to national or international organizations or a network of people for presentation in seminars.

The ESW portfolio has little visibility in regional member countries, and the Bank’s field offices in turn do not have a clear role in disseminating ESW.

The report concluded that AfDB should be more realistic with respect to actual delivery of ESW activities while increasing the corporate oversight to support and manage ESW portfolio, generate and disseminate the knowledge acquired. Thus, AfDB needs to contextualize ESW and its knowledge products and services, and provide implementation mechanisms so that it can clearly positions ESW within the AfDB’s knowledge ecosystem.

On the ESW positioning, the evluation report recommended that much confusion exists among ESW, research, and internal reports (such as portfolio analysis), with unclear audiences for AfDB ESW. As a result, AfDB should revisit its knowledge strategy to clarify the function, purpose, and audience of its analytical and knowledge products. Adapting from the “general framework for knowledge in organizations”, one option is to organize AfDB knowledge products according to a specific and well agreed typology.


41

Asian Development Bank (AsDB, 2012). Knowledge Products and Services: Building a Stronger Knowledge Institution. Special Evaluation Study, Independent Evaluation Department Objective and scope

The main objective of this ASDB evaluation study was to identify lessons that would help AsDB to become a stronger knowledge institution. To do this, the study adopted a two-fold approach: looking backward to assess past accomplishments against expectations, as well as looking forward to determine what features are essential to make ASDB more effective as a knowledge institution and how these features could be adapted to the changing context of the Asia and Pacific region. The AsDB evaluation study also presented an independent evaluation of the implementation of AsDB’s knowledge management The SES drew on the 2011 evaluation of the Asian Development Bank Institute (AsDBI), which was the first evaluation of knowledge management in AsDB.

Methodology The SES used a number of study components, including (i) a literature review of knowledge evaluations, (ii) a comparator assessment of other organizations with comparable knowledge activities (five development partners and one private sector company), (iii) case studies of influential KPS, (iv) institutional assessment, (v) questionnaire surveys of ASDB and DMC stakeholders, (vi) information questionnaire to the chairpersons of the internal communities of practice (CoPs), (vii) key informant interviews, and (viii) external review of 85 knowledge publications by a panel of experts.


This special evaluation study (SES) noted that ASDB’s knowledge role represented an unfinished and underdeveloped agenda. Recognizing the increased client demand for knowledge, ASDB’s knowledge products and services output has grown substantially in the past decade. But there was widespread concern about the multiplicity of approaches, and serious gaps and constraints remained. This has created a sense of confusion and lack of clarity on the direction of knowledge management at the corporate level. In addition, while ASDB was becoming more knowledge oriented, there remained a poor understanding of knowledge management, stemming largely from a lack of clarity and common agreement on key concepts and terms and the respective knowledge management roles of different organizational units. Weak guidance, coordination across departments, and knowledge channels (between headquarters and field operations) were other key constraints. The absence of an underpinning knowledge management conceptual framework was noted as another issue. Recommendations included: Improve the incentive structures to better reward staff doing knowledge work. Staff

must be encouraged to make learning and knowledge identification, generation, sharing, and use a natural feature of daily work.

Improve enabling technologies, particularly for knowledge storage, retrieval, and sharing. Although ASDB has invested substantially in information technology infrastructure, there are a number of information technology gaps that impede the smooth generation, sharing, and use of knowledge.

Strengthen knowledge needs identification by expanding successful approaches undertaken by regional departments and preparing country-specific knowledge plans.

Strengthen knowledge sharing by better capturing and sharing tacit knowledge across DMCs, through South-South cooperation, and through increased use of social interaction processes, in particular through CoPs, trainings, and social media.

Strengthen knowledge use through dissemination of KPS, providing easy-to-access on-time information and using specific feedback mechanisms to gauge client satisfaction.

Prepare a knowledge management strategic directions document building on ongoing work in this respect, and prioritize key areas of focus. Neither a knowledge management policy nor a knowledge management strategy has been prepared in past years. The evaluation stressed that the main rationale for the development of a knowledge management strategic directional document is a need to prioritize areas wherein ASDB can add most value through its knowledge work.

Asian Development Bank (ASDB, 2011). Performance of the Asian Development Bank Institute: Research,


42

Capacity Building and Training, and Outreach and Knowledge Management. Special Evaluation Study, Independent Evaluation Department. Objective and scope

This special evaluation study examined the performance of Asian Development Bank Institute (AsDB) thus far and drew strategic lessons for the future. The evaluation covered three key knowledge products and activities of ASDBI: (i) research products; (ii) capacity building and training (CBT) activities; and (iii) outreach and knowledge management program.

Methodology No monitoring or results frameworks were available to assess the performance of ASDBI. Due to the absence of these frameworks, ASDB deployed its own conceptual framework for evaluating AsDBI‘s organization and processes in research and CBT, and its outreach and knowledge management functions in support of research and CBT activities. Conceptual framework thus supported evaluation criteria and questions in assessing the performance of AsDBI. The evaluation team conducted e-surveys, field interviews with AsDBI partners and CBT course participants, and a full review of relevant ASDBI documents.


Taking into account ASDBI’s performance in the context of research, capacity building and training and outreach and knowledge management program, based on relevance, effectiveness and usefulness, the overall performance of ASDBI was rated successful. The evaluation of the research function showed that overall, ASDBI’s research function is relevant to the topics it researches, effective in providing quality information to the developing member counties (DMCs), and moderately useful in terms of utilization. An evaluation of the research products by an eminent external review panel found that they are generally useful, informative, and appropriate to the mandate or mission of ASDBI. ASDBI is not an academic research institution but is tasked to provide knowledge support to ASDB‘s developing member countries (DMCs). In terms of the performance of the ASDBI’s capacity building and training activities, the program has been successful. It was noted that ASDBI performance has been evolving since its establishment in December 1997. In early years, there was no clear distinction of research focus between ASDBI and ASDB headquarters. When ASDBI was established, ASDB had limited research and training functions in the then Economics and Development Research Center. Given the more prominent and larger coverage of research and training being carried out by ASDB‘s knowledge departments (the Economics and Research Department, the Office of Regional Economic Integration, and the Regional and Sustainable Development Department) in the recent past, the rationale for ASDBI to conduct research and CBT became somewhat different than at the time of its establishment. Discerning a clear distinction between ASDBI and ASDB knowledge departments is a difficult issue. In 2006, ASDBI and ASDB knowledge departments introduced a formal process to coordinate their activities, including engaging in joint undertakings. Against this context, ASDBI has decided to conduct research and CBT activities that contribute to ASDB‘s overarching goal of poverty reduction by applying cutting edge research knowledge to the Asia-specific context and focusing on the region‘s medium - and long-term development issues of strategic importance. ASDBI needs to sharpen its focus by identifying its comparative advantage in line with its strategic vision to become a leading center for the creation and sharing of knowledge on economic development in the Asia and Pacific region. The evaluation study indicated that a clear, long-term, strategic approach of selecting training themes would increase the likelihood that capacity building and training courses will contribute in a sustainable manner to the capacity of DMC institutions. A more structured consultative process with top-level DMC officials could also produce a capacity building and training plan that was more responsive and tailored to specific DMC development needs. Recommendations included: Facilitate institutional changes with the involvement of ASDB headquarters'

knowledge and regional departments to improve the effectiveness of ASDBI's


43

knowledge products, further strengthen its links with ASDB, and increase its impact on the development directions of the DMCs.

Strengthen the quality of research products, including theoretical and conceptual underpinnings and rigorous evidence of research, to an extent that can guide DMC policy makers to formulate appropriate policies.

Align the capacity building and training program with ASDBI's strategic vision of knowledge dissemination to senior-level and top-end of middle-level policy makers, with particular focus on strategic regional issues, and consider a more programmatic approach of long-term strategic partnerships with a small number of selected DMC-based institutions.

Take strategic steps to increase ASDBI's visibility in the Asia and Pacific region, considering multidimensional improvements to reach a wider audience.


44

Evaluation Cooperation Group (December 2012). Evaluating Technical Assistance: Taking Stock of the Practices of International Financial Institutions, Kris Hallberg. Objective and scope

ECG members proceeded with a detailed “stocktaking” exercise of technical assistance (TA). The objective of this report is to take stock of the types of technical assistance offered by ECG members to clients, and members’ practices with regard to evaluation of those technical assistance activities. From these comparisons, the report identifies methodological issues that need to be considered in designing Good Practice Standards (GPS) for Evaluation of Technical Assistance. The definition of TA includes TA to both public sector and private sector clients, TA provided on a non�reimbursable (grant) basis, and TA that is partly or fully covered by fees from clients.

Methodology The methodology was based on: a review of relevant documents provided by ECG members, including Board papers,

resolutions, and discussions; policy papers, staff instructions, guidelines and manuals; interdepartmental memoranda; published evaluation standards; a selection of TA evaluations (internal and external); and other written information.

interviews with staff from IFI CEDs and operational departments responsible for self evaluation. Some of these interviews were conducted by telephone (AsDB, EIB, EBRD, and IsDB); others were done in person during a visit to Washington, D.C. (WBG, IDB, and IMF).

information gathered from other development and research institutions via Web sites and e�mail and telephone communication with evaluation staff, including the Canadian International Development Agency (CIDA), the U.K. Department for International Development (DFID), the International Development Research Center (IDRC), the Netherlands Development Organization (SNV), the Australian Government Overseas Aid Program (AusAid), the OECD�DAC Network on Development Evaluation, and the International Initiative for Impact Evaluation (3IE).


To be summarized in next draft


45

Evaluation Cooperation Group (December 2012). Evaluating Technical Assistance: Taking Stock of the Practices of International Financial Institutions, Kris Hallberg. Objective and scope

ECG members proceeded with a detailed “stocktaking” exercise of technical assistance (TA). The objective of this report is to take stock of the types of technical assistance offered by ECG members to clients, and members’ practices with regard to evaluation of those technical assistance activities. From these comparisons, the report identifies methodological issues that need to be considered in designing Good Practice Standards (GPS) for Evaluation of Technical Assistance. The definition of TA includes TA to both public sector and private sector clients, TA provided on a non�reimbursable (grant) basis, and TA that is partly or fully covered by fees from clients.

Methodology The methodology was based on: a review of relevant documents provided by ECG members, including Board papers,

resolutions, and discussions; policy papers, staff instructions, guidelines and manuals; interdepartmental memoranda; published evaluation standards; a selection of TA evaluations (internal and external); and other written information.

interviews with staff from IFI CEDs and operational departments responsible for self evaluation. Some of these interviews were conducted by telephone (AsDB, EIB, EBRD, and IsDB); others were done in person during a visit to Washington, D.C. (WBG, IDB, and IMF).

information gathered from other development and research institutions via Web sites and e�mail and telephone communication with evaluation staff, including the Canadian International Development Agency (CIDA), the U.K. Department for International Development (DFID), the International Development Research Center (IDRC), the Netherlands Development Organization (SNV), the Australian Government Overseas Aid Program (AusAid), the OECD�DAC Network on Development Evaluation, and the International Initiative for Impact Evaluation (3IE).


To be summarized in next draft


46

Inter-American Development Bank (IDB, 2006). Evaluation of the IDB's Studies (RE-323). Office of Evaluation and Oversight Objective and scope

This evaluation’s objective is to assess the Bank’s efforts in the production, storage, dissemination, and utilization of studies. The IDB recognized the importance of having standards of review, quality control, and evaluation for analytical work, against which to measure the actual output of the IDB in this area. This evaluation aimed at objectively reviewing the purpose of research studies at IDB and assessing whether the organization delivered the results originally expected.

Methodology The methodology is not disclosed.


This evaluation has found a significant number of problems with the way in which the IDB managed research work and the production of studies, and made recommendations for dealing with these process issues. In particular, it was noted that, as of 2006, IDB had established no formal standards to measure impact of research. There was no definition of what constitutes a “study”, there were no operations processing policies for analytical work, there were no quality standards or guidelines to use in assessing proposed or completed analytical work, and there was no system of internal self-evaluation of this important line of activity. The evaluation report pointed out to a weak programming. The IDB was said to be deficient about how it prioritizes potential studies. Many studies were not programmed. Supply was generally adhoc and little attempt was made to validate compliance with programming decisions to ensure that programmed activities actually result in the production of the expected studies. Quality was also reported as a major issue. The report indicated that many studies are of low quality. A substantial number of studies (20%) made no use of primary or secondary data. Over half did not use a replicable, evidence based methodology, and 3/4 did not provide policy implications that were logically consistent given the rest of the study. Quality control process was also qualified as sporadic: IDB’s internal documents and stakeholder interviews suggested that the studies produced by IDB were not systematically quality controlled. Systems were seen as weak and IDB did not systematically track the studies it has in the pipeline, or their costs. Finally, the production incentives were seen as weak and ad hoc: The evaluation has found several shortcomings in the IDB’s system for producing studies. This was uggestive that incentives are not set strategically. And the opinions of stakeholders across the Bank also suggested incentives were not strategically set: Managers in the operational departments had incentives linked to the number and amount of loans approved, not to studies produced. Several recommendations were made: Improve Programming Improve budgeting and monitoring information systems Formalize the quality control process for all studies Ensure the Bank’s stock of studies is stored Integrate and coordinate the storage systems Provide incentives to project team leaders, authors and filers regarding the storage

and dissemination of studies


47

International Fund for Agricultural Development (IFAD), IFAD’s institutional efficiency and efficiency of IFAD-funded operations Corporate-level evaluation, Independent Office of Evaluation (IOE), IFAD, July 2013.84 Objective and scope

This report presents the findings and recommendations of the corporate-level evaluation on IFAD’s efficiency. The evaluation has a number of important findings and recommendations that merit close attention, including on knowledge management (KM).

Methodology Following discussions with Management and the Evaluation Committee (EC), the Board requested IFAD’s IOE to conduct a corporate-level evaluation on IFAD’s efficiency. The draft approach paper of the evaluation was discussed with the EC at its sixty-sixth session in March 2011. In September 2011, a team of consultants was contracted. The broad objectives, scope and coverage of this evaluation were agreed with IFAD Management and the EC at the outset of the process. According to IFAD, this is the first evaluation of its kind carried out in multilateral and bilateral development organizations. It may well be among the most complex and far-reaching type of evaluation ever conducted by IOE. As such, it posed complex methodological challenges.


For much of its history, IFAD has been a project-driven institution with little attention given to policy dialogue, knowledge sharing and partnerships to leverage the lessons emerging from its projects and grants. Still, realizing the full potential of knowledge management, IFAD is now seeking to strengthen its organizational knowledge structure. Several issues were raised: Recent good practices were noted. For example, IFAD established a knowledge

management core team under the leadership of the then Vice President in 2008; a knowledge management communities of practice was also set up the same year. IFAD organized a knowledge management launch initiative in 2008 to raise awareness about the importance of knowledge management, to create space for dialogue and learning from each other, and to stimulate horizontal collaboration across divisions and departments. A knowledge fair on community driven development to share experiences from Africa was organized in 2009. These are some of the initiatives that illustrate the efforts made by IFAD to systematize knowledge management activities within the organization.

Despite the increased attention to KM since the KM strategy was adopted in 2007, the implementation has fallen short of expectations. Among the more notable accomplishments are learning fairs, meetings of project managers at the country and sometimes regional level to share experiences and portfolio reviews that are held annually in various in regions for discussing implementation results and lessons with all staff. Other initiatives are also being taken to improve KM. These include improvements in documentation and providing electronic access to some key documents to staff. For example, 150 project completion reports (PCRs) and their summaries that were reviewed and rated over the last six years were posted on IFAD’s Intranet.

On the other hand, KM in country programs has not yet been sufficiently focused on scaling up IFAD’s impact on the ground, and staff interviews and surveys are clear that staff believe that more can be done. More generally, the knowledge products that were to have been key outputs under the strategy (i.e. learning notes and technical advisory notes) have been produced only sporadically since 2008. Thematic groups and communities of practice have progressed little beyond their status at the time of the strategy. More attention was given in the strategy to what was to be produced rather than to how staff work and cooperate with each other.

In terms of quality, IFAD also revised and introduced a comprehensive quality-at-

84 For a discussion of IFAD knowledge management, see also the two reports titled (1) International Fund for Agricultural Development (IFAD), IFAD's Private-Sector Development and Partnership Strategy, Corporate-level evaluation, Independent Office of Evaluation of IFAD, June 2011. And (2) International Fund for Agricultural Development (IFAD), IFAD’s Capacity to Promote Innovation and Scaling Up, Corporate-level evaluation, Independent Office of Evaluation of IFAD, June 2010.


48

entry system in 2007/2008, aimed at improving the quality and process of project design. This function was assigned in 2012 to the Strategy and Knowledge Management Department. However, the practice of a heavy quality enhancement review is costly and cannot substitute for building quality into original designs by injecting quality field input by IFAD technical staff, both at the design stage and during implementation.

IFAD activities in MICs should be expected to emphasize knowledge-sharing content. It was noted that IFAD’s country managers still tend to work in silos, and they have few opportunities to share their knowledge.

IFAD’s country programs show insufficient customization to the respective country contexts and therefore increased use of knowledge services to complement financial services, especially in MICs and poorly performing, should be expected.

It was also noted that overreliance on consultants can also limit institutional learning since knowledge gained on consulting assignments is retained by the consultant and may be lost to the institution.

In summary, the report concluded that while IFAD has made some progress in systematic learning and knowledge management, this remains an area where improvements are needed. There is room for further improvement, as relatively few resources have been allocated to KM, little formal opportunity exists to share knowledge among country program managers, and efforts to learn from failures can be further expanded. The reported concluded by arguing that in IFAD there is insufficient mining of the rich knowledge embedded within the operations it funds.


49

International Monetary Fund (IMF, 2011). Research at the IMF: Relevance and Utilization. Independent Evaluation Office (IEO) Objective and scope

This study evaluated the relevance and utilization of IMF research to member country authorities, to IMF staff and to other stakeholders between 1999 and 2008. The IMF produced a large body of research, ranging from background studies for bilateral surveillance activities to working papers and external publications dealing with topics of more general interest. Research was defined broadly to capture most analytical publications of the IMF, ranging from surveillance-oriented output, for example, selected issues papers (SIPs) prepared for Article IV consultations and the analytical chapters of the World Economic Outlook (WEO) and Global Financial Stability Report (GFSR), to more academically-oriented output, for example, working papers (WPs) and publications in external journals. These outputs comprised a large body of research, about 650 publications annually, at a cost of about 10 percent of the IMF budget.

Methodology The evaluation used a variety of methods of analysis and several sources of evidence, which allowed for triangulation of findings and contributed to the robustness of conclusions. The main sources of information for this report were: (i) Document review—mainly reflected in six background papers presenting the assessment of peer review panels on specific product lines, for example, WEO/GFSR and SIPs; (ii) Interviews—more than 350 semi-structured and open interviews of authorities, staff, and other stakeholders; (iii) Surveys—two surveys were conducted—one of authorities and one of staff; and (iii) Technical and statistical work, including a citation analysis.


This evaluation found that the vast body of research produced by the IMF included high-quality products, many of which are widely read in member countries and play a significant role in policy making; this was particularly true for the World Economic Outlook and Global Financial Stability Report, but also for other publications. At the same time, the evaluation identified a number of shortcomings in IMF research. Of particular importance, there is a widespread perception that Fund research is message-driven. The evaluation also found that the relevance of research suffered from lack of consultation with authorities on research topics and inadequate country and institutional context; it also found insufficient quality control. For instance, authorities indicated that some important issues, such as macro-financial linkages and aspects of monetary policy, were not adequately covered. Moreover, the technical quality of IMF research publications was said to be quite diverse. To enhance quality, adequate time and resources should be allocated to each research project, even if this leads to fewer publications. The review of research products should also be strengthened to improve quality and to prevent the publication of low-quality products. Finally, there was a need for greater prioritization and coordination of research across the IMF. In terms of recommendations, the IMF evaluation study pointed out to four areas to (a) enhance the relevance of research, (b) enhance the technical quality of analytical work, (c) promote openness to alternative perspectives and (d) improve the management of IMF research. In particular, the evaluation study emphasized the need for IMF Management and its Executive Board to cultivate an open, independent, and innovative research environment, explicitly encouraging staff to explore differing and alternative views. It was recommended greater consultation and cooperation with country authorities, and an enhanced quality review process—reforms that would bring greater diversity of research methods and perspectives.


50

The World Bank (2013). Knowledge-Based Country Programs, An Evaluation of the World Bank Group Experience. Independent Evaluation Group (IEG) Objective and scope

The main objective of the evaluation is to learn lessons from practices in a focus group of high-income and upper-middle-income countries that have knowledge based programs with the Bank Group. Over the past 15 years, Bank Group country programs have shifted toward more intensive delivery of knowledge services relative to lending, and this trend is expected to continue. The lessons from this evaluation could help leverage the Bank Group’s global knowledge to meet the needs of countries that mainly rely on knowledge services and are not pressed for financing. The nine selected countries are high-income (Kuwait) and upper-middle income countries with a high share of knowledge services in their programs, a diversified economic structure, no or moderate Bank lending, and fee-based knowledge services.

Methodology The evaluation categorized Bank Group country programs according to the preponderance of knowledge services in program interventions. At one end of the spectrum were lending-based programs with a predominant role for finance and a relatively lower presence of knowledge services. At the other end were knowledge-based programs where knowledge products are at the core of the relationship. The categorization was then used to purposively select focus countries that make relatively intensive use of the Bank’s core knowledge services. The selected countries were Bulgaria, Chile, China, Kazakhstan, Kuwait, Malaysia, Russian Federation, South Africa, and Thailand. To probe the synergy of the Bank’s knowledge services with IFC Advisory Services, the evaluation also examined those services in the focus countries. The selected knowledge activities in the nine focus countries were assessed against four criteria: relevance of the knowledge activities to the priority needs of the recipients and the key development goals of the client country; technical quality of the activities in leveraging the Bank’s global knowledge and conveying relevant and customized expertise to recipients; results achieved; and sustainability of results.


The main findings were summarized according to areas of strength and areas of weakness or risk. They are as follows: Areas of strength The Bank remained relevant and a strategic partner in the focus countries by

providing knowledge services that addressed one or more client needs. Customized development solutions filled a knowledge gap in an area where counterparts needed timely and actionable recommendations to develop a strategy or take action.

The Bank’s main strength, which reflected recommendations from previous IEG knowledge services evaluations, was its ability to fulfill in a timely manner client requests for state-of-the-art advice.

Another key strength was linked to its role as “knowledge connector.” The Bank’s convening power was often used to mobilize top international experts for brainstorming sessions and seminars with high-level government officials, or for TA and working sessions with government agencies.

Bank knowledge services and IFC Advisory Services generally complemented one another in contributing to results, despite some gaps. In a few cases there were well-defined programs of joint World Bank and IFC knowledge activities. For example in South Africa, to ensure synergy and coordination, IFC used experienced Bank staff to manage projects on enterprise tax burden and compliance.

Areas of weakness or risk Poor achievement of outcomes was associated with weaknesses in relevance of

design, quality, timeliness of delivery, or client participation, and little use of local expertise. Knowledge services that lagged in the achievement of outcomes were also weak in conveying international best practice, providing relevant examples, producing new evidence and data useful for policy making, formulating actionable recommendations, and discussing the capacity requirements and administrative feasibility of implementing recommendations.


51

The Bank’s ability to customize knowledge services to the local context and to deliver multisectoral solutions is at risk of eroding where country knowledge is too shallow or too narrow. This risk arises mainly when the Bank works through Reimbursable Advisory Services (RAS) and does not maintain a local presence.

Monitoring of Bank knowledge services results was weak—both for individual activities and for country programs. In only 17 percent of the knowledge activities assessed was there at least a partial results framework in the CPS, allowing a tracking of the contribution of the activity to the broader development outcomes sought by the CPS.


52

The World Bank (2012). The Matrix System at Work: An Evaluation of the World Bank's Organizational Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG) Objective and scope

The objective of the evaluation was to assess the extent to which the dual objectives of the matrix system—enhancing client responsiveness and establishing strong technical networks to deliver quality services—have been attained, and have enhanced the World Bank’s development effectiveness. This evaluation is not about knowledge only but covers a wide spectrum of issues. Therefore, we will focus our review on Chapter 3 only (“The Promise of a Knowledge Bank”).

Methodology In addition to a literature review and commissioned background papers, evidence for this evaluation came from a wide range of available documents and data sources, including relevant World Bank management assessments of the matrix, portfolio data, human resources data, and budget data. The evaluation drew on findings of previous IEG assessments, including Country Assistance Evaluations (CAEs), Country Assistance Strategy Completion Report (CASCR) reviews, and sector and corporate evaluations to identify matrix issues that had a bearing on the evaluation questions. The portfolio review covered the period 1991–2010.


In 1996, the concept of a Knowledge Bank was put forward both as the means to enhance the quality of World Bank lending and to provide a new stream of knowledge services to developing countries. The concept implied a shift toward capacity building and knowledge creation drawing on global and local knowledge to improve the quality of client services and achieve better results. The evidence from the flow of virtual knowledge and tacit expertise indicated that, rather than functioning as a global organization, the World Bank was at risk of evolving into a group of regional banks interconnected by fraying ties among the Regions and between the Regions and networks. Decentralization had created further impediments to the flows of knowledge and expertise, and until the underlying incentives and constraints inhibiting effective knowledge flows were addressed, further decentralization would likely aggravate these problems. The impact of analytical and advisory activities (AAA) is undermined by less effective dialogue and dissemination. Indeed, the report pointed out that the World Bank generated a large volume of knowledge embedded in its operations but lacked the ability to efficiently capture and share that knowledge. In addition, most staff, particularly those in the Regions and country offices, were unable to draw efficiently on knowledge generated inside and outside the World Bank. Knowledge products were not stored in an easily searchable and retrievable form and were rarely used by staff outside the units where they are produced. As a result, and notwithstanding the analytical quality of the Bank’s AAA, much of the Bank’s knowledge had limited shelf life and use value. Furthermore, innovation in the field was not well captured across the institution, and country operations did not draw efficiently on global knowledge to meet client needs. Operational staff in the Regions, especially those in country offices, thus complained about the ineffectiveness of the World Bank’s role as a connector of knowledge. Finally, in the absence of efficient search engines for knowledge retrieval and mechanisms for knowledge sharing, decentralization creates further challenges.


53

The World Bank (2011). The State of World Bank Knowledge Services: Knowledge for Development 2011Objective and scope

This Report is not an evaluation report but a World Bank Knowledge Report seeking to provide a comprehensive overview of the Bank’s knowledge work. It takes stock of the various types of World Bank’s knowledge activities and their evolution in response to the fast changing world of development and multiple sources of knowledge flows. Stemming from the 2010 Knowledge Strategy, the report aimed at strengthening the management and development impact of the Bank’s knowledge services. It identified actions to improve the Bank’s ability to respond to the changing needs of our clients and ensured a greater outcome orientation for our knowledge services.

Methodology This is not an evaluation report and the methodology is not disclosed.


This Report demonstrated the growing importance of knowledge services among the services provided by the World Bank to its clients. Over the last nine years the World Bank management has steadily allocated a larger share of its administrative budget for core knowledge work. In 2011 this came to 31 percent of the World Bank’s budget, compared with 24 percent in 2002. The report is relevant to our work as it recognized the globally proliferating sources of knowledge, the changing demands of the Bank’s clients, and the role of technology. The report showed how World Bank was drawing on its roles as a producer, customizer, and connector of knowledge to respond to client needs. The World Bank worked with clients to customize policies, programs, and products to meet specific challenges, based on the best knowledge available. Although clients cite knowledge services as World Bank’s most valuable contribution (more than twice as often as financial resources), knowledge work was said not to be seen internally, or by independent evaluators, as having the impact it could. The report indicated that managers and staff saw limited internal support for their knowledge work and some of them feel that such work is undervalued. Most staff felt, despite the growing importance of knowledge work, that the World Bank’s main internal incentives were still related to lending. One reason for this apparent contradiction may be the lack of robust and systematic evidence that knowledge work brought demonstrable and measurable returns. By contrast, lending has built-in metrics, e.g. volumes and disbursement rates. To address this paradox, the Report pointed to three directions for improving the way World Bank manages its knowledge services: Establishing consistent standards for governance and stronger results frameworks Strengthening connectivity across the core knowledge product lines and developing a

framework for a comprehensive approach to managing knowledge as a portfolio— setting strategic priorities and ensuring complementarities,

Moving toward a more open and collaborative approach to knowledge services.


54

The World Bank Institute and the Korea Development Institute. Using Knowledge Exchange for Capacity Development: What Works in Global Practice? Three case studies in assessment of knowledge exchange programs using a results-focused methodology, November 2011 Objective and scope

In this joint study, the Korea Development Institute (KDI) and World Bank Institute (WBI) assess three of their South-South knowledge exchange programs to analyze their effectiveness as well as gather lessons that can inform global practice. WBI’s Capacity Development and Results Framework (CDRF) provides the conceptual foundation and methodology for facilitating the assessment. The framework supplies logic for understanding how institutions change, as well as sets of institutional and intermediate indicators that can be flexibly applied to measure and analyze capacity development outcomes across sectors and countries. He main objective is to draw lessons drawn from the practical experience of others on South-South knowledge exchange and assess how such knowledge can improve stakeholder buy-in and subsequently create an enabling environment for designing and implementing difficult development initiatives and reforms. The assessment also seek to identify good practices in managing knowledge exchange for results, by demonstrating the value of a results focus, clear change logic, and indicators to monitor and measure progress.

Methodology The study examines three programs: (1) KDI’s Knowledge Sharing Program (KSP) for advancing export development in the Dominican Republic; (2) KDI’s KSP for promoting public-private partnerships and prefeasibility studies in Mongolia, along with providing support for deposit insurance reform; and (3) the World Bank’s South-South Experience Exchange Facility for the New Economy Skills for Africa Program (NESAP) in India for sharing knowledge on information and communications technology with African countries, including Ghana and Nigeria.


The most important findings can be summarized as follows: The study revealed that, to develop an efficient knowledge exchange, knowledge

seekers must be able to identify their needs and describe their demands. In this respect, non-governmental stakeholders, such as academia and the private sector, play an important role.

A key challenge in the knowledge exchange process is the matching of demand and supply of knowledge. A good practice is noted in the case of the Dominican Republic, the government expressed strong demand for the KSP on export development and articulated its demands based on a needs assessment conducted in the local context together with Korean consultants.

The study indicated that participant selection is critical to both sides of the knowledge exchange. On the supply side, knowledge exchange programs should involve the right staff and practitioners. On the demand side in the knowledge seeking country, programs must target and engage the appropriate change agents—the individuals or groups best positioned to initiate and manage needed changes.

Successful knowledge exchange activities are characterized by continued engagement, regular consultations, and evidence-based recommendations. Acquiring, adapting, and applying knowledge is a medium-term process. This is why recurrent interaction is necessary in both vertical (within-country) and horizontal (cross-country) relationships.

According to the study, maintaining peer-to-peer interaction is the most effective way to exchange tacit knowledge and sustain horizontal partnerships.

Developing indicators for monitoring knowledge exchange practices is needed to facilitate adaptive management and assessments of outcomes. In particular, the study suggests to improve the approach applied to intermediate outcomes. Monitoring should take place to capture improvements in the ability or disposition of stakeholders to take action in the form of raised awareness enhanced knowledge and skills, improved consensus and teamwork, stronger coalitions, enhanced networks, and increased implementation know-how.

The absence of high-quality, precise results measures and failure to effectively track them limit the possibility to make needed, timely adjustments to knowledge exchange programs.

Disseminating practical lessons learned is critical for future knowledge exchange


55

programming by the development community Finally, the case studies demonstrated the importance of having a clear change logic:

picturing the flow of how knowledge services and activities affect agents and their capacities (i.e., intermediate capacity outcomes), then how they affect change in institutional capacities (i.e., stakeholder ownership, policy instruments, and organizational arrangements), which finally affect development goals.


56

The World Bank. Knowledge for Private Sector Development. Independent Evaluation of IFC’s Development Results. Independent Evaluation Group (IEG), 2009 Objective and scope

In this report, IEG provided a first global review of IFC advisory and knowledge services to private firms and governments in support of private sector development, such as customized training or advice on investment promotion and policy. IFC's advisory portfolio has grown tenfold, to nearly USD1 billion, in the last seven years, and IFC now has more advisory than investment staff in the field. The substantial growth of this business line raises important questions related to the appropriate balance of advisory and investment services to ensure maximum development impact.

Methodology The evaluation methodology is not disclosed.


Knowledge products and services have been growing rapidly; Advisory Services (AS) teams now dominate IFC’s presence on the ground. This rapid growth had happened in largely uncontrolled manner and raised some important strategic questions. These included whether, in drafting a consulting business onto a bank, IFC had the right balance between AS and investment operations. Recommendations included: Effectively manage the tension between protecting the portfolio and responding to

opportunities during the crisis Set out a clear vision and business framework for Advisory Services that is more

closely linked with IFC’s global corporate strategy Pursue more programmatic Advisory Services interventions Over the long term, pricing of Advisory Services should reflect the value and impact

(that is, not just the cost) Strengthen Advisory Services performance measurement and internal knowledge

management.


57

The World Bank. Using Knowledge to Improve Development Effectiveness, An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006. Independent Evaluation Group (IEG), 2008 Objective and scope

This evaluation focused on two of the analytical and advisory activities through which the World Bank provides knowledge to its client countries: economic and sector work (ESW) and nonlending technical assistance (TA). The World Bank has committed itself to becoming a “global knowledge bank”, using knowledge to improve the development effectiveness of its work. ESW and TA are an essential part of the Bank's engagement with its clients—it spent $910 million (26% of its spending on country services) on these products during fiscal 2000-06. This evaluation assesses the extent to which the stated objectives of ESW and TA have been met. It also assesses whether the way ESW and TA are originated, partnership with clients in production, technical quality, and dissemination of these products influence the extent to which the stated objectives are met.

Methodology Twelve countries were selected for in-depth reviews. In-depth reviews entailed structured interviews of World Bank staff, World Bank management, and in-country stakeholders in the field (government officials and representatives from the private sector, civil society, academia, the media, and the legislature). The structured interviews sought stakeholder views (with quantitative ratings) on various dimensions of a portfolio of World Bank ESW and TA and the extent to which these tasks met various objectives (lending, policy, institution and capacity building, public debate, donor activities, and knowledge exchange). The interviews also sought the views of stakeholders on the relative importance (with rankings) of the following for the country concerned: (i) the different World Bank instruments (lending and nonlending), (ii) the different types of World Bank ESW and TA, and (iii) the various dimensions of ESW and TA. The country reviews also included desk reviews of ESW, documents on TA and loans, country assistance strategies, and poverty-reduction strategies.


The evaluation underscores the importance of ESW and TA and provides a broad endorsement of the quality and relevance of these Bank products. Overall, the evaluation notes, the ESW reforms of fiscal 1999 were accompanied by an increase in the quality of ESW. It also notes that in-country stakeholders validated the Quality Assurance Group’s high ratings on internal quality, although stakeholders rated dissemination activities lower than technical quality. In terms of effectiveness, the evaluation concludes that most ESW and TA tasks met their stated objectives and were effective in shaping lending. It also corroborates the link between the existence of relevant ESW and the quality of loan design. From a client perspective, stakeholders in most countries indicated that ESW and TA tasks had made a difference to the reforms in their countries, although the effectiveness of these activities differed across countries. Management finds that IEG’s discussion of the several factors that influence ESW and TA effectiveness and the summary of the preferences expressed by clients regarding different Bank instruments and their delivery modes provide useful and welcome insights. The recommendations included: Reinvigorate the Bank’s mandate to maintain a strong knowledge base Ensure that ESW tasks in countries funded by the International Development

Association (IDA) are adequately resourced, even if it means fewer economic and sector work

(ESW) tasks in some countries Enhance the institutional arrangements for undertaking ESW and technical

assistance by ensuring substantive task team presence in country offices, particularly in countries with low institutional capacity, and by formulating a dissemination strategy at the concept paper stage

Build on client preferences, including clients’ feedback after completing the task Take the results tracking framework seriously.

The World Bank (2003). Sharing Knowledge: Innovations and Remaining Challenges. Operations Evaluation Department


58

Objective and scope

In 1996, the World Bank made a commitment to become a global knowledge bank. This evaluation examined the relevance of that strategy and the institutional infrastructure put in place to implement it. It also reviewed the effectiveness of the strategy’s three main areas of innovation: (i) network and regional internal knowledge-sharing activities among the World Bank staff; (ii) regional and country external knowledge sharing with clients; and (iii) the three Bank-supported global knowledge initiatives that have the broadest knowledge-sharing scope—the Development Gateway, the GDLN, and the GDN.

Methodology The evaluation framework used the characteristics of strategic alignment, quality of shared knowledge, accessibility, and operational usefulness to measure the World Bank’s knowledge initiative against the standard OED evaluation criteria of relevance, efficacy, efficiency, and adequacy. The review used several approaches: review of the literature on knowledge management and transfer; desk reviews of relevant World Bank policy and strategy documents and program reports; surveys of 15 network advisory services and 28 thematic group leaders; structured interviews of 25 task team members to obtain information on the actual use of various knowledge-sharing activities, and the extent of knowledge capacity building as an explicit project objective; more than 30 additional interviews with World Bank management and staff; expert reviews of the degree of innovation, quality, and relevance of the World Bank’s knowledge and knowledge-sharing efforts in four issue areas—education, power, water, and public expenditure management; reviews of GDLN and the Development Gateway; and a survey of the views of officials, academics, nongovernment organizations, journalists, and representatives from the private sector.


The study concluded that the World Bank had made more progress in establishing the architecture to support its knowledge initiative than in creating the governance arrangements and work processes for carrying it out. As a result, the strategic intent of making knowledge sharing a way of doing business had been only partly realized—a process that in other leading knowledge management organizations has tended to take from three to five years. The World Bank needed to move deliberately to embed knowledge sharing in its core operational processes by providing more direct support to task teams and more knowledge capacity enhancement for clients, and it needed to manage its knowledge services for results. This review recommended that the World Bank took three sets of actions: Management should exercise more strategic direction and oversight over the Bank’s

knowledge processes. To accomplish this, management should: define clear responsibilities and accountabilities of corporate, network, and Regional units for integrating knowledge sharing into the Bank’s core business processes; ensure that incentives are aligned with responsibilities, especially at the task-manager level; and establish a strategic approach to the Bank’s role in existing and any new global knowledge initiatives.

Network and Regional units should tightly link their knowledge-sharing activities to lending and nonlending processes. To achieve this, networks should set clear objectives for anchor, thematic group, and advisory service support of operational teams; and Regional and country units should make explicit the knowledge objectives and strategies of Country Assistance Strategies (CASs) and projects.

Vice-presidential units should set outcome objectives and supporting performance indicators for their respective knowledge sharing programs and activities, and they should agree, Bankwide, on procedures to be established for monitoring and evaluating Bank knowledge-sharing programs and activities.

Appendix 4. Bibliography Evaluations reviewed in this study


59

[1] The African Development Bank (AfDB). Review of the African Development Bank’s Economic and Sector Work (2005−2010). Operations Evaluation Department (OPEV), 2013.

[2] Asian Development Bank (ASDB). Knowledge Products and Services: Building a Stronger Knowledge Institution. Special Evaluation Study, Independent Evaluation Department, 2012.

[3] Asian Development Bank (ASDB). Performance of the Asian Development Bank Institute:

Research, Capacity Building and Training, and Outreach and Knowledge Management. Special Evaluation Study, Independent Evaluation Department, 2011.

[4] Evaluation Cooperation Group. Evaluating Technical Assistance: Taking Stock of the Practices of

International Financial Institutions, December 17, 2012. [5] Inter-American Development Bank (IDB). Evaluation of the IDB's Studies (RE-323). Office of

Evaluation and Oversight, 2006. [6] International Fund for Agricultural Development (IFAD). IFAD’s institutional efficiency and

efficiency of IFAD-funded operations Corporate-level evaluation, Independent Office of Evaluation of IFAD, July 2013.

[7] International Monetary Fund (IMF). Research at the IMF: Relevance and Utilization. Independent

Evaluation Office (IEO), 2011. [8] The World Bank. Knowledge-Based Country Programs, An Evaluation of the World Bank Group

Experience. Independent Evaluation Group (IEG), July 2013. [9] The World Bank. The Matrix System at Work, An Evaluation of the World Bank’s Organizational

Effectiveness – Chapter 3: The Promise of a Knowledge Bank (pages 41-65). Independent Evaluation Group (IEG), April 2012.

[10] The World Bank. The State of World Bank Knowledge Services: Knowledge for Development,

2011. [11] The World Bank Institute and the Korea Development Institute. Using Knowledge Exchange for

Capacity Development: What Works in Global Practice? Three case studies in assessment of knowledge exchange programs using a results-focused methodology, November 2011.

[12] The World Bank. Knowledge for Private Sector Development. Independent Evaluation Group

(IEG), 2009. [13] The World Bank. Using Knowledge to Improve Development Effectiveness, An Evaluation of

World Bank Economic and Sector Work and Technical Assistance, 2000–2006. Independent Evaluation Group (IEG), 2008.

[14] The World Bank. Sharing Knowledge: Innovations and Remaining Challenges. Operations

Evaluation Department, 2003. Other reports [1] Asian Development Bank (ASDB). Regional and Sustainable Development Department’s Self-

Assessment of ASDB's Knowledge Management Implementation Framework (ASDB Knowledge Management MAKE Surveys by Teleos), 2005–2011.

[2] Banerjee, Abhijit, Deaton, Angus, Lustig, Nora, and Rogoff, Ken. 2006. An Evaluation of World Bank Research, 1998–2005. Washington, DC: The World Bank.


60

[3] Chioda, Laura, Augusto De la Torre, and William F. Maloney. 2011. “Towards a Conceptual

Framework for the Knowledge Bank.” Background paper for 2011 Knowledge Report. [4] Fries, Tom, and Peter Walkenhorst. 2011. “Sharing Global Governance: The Role of Civil Society

Organizations.” Bertelsmann Stiftung, Gütersloh, Germany. [1] International Fund for Agricultural Development (IFAD). Knowledge Management Strategy, IFAD

2007. [2] International Fund for Agricultural Development (IFAD). IFAD's Private-Sector Development and

Partnership Strategy, Corporate-level evaluation, Independent Office of Evaluation of IFAD, June 2011.

[3] International Fund for Agricultural Development (IFAD). IFAD’s Capacity to Promote Innovation

and Scaling Up, Corporate-level evaluation, Independent Office of Evaluation of IFAD, June 2010. [4] International Monetary Fund. Learning from Experience at the IMF: An IEO Assessment of Self-

Evaluation Systems. Independent Evaluation Office (IEO), Draft Issues Paper, 2011. [5] Iftikhar, Zuhair, Eriksson, Inger V., and Dickson, Gary W. 2003. Developing an Instrument for

Knowledge Management Project Evaluation. Electric Journal of Knowledge Management. Vol. 1, Issue 1: 55–62.

[6] Khattir, Nidhi. 2004. Quality and Impact of Programs Facilitated by the Global Development

Learning Network. WBI Evaluation Briefs. December. [7] Lusthaus, Charles, Adrien, Marie-Helene, Anderson, Gary, Carden, Fred, and Plinio Montalvan,

George. 2002. Organizational Assessment: A Framework for Improving Performance. Washington, DC and Ottawa: Inter-American Development Bank and International Development Research Centre.

[8] Minonne, Clemente and Turner, Geoff. 2009. "Evaluating Knowledge Management Performance."

Electric Journal of Knowledge Management. Vol. 7, Issue 5: 583–592. OED. 2003. Sharing Knowledge: Innovations and Remaining Challenges. Washington, DC: The World Bank.

[9] OECD. The Knowledge-Based Economy. OCDE/GD(96)102, 1996. [10] QAG (Quality Assurance Group). 2009a. “Internal Knowledge Management Products: a QAG

Learning Review.” World Bank, QAG, Washington, DC. [11] QAG (Quality Assurance Group). 2009b. “Synthesis of Past Assessments of Analytical and

Advisory Activities (AAA): A QAG Review.” World Bank, QAG, Washington, DC. [12] Ravallion, Martin. 2011. Knowledgeable Bankers? The Demand for Research in World Bank

Operations. Policy Research Working Paper 5892. The World Bank: Development Research Group.

[13] Sagasti, F., Knowledge and Innovation for Development: The Sisyphus Challenge of the 21st

Century. Cheltenham, Edward Elgar Publishing Ltd, 2004. [14] Talisayon, Serafin D. 2009. Monitoring and Evaluation in Knowledge Management for

Development. Information and Knowledge Management Working Paper No. 3.


61

[15] Ortiz, Alfredo and Peter Taylor. 2009. Learning Purposefully in Capacity Development: Why, What and When to Measure? United Kingdom

[16] Otoo, Samuel, Natalia Agapitova and Joy Behrens. 2009. Capacity Development and Results

Framework: A Strategic and Results�Oriented Approach to Learning for Capacity Development [17] United Nations. 2003. World Summit on the Information Society: Declaration of Principles [18] World Bank Institute. 2010. Consultations in Results-Focused Capacity Development Project

Experiences in East Africa Using the Capacity Development and Results Framework. (Discussion Version)

[19] The World Bank. Report on the World Bank Research Program for Fiscal Years 2006, 2007, and

2008 and Future Directions, 2009. [20] UNESCO. Towards Knowledge Societies, 2005.


62

Appendix 5. List of interviews To be completed in next draft.


63

Appendix 6. Consultant’s background and expertise Thierry Senechal is an expert specializing in evaluation, information and data management, and knowledge management. For the past two decades, he has advised governments, bi/multilateral organizations and corporates on how to develop management information systems and performance and evaluation frameworks to influence change throughout the organization. His experience encompasses complex strategic and corporate-wide evaluations, country portfolio evaluations and program/project evaluations in a vast array of sectors. He combines his interests in research, teaching, and business management as a member of the Faculty of Public Affairs at the Sciences Po Institute in Paris, where he teaches the second-year graduate seminar “Managing & Anticipating Change”. Between 2007 and 2013, he served as a senior consultant to the International Chamber of Commerce (ICC) for which he developed several flagship knowledge products and services, including the “Global Risks” series and the annual “Rethinking Trade & Finance”, a project done in partnership with multilateral development banks. He is also one of the founding members of the ICC Trade Register, a high-volume dataset capturing risk characteristics of financial products worldwide. He has advised many international organizations on knowledge management, complex data analysis and development of e-learning tools and trainings. For the Mazars Group (13,000 professionals in 69 countries), he led the advisory services in the field of evaluation and policy analysis (2002-2005). He gained extensive expertise in knowledge management and evaluation while leading a United Nations program related to war damages from the Gulf war (1997-2001). Between 1992-1997, he worked for a risk management consulting firm and developed advisory services in the fields of business forecasting and decision technology for clients in various industries. A national of France, he obtained a bachelor’s degree with honors in Economics and Philosophy from Columbia University. He holds a Master in Science degree from the University of London (London Business School) in financial engineering with research work on artificial neural networks and business modeling. He received a M.B.A. from the Massachusetts Institute of Technology (management of technology and innovation). Thierry Senechal Consultant [email protected]

Documents

Knowledge Generation and Sharing by Multilateral ... · Knowledge Generation and Sharing by MDBs: Evidence from Evaluation 7 knowledge capabilities for local audiences at country