elf

Evaluating the Evaluators: Some Lessons from a Recent World Bank Self-Evaluation


Editor's Note: The World Bank’s Independent Evaluation Group (IEG) recently published a self-evaluation of its activities. Besides representing current thinking among evaluation experts at the World Bank, it also more broadly reflects some of the strengths and gaps in the approaches that evaluators use to assess and learn from the performance of the international institutions with which they work. The old question “Quis custodet ipsos custodes?” – loosely translated as “Who evaluates the evaluators?” – remains as relevant as ever. Johannes Linn served as an external peer reviewer of the self-evaluation and provides a bird’s-eye view on the lessons learned.

An Overview of the World Bank’s IEG Self-Evaluation Report

In 2011 the World Bank’s Independent Evaluation Group (IEG) carried out and published a self-evaluation of its activities. The self-evaluation team was led by an internal manager, but involved a respected external evaluation expert as the principal author and also an external peer reviewer.

The IEG self-evaluation follows best professional practices as codified by the Evaluation Cooperation Group (ECG). This group brings together the evaluation offices of seven major multilateral financial institutions in joint efforts designed to enhance evaluation performance and cooperation among their evaluators. One can therefore infer that the approach and focus of the IEG self-evaluation is representative of a broader set of practices that are currently used by the evaluation community of international financial organizations.

At the outset the IEG report states that “IEG is the largest evaluation department among Evaluation Capacity Group (ECG) members and is held in high regard by the international evaluation community. Independent assessments of IEG’s role as an independent evaluation function for the Bank and IFC rated it above the evaluation functions in most other ECG members, international nongovernmental organizations, and transnational corporations and found that IEG follows good practice evaluation principles.”

The self-evaluation report generally confirms this positive assessment. For four out of six areas of its mandate IEG gives itself the second highest rating (“good”) out of six possible rating categories. This includes (a) the professional quality of its evaluations, (b) its reports on how the World Bank’s management follows up on IEG recommendations, (c) cooperation with other evaluation offices, and (d) assistance to borrowing countries in improving their own evaluation capacity. In the area of appraising the World Bank’s self-evaluation and risk management practices, the report offers the third highest rating (“satisfactory”), while it gives the third lowest rating (“modest”) for IEG’s impact on the Bank’s policies, strategies and operations. In addition the self-evaluation concludes that overall the performance of IEG has been “good” and that it operates independently, effectively and efficiently.

The report makes a number of recommendations for improvement, which are likely to be helpful, but have limited impact on its activities. They cover measures to further enhance the independence of IEG and the consistency of evaluation practices as applied across the World Bank Group’s branches – the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) –; to improve the design of evaluations and the engagement with Bank management upstream for greater impact; and monitoring the impact of recent organizational changes in IEG in terms of results achieved. The report also recommends that more be done to evaluate the Bank’s analytical work and that evaluations draw on comparative evidence.

Assessment

In terms of the parameters of self-evaluation set by the prevailing practice among the evaluators on international financial agencies, the IEG self-evaluation is accurate and helpful. From my own experience as an operational manager in the Bank whose activities were evaluated by IEG in years past, and as a user of IEG evaluations (and of evaluations of other international aid organizations) for my research on aid effectiveness, I concur that IEG is independent and effective in meeting its mandate as defined. Moreover, the self-evaluation produces useful quantitative evidence (including survey results, budget analysis, etc.) to corroborate qualitative judgments.

However, the self-evaluation suffers from a number of limitations in approach and gaps in focus, which are broadly representative of the practices prevalent among many of the evaluation offices of international aid agencies.

Approach of the IEG self-evaluation

The core of the self-evaluation report is about the evaluation process followed by IEG, with very little said about the substance of IEG’s evaluations. The following questions could have usefully been raised, but were not: do evaluations cover the right issues with the right intensity, such as growth and poverty; environmental, governance, and gender impacts; regional dimensions versus exclusive country or project focus; effectiveness in addressing the problems of fragile and conflict states; effectiveness in dealing with global public goods; sustainability and scaling up; etc. Therefore the report does not deal with the question of whether IEG effectively responds in its evaluations to the many important strategic debates and issues with which the development community is grappling.

Related to this limitation is the fact that the report assessed the quality of IEG’s mostly in terms of (a) whether its approach and processes meet certain standards established by the Evaluation Cooperation Group; and (b) how it is judged by stakeholders in response to a survey commissioned for this evaluation. Both these approaches are useful, but they do not have any basis in professional assessments of the quality of individual products. This is equivalent to IEG evaluating the World Bank’s projects on the quality of its processes (e.g., appraisal and supervision processes) and on the basis of stakeholder surveys, without evaluating individual products and their impacts.

Gaps in the Self-Evaluation and in Evaluation Practice

Careful reading of the report reveals six important gaps in the IEG self-evaluation, in the prevailing evaluation practice in the World Bank, and more generally in the way international financial organizations evaluate their own performance. The first three gaps relate to aspects of the evaluation approach used and the second three gaps relate to lack of focus in the self-evaluation on key internal organizational issues:

1. Impact Evaluations: The report notes that IEG carries out two to three impact evaluations per year, but it sidesteps the debate in the current evaluation literature and practice as to what extent the “gold standard” of randomized impact evaluation should occupy a much more central role. Given the importance of this debate and divergence of views, it would have been appropriate for the self-evaluation to assess IEG’s current practice of very limited use of randomized evaluations.

2. Evaluation of Scaling Up: The report does not address the question of to what extent current IEG practice not only assesses the performance of individual projects in terms of their outcomes and sustainability, but also in terms of whether the Bank has systematically built on its experience in specific projects to help scale up their impact through support for expansion or replication in follow-up operations or through effective hand-off to the government or other partners. In fact, currently IEG does not explicitly and systematically consider scaling up in its project and program evaluations. For example, in a recent IEG evaluation of World Bank funded municipal development projects (MDPs) , IEG found that the Bank has supported multiple MDPs in many countries over the years, but the evaluation did not address the obvious question whether the Bank systematically planned for the project sequence or built on its experience from prior projects in subsequent operations. While most other evaluation offices like IEG do not consider scaling up, some (in particular those of the International Fund for Agricultural Development and the United Nations Development Program) have started doing so in recent years.

3. Drawing on the Experience of and Benchmarking Against Other Institutions: The self-evaluation report does a good job in benchmarking IEG performance in a number of respects against that of other multilateral institutions. In the main text of the report it states that “IEG plans to develop guidelines for approach papers to ensure greater quality, in particular in drawing on comparative information from other sources and benchmarking against other institutions.” This is a welcome intention, but it is inadequately motivated in the rest of the report and not reflected in the Executive Summary. The reality is that IEG, like most multilateral evaluation offices, so far has not systematically drawn on the evaluations and relevant experience of other aid agencies in its evaluations of World Bank performance. This has severely limited the learning impact of the evaluations.

4. Bank Internal Policies, Management Processes and Incentives: IEG evaluations traditionally do not focus on how the Bank’s internal policies, management and incentives affect the quality of Bank engagement in countries. Therefore evaluations cannot offer any insights into whether and how Bank-internal operating modalities contribute to results. Two recent exceptions are notable exceptions. First, the IEG evaluation of the Bank’s approach to harmonization with other donors and alignment with country priorities assesses the incentives for staff to support harmonization and alignment. The evaluation concludes that there are insufficient incentives, a finding disputed by management. Second, is the evaluation of the Bank’s internal matrix management arrangements, which is currently under way. The self-evaluation notes that Bank management tried to quash the matrix evaluation on the grounds that it did not fall under the mandate of IEG. This is an unfortunate argument, since an assessment of the institutional reasons for the Bank’s performance is an essential component of any meaningful evaluation of Bank-supported programs. While making a good case for the specific instance of the matrix evaluation, the self-evaluation report shies away from a more general statement in support of engaging IEG on issues of Bank-internal policies, management processes and incentives. It is notable that IFAD’s Independent Office of Evaluation appears to be more aggressive in this regard: It currently is carrying out a full evaluation of IFAD’s internal efficiency and previous evaluations (e.g., an evaluation of innovation and scaling up) did not shy away from assessing internal institutional dimensions.

5. World Bank Governance: The IEG self-evaluation is even more restrictive in how it interprets its mandate regarding the evaluation of the World Bank’s governance structures and processes (including its approach to members’ voice and vote, the functioning of its board of directors, the selection of its senior management, etc.). It considers these topics beyond IEG’s mandate. This is unfortunate, since the way the Bank’s governance evolves will substantially affect its long-term legitimacy, effectiveness and viability as an international financial institution. Since IEG reports to the Bank’s board of directors, and many of the governance issues involve questions of the board’s composition, role and functioning, there is a valid question of how effectively IEG could carry out such an evaluation. However, it is notable that the IMF’s Independent Evaluation Office, which similarly reports to the IMF board of directors, published a full evaluation of the IMF’s governance in 2008, which effectively addressed many of the right questions.

6. Synergies between World Bank, IFC and MIGA: The self-evaluation report points out that the recent internal reorganization of IEG aimed to assure more effective and consistent evaluations across the three member branches of the World Bank Group. This is welcome, but the report does not assess how past evaluations addressed the question of whether the World Bank, IFC and MIGA effectively capitalized on the potential synergies among the three organizations. The recent evaluation of the World Bank Group’s response to the global economic crisis of 2008/9 provided parallel assessments of each agency’s performance, but did not address whether they work together effectively in maximizing their synergies. The reality is that the three organizations have deeply engrained institutional cultures and generally go their own ways rather than closely coordinating their activities on the ground. Future evaluations should explicitly consider whether the three effectively cooperate or not. While the World Bank is unique in the way it has organizationally separated its private sector and guarantee operations, other aid organizations also have problems of a lack of cooperation, coordination and synergy among different units within the agency. Therefore, the same comment also applies to their evaluation approaches.

Conclusions

Self-evaluations are valuable tools for performance assessment and IEG is to be congratulated for carrying out and publishing such an evaluation of its own activities. As for all self-evaluations, it should be seen as an input to an independent external evaluation, a decision that, for now, has apparently been postponed by the Bank’s board of directors.

IEG’s self-evaluation has many strengths and provides an overall positive assessment of IEG’s work. However, it does reflect some important limitations of analysis and of certain gaps in approach and coverage, which an independent external review should consider explicitly, and which IEG’s management should address. Since many of these issues also likely apply to most of the other evaluation approaches by other evaluation offices, the lessons have relevance beyond IEG and the World Bank.

Key lessons include:

  • An evaluation of evaluations should focus not only on process, but also on the substantive issues that the institution is grappling with.
  • An evaluation of the effectiveness of evaluations should include a professional assessment of the quality of evaluation products.
  • An evaluation of evaluations should assess:
    o How effectively impact evaluations are used;
    o How scaling up of successful interventions is treated;
    o How the experience of other comparable institutions is utilized;
    o Whether and how the internal policies, management practices and incentives of the institution are effectively assessed;
    o Whether and how the governance of the institution is evaluated; and
    o Whether and how internal coordination, cooperation and synergy among units within the organizations are assessed.

Evaluations play an essential role in the accountability and learning of international aid organizations. Hence it is critical that evaluations address the right issues and use appropriate techniques. If the lessons above were reflected in the evaluation practices of the aid institutions, this would represent a significant step forward in the quality, relevance and likely impact of evaluations.

Image Source: © Christian Hartmann / Reuters
      
 
 




elf

The National Effort at Self-Exoneration on Torture

The nation's current attempt at catharsis through a gargantuan report prepared by the Democratic staff of a Senate committee exhibits some familiar patterns. Most of them involve treating a government agency as if it were Dorian Gray's portrait, which can take on all the hideous marks of our own transgressions while we present ourselves as…

      
 
 




elf

The United States can’t save Egypt from itself


Editors’ Note: On March 23, the Working Group on Egypt sent a letter to President Obama urging him to publicly and privately object to Egyptian President Abdel Fattah el-Sissi’s accelerating crackdown on human rights and civil society organizations. Brookings senior fellow and director of the Center for Middle East Policy Tamara Wittes was among the letter’s signers, and she explains her decision to do so. The letter was originally published by the Project on Middle East Democracy (POMED).

Tamara Wittes: In a disordered Middle East, America needs anchors of stability and reliable partners to help it achieve its goals. Both are in sadly short supply. 

For more than thirty years, Egypt was an anchor of stability and a reliable American partner in regional security. From the time Sadat expelled Soviet advisers and broached peace with Israel, ties with Egypt have been a core pillar of American Middle East policy. But, as my colleague Steven Cook presciently noted way back in February 2012, Egypt’s revolution accelerated the launch of what he calls a “long goodbye” between these two formerly indispensable partners. He argued back then that shifting from a “special relationship” to something more transaction would have four concrete benefits for Washington:

First, Washington will no longer be in the unseemly position of providing taxpayer largesse—however small in the grand scheme of things—to a government that resents the United States and clearly does not share its values. Second, it will provide an opportunity for a much-needed change in military-to-military relations in which the United States merely pays for the services it needs like expedited transit through the Suez Canal. Third, it is consistent with this moment of empowerment and dignity for Egyptians many of whom do not want U.S. assistance either because they believe it actually stands in the way of a democratic transition or accept Aboul Naga’s argument along with those who couldn’t care less about U.S. assistance because it doesn’t touch their lives. Finally, it will free up funds for the United States to help others who actually might want Washington’s help, perhaps the Tunisians, Moroccans, or some sub-Saharan African countries would be grateful for development assistance.

Since that blog post went up, Egypt has had three different governments and lost its place as a diplomatic and security leader in the region; while the United States has withdrawn from Iraq and begun to do the same in Afghanistan, while emphasizing burden-sharing in its new fight against ISIS. All of these shifts strengthen the argument for a more distant and transactional U.S.-Egyptian relationship. 

Moreover, since his accession to power (first in a military coup in July 2013 and then in a highly constrained election in 2014), President Abdel Fattah el-Sisi has made decisions that are undermining both Egypt’s domestic stability and key American policy goals in the region. 

  • Sisi’s failure to move forward on economic reforms (recommended by leading Egyptian voices, regional supporters, and international donors) has left his country in a spiral of shrinking cash reserves, capital flight and currency devaluation that together threaten the government’s ability to import needed food and medicine and to carry out core government functions. 
  • Sisi’s counterterrorism campaign in the Sinai has succeeded in “making the sand jump,” as one regional security official told me, but it seems to have stoked more than tamped down the fire of violent extremism threatening both Egypt and Israel; meanwhile, its alleged military abuses have sparked a Senate request for investigation.
  • The intense political polarization and relentless repression of post-coup Egypt are producing other destabilizing effects, which are detailed in the Egypt Working Group’s newest letter to President Obama posted below (I am a member of the Working Group). 

To top it all off, the Egyptian government continues to throw obstacles in the road of U.S.-Egyptian cooperation. Its military resists learning from the hard-won American experience in effective counterinsurgency. Its leadership has resolutely refused to allow core bilateral aid programs, like those supporting higher education, to move forward. And at the same time, the Egyptian government continues to promote conspiracy theories about the United States to its public through media smears and show trials, and now, apparently, to its newly elected parliamentarians. 

It’s long past time for the United States to undertake a strategic review of its approach to the Middle East, one focused on building anchors of stability and sustaining reliable partners in pursuit of American priorities. Egypt, as I told The New York Times, no longer qualifies as either one. That doesn’t mean the two countries can’t continue to work together in those narrow areas where they agree on interests, priorities, and approaches. 

But Secretary of State Kerry’s public embrace last week of Egyptian Foreign Minister Shoukry cannot hide the facts—there is no “back to business” option for the U.S.-Egyptian relationship, and it seems increasingly clear that even direct White House engagement would not shift Egypt’s leadership off of its self-destructive trajectory. Egypt's looming instability demands that the United States take steps now to safeguard itself from reliance on a country we cannot rescue, not least from its own leaders' worst impulses. 


March 23, 2016

Dear Mr. President,

We are writing to urge you to speak directly with Egyptian President Abdel Fattah el-Sissi and to express both publicly and privately your objection to his accelerating crackdown on human rights, including recent moves to prosecute civil society organizations. You were correct to declare in September 2014 that “America’s support for civil society is a matter of national security,” and nowhere is that more true than in Egypt today.

President el-Sissi’s campaign against civil society takes place against the backdrop of unprecedented abuses by Egyptian security forces, including extrajudicial killings, the detention of tens of thousands of political prisoners, the widespread documented use of torture, and the forced disappearances of hundreds of Egyptians. The killing of Italian student Giulio Regeni, whose tortured body appeared on a roadside near Cairo a week after his abduction in late January, has come to international attention, but many Egyptians have shared his fate since President el-Sissi came to power.

On March 24, an Egyptian court will hear a request to freeze the bank accounts and other assets of two internationally-respected human rights defenders, Hossam Bahgat and Gamal Eid, along with members of Eid’s family. Mr. Bahgat and Mr. Eid and other activists may soon be indicted and face trial for illegally accepting foreign funding—a criminal charge that violates their right to free association and could carry a sentence of up to 25 years in prison.

The imminent proceedings are a major step in Egyptian authorities’ campaign to crush the last remnants of Egypt’s independent civil society and human rights community. Egypt’s media has recently reported that dozens of organizations are under criminal investigation, essentially for their peaceful work to monitor abuses and to hold Egypt’s government accountable to its own constitution and international human rights commitments. In recent weeks, Egyptian authorities have ordered the closure of a prominent anti-torture organization, the Nadeem Center; summoned staff from several human rights organizations for interrogation; banned prominent rights activists and advocates from traveling outside Egypt in violation of the Egyptian constitution; and harassed and threatened human rights activists with arrest and violence. The media regularly propagate vitriol against human rights defenders, portraying them as traitors and security threats.

If this crackdown is allowed to reach its conclusion, it will silence an indigenous human rights community that has survived more than 30 years of authoritarian rule, leaving few if any Egyptians free to investigate mounting abuses by the state.

The current attacks on Egypt’s rights advocates are a continuation of the same criminal prosecution of American and German NGO workers in Egypt that began in 2011. That prosecution, driven by senior members of the Egyptian government still in high office today, resulted in the June 2013 criminal convictions, in a deeply flawed trial, of 43 Egyptian and international NGO staff, including 17 American citizens. President el-Sissi, who was the head of military intelligence in 2011 when Egypt’s military government launched the investigation, has refused repeated requests to overturn the convictions.

While the current crackdown is primarily targeting domestic organizations, there are indications that international NGOs may also face increased pressure, including some that currently do not even have offices or staff working in Egypt. On March 20, the newspaper Al Masry Al Youm published the names of more than 150 individuals and civil society organizations reportedly under investigation for receiving foreign funding, including prominent American and European organizations such as the Center for International Private Enterprise, the Solidarity Center, Transparency International, Save the Children, Catholic Relief Services, CARE, AMIDEAST, the National Democratic Institute, and the International Republican Institute.

Mr. President, in your September 2014 Presidential Memorandum on Civil Society, you pledged that the United States government—including you personally—would stand firmly with those in civil society facing pressure or harassment from their governments. While the past five years have been tumultuous and challenging for U.S. policy toward Egypt, this is another defining moment for the United States, a moment that tests your pledge to “stand with civil society.” Secretary Kerry’s March 18 statement of concern was welcome, but further action is urgently needed. Past practice demonstrates that when the United States government speaks clearly, in one voice, and consistently on NGO freedom and human rights in Egypt, the government in Cairo listens.

It is essential that you act to stand up for human rights, freedom of association, and the rights of both Egyptian and international civil society organizations to work together on behalf of common goals. You must make crystal clear to President el-Sissi that continued assaults on civil society, including harassment of U.S. organizations, will make it difficult for the administration to cooperate across a range of issues, including your administration’s efforts to promote American investment in Egypt and to provide financial assistance to the Egyptian government and military. If Egypt’s government continues down a path to destroy its own civil society, American support and assistance will become, in both principled and practical terms, impossible.

Sincerely,

The Working Group on Egypt

Publication: Project on Middle East Democracy (POMED)
      
 
 




elf

Mitt Romney changed the impeachment story, all by himself. Here are 3 reasons that matters.

       




elf

What COVID-19 means for America’s child welfare system

The COVID-19 crisis has allowed a revealing look into the shortcomings of the U.S.’s child welfare system. While no institution has proved strong enough to operate effectively and efficiently under the unprecedented circumstances brought on by COVID-19, the crisis has unveiled holes in the child welfare system that call for both immediate and long-term action.…

       




elf

Welfare Reform and Beyond

The Brookings Institution's Welfare Reform & Beyond Initiative was created to inform the critical policy debates surrounding the upcoming congressional reauthorization of the Temporary Assistance for Needy Families (TANF) program and a number of related programs that were created or dramatically altered by the 1996 landmark welfare reform legislation. The goal of the project has…

       




elf

American self-criticism borders on narcissism

Those who said there will be war may not have realized there already was war. This doesn’t mean killing Iranian General Qassem Soleimani was good. It almost certainly wasn’t. Iran quickly retaliated by targeting two American military bases in Iraq and may find new ways to escalate, but Iran had already been escalating. The regime of the Syrian…

       




elf

Evaluating the Evaluators: Some Lessons from a Recent World Bank Self-Evaluation


Editor's Note: The World Bank’s Independent Evaluation Group (IEG) recently published a self-evaluation of its activities. Besides representing current thinking among evaluation experts at the World Bank, it also more broadly reflects some of the strengths and gaps in the approaches that evaluators use to assess and learn from the performance of the international institutions with which they work. The old question “Quis custodet ipsos custodes?” – loosely translated as “Who evaluates the evaluators?” – remains as relevant as ever. Johannes Linn served as an external peer reviewer of the self-evaluation and provides a bird’s-eye view on the lessons learned.

An Overview of the World Bank’s IEG Self-Evaluation Report

In 2011 the World Bank’s Independent Evaluation Group (IEG) carried out and published a self-evaluation of its activities. The self-evaluation team was led by an internal manager, but involved a respected external evaluation expert as the principal author and also an external peer reviewer.

The IEG self-evaluation follows best professional practices as codified by the Evaluation Cooperation Group (ECG). This group brings together the evaluation offices of seven major multilateral financial institutions in joint efforts designed to enhance evaluation performance and cooperation among their evaluators. One can therefore infer that the approach and focus of the IEG self-evaluation is representative of a broader set of practices that are currently used by the evaluation community of international financial organizations.

At the outset the IEG report states that “IEG is the largest evaluation department among Evaluation Capacity Group (ECG) members and is held in high regard by the international evaluation community. Independent assessments of IEG’s role as an independent evaluation function for the Bank and IFC rated it above the evaluation functions in most other ECG members, international nongovernmental organizations, and transnational corporations and found that IEG follows good practice evaluation principles.”

The self-evaluation report generally confirms this positive assessment. For four out of six areas of its mandate IEG gives itself the second highest rating (“good”) out of six possible rating categories. This includes (a) the professional quality of its evaluations, (b) its reports on how the World Bank’s management follows up on IEG recommendations, (c) cooperation with other evaluation offices, and (d) assistance to borrowing countries in improving their own evaluation capacity. In the area of appraising the World Bank’s self-evaluation and risk management practices, the report offers the third highest rating (“satisfactory”), while it gives the third lowest rating (“modest”) for IEG’s impact on the Bank’s policies, strategies and operations. In addition the self-evaluation concludes that overall the performance of IEG has been “good” and that it operates independently, effectively and efficiently.

The report makes a number of recommendations for improvement, which are likely to be helpful, but have limited impact on its activities. They cover measures to further enhance the independence of IEG and the consistency of evaluation practices as applied across the World Bank Group’s branches – the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) –; to improve the design of evaluations and the engagement with Bank management upstream for greater impact; and monitoring the impact of recent organizational changes in IEG in terms of results achieved. The report also recommends that more be done to evaluate the Bank’s analytical work and that evaluations draw on comparative evidence.

Assessment

In terms of the parameters of self-evaluation set by the prevailing practice among the evaluators on international financial agencies, the IEG self-evaluation is accurate and helpful. From my own experience as an operational manager in the Bank whose activities were evaluated by IEG in years past, and as a user of IEG evaluations (and of evaluations of other international aid organizations) for my research on aid effectiveness, I concur that IEG is independent and effective in meeting its mandate as defined. Moreover, the self-evaluation produces useful quantitative evidence (including survey results, budget analysis, etc.) to corroborate qualitative judgments.

However, the self-evaluation suffers from a number of limitations in approach and gaps in focus, which are broadly representative of the practices prevalent among many of the evaluation offices of international aid agencies.

Approach of the IEG self-evaluation

The core of the self-evaluation report is about the evaluation process followed by IEG, with very little said about the substance of IEG’s evaluations. The following questions could have usefully been raised, but were not: do evaluations cover the right issues with the right intensity, such as growth and poverty; environmental, governance, and gender impacts; regional dimensions versus exclusive country or project focus; effectiveness in addressing the problems of fragile and conflict states; effectiveness in dealing with global public goods; sustainability and scaling up; etc. Therefore the report does not deal with the question of whether IEG effectively responds in its evaluations to the many important strategic debates and issues with which the development community is grappling.

Related to this limitation is the fact that the report assessed the quality of IEG’s mostly in terms of (a) whether its approach and processes meet certain standards established by the Evaluation Cooperation Group; and (b) how it is judged by stakeholders in response to a survey commissioned for this evaluation. Both these approaches are useful, but they do not have any basis in professional assessments of the quality of individual products. This is equivalent to IEG evaluating the World Bank’s projects on the quality of its processes (e.g., appraisal and supervision processes) and on the basis of stakeholder surveys, without evaluating individual products and their impacts.

Gaps in the Self-Evaluation and in Evaluation Practice

Careful reading of the report reveals six important gaps in the IEG self-evaluation, in the prevailing evaluation practice in the World Bank, and more generally in the way international financial organizations evaluate their own performance. The first three gaps relate to aspects of the evaluation approach used and the second three gaps relate to lack of focus in the self-evaluation on key internal organizational issues:

1. Impact Evaluations: The report notes that IEG carries out two to three impact evaluations per year, but it sidesteps the debate in the current evaluation literature and practice as to what extent the “gold standard” of randomized impact evaluation should occupy a much more central role. Given the importance of this debate and divergence of views, it would have been appropriate for the self-evaluation to assess IEG’s current practice of very limited use of randomized evaluations.

2. Evaluation of Scaling Up: The report does not address the question of to what extent current IEG practice not only assesses the performance of individual projects in terms of their outcomes and sustainability, but also in terms of whether the Bank has systematically built on its experience in specific projects to help scale up their impact through support for expansion or replication in follow-up operations or through effective hand-off to the government or other partners. In fact, currently IEG does not explicitly and systematically consider scaling up in its project and program evaluations. For example, in a recent IEG evaluation of World Bank funded municipal development projects (MDPs) , IEG found that the Bank has supported multiple MDPs in many countries over the years, but the evaluation did not address the obvious question whether the Bank systematically planned for the project sequence or built on its experience from prior projects in subsequent operations. While most other evaluation offices like IEG do not consider scaling up, some (in particular those of the International Fund for Agricultural Development and the United Nations Development Program) have started doing so in recent years.

3. Drawing on the Experience of and Benchmarking Against Other Institutions: The self-evaluation report does a good job in benchmarking IEG performance in a number of respects against that of other multilateral institutions. In the main text of the report it states that “IEG plans to develop guidelines for approach papers to ensure greater quality, in particular in drawing on comparative information from other sources and benchmarking against other institutions.” This is a welcome intention, but it is inadequately motivated in the rest of the report and not reflected in the Executive Summary. The reality is that IEG, like most multilateral evaluation offices, so far has not systematically drawn on the evaluations and relevant experience of other aid agencies in its evaluations of World Bank performance. This has severely limited the learning impact of the evaluations.

4. Bank Internal Policies, Management Processes and Incentives: IEG evaluations traditionally do not focus on how the Bank’s internal policies, management and incentives affect the quality of Bank engagement in countries. Therefore evaluations cannot offer any insights into whether and how Bank-internal operating modalities contribute to results. Two recent exceptions are notable exceptions. First, the IEG evaluation of the Bank’s approach to harmonization with other donors and alignment with country priorities assesses the incentives for staff to support harmonization and alignment. The evaluation concludes that there are insufficient incentives, a finding disputed by management. Second, is the evaluation of the Bank’s internal matrix management arrangements, which is currently under way. The self-evaluation notes that Bank management tried to quash the matrix evaluation on the grounds that it did not fall under the mandate of IEG. This is an unfortunate argument, since an assessment of the institutional reasons for the Bank’s performance is an essential component of any meaningful evaluation of Bank-supported programs. While making a good case for the specific instance of the matrix evaluation, the self-evaluation report shies away from a more general statement in support of engaging IEG on issues of Bank-internal policies, management processes and incentives. It is notable that IFAD’s Independent Office of Evaluation appears to be more aggressive in this regard: It currently is carrying out a full evaluation of IFAD’s internal efficiency and previous evaluations (e.g., an evaluation of innovation and scaling up) did not shy away from assessing internal institutional dimensions.

5. World Bank Governance: The IEG self-evaluation is even more restrictive in how it interprets its mandate regarding the evaluation of the World Bank’s governance structures and processes (including its approach to members’ voice and vote, the functioning of its board of directors, the selection of its senior management, etc.). It considers these topics beyond IEG’s mandate. This is unfortunate, since the way the Bank’s governance evolves will substantially affect its long-term legitimacy, effectiveness and viability as an international financial institution. Since IEG reports to the Bank’s board of directors, and many of the governance issues involve questions of the board’s composition, role and functioning, there is a valid question of how effectively IEG could carry out such an evaluation. However, it is notable that the IMF’s Independent Evaluation Office, which similarly reports to the IMF board of directors, published a full evaluation of the IMF’s governance in 2008, which effectively addressed many of the right questions.

6. Synergies between World Bank, IFC and MIGA: The self-evaluation report points out that the recent internal reorganization of IEG aimed to assure more effective and consistent evaluations across the three member branches of the World Bank Group. This is welcome, but the report does not assess how past evaluations addressed the question of whether the World Bank, IFC and MIGA effectively capitalized on the potential synergies among the three organizations. The recent evaluation of the World Bank Group’s response to the global economic crisis of 2008/9 provided parallel assessments of each agency’s performance, but did not address whether they work together effectively in maximizing their synergies. The reality is that the three organizations have deeply engrained institutional cultures and generally go their own ways rather than closely coordinating their activities on the ground. Future evaluations should explicitly consider whether the three effectively cooperate or not. While the World Bank is unique in the way it has organizationally separated its private sector and guarantee operations, other aid organizations also have problems of a lack of cooperation, coordination and synergy among different units within the agency. Therefore, the same comment also applies to their evaluation approaches.

Conclusions

Self-evaluations are valuable tools for performance assessment and IEG is to be congratulated for carrying out and publishing such an evaluation of its own activities. As for all self-evaluations, it should be seen as an input to an independent external evaluation, a decision that, for now, has apparently been postponed by the Bank’s board of directors.

IEG’s self-evaluation has many strengths and provides an overall positive assessment of IEG’s work. However, it does reflect some important limitations of analysis and of certain gaps in approach and coverage, which an independent external review should consider explicitly, and which IEG’s management should address. Since many of these issues also likely apply to most of the other evaluation approaches by other evaluation offices, the lessons have relevance beyond IEG and the World Bank.

Key lessons include:

  • An evaluation of evaluations should focus not only on process, but also on the substantive issues that the institution is grappling with.
  • An evaluation of the effectiveness of evaluations should include a professional assessment of the quality of evaluation products.
  • An evaluation of evaluations should assess:
    o How effectively impact evaluations are used;
    o How scaling up of successful interventions is treated;
    o How the experience of other comparable institutions is utilized;
    o Whether and how the internal policies, management practices and incentives of the institution are effectively assessed;
    o Whether and how the governance of the institution is evaluated; and
    o Whether and how internal coordination, cooperation and synergy among units within the organizations are assessed.

Evaluations play an essential role in the accountability and learning of international aid organizations. Hence it is critical that evaluations address the right issues and use appropriate techniques. If the lessons above were reflected in the evaluation practices of the aid institutions, this would represent a significant step forward in the quality, relevance and likely impact of evaluations.

Image Source: © Christian Hartmann / Reuters
     
 
 




elf

High Levels Of BPA Found In Cash Register Receipts, What You Can Do To Protect Yourself

Image Source: red5standingby Environmental Working Group (EWG), a nonprofit research organization based in Washington, DC, has discovered that many cash register receipts contain levels of Bisphenol-A (BPA) hundreds of times higher than those found in




elf

Plywood homes were lighter and cheaper, and you could build them yourself

Another look back at some great designs for inexpensive homes.




elf

From bike lanes to solar, the UK has transformed itself since 2006

What a difference a decade makes.




elf

7 questions to ask yourself when planning a wedding

Challenge the notion that you have to blow the bank account just to throw a good party. There are smarter ways of doing it.




elf

There's a story behind that kimchi on the supermarket shelf

Many exotic ingredients aren't on shelves because people ask for them, but more so because the governments of those countries are actively promoting them.




elf

'Achtung Baby: An American Mom on the German Art of Raising Self-Reliant Children' (book review)

Author Sara Zaske shows that it all comes down to respecting a child's right to independence.




elf

Compact self-sufficient house has a terraced roof garden

This proposed home for a young family incorporates an efficient roof garden.




elf

Hey Google, this rooftop solar cooker is perfect for your self-driving car

Why turn your car into a living room when it can also be a kitchen?




elf

Elfin mountain toad discovered in misty, mossy elfin forest

The newly discovered horned mountain toad found in Southern Vietnam's elfin forest is the smallest of its species – and is already considered endangered.




elf

NASA's Advanced Space Tech Gets Turned Into Self-Aware Eco Building

NASA is using decades of space exploration technology to build a new eco-aware base here in California.




elf

The First 3290% Energy Self-Sufficient Town In Japan

Yanaizu Town, Kawanuma District, Fukushima Prefecture is rural-at-best and about as far away as you can get from the big cities. The town recently made the news when Chiba




elf

Built on a tilt: climbable bookshelf is also earthquake resistant

It's wood, it's small, it's resilient.




elf

Questions to ask yourself when buying clothes

Following these guidelines can help you to build a higher quality, longer-lasting wardrobe.




elf

Flip Side of Local - New Hampshire Sees Opportunity In Food Self-Sufficiency

University of New Hampshire researchers have analyzed the economic impact of their State's locally-produced food system. The good news: farm market sales - known as 'direct marketing' of food -




elf

Six Selfish Reasons You Don't Want Dead Oceans

TreeHugger asked Andrew Sharpless, CEO for the Oceana ocean protection organization, why we really personally care about the health and fate of the world's big water bodies. Many of us, after all, live far from




elf

Selfie-takers are trampling Dutch tulip fields

After thousands of euros' worth of damage, the tourism board is begging young people to be more respectful.




elf

The Floyd Shelf is a smart bracket that could be a smart shelf or standing desk

But there is a real question of how much one should pay for an artisanal shelf bracket.




elf

Accordion-like flat pack bookshelf unfolds in seconds

Inspired by origami and looking a bit like an accordion, this no-tools-required bookshelf sets up in seconds to adapt to any space.




elf

How Internet-driven activism has improved animal welfare

Over the past 15 years, awareness and interest in livestock conditions and wellbeing have increased exponentially, leading to significant improvements. How much further can it go?




elf

Nutrition and animal welfare were winners on Election Day

Several monumental measures were passed in last week's U.S. election.




elf

Transformer Furniture: A Bookshelf Hides Table and Chairs

From Japanese designer Sakura Adachi comes this marvel of space-saving design: "Trick," the bookshelf that hides a table and two chairs. Featured by Italian manufacturer Campeggi at the 2010 Salone del Mobile in Milan, this is a great




elf

An Adjustable Book Shelf Lets You Put Your Own Spin On Your Living Room

From South Korean design studio DesignJoo comes the Giro One, the piece that lets you put your own spin on your shelves, lighting, and side tables. The concept is simple: shelves can be added and their height adjusted by spinning them




elf

This high-tech, grow-it-yourself medical cannabis kit is as easy as using an app

Seedo, creators of automated consumer grow labs, will shortly plant a seed in the U.S. medical marijuana market.




elf

Self-powered fish tag tracks fish for as long as they swim

Long-living fish can now be tracked for their whole lives with a single tag.




elf

It's not just self-driving cars, it is a whole New Mobility Now

A new study looks at how AVs are part of a much bigger picture of how we get around.




elf

The Honda IeMobi is a mobile autonomous living room and the future of self-driving cars

The car changed the design of our houses; the AV will change our houses and the way we live.




elf

How will self-driving cars affect our cities? Two views

Phil Levin thinks it will change real estate completely; Christian Wolmar thinks it is ludicrous tech nerd fantasy.




elf

Bike industry bigwig says bikes will need to communicate with self-driving cars

They want to build beacons into bikes. Will implants in pedestrians be next?




elf

Conservatives and pinkos unite in hating self-driving cars

Judging from what I am reading all over the political map, I suspect that autonomous cars are the new Juiceroo.




elf

There's a Ford self-driving round games/meeting room in your future

A recent patent shows some of the thinking about fully autonomous cars, which might look nothing like cars do now.




elf

Pedestrians will have to be "lawful and considerate" in a world of self-driving cars

It may be decades before AVs are good enough, so in the meantime everyone will have to keep out of their way.




elf

Self-driving car gets stuck in the slough of despond

In which we learn a new and useful word: techno-chauvinism.




elf

$80 billion has been spent on self-driving cars with nothing to show for it.

We are wasting too much time, energy and money on autonomous vehicles. We know what to do and it's not AVs.




elf

This robotic lawnmower fuels itself with the grass it cuts, then harvests excess biomass for later use

Could this be the lawnmower we've all been waiting for?




elf

If More of Us Love Nature, The Environment Will Take Care of Itself

Like the word 'sustainability', 'environmentalism' has become a hollow word, a word divorced from meaning, a word lacking in soul, separated from its essence.




elf

Google's self-driving car hits the road this summer

These may be limited to 25 MPH, but the era of autonomous cars is coming at us really fast.




elf

The troubling link between self-care and capitalism

The covering-yourself-with-blankets movement isn't nearly as cuddly as it seems.




elf

Solar wagon delivers a self-contained portable water purification system

One solution for purifying water in remote off-grid areas comes in the form of a wagon, complete with solar panel and water purification system.




elf

Self-filtering water bottle also acts as a flashlight

The ÖKO Odyssey water bottle actually has six functions!




elf

'Whimsical scientist' seals himself inside greenhouse for climate change experiment (Video)

This man wants to prove a point about rising global carbon dioxide levels by locking himself inside this DIY greenhouse for the next three days.




elf

Giant Antarctica ice shelf about to disintegrate, says NASA

The 10,000 year-old ice shelf is half the size of Rhode Island; its collapse will add greatly to global sea level rise.




elf

Inquisitive Emperor penguins star in delightful selfie (video)

Australian Antarctic Division captures a vision of curious penguins ready for their close-up.