Event Information

Under the Food and Drug Administration Amendments Act (FDAAA) of 2007, the FDA has the authority to require pharmaceutical manufacturers to develop Risk Evaluation and Mitigation Strategies (REMS) for drugs or biologics that carry serious potential or known risks. Since that time, the REMS program has become an important tool in ensuring that riskier drugs are used safely, and it has allowed FDA to facilitate access to a host of drugs that may not otherwise have been approved. However, concerns have arisen regarding the effects of REMS programs on patient access to products, as well as the undue burden that the requirements place on the health care system. In response to these concerns, FDA has initiated reform efforts aimed at improving the standardization, assessment, and integration of REMS within the health care system. As part of this broader initiative, the agency is pursuing four priority projects, one of which focuses on improving provider-patient benefit-risk counseling for drugs that have a REMS attached.

Under a cooperative agreement with FDA, the Center for Health Policy at Brookings held an expert workshop on July 24 titled, “Risk Evaluation and Mitigation Strategies (REMS): Building a Framework for Effective Patient Counseling on Medication Risks and Benefits”. This workshop was the first in a series of convening activities that will seek input from stakeholders across academia, industry, health systems, and patient advocacy groups, among others. Through these activities, Brookings and FDA will further develop and refine an evidence-based framework of best practices and principles that can be used to inform the development and effective use of REMS tools and processes.

Event Materials

• REMS_PBRC_Meeting_Agenda
• REMS BR Speaker Bios
• REMS BenefitRisk Meeting Summary
• REMS BenefitRisk communication white paper

INTRODUCTION

In August, 2010 the government of Kenya adopted a new constitution. This followed a referendum in which an overwhelming majority of Kenyans voted for change. The decisive impetus for reform came from the widespread violence and political crisis that followed the 2007 election. While claims of electoral fraud provided the immediate catalyst for violence, the deeper causes were to be found in the interaction of a highly centralized ‘winner-take-all’ political system with deep social disparities based in part on group identity (Hanson 2008).

Provisions for equity figure prominently in the new constitution. Backed by a bill of rights that opens the door to legal enforcement, citizenship rights have been strengthened in many areas,including access to basic services. ‘Equitable sharing’ has been introduced as a guiding principle for public spending. National and devolved governments are now constitutionally required to redress social disparities, target disadvantaged areas and provide affirmative action for marginalized groups.

Translating these provisions into tangible outcomes will not be straightforward. Equity is a principle that would be readily endorsed by most policymakers in Kenya and Kenya’s citizens have provided their own endorsement through the referendum. However, there is an ongoing debate over what the commitment to equity means in practice, as well as over the pace and direction of reform. Much of that debate has centered on the constitutional injunction requiring ‘equitable sharing’ in public spending.

On most measures of human development, Kenya registers average outcomes considerably above those for sub-Saharan Africa as a region. Yet the national average masks extreme disparities—and the benefits of increased prosperity have been unequally shared.

There are compelling grounds for a strengthened focus on equity in Kenya. In recent years, the country has maintained a respectable, if less than spectacular, record on economic growth. Social indicators are also on an upward trend. On most measures of human development, Kenya registers average outcomes considerably above those for sub-Saharan Africa as a region. Yet the national average masks extreme disparities—and the benefits of increased prosperity have been unequally shared. Some regions and social groups face levels of deprivation that rank alongside the worst in Africa. Moreover, the deep fault lines running through society are widely perceived as a source of injustice and potential political instability.

High levels of inequality in Kenya raise wider concerns. There has been a tendency in domestic debates to see ‘equitable sharing’ as a guiding principle for social justice, rather than as a condition for accelerated growth and enhanced economic efficiency. Yet international evidence strongly suggests that extreme inequality—especially in opportunities for education— is profoundly damaging for economic growth. It follows that redistributive public spending has the potential to support growth.

The current paper focuses on a group of 12 counties located in Kenya’s Arid and Semi-Arid Lands (ASALs). They are among the most disadvantaged in the country. Most are characterized by high levels of income poverty, chronic food insecurity and acute deprivation across a wide range of social indicators.

Nowhere is the deprivation starker than in education. The ASAL counties account for a disproportionately large share of Kenya’s out-of-school children, pointing to problems in access and school retention. Gender disparities in education are among the widest in the country. Learning outcomes for the small number of children who get through primary school are for the most part abysmal, even by the generally low national average standards.

Unequal public spending patterns have played no small part in creating the disparities that separate the ASAL counties from the rest of Kenya—and ‘equitable sharing’ could play a role in closing the gap. But what would a more equitable approach to public spending look like in practice?

This paper addresses that question. It looks in some detail at education for two reasons. First, good quality education is itself a powerful motor of enhanced equity. It has the potential to equip children and youth with the skills and competencies that they need to break out of cycles of poverty and to participate more fully in national prosperity. If Kenya is to embark on a more equitable pattern of development, there are strong grounds for prioritizing the creation of more equal opportunities in education. Second, the education sector illustrates many of the wider challenges and debates that Kenya’s policymakers will have to address as they seek to translate constitutional provisions into public spending strategies. In particular, it highlights the importance of weighting for indicators that reflect need in designing formulae for budget allocations.

Our broad conclusion is that, while Kenya clearly needs to avoid public spending reforms that jeopardize service delivery in wealthier counties, redistributive measures are justified on the grounds of efficiency and equity.

The paper is organized as follows. Part 1 provides an overview of the approach to equity enshrined in the constitution. While the spirit of the constitution is unequivocal, the letter is open to a vast array of interpretations. We briefly explore the implications of a range of approaches. Our broad conclusion is that, while Kenya clearly needs to avoid public spending reforms that jeopardize service delivery in wealthier counties, redistributive measures are justified on the grounds of efficiency and equity. Although this paper focuses principally on basic services, we caution against approaches that treat equity as a matter of social sector financing to the exclusion of growth-oriented productive investment.

Part 2 provides an analysis of some key indicators on poverty, health and nutrition. Drawing on household expenditure data, the report locates the 12 ASAL counties in the national league table for the incidence and depth of poverty. Data on health outcomes and access to basic services provide another indicator of the state of human development. While there are some marked variations across counties and indicators, most of the 12 counties register levels of deprivation in poverty and basic health far in excess of those found in other areas.

Part 3 shifts the focus to education. Over the past decade, Kenya has made considerable progress in improving access to basic education. Enrollment rates in primary education have increased sharply since the elimination of school fees in 2003. Transition rates to secondary school are also rising. The record on learning achievement is less impressive. While Kenya lacks a comprehensive national learning assessment, survey evidence points to systemic problems in education quality. In both access and learning, children in the ASAL counties—especially female children—are at a considerable disadvantage. After setting out the national picture, the paper explores the distinctive problems facing these counties.

In Part 4 we look beyond Kenya to wider international experience. Many countries have grappled with the challenge of reducing disparities between less-favored and more-favored regions. There are no blueprints on offer. However, there are some useful lessons and guidelines that may be of some relevance to the policy debate in Kenya. The experience of South Africa may be particularly instructive given the weight attached to equity in the post-apartheid constitution.

Part 5 of the paper explores a range of approaches to financial allocations. Converting constitutional principle into operational practice will require the development of formulae-based approaches. From an equitable financing perspective there is no perfect model. Any formula that is adopted will involve trade-offs between different goals. Policymakers have to determine what weight to attach to different dimensions of equity (for example, gender, income, education and health), the time frame for achieving stated policy goals, and whether to frame targets in terms of outcomes or inputs. These questions go beyond devolved financing. The Kenyan constitution is unequivocal in stipulating that the ‘equitable sharing’ provision applies to all public spending. We therefore undertake a series of formula-based exercises illustrating the allocation patterns that would emerge under different formulae, with specific reference to the 12 ASAL focus counties and to education.

Downloads

• 08 financing kenya watkins

Editor's Note: The World Bank’s Independent Evaluation Group (IEG) recently published a self-evaluation of its activities. Besides representing current thinking among evaluation experts at the World Bank, it also more broadly reflects some of the strengths and gaps in the approaches that evaluators use to assess and learn from the performance of the international institutions with which they work. The old question “Quis custodet ipsos custodes?” – loosely translated as “Who evaluates the evaluators?” – remains as relevant as ever. Johannes Linn served as an external peer reviewer of the self-evaluation and provides a bird’s-eye view on the lessons learned.

An Overview of the World Bank’s IEG Self-Evaluation Report

In 2011 the World Bank’s Independent Evaluation Group (IEG) carried out and published a self-evaluation of its activities. The self-evaluation team was led by an internal manager, but involved a respected external evaluation expert as the principal author and also an external peer reviewer.

The IEG self-evaluation follows best professional practices as codified by the Evaluation Cooperation Group (ECG). This group brings together the evaluation offices of seven major multilateral financial institutions in joint efforts designed to enhance evaluation performance and cooperation among their evaluators. One can therefore infer that the approach and focus of the IEG self-evaluation is representative of a broader set of practices that are currently used by the evaluation community of international financial organizations.

At the outset the IEG report states that “IEG is the largest evaluation department among Evaluation Capacity Group (ECG) members and is held in high regard by the international evaluation community. Independent assessments of IEG’s role as an independent evaluation function for the Bank and IFC rated it above the evaluation functions in most other ECG members, international nongovernmental organizations, and transnational corporations and found that IEG follows good practice evaluation principles.”

The self-evaluation report generally confirms this positive assessment. For four out of six areas of its mandate IEG gives itself the second highest rating (“good”) out of six possible rating categories. This includes (a) the professional quality of its evaluations, (b) its reports on how the World Bank’s management follows up on IEG recommendations, (c) cooperation with other evaluation offices, and (d) assistance to borrowing countries in improving their own evaluation capacity. In the area of appraising the World Bank’s self-evaluation and risk management practices, the report offers the third highest rating (“satisfactory”), while it gives the third lowest rating (“modest”) for IEG’s impact on the Bank’s policies, strategies and operations. In addition the self-evaluation concludes that overall the performance of IEG has been “good” and that it operates independently, effectively and efficiently.

The report makes a number of recommendations for improvement, which are likely to be helpful, but have limited impact on its activities. They cover measures to further enhance the independence of IEG and the consistency of evaluation practices as applied across the World Bank Group’s branches – the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) –; to improve the design of evaluations and the engagement with Bank management upstream for greater impact; and monitoring the impact of recent organizational changes in IEG in terms of results achieved. The report also recommends that more be done to evaluate the Bank’s analytical work and that evaluations draw on comparative evidence.

Assessment

In terms of the parameters of self-evaluation set by the prevailing practice among the evaluators on international financial agencies, the IEG self-evaluation is accurate and helpful. From my own experience as an operational manager in the Bank whose activities were evaluated by IEG in years past, and as a user of IEG evaluations (and of evaluations of other international aid organizations) for my research on aid effectiveness, I concur that IEG is independent and effective in meeting its mandate as defined. Moreover, the self-evaluation produces useful quantitative evidence (including survey results, budget analysis, etc.) to corroborate qualitative judgments.

However, the self-evaluation suffers from a number of limitations in approach and gaps in focus, which are broadly representative of the practices prevalent among many of the evaluation offices of international aid agencies.

Approach of the IEG self-evaluation

The core of the self-evaluation report is about the evaluation process followed by IEG, with very little said about the substance of IEG’s evaluations. The following questions could have usefully been raised, but were not: do evaluations cover the right issues with the right intensity, such as growth and poverty; environmental, governance, and gender impacts; regional dimensions versus exclusive country or project focus; effectiveness in addressing the problems of fragile and conflict states; effectiveness in dealing with global public goods; sustainability and scaling up; etc. Therefore the report does not deal with the question of whether IEG effectively responds in its evaluations to the many important strategic debates and issues with which the development community is grappling.

Related to this limitation is the fact that the report assessed the quality of IEG’s mostly in terms of (a) whether its approach and processes meet certain standards established by the Evaluation Cooperation Group; and (b) how it is judged by stakeholders in response to a survey commissioned for this evaluation. Both these approaches are useful, but they do not have any basis in professional assessments of the quality of individual products. This is equivalent to IEG evaluating the World Bank’s projects on the quality of its processes (e.g., appraisal and supervision processes) and on the basis of stakeholder surveys, without evaluating individual products and their impacts.

Gaps in the Self-Evaluation and in Evaluation Practice

Careful reading of the report reveals six important gaps in the IEG self-evaluation, in the prevailing evaluation practice in the World Bank, and more generally in the way international financial organizations evaluate their own performance. The first three gaps relate to aspects of the evaluation approach used and the second three gaps relate to lack of focus in the self-evaluation on key internal organizational issues:

1. Impact Evaluations: The report notes that IEG carries out two to three impact evaluations per year, but it sidesteps the debate in the current evaluation literature and practice as to what extent the “gold standard” of randomized impact evaluation should occupy a much more central role. Given the importance of this debate and divergence of views, it would have been appropriate for the self-evaluation to assess IEG’s current practice of very limited use of randomized evaluations.

2. Evaluation of Scaling Up: The report does not address the question of to what extent current IEG practice not only assesses the performance of individual projects in terms of their outcomes and sustainability, but also in terms of whether the Bank has systematically built on its experience in specific projects to help scale up their impact through support for expansion or replication in follow-up operations or through effective hand-off to the government or other partners. In fact, currently IEG does not explicitly and systematically consider scaling up in its project and program evaluations. For example, in a recent IEG evaluation of World Bank funded municipal development projects (MDPs) , IEG found that the Bank has supported multiple MDPs in many countries over the years, but the evaluation did not address the obvious question whether the Bank systematically planned for the project sequence or built on its experience from prior projects in subsequent operations. While most other evaluation offices like IEG do not consider scaling up, some (in particular those of the International Fund for Agricultural Development and the United Nations Development Program) have started doing so in recent years.

3. Drawing on the Experience of and Benchmarking Against Other Institutions: The self-evaluation report does a good job in benchmarking IEG performance in a number of respects against that of other multilateral institutions. In the main text of the report it states that “IEG plans to develop guidelines for approach papers to ensure greater quality, in particular in drawing on comparative information from other sources and benchmarking against other institutions.” This is a welcome intention, but it is inadequately motivated in the rest of the report and not reflected in the Executive Summary. The reality is that IEG, like most multilateral evaluation offices, so far has not systematically drawn on the evaluations and relevant experience of other aid agencies in its evaluations of World Bank performance. This has severely limited the learning impact of the evaluations.

4. Bank Internal Policies, Management Processes and Incentives: IEG evaluations traditionally do not focus on how the Bank’s internal policies, management and incentives affect the quality of Bank engagement in countries. Therefore evaluations cannot offer any insights into whether and how Bank-internal operating modalities contribute to results. Two recent exceptions are notable exceptions. First, the IEG evaluation of the Bank’s approach to harmonization with other donors and alignment with country priorities assesses the incentives for staff to support harmonization and alignment. The evaluation concludes that there are insufficient incentives, a finding disputed by management. Second, is the evaluation of the Bank’s internal matrix management arrangements, which is currently under way. The self-evaluation notes that Bank management tried to quash the matrix evaluation on the grounds that it did not fall under the mandate of IEG. This is an unfortunate argument, since an assessment of the institutional reasons for the Bank’s performance is an essential component of any meaningful evaluation of Bank-supported programs. While making a good case for the specific instance of the matrix evaluation, the self-evaluation report shies away from a more general statement in support of engaging IEG on issues of Bank-internal policies, management processes and incentives. It is notable that IFAD’s Independent Office of Evaluation appears to be more aggressive in this regard: It currently is carrying out a full evaluation of IFAD’s internal efficiency and previous evaluations (e.g., an evaluation of innovation and scaling up) did not shy away from assessing internal institutional dimensions.

5. World Bank Governance: The IEG self-evaluation is even more restrictive in how it interprets its mandate regarding the evaluation of the World Bank’s governance structures and processes (including its approach to members’ voice and vote, the functioning of its board of directors, the selection of its senior management, etc.). It considers these topics beyond IEG’s mandate. This is unfortunate, since the way the Bank’s governance evolves will substantially affect its long-term legitimacy, effectiveness and viability as an international financial institution. Since IEG reports to the Bank’s board of directors, and many of the governance issues involve questions of the board’s composition, role and functioning, there is a valid question of how effectively IEG could carry out such an evaluation. However, it is notable that the IMF’s Independent Evaluation Office, which similarly reports to the IMF board of directors, published a full evaluation of the IMF’s governance in 2008, which effectively addressed many of the right questions.

6. Synergies between World Bank, IFC and MIGA: The self-evaluation report points out that the recent internal reorganization of IEG aimed to assure more effective and consistent evaluations across the three member branches of the World Bank Group. This is welcome, but the report does not assess how past evaluations addressed the question of whether the World Bank, IFC and MIGA effectively capitalized on the potential synergies among the three organizations. The recent evaluation of the World Bank Group’s response to the global economic crisis of 2008/9 provided parallel assessments of each agency’s performance, but did not address whether they work together effectively in maximizing their synergies. The reality is that the three organizations have deeply engrained institutional cultures and generally go their own ways rather than closely coordinating their activities on the ground. Future evaluations should explicitly consider whether the three effectively cooperate or not. While the World Bank is unique in the way it has organizationally separated its private sector and guarantee operations, other aid organizations also have problems of a lack of cooperation, coordination and synergy among different units within the agency. Therefore, the same comment also applies to their evaluation approaches.

Conclusions

Self-evaluations are valuable tools for performance assessment and IEG is to be congratulated for carrying out and publishing such an evaluation of its own activities. As for all self-evaluations, it should be seen as an input to an independent external evaluation, a decision that, for now, has apparently been postponed by the Bank’s board of directors.

IEG’s self-evaluation has many strengths and provides an overall positive assessment of IEG’s work. However, it does reflect some important limitations of analysis and of certain gaps in approach and coverage, which an independent external review should consider explicitly, and which IEG’s management should address. Since many of these issues also likely apply to most of the other evaluation approaches by other evaluation offices, the lessons have relevance beyond IEG and the World Bank.

Key lessons include:

• An evaluation of evaluations should focus not only on process, but also on the substantive issues that the institution is grappling with.
• An evaluation of the effectiveness of evaluations should include a professional assessment of the quality of evaluation products.
• An evaluation of evaluations should assess:
  o How effectively impact evaluations are used;
  o How scaling up of successful interventions is treated;
  o How the experience of other comparable institutions is utilized;
  o Whether and how the internal policies, management practices and incentives of the institution are effectively assessed;
  o Whether and how the governance of the institution is evaluated; and
  o Whether and how internal coordination, cooperation and synergy among units within the organizations are assessed.

Evaluations play an essential role in the accountability and learning of international aid organizations. Hence it is critical that evaluations address the right issues and use appropriate techniques. If the lessons above were reflected in the evaluation practices of the aid institutions, this would represent a significant step forward in the quality, relevance and likely impact of evaluations.

Editors’ Note: On March 23, the Working Group on Egypt sent a letter to President Obama urging him to publicly and privately object to Egyptian President Abdel Fattah el-Sissi’s accelerating crackdown on human rights and civil society organizations. Brookings senior fellow and director of the Center for Middle East Policy Tamara Wittes was among the letter’s signers, and she explains her decision to do so. The letter was originally published by the Project on Middle East Democracy (POMED).

Tamara Wittes: In a disordered Middle East, America needs anchors of stability and reliable partners to help it achieve its goals. Both are in sadly short supply. 

For more than thirty years, Egypt was an anchor of stability and a reliable American partner in regional security. From the time Sadat expelled Soviet advisers and broached peace with Israel, ties with Egypt have been a core pillar of American Middle East policy. But, as my colleague Steven Cook presciently noted way back in February 2012, Egypt’s revolution accelerated the launch of what he calls a “long goodbye” between these two formerly indispensable partners. He argued back then that shifting from a “special relationship” to something more transaction would have four concrete benefits for Washington:

First, Washington will no longer be in the unseemly position of providing taxpayer largesse—however small in the grand scheme of things—to a government that resents the United States and clearly does not share its values. Second, it will provide an opportunity for a much-needed change in military-to-military relations in which the United States merely pays for the services it needs like expedited transit through the Suez Canal. Third, it is consistent with this moment of empowerment and dignity for Egyptians many of whom do not want U.S. assistance either because they believe it actually stands in the way of a democratic transition or accept Aboul Naga’s argument along with those who couldn’t care less about U.S. assistance because it doesn’t touch their lives. Finally, it will free up funds for the United States to help others who actually might want Washington’s help, perhaps the Tunisians, Moroccans, or some sub-Saharan African countries would be grateful for development assistance.

Since that blog post went up, Egypt has had three different governments and lost its place as a diplomatic and security leader in the region; while the United States has withdrawn from Iraq and begun to do the same in Afghanistan, while emphasizing burden-sharing in its new fight against ISIS. All of these shifts strengthen the argument for a more distant and transactional U.S.-Egyptian relationship. 

Moreover, since his accession to power (first in a military coup in July 2013 and then in a highly constrained election in 2014), President Abdel Fattah el-Sisi has made decisions that are undermining both Egypt’s domestic stability and key American policy goals in the region. 

• Sisi’s failure to move forward on economic reforms (recommended by leading Egyptian voices, regional supporters, and international donors) has left his country in a spiral of shrinking cash reserves, capital flight and currency devaluation that together threaten the government’s ability to import needed food and medicine and to carry out core government functions. 
• Sisi’s counterterrorism campaign in the Sinai has succeeded in “making the sand jump,” as one regional security official told me, but it seems to have stoked more than tamped down the fire of violent extremism threatening both Egypt and Israel; meanwhile, its alleged military abuses have sparked a Senate request for investigation.
• The intense political polarization and relentless repression of post-coup Egypt are producing other destabilizing effects, which are detailed in the Egypt Working Group’s newest letter to President Obama posted below (I am a member of the Working Group). 

To top it all off, the Egyptian government continues to throw obstacles in the road of U.S.-Egyptian cooperation. Its military resists learning from the hard-won American experience in effective counterinsurgency. Its leadership has resolutely refused to allow core bilateral aid programs, like those supporting higher education, to move forward. And at the same time, the Egyptian government continues to promote conspiracy theories about the United States to its public through media smears and show trials, and now, apparently, to its newly elected parliamentarians. 

It’s long past time for the United States to undertake a strategic review of its approach to the Middle East, one focused on building anchors of stability and sustaining reliable partners in pursuit of American priorities. Egypt, as I told The New York Times, no longer qualifies as either one. That doesn’t mean the two countries can’t continue to work together in those narrow areas where they agree on interests, priorities, and approaches. 

But Secretary of State Kerry’s public embrace last week of Egyptian Foreign Minister Shoukry cannot hide the facts—there is no “back to business” option for the U.S.-Egyptian relationship, and it seems increasingly clear that even direct White House engagement would not shift Egypt’s leadership off of its self-destructive trajectory. Egypt's looming instability demands that the United States take steps now to safeguard itself from reliance on a country we cannot rescue, not least from its own leaders' worst impulses. 

March 23, 2016

Dear Mr. President,

We are writing to urge you to speak directly with Egyptian President Abdel Fattah el-Sissi and to express both publicly and privately your objection to his accelerating crackdown on human rights, including recent moves to prosecute civil society organizations. You were correct to declare in September 2014 that “America’s support for civil society is a matter of national security,” and nowhere is that more true than in Egypt today.

President el-Sissi’s campaign against civil society takes place against the backdrop of unprecedented abuses by Egyptian security forces, including extrajudicial killings, the detention of tens of thousands of political prisoners, the widespread documented use of torture, and the forced disappearances of hundreds of Egyptians. The killing of Italian student Giulio Regeni, whose tortured body appeared on a roadside near Cairo a week after his abduction in late January, has come to international attention, but many Egyptians have shared his fate since President el-Sissi came to power.

On March 24, an Egyptian court will hear a request to freeze the bank accounts and other assets of two internationally-respected human rights defenders, Hossam Bahgat and Gamal Eid, along with members of Eid’s family. Mr. Bahgat and Mr. Eid and other activists may soon be indicted and face trial for illegally accepting foreign funding—a criminal charge that violates their right to free association and could carry a sentence of up to 25 years in prison.

The imminent proceedings are a major step in Egyptian authorities’ campaign to crush the last remnants of Egypt’s independent civil society and human rights community. Egypt’s media has recently reported that dozens of organizations are under criminal investigation, essentially for their peaceful work to monitor abuses and to hold Egypt’s government accountable to its own constitution and international human rights commitments. In recent weeks, Egyptian authorities have ordered the closure of a prominent anti-torture organization, the Nadeem Center; summoned staff from several human rights organizations for interrogation; banned prominent rights activists and advocates from traveling outside Egypt in violation of the Egyptian constitution; and harassed and threatened human rights activists with arrest and violence. The media regularly propagate vitriol against human rights defenders, portraying them as traitors and security threats.

If this crackdown is allowed to reach its conclusion, it will silence an indigenous human rights community that has survived more than 30 years of authoritarian rule, leaving few if any Egyptians free to investigate mounting abuses by the state.

The current attacks on Egypt’s rights advocates are a continuation of the same criminal prosecution of American and German NGO workers in Egypt that began in 2011. That prosecution, driven by senior members of the Egyptian government still in high office today, resulted in the June 2013 criminal convictions, in a deeply flawed trial, of 43 Egyptian and international NGO staff, including 17 American citizens. President el-Sissi, who was the head of military intelligence in 2011 when Egypt’s military government launched the investigation, has refused repeated requests to overturn the convictions.

While the current crackdown is primarily targeting domestic organizations, there are indications that international NGOs may also face increased pressure, including some that currently do not even have offices or staff working in Egypt. On March 20, the newspaper Al Masry Al Youm published the names of more than 150 individuals and civil society organizations reportedly under investigation for receiving foreign funding, including prominent American and European organizations such as the Center for International Private Enterprise, the Solidarity Center, Transparency International, Save the Children, Catholic Relief Services, CARE, AMIDEAST, the National Democratic Institute, and the International Republican Institute.

Mr. President, in your September 2014 Presidential Memorandum on Civil Society, you pledged that the United States government—including you personally—would stand firmly with those in civil society facing pressure or harassment from their governments. While the past five years have been tumultuous and challenging for U.S. policy toward Egypt, this is another defining moment for the United States, a moment that tests your pledge to “stand with civil society.” Secretary Kerry’s March 18 statement of concern was welcome, but further action is urgently needed. Past practice demonstrates that when the United States government speaks clearly, in one voice, and consistently on NGO freedom and human rights in Egypt, the government in Cairo listens.

It is essential that you act to stand up for human rights, freedom of association, and the rights of both Egyptian and international civil society organizations to work together on behalf of common goals. You must make crystal clear to President el-Sissi that continued assaults on civil society, including harassment of U.S. organizations, will make it difficult for the administration to cooperate across a range of issues, including your administration’s efforts to promote American investment in Egypt and to provide financial assistance to the Egyptian government and military. If Egypt’s government continues down a path to destroy its own civil society, American support and assistance will become, in both principled and practical terms, impossible.

Sincerely,

The Working Group on Egypt

Editor's Note: The World Bank’s Independent Evaluation Group (IEG) recently published a self-evaluation of its activities. Besides representing current thinking among evaluation experts at the World Bank, it also more broadly reflects some of the strengths and gaps in the approaches that evaluators use to assess and learn from the performance of the international institutions with which they work. The old question “Quis custodet ipsos custodes?” – loosely translated as “Who evaluates the evaluators?” – remains as relevant as ever. Johannes Linn served as an external peer reviewer of the self-evaluation and provides a bird’s-eye view on the lessons learned.

An Overview of the World Bank’s IEG Self-Evaluation Report

In 2011 the World Bank’s Independent Evaluation Group (IEG) carried out and published a self-evaluation of its activities. The self-evaluation team was led by an internal manager, but involved a respected external evaluation expert as the principal author and also an external peer reviewer.

The IEG self-evaluation follows best professional practices as codified by the Evaluation Cooperation Group (ECG). This group brings together the evaluation offices of seven major multilateral financial institutions in joint efforts designed to enhance evaluation performance and cooperation among their evaluators. One can therefore infer that the approach and focus of the IEG self-evaluation is representative of a broader set of practices that are currently used by the evaluation community of international financial organizations.

At the outset the IEG report states that “IEG is the largest evaluation department among Evaluation Capacity Group (ECG) members and is held in high regard by the international evaluation community. Independent assessments of IEG’s role as an independent evaluation function for the Bank and IFC rated it above the evaluation functions in most other ECG members, international nongovernmental organizations, and transnational corporations and found that IEG follows good practice evaluation principles.”

The self-evaluation report generally confirms this positive assessment. For four out of six areas of its mandate IEG gives itself the second highest rating (“good”) out of six possible rating categories. This includes (a) the professional quality of its evaluations, (b) its reports on how the World Bank’s management follows up on IEG recommendations, (c) cooperation with other evaluation offices, and (d) assistance to borrowing countries in improving their own evaluation capacity. In the area of appraising the World Bank’s self-evaluation and risk management practices, the report offers the third highest rating (“satisfactory”), while it gives the third lowest rating (“modest”) for IEG’s impact on the Bank’s policies, strategies and operations. In addition the self-evaluation concludes that overall the performance of IEG has been “good” and that it operates independently, effectively and efficiently.

The report makes a number of recommendations for improvement, which are likely to be helpful, but have limited impact on its activities. They cover measures to further enhance the independence of IEG and the consistency of evaluation practices as applied across the World Bank Group’s branches – the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) –; to improve the design of evaluations and the engagement with Bank management upstream for greater impact; and monitoring the impact of recent organizational changes in IEG in terms of results achieved. The report also recommends that more be done to evaluate the Bank’s analytical work and that evaluations draw on comparative evidence.

Assessment

In terms of the parameters of self-evaluation set by the prevailing practice among the evaluators on international financial agencies, the IEG self-evaluation is accurate and helpful. From my own experience as an operational manager in the Bank whose activities were evaluated by IEG in years past, and as a user of IEG evaluations (and of evaluations of other international aid organizations) for my research on aid effectiveness, I concur that IEG is independent and effective in meeting its mandate as defined. Moreover, the self-evaluation produces useful quantitative evidence (including survey results, budget analysis, etc.) to corroborate qualitative judgments.

However, the self-evaluation suffers from a number of limitations in approach and gaps in focus, which are broadly representative of the practices prevalent among many of the evaluation offices of international aid agencies.

Approach of the IEG self-evaluation

The core of the self-evaluation report is about the evaluation process followed by IEG, with very little said about the substance of IEG’s evaluations. The following questions could have usefully been raised, but were not: do evaluations cover the right issues with the right intensity, such as growth and poverty; environmental, governance, and gender impacts; regional dimensions versus exclusive country or project focus; effectiveness in addressing the problems of fragile and conflict states; effectiveness in dealing with global public goods; sustainability and scaling up; etc. Therefore the report does not deal with the question of whether IEG effectively responds in its evaluations to the many important strategic debates and issues with which the development community is grappling.

Related to this limitation is the fact that the report assessed the quality of IEG’s mostly in terms of (a) whether its approach and processes meet certain standards established by the Evaluation Cooperation Group; and (b) how it is judged by stakeholders in response to a survey commissioned for this evaluation. Both these approaches are useful, but they do not have any basis in professional assessments of the quality of individual products. This is equivalent to IEG evaluating the World Bank’s projects on the quality of its processes (e.g., appraisal and supervision processes) and on the basis of stakeholder surveys, without evaluating individual products and their impacts.

Gaps in the Self-Evaluation and in Evaluation Practice

Careful reading of the report reveals six important gaps in the IEG self-evaluation, in the prevailing evaluation practice in the World Bank, and more generally in the way international financial organizations evaluate their own performance. The first three gaps relate to aspects of the evaluation approach used and the second three gaps relate to lack of focus in the self-evaluation on key internal organizational issues:

1. Impact Evaluations: The report notes that IEG carries out two to three impact evaluations per year, but it sidesteps the debate in the current evaluation literature and practice as to what extent the “gold standard” of randomized impact evaluation should occupy a much more central role. Given the importance of this debate and divergence of views, it would have been appropriate for the self-evaluation to assess IEG’s current practice of very limited use of randomized evaluations.

2. Evaluation of Scaling Up: The report does not address the question of to what extent current IEG practice not only assesses the performance of individual projects in terms of their outcomes and sustainability, but also in terms of whether the Bank has systematically built on its experience in specific projects to help scale up their impact through support for expansion or replication in follow-up operations or through effective hand-off to the government or other partners. In fact, currently IEG does not explicitly and systematically consider scaling up in its project and program evaluations. For example, in a recent IEG evaluation of World Bank funded municipal development projects (MDPs) , IEG found that the Bank has supported multiple MDPs in many countries over the years, but the evaluation did not address the obvious question whether the Bank systematically planned for the project sequence or built on its experience from prior projects in subsequent operations. While most other evaluation offices like IEG do not consider scaling up, some (in particular those of the International Fund for Agricultural Development and the United Nations Development Program) have started doing so in recent years.

3. Drawing on the Experience of and Benchmarking Against Other Institutions: The self-evaluation report does a good job in benchmarking IEG performance in a number of respects against that of other multilateral institutions. In the main text of the report it states that “IEG plans to develop guidelines for approach papers to ensure greater quality, in particular in drawing on comparative information from other sources and benchmarking against other institutions.” This is a welcome intention, but it is inadequately motivated in the rest of the report and not reflected in the Executive Summary. The reality is that IEG, like most multilateral evaluation offices, so far has not systematically drawn on the evaluations and relevant experience of other aid agencies in its evaluations of World Bank performance. This has severely limited the learning impact of the evaluations.

4. Bank Internal Policies, Management Processes and Incentives: IEG evaluations traditionally do not focus on how the Bank’s internal policies, management and incentives affect the quality of Bank engagement in countries. Therefore evaluations cannot offer any insights into whether and how Bank-internal operating modalities contribute to results. Two recent exceptions are notable exceptions. First, the IEG evaluation of the Bank’s approach to harmonization with other donors and alignment with country priorities assesses the incentives for staff to support harmonization and alignment. The evaluation concludes that there are insufficient incentives, a finding disputed by management. Second, is the evaluation of the Bank’s internal matrix management arrangements, which is currently under way. The self-evaluation notes that Bank management tried to quash the matrix evaluation on the grounds that it did not fall under the mandate of IEG. This is an unfortunate argument, since an assessment of the institutional reasons for the Bank’s performance is an essential component of any meaningful evaluation of Bank-supported programs. While making a good case for the specific instance of the matrix evaluation, the self-evaluation report shies away from a more general statement in support of engaging IEG on issues of Bank-internal policies, management processes and incentives. It is notable that IFAD’s Independent Office of Evaluation appears to be more aggressive in this regard: It currently is carrying out a full evaluation of IFAD’s internal efficiency and previous evaluations (e.g., an evaluation of innovation and scaling up) did not shy away from assessing internal institutional dimensions.

5. World Bank Governance: The IEG self-evaluation is even more restrictive in how it interprets its mandate regarding the evaluation of the World Bank’s governance structures and processes (including its approach to members’ voice and vote, the functioning of its board of directors, the selection of its senior management, etc.). It considers these topics beyond IEG’s mandate. This is unfortunate, since the way the Bank’s governance evolves will substantially affect its long-term legitimacy, effectiveness and viability as an international financial institution. Since IEG reports to the Bank’s board of directors, and many of the governance issues involve questions of the board’s composition, role and functioning, there is a valid question of how effectively IEG could carry out such an evaluation. However, it is notable that the IMF’s Independent Evaluation Office, which similarly reports to the IMF board of directors, published a full evaluation of the IMF’s governance in 2008, which effectively addressed many of the right questions.

6. Synergies between World Bank, IFC and MIGA: The self-evaluation report points out that the recent internal reorganization of IEG aimed to assure more effective and consistent evaluations across the three member branches of the World Bank Group. This is welcome, but the report does not assess how past evaluations addressed the question of whether the World Bank, IFC and MIGA effectively capitalized on the potential synergies among the three organizations. The recent evaluation of the World Bank Group’s response to the global economic crisis of 2008/9 provided parallel assessments of each agency’s performance, but did not address whether they work together effectively in maximizing their synergies. The reality is that the three organizations have deeply engrained institutional cultures and generally go their own ways rather than closely coordinating their activities on the ground. Future evaluations should explicitly consider whether the three effectively cooperate or not. While the World Bank is unique in the way it has organizationally separated its private sector and guarantee operations, other aid organizations also have problems of a lack of cooperation, coordination and synergy among different units within the agency. Therefore, the same comment also applies to their evaluation approaches.

Conclusions

Self-evaluations are valuable tools for performance assessment and IEG is to be congratulated for carrying out and publishing such an evaluation of its own activities. As for all self-evaluations, it should be seen as an input to an independent external evaluation, a decision that, for now, has apparently been postponed by the Bank’s board of directors.

IEG’s self-evaluation has many strengths and provides an overall positive assessment of IEG’s work. However, it does reflect some important limitations of analysis and of certain gaps in approach and coverage, which an independent external review should consider explicitly, and which IEG’s management should address. Since many of these issues also likely apply to most of the other evaluation approaches by other evaluation offices, the lessons have relevance beyond IEG and the World Bank.

Key lessons include:

• An evaluation of evaluations should focus not only on process, but also on the substantive issues that the institution is grappling with.
• An evaluation of the effectiveness of evaluations should include a professional assessment of the quality of evaluation products.
• An evaluation of evaluations should assess:
  o How effectively impact evaluations are used;
  o How scaling up of successful interventions is treated;
  o How the experience of other comparable institutions is utilized;
  o Whether and how the internal policies, management practices and incentives of the institution are effectively assessed;
  o Whether and how the governance of the institution is evaluated; and
  o Whether and how internal coordination, cooperation and synergy among units within the organizations are assessed.

Evaluations play an essential role in the accountability and learning of international aid organizations. Hence it is critical that evaluations address the right issues and use appropriate techniques. If the lessons above were reflected in the evaluation practices of the aid institutions, this would represent a significant step forward in the quality, relevance and likely impact of evaluations.