Introduction: Despite the launch of indirect, “proximity” talks between Palestinians and Israelis, Palestinian President Mahmoud Abbas continues to resist a resumption of direct negotiations with Israel absent a full settlement freeze. As chairman of the Palestine Liberation Organization (PLO) and president of the Palestinian Authority (PA), Abbas also insists that any new negotiations pick up where previous talks left off in December 2008 and that the parties spell out ahead of time a clear “endgame,” including a timetable for concluding negotiations. While these may seem like unreasonable preconditions, Palestinian reluctance to dive headfirst into yet another round of negotiations is rooted in some genuine, hard-learned lessons drawn from nearly two decades of repeated failures both at the negotiating table and on the ground.

Not only have negotiations failed to bring Palestinians closer to their national aspirations but the peace process itself has presided over (and in some ways facilitated) a deepening of Israel’s occupation and an unprecedented schism within the Palestinian polity. Such failures have cost the Palestinian leadership dearly in terms of both its domestic legitimacy and its international credibility. While it remains committed to a negotiated settlement with Israel based on a two-state solution, the PLO/PA leadership has been forced to rethink previous approaches to the peace process and to negotiations, as much for its own survival as out of a desire for peace.

Haunted by past failures, Palestinian negotiators are now guided, to varying degrees, by six overlapping and sometimes conflicting lessons:

1. Realities on the ground must move in parallel with negotiations at the table.

2. Don’t engage in negotiations for their own sake.

3. Agreements are meaningless without implementation.

4. Incrementalism does not work.

5. Avoid being blamed at all costs.

6. Don’t go it alone.

Downloads

• Download Full Paper - English

On April 10th and 11th, the heads of state and government from nearly every state in the Americas will meet in Panama City for the Seventh Summit of the Americas. The leaders present in Panama preside over a region that has advanced far and fast on key political and economic indicators since the first of these meetings was held in Miami in 1994. At the Miami Summit, the legacy of the Cold War was very much present, and the specter of war, military dictatorship, armed revolution, financial crises, and political instability still hung in the air. 

In 2015, the region is by and large more democratic, economically prosperous, free from war, and the last insurgency in the region—Colombia’s—is winding down as peace is discussed between the government and its opponents at talks hosted by Havana. The beginning of a rapprochement between the United States and Cuba in December 2014 broke down one of the last remaining obstacles to an event that is truly inclusive of every country in the Western Hemisphere. 

In comparison to the rest of the world—where in the past year we have witnessed terrorist attacks in Paris, war in Ukraine, insurgency in Yemen, and saber-rattling around the South China Sea—the Western Hemisphere appears to be relatively better off. While there are a small number of countries that face challenging circumstances, especially among the fragile states of the Caribbean basin, these problems mostly threaten local rather than regional order. Given this picture, what lessons can we learn from the Western Hemisphere, and from U.S. policy towards the region, as we contemplate how best to improve global order?

Drawing the right lessons from history

The Americas have a long history of developing regional norms that promote the peaceful resolution of conflicts. Since the founding of the Panamerican Union in 1890, which transformed into the Organization of American States (OAS) in 1948, the countries of the hemisphere have embedded these norms of peace into their multilateral institutions. While frequently criticized, it is important to remember that the OAS has presided over the elimination of inter-state conflict in the Americas. 

Today, Latin American states resolve territorial disputes at the International Court of Justice rather than on the battlefield. The last war in the region, between Peru and Ecuador in 1995, occurred two decades ago. Given how rare militarized disputes are at home, Latin American soldiers frequently serve as peacekeepers in United Nations missions around the world. Latin Americans have become good at peacemaking and peacekeeping, something that other regions of the world would do well to emulate.

When it comes to domestic politics, most leaders in the region now understand the political and economic principles that contribute to stability. Governments have become much better about economic governance, which means that as South America’s economy cools off this year, fiscal problems will be manageable and localized rather than region-wide and existential, a sharp contrast with the 1980s and 1990s. 

Leaders in the region have learned that promoting polarization for short-term political advantage is all too likely to produce instability, coups, and revolution. To minimize the risk that domestic political violence might reoccur in the future, states in the region have self-consciously examined the legacy of their authoritarian pasts, using innovative processes such as truth and reconciliation commissions—initially in Argentina in 1983—but also drawing on traditional courts to prosecute perpetrators of past abuses.  

In the 21st century, successful coups d’état have become rare, and when they do occur, as was the case in Honduras in 2009, the region collaborates to ensure a return to democracy. Here again is an area where Latin America has led the way through policies that reduce the likelihood of domestic conflicts that threaten internal stability or global order.

The importance of revisiting unworkable U.S. policies 

At this Summit in Panama, President Barack Obama will be able to credibly claim that he has listened to his Latin American counterparts and has begun to change policies that had become obstacles to improving regional order. At the 2009 and 2012 Summits (they occur every three years), U.S. policies on drugs, immigration, and Cuba had made President Obama the target of growing criticism from other leaders. In fact, many governments had made it clear that they would not attend the 2015 Summit if Cuba was not invited. 

Since 2012, the Obama administration has taken steps to address these concerns. It has taken executive action to reform immigration policy, signaled greater openness to drug policy liberalization by states such as Uruguay, and initiated a historic normalization of relations with Cuba. In each of these areas, the United States has shifted from policies that were largely unilateral towards its neighbors to policies that emphasize collaboration and partnership. This reflects U.S. learning that unilateralism produces blowback, strengthens its political adversaries in the region, and undermines its interests in the long run. This is a lesson worth considering as we think about our policies towards troubled regions of the world.

The risk of forgetting lessons learned

Yet not all countries and all politicians have remembered these lessons, and some of them have learned the wrong ones. In Argentina, macroeconomic stability is at risk due to a feud between the government and its international creditors. The result is a country cut off from international capital markets at a time when its economy is suffering the effect of declining commodity prices. Venezuela faces a deep crisis that has at its heart the highly polarizing politics practiced by the governing party and an unreasoning attachment to an unworkable economic model. Key countries such as Brazil have lost interest in hemisphere-wide institutions, as indicated by their refusal to appoint an ambassador to the OAS or pay their membership dues. And the region as a whole has become so attached to multilateralism and politics by consensus that is has forgotten how to work together when individual member states deviate from regional norms of democracy and human rights, as is occurring today in Venezuela.

So while the recent history of the Americas offers insights into policies that contribute to a peaceful and stable regional order, it also illustrates that these achievements are not irreversible. Let us hope that future generations do not have the relearn these lessons anew. At this and future Summits, there must be a commitment to preserving the gains made in peace, democracy, human rights, and economic prosperity, but also a new emphasis on developing workable mechanisms to address deviations from the norms and practices that have contributed to making the Americas a relative safe and orderly region of the world.

For more information, check out Emily Miller's post on U.S. priorities at the Seventh Summit of the Americas.

For nearly fifteen hours between Wednesday morning and early Thursday, Senator Chris Murphy (D-CT), along with his Connecticut colleague Senator Richard Blumenthal (D) and Senator Cory Booker (D-NJ), led a filibuster on the floor of the Senate aimed at addressing gun control issues in the aftermath of last weekend’s mass shooting in Orlando. Other than learning that Wednesday is pizza night in the Murphy household, what else should we take away from this Mr. Smith Goes to Washington­-style exercise? Here are three lessons:

1. The real meaning of “I” in “I hold the floor until I yield the floor.”

Anyone who tuned into yesterday’s filibuster joined Senate procedure wonks (and faithful viewers of the West Wing) in the knowledge that a senator who holds the floor can yield to another senator for a question without yielding the floor. Indeed, 38 of Murphy’s 45 Democratic colleagues (as well as two Republicans, Senators Ben Sasse (R-NE) and Pat Toomey (R-PA)), came to the chamber yesterday to ask “questions.” In many cases, these were lengthy speeches—Senator Tammy Baldwin (D-WI), for example, read brief biographies of all 49 Orlando victims—in which the speaker satisfied the question requirement with a conclusion that asked Murphy for his reactions to their statement.

This kind of teamwork on extended speech-making is not unusual. When Senator Ted Cruz (R-TX) took the floor to talk for 21 hours about the Affordable Care Act in 2013, he took questions from nine fellow Republicans (as well as two Democrats). Last May, Senator Rand Paul (R-KY) got an assist from ten colleagues, including seven Democrats, during his filibuster of a bill extending the PATRIOT Act. The depth of Murphy’s bench not only reduced the energy he had to expend speaking, but also helped guarantee that the entire discussion was on-message and focused on the topic at hand; Murphy did not have to resort to reading the phone book to fill the hours.

2. In policy terms, it’s hard to know if the filibuster was a success…

When Murphy left the floor early Thursday morning, it was reported that Senate leaders had agreed to consider two gun control amendments: one that would address the ability of suspected terrorists to purchase guns and a second that would expand background checks for gun purchases. Details of the deal ensuring consideration are still emerging, but it is difficult to know if Murphy’s filibuster caused Senate leaders to agree to hold votes on them. It is possible that, had Democrats simply threatened to object to the motion to proceed to debate on the underlying spending bill, Republican leaders would have been forced to agree to consider the amendments for which Murphy and his allies were pushing. In the contemporary Senate, this is often how obstruction proceeds: without extended speeches and off the floor, with the two sides negotiating behind the scenes.

3. …but the political victory is perhaps more important

As my colleague Sarah Binder and her co-author Steve Smith wrote in their 1997 book on the filibuster, “encouragement from external groups…has given senators an incentive to exploit their procedural rights, sometimes leading them to block legislation with the filibuster or with holds and at other times leading them to use procedural prerogatives to force the Senate to consider issues of importance to parochial, partisan, or national constituencies.” On these grounds, Murphy’s filibuster was unequivocally a success in the eyes of its supporters. As the filibuster neared its end, Murphy reported that his office had received 10,000 phone calls supporting his efforts, and the hashtag #filibuster was trending on Twitter for much of the day. Even if the underlying amendments are not adopted—a real possibility that Murphy acknowledged in one of his final speeches of the evening—the visibility of the exercise is likely to pay political dividends for Democrats in the coming weeks.

In 2015 the international community agreed on a set of ambitious sustainable development goals (SDGs) for the global society, to be achieved by 2030. One of the lessons that the implementation of the Millennium Development Goals (MDG s) has highlighted is the importance of a systematic approach to identify and sequence development interventions—policies, programs, and projects—to achieve such goals at a meaningful scale. The Chinese approach to development, which consists of identifying a problem and long-term goal, testing alternative solutions, and then implementing those that are promising in a sustained manner, learning and adapting as one proceeds—Deng Xiaoping’s “crossing the river by feeling the stones”—is an approach that holds promise for successful achievement of the SDGs.

Having observed the Chinese way, then World Bank Group President James Wolfensohn in 2004, together with the Chinese government, convened a major international conference in Shanghai on scaling up successful development interventions, and in 2005 the World Bank Group (WBG ) published the results of the conference, including an assessment of the Chinese approach. (Moreno-Dodson 2005). Some ten years later, the WBG once again is addressing the question of how to support scaling up of successful development interventions, at a time when the challenge and opportunity of scaling up have become a widely recognized issue for many development institutions and experts.

Since traditional private and public service providers frequently do not reach the poorest people in developing countries, social enterprises can play an important role in providing key services to those at the “base of the pyramid.”

In parallel with the recognition that scaling up matters, the development community is now also focusing on social enterprises (SEs), a new set of actors falling between the traditionally recognized public and private sectors. We adopt here the World Bank’s definition of “social enterprises” as a social-mission-led organization that provides sustainable services to Base of the Pyramid (BoP) populations. This is broadly in line with other existing definitions for the sector and reflects the World Bank’s primary interest in social enterprises as a mechanism for supporting service delivery for the poor. Although social enterprises can adopt various organizational forms—business, nongovernmental organizations (NGOs), and community-based organizations are all forms commonly adopted by social enterprises—they differ from private providers principally by combining three features: operating with a social purpose, adhering to business principles, and aiming for financial sustainability. Since traditional private and public service providers frequently do not reach the poorest people in developing countries, social enterprises can play an important role in providing key services to those at the “base of the pyramid.” (Figure 1)

Figure 1. Role of SE sector in public service provision

Social enterprises often start at the initiative of a visionary entrepreneur who sees a significant social need, whether in education, health, sanitation, or microfinance, and who responds by developing an innovative way to address the perceived need, usually by setting up an NGO, or a for-profit enterprise. Social enterprises and their innovations generally start small. When successful, they face an important challenge: how to expand their operations and innovations to meet the social need at a larger scale. 

Development partner organizations—donors, for short—have recognized the contribution that social enterprises can make to find and implement innovative ways to meet the social service needs of people at the base of the pyramid, and they have started to explore how they can support social enterprises in responding to these needs at a meaningful scale. 

The purpose of this paper is to present a menu of approaches for addressing the challenge of scaling up social enterprise innovations, based on a review of the literature on scaling up and on social enterprises. The paper does not aim to offer specific recommendations for entrepreneurs or blueprints and guidelines for the development agencies. The range of settings, problems, and solutions is too wide to permit that. Rather, the paper provides an overview of ways to think about and approach the scaling up of social enterprise innovations. Where possible, the paper also refers to specific tools that can be helpful in implementing the proposed approaches. 

Note that we talk about scaling up social enterprise innovations, not about social enterprises. This is because it is the innovations and how they are scaled up that matter. An innovation may be scaled up by the social enterprise where it originated, by handoff to a public agency for implementation at a larger scale, or by other private enterprises, small or large. 

This paper is structured in three parts: Part I presents a general approach to scaling up development interventions. This helps establish basic definitions and concepts. Part II considers approaches for the scaling up of social enterprise innovations. Part III provides a summary of the main conclusions and lessons from experience. A postscript draws out implications for external aid donors. Examples from actual practice are used to exemplify the approaches and are summarized in Annex boxes.

Downloads

• Download the full paper (PDF)

Economist Albert Hirschman’s marvelously perceptive little book with big ideas written in 1970 titled “Exit, Voice and Loyalty: Responses to Decline in Firms, Organizations and States” provides a cornucopia of insights into understanding Brexit and the current state of global governance.  When it emerged American economist Kenneth Arrow marveled at its extraordinary richness, and political scientist Karl Deutsch, in his presidential address to the American Political Science Association, called it an “outstanding contribution to political theory.”

Economists assume exit to mean dissatisfaction with an organization’s product or the service leading to decline in demand for it. The value of exit lies in the certainty it provides in terms of the relationship between the customer or member and the firm. Political scientists think of how a firm handles its response to customer dissatisfaction as the exercise of voice by stakeholders. The value of voice is that it can lead to reform that ultimately determines the firm’s revival, an idea also advanced by scholar Clayton Christiansen in his book “The Innovator’s Dilemma.” An understanding of the conditions under which exit and voice are exercised requires the incorporation of the concept of loyalty. Loyalty makes voice more probable and exit less likely. But loyalty does not by itself make the exercise of voice more effective. That depends on the extent to which customers or members are willing to trade off the certainty of exit against the uncertainties of improvement in the deteriorating product, and their ability to influence the organization.  

Applying these ideas to Brexit suggests that the option of a U.K. exit was made more likely because of the limited voice of the U.K. in achieving reforms, coupled with the fact that Britain’s loyalty to the European Union was mixed at best. Its self-perception as “special people” was accompanied by long-standing skepticism about foreigners, including other Europeans.

Some have attributed Brexit to misjudgment by Prime Minister David Cameron about holding a referendum, poor management of migration policy by the EU including procrastination and downright misjudgment on migration, and they have termed the historic vote as nothing short of the beginning of the end of the post-World War II institutional frameworks, including the Bretton Woods institutions. They fear that the longest and most prosperous period of sustained peace in modern human history, enabled by post-war global architecture, may have come to an end.

The Economist is one proponent of this view, describing Brexit as multiple calamities. The British economy and polity are wildly off the rails, the newspaper notes. The prime minister has resigned with no obvious successor. The leader of the opposition is struggling to survive a coup. The pound hit a 31-year low against the dollar and banks lost a third of their value before stabilizing. Meanwhile there is talk in Scotland and Northern Ireland of secession.

But my own English friends, some of whom favored Brexit, talk about the high tax payments to the EU, oppressive overreach of the EU bureaucracy, and the fear of open borders leading to uncontrollable immigration from Eastern Europe, Turkey, and the Middle East. In short they see EU membership as all pain and no gain. On the surface Brexit has all the flavors ranging from nostalgia of self-rule to xenophobia.

There are already signs that exit is becoming the preferred option in various global governance organizations. Global loyalties are split, not just among great powers, but also between developed and developing countries. Voice and reform have not been effective.

Hirschman mentions leadership and timely action in sharing power with the next generation as a behavioral trait (often found in the animal kingdom) favoring voice. He contrasts that with exit, which he describes as a human behavior which assumes markets, including political markets, will solve problems.

Hirschman’s chapter “Exit and Voice in American Ideology and Practice” helps us to better understand the U.S. role in global governance. He notes that exit has been accorded “an extraordinarily privileged position in the American tradition” founded in its very creation as a land of immigrants, who, he reminds us, were opting for exit.  Indeed, like in Britain, “the neatness of exit over the messiness and heartbreak of voice” has persisted throughout U.S. history. In his last chapter, “Elusive Optimal Mix of Exit and Voice,” he does not come up with a recipe for some optimum mix of the two, nor does he recommend each institution has its own optimum mix, instead arguing conditions are seldom ripe for their optimum and stable mix—although it is possible to say there is deficiency of one or the other at a given point in time.

Today, it seems that the dominant mode of the post-World War II era, namely voice, is plainly revealing its inadequacy, so the other mode, exit, will eventually be injected once again.

Having had a leading role in founding the global architecture of the United Nation, Food and Agriculture Organization, and Bretton Woods institutions, the U.S. has had a strong voice in and loyalty to the Bretton Woods institutions as well as leadership roles commensurate with its historic roles. U.S. loyalty to the U.N. outside of the Security Council has varied among administrations, since voice in U.N. organizations is distributed more equally. The U.S. has opted for exit from specific U.N. organizations from time to time when it has disliked the dissenting views of other members. 

Others are also choosing to exit. China’s slightly increased shares in the International Monetary Fund and the World Bank after the financial crisis are nowhere near its weight in the global economy, thanks to European reluctance to accept a reduced voice. China and other emerging countries have exercised a partial exit option by establishing the Asian Infrastructure Investment Bank and the New Development Bank to meet the investment needs of developing countries.The U.S. considered the establishment of the two as a threat to its leadership and to the Bretton Woods institutions, viewing the acts as verging on disloyalty, whereas most U.S. allies have embraced membership in both. And yet the Asian Infrastructure Investment Bank is following on the footsteps of the Bretton Woods institutions as regards norms and rules.

To strengthen global governance requires strengthening “voice” and weakening incentives for “exit” from the U.N. and Bretton Woods institutions and other forums of global governance. The U.S. needs to also lead the effort to increase the rewards and reduce the cost of exercising voice. This would be a timely reminder, when politics seems to thrive on divisions, that leadership means forging inclusive institutions that serve all members. 

In 2015 the international community agreed on a set of ambitious sustainable development goals (SDGs) for the global society, to be achieved by 2030. One of the lessons that the implementation of the Millennium Development Goals (MDG s) has highlighted is the importance of a systematic approach to identify and sequence development interventions—policies, programs, and projects—to achieve such goals at a meaningful scale. The Chinese approach to development, which consists of identifying a problem and long-term goal, testing alternative solutions, and then implementing those that are promising in a sustained manner, learning and adapting as one proceeds—Deng Xiaoping’s “crossing the river by feeling the stones”—is an approach that holds promise for successful achievement of the SDGs.

Having observed the Chinese way, then World Bank Group President James Wolfensohn in 2004, together with the Chinese government, convened a major international conference in Shanghai on scaling up successful development interventions, and in 2005 the World Bank Group (WBG ) published the results of the conference, including an assessment of the Chinese approach. (Moreno-Dodson 2005). Some ten years later, the WBG once again is addressing the question of how to support scaling up of successful development interventions, at a time when the challenge and opportunity of scaling up have become a widely recognized issue for many development institutions and experts.

Since traditional private and public service providers frequently do not reach the poorest people in developing countries, social enterprises can play an important role in providing key services to those at the “base of the pyramid.”

In parallel with the recognition that scaling up matters, the development community is now also focusing on social enterprises (SEs), a new set of actors falling between the traditionally recognized public and private sectors. We adopt here the World Bank’s definition of “social enterprises” as a social-mission-led organization that provides sustainable services to Base of the Pyramid (BoP) populations. This is broadly in line with other existing definitions for the sector and reflects the World Bank’s primary interest in social enterprises as a mechanism for supporting service delivery for the poor. Although social enterprises can adopt various organizational forms—business, nongovernmental organizations (NGOs), and community-based organizations are all forms commonly adopted by social enterprises—they differ from private providers principally by combining three features: operating with a social purpose, adhering to business principles, and aiming for financial sustainability. Since traditional private and public service providers frequently do not reach the poorest people in developing countries, social enterprises can play an important role in providing key services to those at the “base of the pyramid.” (Figure 1)

Figure 1. Role of SE sector in public service provision

Social enterprises often start at the initiative of a visionary entrepreneur who sees a significant social need, whether in education, health, sanitation, or microfinance, and who responds by developing an innovative way to address the perceived need, usually by setting up an NGO, or a for-profit enterprise. Social enterprises and their innovations generally start small. When successful, they face an important challenge: how to expand their operations and innovations to meet the social need at a larger scale. 

Development partner organizations—donors, for short—have recognized the contribution that social enterprises can make to find and implement innovative ways to meet the social service needs of people at the base of the pyramid, and they have started to explore how they can support social enterprises in responding to these needs at a meaningful scale. 

The purpose of this paper is to present a menu of approaches for addressing the challenge of scaling up social enterprise innovations, based on a review of the literature on scaling up and on social enterprises. The paper does not aim to offer specific recommendations for entrepreneurs or blueprints and guidelines for the development agencies. The range of settings, problems, and solutions is too wide to permit that. Rather, the paper provides an overview of ways to think about and approach the scaling up of social enterprise innovations. Where possible, the paper also refers to specific tools that can be helpful in implementing the proposed approaches. 

Note that we talk about scaling up social enterprise innovations, not about social enterprises. This is because it is the innovations and how they are scaled up that matter. An innovation may be scaled up by the social enterprise where it originated, by handoff to a public agency for implementation at a larger scale, or by other private enterprises, small or large. 

This paper is structured in three parts: Part I presents a general approach to scaling up development interventions. This helps establish basic definitions and concepts. Part II considers approaches for the scaling up of social enterprise innovations. Part III provides a summary of the main conclusions and lessons from experience. A postscript draws out implications for external aid donors. Examples from actual practice are used to exemplify the approaches and are summarized in Annex boxes.

Downloads

• Download the full paper (PDF)

The breakthrough therapy designation (BTD) program was initiated by the U.S. Food and Drug Administration (FDA) in 2012 to expedite the development of treatments for serious or life-threatening illness that demonstrate “substantial improvement” over existing therapies. The program has since become a widely supported mechanism for accelerating patient access to new drugs. As of March 2015, FDA has received a total of 293 requests for BTD. However, it has granted just  82 (28%), which indicates an ongoing lack of clarity over what exactly meets the criteria for the designation.

On April 24, the Center for Health Policy at Brookings convened a public meeting to explore the designation’s qualifying criteria and how FDA applies those criteria across therapeutic areas. Panelists used real-world and hypothetical case studies to frame the discussion, and highlighted major considerations for the application process, the FDA’s evaluation of the evidence, and the key factors for acceptance or rejection. The discussion also identified strategies to ensure that qualifying criteria are well understood. Here are the five big takeaways:

1.  The BTD program is viewed positively by drug companies, researchers, advocates, and others 

Across the board, participants expressed enthusiasm for the BTD program. Industry representatives noted that their experience had been extremely positive, and that the increased cooperation with and guidance from FDA were very helpful in streamlining their development programs. Receiving the designation can also raise a drug company’s profile, which can facilitate additional investment as well as clinical trial patient recruitment; this is particularly important for smaller companies with limited resources.

Patient and disease advocates were likewise supportive, and expressed hope that the early lessons learned from successful breakthrough therapy approvals (which have been mostly concentrated in the oncology and antiviral fields) could be translated to other disease areas with less success. However, while BTD is an important tool in expediting the development of new drugs, it is just one piece of broader scientific and regulatory policy landscape. Accelerating the pace of discovery and development of truly innovative new drugs will depend on a range of other factors, such as developing and validating new biomarkers that can be used to measure treatment effects at an earlier stage, as well as establishing networks that can streamline the clinical trial process. It will also be important to develop effective new approaches to collecting, analyzing, and communicating information about these treatments once they are on the market, as this information can potentially be used by FDA, providers, and patients to  further improve prescription drug policy and medical decision-making.

2.  BTD requests far outnumber those that actually meet the qualifying criteria

Since the program began, less than 30 percent of requests have received BTD designation. A substantial majority were denied at least in part due to either a lack of data or problems with the quality of the data, or some combination of the two. For example, some sponsors requested the designation before they had any clinical data, or submitted the request using clinical data that was incomplete or based on flawed study designs. Many requests also failed to meet the Agency’s bar for “substantial improvement” over existing therapies.

One reason for the high denial rate may be a lack of a clear regulatory or statutory bar that could be used as a definitive guide for sponsors to know what is needed to qualify for the designation. BTD denials are also confidential, which means that sponsors effectively have nothing to lose by submitting a request. Going forward, manufacturers may need to exercise more discretion in deciding to request the designation, as the process can be resource- and time-intensive for both sides.

3.  There is no single threshold for determining what defines a breakthrough therapy

About 53 percent of the 109 total BTD denials were due at least in part to the fact that the drug did not represent a substantial improvement over existing therapies. During the day’s discussion, FDA and sponsors both noted that this is likely because the criteria for BTD are inherently subjective. In practice, this means there is no clear threshold for determining when a new therapy represents a “substantial improvement” over existing therapies. Designation decisions are complex and highly dependent on the context, including the disease or condition being targeted, the availability of other treatments, the patient population, the outcomes being studied, and the overall reliability of the data submitted. Given the multiple factors at play, it can be difficult in some cases to determine when a new product is potentially “transformational” as opposed to “better,” especially for conditions that are poorly understood or have few or no existing treatments. In making its determinations, FDA considers the totality of the evidence submitted, rather than focusing on specific evidentiary requirements.

4.  Early communication with FDA is strongly recommended for BTD applicants

Roughly 72 percent of the BTD denials related at least in part to trial design or analysis problems, which led several people to suggest that sponsors engage with FDA prior to submitting their request. Though there are several formal mechanisms for interacting with the agency, informal consultations with the relevant review division could help sponsors to get a better  and much earlier sense of what kind of data FDA might need. This early communication could both strengthen viable BTD requests and reduce the number of frivolous requests.

5.  FDA may need more resources for implementing the BTD program

It has been hailed as the most significant discovery in biology since polymerase chain reaction allowed for the mass replication of DNA samples. CRISPR-Cas9 is an inexpensive and easy-to-use gene-editing method that promises applications ranging from medicine to industrial agriculture to biofuels. Currently, applications to treat leukemia, HIV, and cancer are under experimental development.¹ However, new technical solutions tend to be fraught with old problems, and in this case, ethical and legal questions loom large over the future.

Disagreements on ethics

The uptake of this method has been so fast that many scientists have started to worry about inadequate regulation of research and its unanticipated consequences.² Consider, for instance, the disagreement on research on human germ cells (eggs, sperm, or embryos) where an edited gene is passed onto offspring. Since the emergence of bioengineering applications in the 1970s, the scientific community has eschewed experiments to alter human germline and some governments have even banned them.³ The regulation regimes are expectedly not uniform: for instance, China bans the implantation of genetically modified embryos in women but not the research with embryos.

Last year, a group of Chinese researchers conducted gene-editing experiments on non-viable human zygotes (fertilized eggs) using CRISPR.⁴ News that these experiments were underway prompted a group of leading U.S. geneticists to meet in March 2015 in Napa, California, to begin a serious consideration of ethical and legal dimensions of CRISPR and called for a moratorium on research editing genes in human germline.⁵ Disregarding that call, the Chinese researchers published their results later in the year largely reporting a failure to precisely edit targeted genes without accidentally editing non-targets. CRISPR is not yet sufficiently precise.

CRISPR reignited an old debate on human germline research that is one of the central motivations (but surely not the only one) for an international summit on gene editing hosted by the U.S. National Academies of Sciences, the Chinese Academy of Sciences, and the U.K.'s Royal Society in December 2015. About 500 scientists as well as experts in the legal and ethical aspects of bioengineering attended.⁶ Rather than consensus, the meeting highlighted the significant contrasts among participants about the ethics of inquiry, and more generally, about the governance of science. Illustrative of these contrasts are the views of prominent geneticists Francis Collins, Director of the National Institutes of Health, and George Church, professor of genetics at Harvard. Collins argues that the “balance of the debate leans overwhelmingly against human germline engineering.” In turn, Church, while a signatory of the moratorium called by the Napa group, has nevertheless suggested reasons why CRISPR is shifting the balance in favor of lifting the ban on human germline experiments.⁷

The desire to speed up discovery of cures for heritable diseases is laudable. But tinkering with human germline is truly a human concern and cannot be presumed to be the exclusive jurisdictions of scientists, clinicians, or patients. All members of society have a stake in the evolution of CRISPR and must be part of the conversation about what kind of research should be permitted, what should be discouraged, and what disallowed. To relegate lay citizens to react to CRISPR applications—i.e. to vote with their wallets once applications hit the market—is to reduce their citizenship to consumer rights, and public participation to purchasing power.⁸ Yet, neither the NAS summit nor the earlier Napa meeting sought to solicit the perspectives of citizens, groups, and associations other than those already tuned in the CRISPR debates.⁹

The scientific community has a bond to the larger society in which it operates that in its most basic form is the bond of the scientist to her national community, is the notion that the scientist is a citizen of society before she is a denizen of science. This bond entails liberties and responsibilities that transcend the ethos and telos of science and, consequently, subordinates science to the social compact. It is worth recalling this old lesson from the history of science as we continue the public debate on gene editing. Scientists are free to hold specific moral views and prescriptions about the proper conduct of research and the ethical limits of that conduct, but they are not free to exclude the rest of society from weighing in on the debate with their own values and moral imaginations about what should be permitted and what should be banned in research. The governance of CRISPR is a question of collective choice that must be answered by means of democratic deliberation and, when irreconcilable differences arise, by the due process of democratic institutions.

Patent disputes

More heated than the ethical debate is the legal battle for key CRISPR patents that has embroiled prominent scientists involved in perfecting this method. The U.S. Patent and Trademark Office initiated a formal contestation process, called interference, in March 2016 to adjudicate the dispute. The process is likely to take years and appeals are expected to extend further in time. Challenges are also expected to patents filed internationally, including those filed with the European Patent Office.

To put this dispute in perspective, it is instructive to consider the history of CRISPR authored by one of the celebrities in gene science, Eric Lander.¹⁰ This article ignited a controversy because it understated the role of one of the parties to the patent dispute (Jennifer Doudna and Emmanuelle Charpentier), while casting the other party as truly culminating the development of this technology (Feng Zhang, who is affiliated to Lander’s Broad Institute). Some gene scientists accused Lander of tendentious inaccuracies and of trying to spin a story in a manner that favors the legal argument (and economic interest) of Zhang.

Ironically, the contentious article could be read as an argument against any particular claim to the CRISPR patents as it implicitly questions the fairness of granting exclusive rights to an invention. Lander tells the genesis of CRISPR that extends through a period of two decades and over various countries, where the protagonists are the many researchers who contributed to the cumulative knowledge in the ongoing development of the method. The very title of Lander’s piece, “The Heroes of CRISPR” highlights that the technology has not one but a plurality of authors.

A patent is a legal instrument that recognizes certain rights of the patent holder (individual, group, or organization) and at the same time denies those rights to everyone else, including those other contributors to the invention. Patent rights are thus arbitrary under the candle of history. I am not suggesting that the bureaucratic rules to grant a patent or to determine its validity are arbitrary; they have logical rationales anchored in practice and precedent. I am suggesting that in principle any exclusive assignation of rights that does not include the entire community responsible for the invention is arbitrary and thus unfair. The history of CRISPR highlights this old lesson from the history of technology: an invention does not belong to its patent holder, except in a court of law.

Some scientists may be willing to accept with resignation the unfair distribution of recognition granted by patents (or prizes like the Nobel) and find consolation in the fact that their contribution to science has real effects on people’s lives as it materializes in things like new therapies and drugs. Yet patents are also instrumental in distributing those real effects quite unevenly. Patents create monopolies that, selling their innovation at high prices, benefit only those who can afford them. The regular refrain to this charge is that without the promise of high profits, there would be no investments in innovation and no advances in life-saving medicine. What’s more, the biotech industry reminds us that start-ups will secure capital injections only if they have exclusive rights to the technologies they are developing. Yet, Editas Medicine, a biotech start-up that seeks to exploit commercial applications of CRISPR (Zhang is a stakeholder), was able to raise $94 million in its February 2016 initial public offering. That some of Editas’ key patents are disputed and were entering interference at USPTO was patently not a deterrent for those investors.

Towards a CRISPR democratic debate

Neither the governance of gene-editing research nor the management of CRISPR patents should be the exclusive responsibility of scientists. Yet, they do enjoy an advantage in public deliberations on gene editing that is derived from their technical competence and from the authority ascribed to them by society. They can use this advantage to close the public debate and monopolize its terms, or they could turn it into stewardship of a truly democratic debate about CRISPR.

The latter choice can benefit from three steps. A first step would be openness: a public willingness to consider and internalize public values that are not easily reconciled with research values. A second step would be self-restraint: publicly affirming a self-imposed ban on research with human germline and discouraging research practices that are contrary to received norms of prudence. A third useful step would be a public service orientation in the use of patents: scientists should pressure their universities, who hold title to their inventions, to preserve some degree of influence over research commercialization so that the dissemination and access to innovations is consonant with the noble aspirations of science and the public service mission of the university. Openness, self-restraint, and an orientation to service from scientists will go a long way to make of CRISPR a true servant of society and an instrument of democracy.

Other reading: See media coverage compiled by the National Academies of Sciences.

¹Nature: an authoritative and accessible primer. A more technical description of applications in Hsu, P. D. et al. 2014. Cell, 157(6): 1262–1278.

³More about ethical concerns on gene editing here: http://www.geneticsandsociety.org/article.php?id=8711

⁴Liang, P. et al. 2015. Protein & Cell, 6, 363–372

⁵Science: A prudent path forward for genomic engineering and germline gene modification.

⁶Nature: NAS Gene Editing Summit.

⁷While Collins and Church participated in the summit, their views quoted here are from StatNews.com: A debate: Should we edit the human germline. See also Sciencenews.org: Editing human germline cells sparks ethics debate.

⁸Hurlbut, J. B. 2015. Limits of Responsibility, Hastings Center Report, 45(5): 11-14.

⁹This point is forcefully made by Sheila Jasanoff and colleagues: CRISPR Democracy, 2015 Issues in S&T, 22(1).

¹⁰Lander, E. 2016. The Heroes of CRISPR. Cell, 164(1-2): 18-28.

Editor's Note: This article originally appeared on Lawfare.

Faced with the breakdown of national armies in Iraq, Libya, Syria, and Yemen, Arab states have increasingly turned toward alliances with armed militias to ensure security. Popular, anti-government protests and insurgencies for the most part precipitated the breakdown of regime military institutions, yet pre-existing internal ethnic, clan, and ideological cleavages helped to hasten the breakdown. The beleaguered state security forces have now entered into a variety of alliances—tacit or active—with militias they deem sympathetic to their interests, often organized on the basis of entrenched ethno-sectarian or tribal identities. Such militia forces supplement and at times even stand in for the weak or absent army and police as providers of local security.

On the one hand, militia forces have in certain circumstances proven effective at counterinsurgency and counterterrorism. On the other hand, they have also committed atrocities against civilians that hamper long-term efforts to build trust and stability. Their greatest risk is that, by eroding the central government’s monopolization on force, they jeopardize the territorial cohesion of the state.

In Iraq, the rise of powerful communal militias has paralleled the growth of the threat from the Islamic State. This has presented the United States with a quandary: how to combat the Islamic State by mobilizing local Sunnis while at the same time safeguarding the broader integrity of the Iraqi state and its security institutions. The national guard concept, which successive Iraqi governments have tried in the past, was seen as one way to do this. A national guard force would retain the militias’ local knowledge and roots, both unique tools necessary for a successful counterinsurgency against the Islamic State. At the same time, the guard would (at least in theory) be subject to increased oversight and control by the central government.

Other fractured Arab states, most notably Libya, have tried to implement a national guard model as a way to harness militia power, but this too has failed. Variations of hybrid, provincially-organized military forces exist in Yemen and Syria. While each case is different, the failure of national guards bears certain similarities. Examining the Iraqi case in particular can highlight the potential utility of national guards but also the parallel political and institutional reforms that are necessary to make the concept work.

The idea of creating a national guard in Iraq has been a centerpiece of U.S. engagement since the dramatic advance of the Islamic State on Tikrit and Mosul in 2014. President Obama specifically mentioned U.S. support for a national guard as a means to help Iraqi Sunnis “secure their own freedom” from the Islamic State. Much of U.S. thinking about the Iraqi National Guard (ING) was guided by the example of the Sunni Awakening of 2006 and 2007, when the United States actively recruited and “flipped” Sunni tribes that had supported the al-Qaeda-inspired insurgency. In return for guarantees of autonomy and military, financial, and political backing, the Sunni tribes were able to turn the tables on the insurgent fighters and impose a measure of peace and stability. The 2014 initiative essentially sought to reproduce this arrangement. The idea was that given proper incentives, the Sunni tribes would again fight the radical Islamists who threatened their supremacy. Over the long term, such national guard forces could be integrated formally as auxiliary troops in a federal structure, comparable in many ways to the U.S. National Guard.

Yet the Awakening analogy failed on a number of levels. The Shi’i-dominated Iraqi central government had never been enthusiastic about empowering Sunni tribes in the first place. With the dismantling of the Iraqi army in 2003, security had effectively devolved to party, tribal, and sectarian militias. Many Iraqis wondered why the United States would seek to create new militias, especially ones recently tied to al-Qaeda and other terrorists. As Iraq scholar Adeed Dawisha described, the gains in security came“not because of the state, but in spite of it.”

As the U.S. began withdrawing from Iraq in 2009 and 2010, then-Prime Minister Nuri al-Maliki quickly moved to dismantle the Awakening-associated militias. Only a handful of former militia fighters received their promised positions in the police, army, or civil services. Some former militia leaders were arrested on seemingly politically-motivated charges of terrorism or subversion. Efforts to enact a Sunni-dominated super-region comparable to the federal status of the Kurdish Regional Government in the north were rebuffed, despite the provisions of Iraq’s constitution that allowed for the creation of such an entity. Politically marginalized, some Sunnis returned to their alliance with the radical mujahideen.

The election of the new prime minister Haydar al-Abadi in 2014 raised the promise of renewed Sunni-Shi’i reconciliation. Abadi expressed support for the national guard initiative and forwarded a bill to parliament in 2014. Thousands of volunteers came forward from the Sunni tribes in the west and U.S. and Iraqi officials met with tribal leaders to help solidify support. The United States began to enlist support from Iraq’s Sunni neighbors to provide training and support for the ING.

Yet resistance within Abadi’s own political coalition stymied these efforts. The National Guard bill foundered in parliamentary committee, with open questions about the extent of control vested in provincial governors and the chain of command subordinating the ING to the ministries of interior, defense, or the prime minister himself. Officers of the Iraqi Security Forces (ISF) regarded the militias as unfit for duty and as rivals for budget and resources. Iraq’s constitution specifically prohibited the formation of militias outside the framework of the armed forces (with an exception of the peshmergaforces of the Kurdish Regional Government). Moreover, there was concern that once the Sunnis were authorized to organize a militia, other ethno-sectarian communities, such as Christians or Turkomen,might try to follow suit out of fear of falling under the mercy of their more powerful neighbors. The ING, then, could undercut any pretense of the Iraqi state possessing a monopoly over the use of force.

At base, though, many of Iraq’s Shi’i leaders simply believed that they didn’t need Sunni support. With the ING initiative stalled in parliament, the Shi’i factions have actively cultivated Shi’i militias as part of the Popular Mobilization Forces (PMF, or Hashd al-Shaabi). The origins of the PMF can be traced to a statement by Grand Ayatollah Ali Sistani, Iraq’s senior Shi’i cleric, which explicitly called on the faithful to take up arms to defend Iraq in the face of the Islamic State onslaught in 2014. Muqtada al-Sadr’s Jaysh al-Mahdi, the Badr Organization, and other political factions quickly took the opportunity to reconstitute or expand their private armies.

Backed by Iran’s expeditionary al-Qods Force, PMF militias played a prominent role in the spring 2015 offensive against the Islamic State in Tikrit. By spring 2015, PMF counted around 60,000 men under arms. Still, the performance of these militias has been less than stellar. In the spring 2015 offensive on Tikrit, PMF forces failed repeatedly to dislodge Islamic State resistance, despite enjoying superiority in numbers. U.S. air support proved critical to allowing the offensive to proceed. Some PMF units quit the fight instead of working under American air cover. Others were involved in a campaign of terror against Sunnis, looting, kidnapping, and killing those suspected of collaborating with the Islamic State.

The prospects for the mobilization of Iraq’s Sunnis are not dead—yet. A handful of Sunni tribes joined the PMF during the Tikrit offensive. In Anbar, likely the next front in the campaign against the Islamic State, U.S. and Iraqi officials have cultivated ties with local Sunni tribes and organized some 8,000 men into Sunni PMF units. Some tribes have made their service conditional on guarantees of greater autonomy and the removal of Shi’i militia forces. Yet the intake for training programs remains slow and drop-out rates high. On the one hand, tribes continue to resent the central government. On the other hand, they fear retribution should the Islamic State return.

Abadi’s visit to Washington in April 2015 focused on expanding and enhancing security cooperation with the United States. The United States has insisted that the PMF be brought more fully under the control of the Iraqi Security Forces and that PMF units reflect the demographics of the provinces and districts in which they operate. This would mean that in ethnically-mixed areas, such as in Nineveh or Babil, each ethnic group would have its own militia proportional to its size in the locality. The Iraq Train and Equip Program (ITEP) is slowly coming online, funneling American money and weapons to various local militia forces as well as ISF.

Cooperating with the United States has been a delicate balancing act for Abadi. While Kurdish and Sunni leaders see U.S. military support as a means to their own ends, Abadi’s own Shi’i political camp—as well as his allies in Tehran—are far more wary. When the U.S. Congress passed a bill in May 2015 effectively mandating the Defense Department to bypass Baghdad and provide support for Sunni and Kurdish fighters directly, Abadi protested that this constituted a grave violation of Iraqi sovereignty.

Still, reliance on the ragtag PMF alone is not sustainable in the long term. Operating far from home and with limited training, these overwhelmingly Shi’i forces cannot be expected to become an army of occupation in Sunni areas like Tikrit or Fallujah. Ultimately, local partners will be necessary to build and maintain peace and stability. The national guard, then, may well re-emerge as a more sustainable structure for administrative and security devolution.

While analysts and policymakers naturally focus on cases of success, there are important lessons to be learned from Iraq’s failures. For countries like Iraq where central armies have more or less broken down and a bevy of militias has emerged in its stead, as in Libya, Yemen, and Syria, the national guard could represent a path to reconstituting fragile state authority.

But for this to happen, several broad principles need to be heeded:

• National guards cannot simply be conceived as short-term, improvised solutions to immediate security crises. Rather, the creation of national guards is part of the impetus of security-sector reform (SSR) and post-conflict demobilization, disarmament, and reintegration (DDR) of armed groups.
• National guards must overcome the legacies of past authoritarian experiences where pro-government militias were often seen as mere thugs for the regime, not a disciplined professional fighting force. In particular, the older officer class of regular forces may see them as competitors. To build trust among the population and other military institutions, national guards should be accompanied by revisions to chain of command establishing clear relationships of authority between the guards, the police, the army, and other security agencies, and subordinating all security services to civilian authorities.
• National guard initiatives must also be accompanied by moves toward political power-sharing arrangements. The success of national guards ultimately depends not just on their short-term tactical effectiveness but on the degree of local buy-in. Constitutions can provide a structure for bolstering confidence between a central government and subnational militia forces. Since militia membership and cohesion is often based on geographic linkages—to town, municipality or province—national guards may well be a part of federalist power devolution, especially in countries with overlapping ethno-sectarian and regional cleavages.
• Western governments can assist in setting up and training national guards, but they must ensure that proper political and institutional reforms are also undertaken. In many cases, Western states provide models for how decentralized, federally-organized military forces can complement national armies and local police. The United States, for instance, has a great deal of experience with its own federalized national guard structure and can draw on this example in its train-and-equip programs. There are other potentially useful models as well, including the British Territorial Army, a part-time, volunteer force that was integrated into the British Army in the early twentieth century; the Danish Home Guard, which incorporated anti-Nazi resistance militias into a national command structure after World War II; or the Italian Carabineri, which is often discussed as a potential model for dealing with Libya’s unique security challenges.

Outside assistance to national guards must avoid exacerbating existing communal and political fault lines. Helping peripheral and minority groups set up their own armed forces can, on one hand, embolden these groups to resist the central government and, on the other hand, spur resentment from the central government and fear of future disloyalty or rebellion. These concerns become even more acute when national guards are seen as proxies for outside powers. With this in mind, the U.S. and outside powers should calibrate their assistance to both regionally-based national guards and central government forces to ensure rough parity between the two. This could entail making funding, equipment and training for the central security services contingent on a proportional commitment to strengthen the guards.

National guards are political institutions, not just military instruments. They can have far-ranging consequences for political stability and cohesion. While no panacea for the challenge of building effective states, they can play an important role in addressing security concerns and moving toward more meaningful power sharing.

Abstract

Scaling up of development interventions is much debated today as a way to improve their impact and effectiveness. Based on a review of scaling up literature and practice, this paper develops a framework for the key dynamics that allow the scaling up process to happen. The authors explore the possible approaches and paths to scaling up, the drivers of expansion and of replication, the space that has to be created for interventions to grow, and the role of evaluation and of careful planning and implementation. They draw a number of lessons for the development analyst and practitioner. More than anything else, scaling up is about political and organizational leadership, about vision, values and mindset, and about incentives and accountability—all oriented to make scaling up a central element of individual, institutional, national and international development efforts. The paper concludes by highlighting some implications for aid and aid donors.

An annotated bibliography of the literature on scaling up and development aid effectiveness was created by Oksana Pidufala to supplement this working paper. Read more »

Downloads

• Download

Editor's Note: The World Bank’s Independent Evaluation Group (IEG) recently published a self-evaluation of its activities. Besides representing current thinking among evaluation experts at the World Bank, it also more broadly reflects some of the strengths and gaps in the approaches that evaluators use to assess and learn from the performance of the international institutions with which they work. The old question “Quis custodet ipsos custodes?” – loosely translated as “Who evaluates the evaluators?” – remains as relevant as ever. Johannes Linn served as an external peer reviewer of the self-evaluation and provides a bird’s-eye view on the lessons learned.

An Overview of the World Bank’s IEG Self-Evaluation Report

In 2011 the World Bank’s Independent Evaluation Group (IEG) carried out and published a self-evaluation of its activities. The self-evaluation team was led by an internal manager, but involved a respected external evaluation expert as the principal author and also an external peer reviewer.

The IEG self-evaluation follows best professional practices as codified by the Evaluation Cooperation Group (ECG). This group brings together the evaluation offices of seven major multilateral financial institutions in joint efforts designed to enhance evaluation performance and cooperation among their evaluators. One can therefore infer that the approach and focus of the IEG self-evaluation is representative of a broader set of practices that are currently used by the evaluation community of international financial organizations.

At the outset the IEG report states that “IEG is the largest evaluation department among Evaluation Capacity Group (ECG) members and is held in high regard by the international evaluation community. Independent assessments of IEG’s role as an independent evaluation function for the Bank and IFC rated it above the evaluation functions in most other ECG members, international nongovernmental organizations, and transnational corporations and found that IEG follows good practice evaluation principles.”

The self-evaluation report generally confirms this positive assessment. For four out of six areas of its mandate IEG gives itself the second highest rating (“good”) out of six possible rating categories. This includes (a) the professional quality of its evaluations, (b) its reports on how the World Bank’s management follows up on IEG recommendations, (c) cooperation with other evaluation offices, and (d) assistance to borrowing countries in improving their own evaluation capacity. In the area of appraising the World Bank’s self-evaluation and risk management practices, the report offers the third highest rating (“satisfactory”), while it gives the third lowest rating (“modest”) for IEG’s impact on the Bank’s policies, strategies and operations. In addition the self-evaluation concludes that overall the performance of IEG has been “good” and that it operates independently, effectively and efficiently.

The report makes a number of recommendations for improvement, which are likely to be helpful, but have limited impact on its activities. They cover measures to further enhance the independence of IEG and the consistency of evaluation practices as applied across the World Bank Group’s branches – the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) –; to improve the design of evaluations and the engagement with Bank management upstream for greater impact; and monitoring the impact of recent organizational changes in IEG in terms of results achieved. The report also recommends that more be done to evaluate the Bank’s analytical work and that evaluations draw on comparative evidence.

Assessment

In terms of the parameters of self-evaluation set by the prevailing practice among the evaluators on international financial agencies, the IEG self-evaluation is accurate and helpful. From my own experience as an operational manager in the Bank whose activities were evaluated by IEG in years past, and as a user of IEG evaluations (and of evaluations of other international aid organizations) for my research on aid effectiveness, I concur that IEG is independent and effective in meeting its mandate as defined. Moreover, the self-evaluation produces useful quantitative evidence (including survey results, budget analysis, etc.) to corroborate qualitative judgments.

However, the self-evaluation suffers from a number of limitations in approach and gaps in focus, which are broadly representative of the practices prevalent among many of the evaluation offices of international aid agencies.

Approach of the IEG self-evaluation

The core of the self-evaluation report is about the evaluation process followed by IEG, with very little said about the substance of IEG’s evaluations. The following questions could have usefully been raised, but were not: do evaluations cover the right issues with the right intensity, such as growth and poverty; environmental, governance, and gender impacts; regional dimensions versus exclusive country or project focus; effectiveness in addressing the problems of fragile and conflict states; effectiveness in dealing with global public goods; sustainability and scaling up; etc. Therefore the report does not deal with the question of whether IEG effectively responds in its evaluations to the many important strategic debates and issues with which the development community is grappling.

Related to this limitation is the fact that the report assessed the quality of IEG’s mostly in terms of (a) whether its approach and processes meet certain standards established by the Evaluation Cooperation Group; and (b) how it is judged by stakeholders in response to a survey commissioned for this evaluation. Both these approaches are useful, but they do not have any basis in professional assessments of the quality of individual products. This is equivalent to IEG evaluating the World Bank’s projects on the quality of its processes (e.g., appraisal and supervision processes) and on the basis of stakeholder surveys, without evaluating individual products and their impacts.

Gaps in the Self-Evaluation and in Evaluation Practice

Careful reading of the report reveals six important gaps in the IEG self-evaluation, in the prevailing evaluation practice in the World Bank, and more generally in the way international financial organizations evaluate their own performance. The first three gaps relate to aspects of the evaluation approach used and the second three gaps relate to lack of focus in the self-evaluation on key internal organizational issues:

1. Impact Evaluations: The report notes that IEG carries out two to three impact evaluations per year, but it sidesteps the debate in the current evaluation literature and practice as to what extent the “gold standard” of randomized impact evaluation should occupy a much more central role. Given the importance of this debate and divergence of views, it would have been appropriate for the self-evaluation to assess IEG’s current practice of very limited use of randomized evaluations.

2. Evaluation of Scaling Up: The report does not address the question of to what extent current IEG practice not only assesses the performance of individual projects in terms of their outcomes and sustainability, but also in terms of whether the Bank has systematically built on its experience in specific projects to help scale up their impact through support for expansion or replication in follow-up operations or through effective hand-off to the government or other partners. In fact, currently IEG does not explicitly and systematically consider scaling up in its project and program evaluations. For example, in a recent IEG evaluation of World Bank funded municipal development projects (MDPs) , IEG found that the Bank has supported multiple MDPs in many countries over the years, but the evaluation did not address the obvious question whether the Bank systematically planned for the project sequence or built on its experience from prior projects in subsequent operations. While most other evaluation offices like IEG do not consider scaling up, some (in particular those of the International Fund for Agricultural Development and the United Nations Development Program) have started doing so in recent years.

3. Drawing on the Experience of and Benchmarking Against Other Institutions: The self-evaluation report does a good job in benchmarking IEG performance in a number of respects against that of other multilateral institutions. In the main text of the report it states that “IEG plans to develop guidelines for approach papers to ensure greater quality, in particular in drawing on comparative information from other sources and benchmarking against other institutions.” This is a welcome intention, but it is inadequately motivated in the rest of the report and not reflected in the Executive Summary. The reality is that IEG, like most multilateral evaluation offices, so far has not systematically drawn on the evaluations and relevant experience of other aid agencies in its evaluations of World Bank performance. This has severely limited the learning impact of the evaluations.

4. Bank Internal Policies, Management Processes and Incentives: IEG evaluations traditionally do not focus on how the Bank’s internal policies, management and incentives affect the quality of Bank engagement in countries. Therefore evaluations cannot offer any insights into whether and how Bank-internal operating modalities contribute to results. Two recent exceptions are notable exceptions. First, the IEG evaluation of the Bank’s approach to harmonization with other donors and alignment with country priorities assesses the incentives for staff to support harmonization and alignment. The evaluation concludes that there are insufficient incentives, a finding disputed by management. Second, is the evaluation of the Bank’s internal matrix management arrangements, which is currently under way. The self-evaluation notes that Bank management tried to quash the matrix evaluation on the grounds that it did not fall under the mandate of IEG. This is an unfortunate argument, since an assessment of the institutional reasons for the Bank’s performance is an essential component of any meaningful evaluation of Bank-supported programs. While making a good case for the specific instance of the matrix evaluation, the self-evaluation report shies away from a more general statement in support of engaging IEG on issues of Bank-internal policies, management processes and incentives. It is notable that IFAD’s Independent Office of Evaluation appears to be more aggressive in this regard: It currently is carrying out a full evaluation of IFAD’s internal efficiency and previous evaluations (e.g., an evaluation of innovation and scaling up) did not shy away from assessing internal institutional dimensions.

5. World Bank Governance: The IEG self-evaluation is even more restrictive in how it interprets its mandate regarding the evaluation of the World Bank’s governance structures and processes (including its approach to members’ voice and vote, the functioning of its board of directors, the selection of its senior management, etc.). It considers these topics beyond IEG’s mandate. This is unfortunate, since the way the Bank’s governance evolves will substantially affect its long-term legitimacy, effectiveness and viability as an international financial institution. Since IEG reports to the Bank’s board of directors, and many of the governance issues involve questions of the board’s composition, role and functioning, there is a valid question of how effectively IEG could carry out such an evaluation. However, it is notable that the IMF’s Independent Evaluation Office, which similarly reports to the IMF board of directors, published a full evaluation of the IMF’s governance in 2008, which effectively addressed many of the right questions.

6. Synergies between World Bank, IFC and MIGA: The self-evaluation report points out that the recent internal reorganization of IEG aimed to assure more effective and consistent evaluations across the three member branches of the World Bank Group. This is welcome, but the report does not assess how past evaluations addressed the question of whether the World Bank, IFC and MIGA effectively capitalized on the potential synergies among the three organizations. The recent evaluation of the World Bank Group’s response to the global economic crisis of 2008/9 provided parallel assessments of each agency’s performance, but did not address whether they work together effectively in maximizing their synergies. The reality is that the three organizations have deeply engrained institutional cultures and generally go their own ways rather than closely coordinating their activities on the ground. Future evaluations should explicitly consider whether the three effectively cooperate or not. While the World Bank is unique in the way it has organizationally separated its private sector and guarantee operations, other aid organizations also have problems of a lack of cooperation, coordination and synergy among different units within the agency. Therefore, the same comment also applies to their evaluation approaches.

Conclusions

Self-evaluations are valuable tools for performance assessment and IEG is to be congratulated for carrying out and publishing such an evaluation of its own activities. As for all self-evaluations, it should be seen as an input to an independent external evaluation, a decision that, for now, has apparently been postponed by the Bank’s board of directors.

IEG’s self-evaluation has many strengths and provides an overall positive assessment of IEG’s work. However, it does reflect some important limitations of analysis and of certain gaps in approach and coverage, which an independent external review should consider explicitly, and which IEG’s management should address. Since many of these issues also likely apply to most of the other evaluation approaches by other evaluation offices, the lessons have relevance beyond IEG and the World Bank.

Key lessons include:

• An evaluation of evaluations should focus not only on process, but also on the substantive issues that the institution is grappling with.
• An evaluation of the effectiveness of evaluations should include a professional assessment of the quality of evaluation products.
• An evaluation of evaluations should assess:
  o How effectively impact evaluations are used;
  o How scaling up of successful interventions is treated;
  o How the experience of other comparable institutions is utilized;
  o Whether and how the internal policies, management practices and incentives of the institution are effectively assessed;
  o Whether and how the governance of the institution is evaluated; and
  o Whether and how internal coordination, cooperation and synergy among units within the organizations are assessed.

Evaluations play an essential role in the accountability and learning of international aid organizations. Hence it is critical that evaluations address the right issues and use appropriate techniques. If the lessons above were reflected in the evaluation practices of the aid institutions, this would represent a significant step forward in the quality, relevance and likely impact of evaluations.