It has been hailed as the most significant discovery in biology since polymerase chain reaction allowed for the mass replication of DNA samples. CRISPR-Cas9 is an inexpensive and easy-to-use gene-editing method that promises applications ranging from medicine to industrial agriculture to biofuels. Currently, applications to treat leukemia, HIV, and cancer are under experimental development.¹ However, new technical solutions tend to be fraught with old problems, and in this case, ethical and legal questions loom large over the future.

Disagreements on ethics

The uptake of this method has been so fast that many scientists have started to worry about inadequate regulation of research and its unanticipated consequences.² Consider, for instance, the disagreement on research on human germ cells (eggs, sperm, or embryos) where an edited gene is passed onto offspring. Since the emergence of bioengineering applications in the 1970s, the scientific community has eschewed experiments to alter human germline and some governments have even banned them.³ The regulation regimes are expectedly not uniform: for instance, China bans the implantation of genetically modified embryos in women but not the research with embryos.

Last year, a group of Chinese researchers conducted gene-editing experiments on non-viable human zygotes (fertilized eggs) using CRISPR.⁴ News that these experiments were underway prompted a group of leading U.S. geneticists to meet in March 2015 in Napa, California, to begin a serious consideration of ethical and legal dimensions of CRISPR and called for a moratorium on research editing genes in human germline.⁵ Disregarding that call, the Chinese researchers published their results later in the year largely reporting a failure to precisely edit targeted genes without accidentally editing non-targets. CRISPR is not yet sufficiently precise.

CRISPR reignited an old debate on human germline research that is one of the central motivations (but surely not the only one) for an international summit on gene editing hosted by the U.S. National Academies of Sciences, the Chinese Academy of Sciences, and the U.K.'s Royal Society in December 2015. About 500 scientists as well as experts in the legal and ethical aspects of bioengineering attended.⁶ Rather than consensus, the meeting highlighted the significant contrasts among participants about the ethics of inquiry, and more generally, about the governance of science. Illustrative of these contrasts are the views of prominent geneticists Francis Collins, Director of the National Institutes of Health, and George Church, professor of genetics at Harvard. Collins argues that the “balance of the debate leans overwhelmingly against human germline engineering.” In turn, Church, while a signatory of the moratorium called by the Napa group, has nevertheless suggested reasons why CRISPR is shifting the balance in favor of lifting the ban on human germline experiments.⁷

The desire to speed up discovery of cures for heritable diseases is laudable. But tinkering with human germline is truly a human concern and cannot be presumed to be the exclusive jurisdictions of scientists, clinicians, or patients. All members of society have a stake in the evolution of CRISPR and must be part of the conversation about what kind of research should be permitted, what should be discouraged, and what disallowed. To relegate lay citizens to react to CRISPR applications—i.e. to vote with their wallets once applications hit the market—is to reduce their citizenship to consumer rights, and public participation to purchasing power.⁸ Yet, neither the NAS summit nor the earlier Napa meeting sought to solicit the perspectives of citizens, groups, and associations other than those already tuned in the CRISPR debates.⁹

The scientific community has a bond to the larger society in which it operates that in its most basic form is the bond of the scientist to her national community, is the notion that the scientist is a citizen of society before she is a denizen of science. This bond entails liberties and responsibilities that transcend the ethos and telos of science and, consequently, subordinates science to the social compact. It is worth recalling this old lesson from the history of science as we continue the public debate on gene editing. Scientists are free to hold specific moral views and prescriptions about the proper conduct of research and the ethical limits of that conduct, but they are not free to exclude the rest of society from weighing in on the debate with their own values and moral imaginations about what should be permitted and what should be banned in research. The governance of CRISPR is a question of collective choice that must be answered by means of democratic deliberation and, when irreconcilable differences arise, by the due process of democratic institutions.

Patent disputes

More heated than the ethical debate is the legal battle for key CRISPR patents that has embroiled prominent scientists involved in perfecting this method. The U.S. Patent and Trademark Office initiated a formal contestation process, called interference, in March 2016 to adjudicate the dispute. The process is likely to take years and appeals are expected to extend further in time. Challenges are also expected to patents filed internationally, including those filed with the European Patent Office.

To put this dispute in perspective, it is instructive to consider the history of CRISPR authored by one of the celebrities in gene science, Eric Lander.¹⁰ This article ignited a controversy because it understated the role of one of the parties to the patent dispute (Jennifer Doudna and Emmanuelle Charpentier), while casting the other party as truly culminating the development of this technology (Feng Zhang, who is affiliated to Lander’s Broad Institute). Some gene scientists accused Lander of tendentious inaccuracies and of trying to spin a story in a manner that favors the legal argument (and economic interest) of Zhang.

Ironically, the contentious article could be read as an argument against any particular claim to the CRISPR patents as it implicitly questions the fairness of granting exclusive rights to an invention. Lander tells the genesis of CRISPR that extends through a period of two decades and over various countries, where the protagonists are the many researchers who contributed to the cumulative knowledge in the ongoing development of the method. The very title of Lander’s piece, “The Heroes of CRISPR” highlights that the technology has not one but a plurality of authors.

A patent is a legal instrument that recognizes certain rights of the patent holder (individual, group, or organization) and at the same time denies those rights to everyone else, including those other contributors to the invention. Patent rights are thus arbitrary under the candle of history. I am not suggesting that the bureaucratic rules to grant a patent or to determine its validity are arbitrary; they have logical rationales anchored in practice and precedent. I am suggesting that in principle any exclusive assignation of rights that does not include the entire community responsible for the invention is arbitrary and thus unfair. The history of CRISPR highlights this old lesson from the history of technology: an invention does not belong to its patent holder, except in a court of law.

Some scientists may be willing to accept with resignation the unfair distribution of recognition granted by patents (or prizes like the Nobel) and find consolation in the fact that their contribution to science has real effects on people’s lives as it materializes in things like new therapies and drugs. Yet patents are also instrumental in distributing those real effects quite unevenly. Patents create monopolies that, selling their innovation at high prices, benefit only those who can afford them. The regular refrain to this charge is that without the promise of high profits, there would be no investments in innovation and no advances in life-saving medicine. What’s more, the biotech industry reminds us that start-ups will secure capital injections only if they have exclusive rights to the technologies they are developing. Yet, Editas Medicine, a biotech start-up that seeks to exploit commercial applications of CRISPR (Zhang is a stakeholder), was able to raise $94 million in its February 2016 initial public offering. That some of Editas’ key patents are disputed and were entering interference at USPTO was patently not a deterrent for those investors.

Towards a CRISPR democratic debate

Neither the governance of gene-editing research nor the management of CRISPR patents should be the exclusive responsibility of scientists. Yet, they do enjoy an advantage in public deliberations on gene editing that is derived from their technical competence and from the authority ascribed to them by society. They can use this advantage to close the public debate and monopolize its terms, or they could turn it into stewardship of a truly democratic debate about CRISPR.

The latter choice can benefit from three steps. A first step would be openness: a public willingness to consider and internalize public values that are not easily reconciled with research values. A second step would be self-restraint: publicly affirming a self-imposed ban on research with human germline and discouraging research practices that are contrary to received norms of prudence. A third useful step would be a public service orientation in the use of patents: scientists should pressure their universities, who hold title to their inventions, to preserve some degree of influence over research commercialization so that the dissemination and access to innovations is consonant with the noble aspirations of science and the public service mission of the university. Openness, self-restraint, and an orientation to service from scientists will go a long way to make of CRISPR a true servant of society and an instrument of democracy.

Other reading: See media coverage compiled by the National Academies of Sciences.

¹Nature: an authoritative and accessible primer. A more technical description of applications in Hsu, P. D. et al. 2014. Cell, 157(6): 1262–1278.

³More about ethical concerns on gene editing here: http://www.geneticsandsociety.org/article.php?id=8711

⁴Liang, P. et al. 2015. Protein & Cell, 6, 363–372

⁵Science: A prudent path forward for genomic engineering and germline gene modification.

⁶Nature: NAS Gene Editing Summit.

⁷While Collins and Church participated in the summit, their views quoted here are from StatNews.com: A debate: Should we edit the human germline. See also Sciencenews.org: Editing human germline cells sparks ethics debate.

⁸Hurlbut, J. B. 2015. Limits of Responsibility, Hastings Center Report, 45(5): 11-14.

⁹This point is forcefully made by Sheila Jasanoff and colleagues: CRISPR Democracy, 2015 Issues in S&T, 22(1).

¹⁰Lander, E. 2016. The Heroes of CRISPR. Cell, 164(1-2): 18-28.

Editor's Note: This article originally appeared on Lawfare.

Faced with the breakdown of national armies in Iraq, Libya, Syria, and Yemen, Arab states have increasingly turned toward alliances with armed militias to ensure security. Popular, anti-government protests and insurgencies for the most part precipitated the breakdown of regime military institutions, yet pre-existing internal ethnic, clan, and ideological cleavages helped to hasten the breakdown. The beleaguered state security forces have now entered into a variety of alliances—tacit or active—with militias they deem sympathetic to their interests, often organized on the basis of entrenched ethno-sectarian or tribal identities. Such militia forces supplement and at times even stand in for the weak or absent army and police as providers of local security.

On the one hand, militia forces have in certain circumstances proven effective at counterinsurgency and counterterrorism. On the other hand, they have also committed atrocities against civilians that hamper long-term efforts to build trust and stability. Their greatest risk is that, by eroding the central government’s monopolization on force, they jeopardize the territorial cohesion of the state.

In Iraq, the rise of powerful communal militias has paralleled the growth of the threat from the Islamic State. This has presented the United States with a quandary: how to combat the Islamic State by mobilizing local Sunnis while at the same time safeguarding the broader integrity of the Iraqi state and its security institutions. The national guard concept, which successive Iraqi governments have tried in the past, was seen as one way to do this. A national guard force would retain the militias’ local knowledge and roots, both unique tools necessary for a successful counterinsurgency against the Islamic State. At the same time, the guard would (at least in theory) be subject to increased oversight and control by the central government.

Other fractured Arab states, most notably Libya, have tried to implement a national guard model as a way to harness militia power, but this too has failed. Variations of hybrid, provincially-organized military forces exist in Yemen and Syria. While each case is different, the failure of national guards bears certain similarities. Examining the Iraqi case in particular can highlight the potential utility of national guards but also the parallel political and institutional reforms that are necessary to make the concept work.

The idea of creating a national guard in Iraq has been a centerpiece of U.S. engagement since the dramatic advance of the Islamic State on Tikrit and Mosul in 2014. President Obama specifically mentioned U.S. support for a national guard as a means to help Iraqi Sunnis “secure their own freedom” from the Islamic State. Much of U.S. thinking about the Iraqi National Guard (ING) was guided by the example of the Sunni Awakening of 2006 and 2007, when the United States actively recruited and “flipped” Sunni tribes that had supported the al-Qaeda-inspired insurgency. In return for guarantees of autonomy and military, financial, and political backing, the Sunni tribes were able to turn the tables on the insurgent fighters and impose a measure of peace and stability. The 2014 initiative essentially sought to reproduce this arrangement. The idea was that given proper incentives, the Sunni tribes would again fight the radical Islamists who threatened their supremacy. Over the long term, such national guard forces could be integrated formally as auxiliary troops in a federal structure, comparable in many ways to the U.S. National Guard.

Yet the Awakening analogy failed on a number of levels. The Shi’i-dominated Iraqi central government had never been enthusiastic about empowering Sunni tribes in the first place. With the dismantling of the Iraqi army in 2003, security had effectively devolved to party, tribal, and sectarian militias. Many Iraqis wondered why the United States would seek to create new militias, especially ones recently tied to al-Qaeda and other terrorists. As Iraq scholar Adeed Dawisha described, the gains in security came“not because of the state, but in spite of it.”

As the U.S. began withdrawing from Iraq in 2009 and 2010, then-Prime Minister Nuri al-Maliki quickly moved to dismantle the Awakening-associated militias. Only a handful of former militia fighters received their promised positions in the police, army, or civil services. Some former militia leaders were arrested on seemingly politically-motivated charges of terrorism or subversion. Efforts to enact a Sunni-dominated super-region comparable to the federal status of the Kurdish Regional Government in the north were rebuffed, despite the provisions of Iraq’s constitution that allowed for the creation of such an entity. Politically marginalized, some Sunnis returned to their alliance with the radical mujahideen.

The election of the new prime minister Haydar al-Abadi in 2014 raised the promise of renewed Sunni-Shi’i reconciliation. Abadi expressed support for the national guard initiative and forwarded a bill to parliament in 2014. Thousands of volunteers came forward from the Sunni tribes in the west and U.S. and Iraqi officials met with tribal leaders to help solidify support. The United States began to enlist support from Iraq’s Sunni neighbors to provide training and support for the ING.

Yet resistance within Abadi’s own political coalition stymied these efforts. The National Guard bill foundered in parliamentary committee, with open questions about the extent of control vested in provincial governors and the chain of command subordinating the ING to the ministries of interior, defense, or the prime minister himself. Officers of the Iraqi Security Forces (ISF) regarded the militias as unfit for duty and as rivals for budget and resources. Iraq’s constitution specifically prohibited the formation of militias outside the framework of the armed forces (with an exception of the peshmergaforces of the Kurdish Regional Government). Moreover, there was concern that once the Sunnis were authorized to organize a militia, other ethno-sectarian communities, such as Christians or Turkomen,might try to follow suit out of fear of falling under the mercy of their more powerful neighbors. The ING, then, could undercut any pretense of the Iraqi state possessing a monopoly over the use of force.

At base, though, many of Iraq’s Shi’i leaders simply believed that they didn’t need Sunni support. With the ING initiative stalled in parliament, the Shi’i factions have actively cultivated Shi’i militias as part of the Popular Mobilization Forces (PMF, or Hashd al-Shaabi). The origins of the PMF can be traced to a statement by Grand Ayatollah Ali Sistani, Iraq’s senior Shi’i cleric, which explicitly called on the faithful to take up arms to defend Iraq in the face of the Islamic State onslaught in 2014. Muqtada al-Sadr’s Jaysh al-Mahdi, the Badr Organization, and other political factions quickly took the opportunity to reconstitute or expand their private armies.

Backed by Iran’s expeditionary al-Qods Force, PMF militias played a prominent role in the spring 2015 offensive against the Islamic State in Tikrit. By spring 2015, PMF counted around 60,000 men under arms. Still, the performance of these militias has been less than stellar. In the spring 2015 offensive on Tikrit, PMF forces failed repeatedly to dislodge Islamic State resistance, despite enjoying superiority in numbers. U.S. air support proved critical to allowing the offensive to proceed. Some PMF units quit the fight instead of working under American air cover. Others were involved in a campaign of terror against Sunnis, looting, kidnapping, and killing those suspected of collaborating with the Islamic State.

The prospects for the mobilization of Iraq’s Sunnis are not dead—yet. A handful of Sunni tribes joined the PMF during the Tikrit offensive. In Anbar, likely the next front in the campaign against the Islamic State, U.S. and Iraqi officials have cultivated ties with local Sunni tribes and organized some 8,000 men into Sunni PMF units. Some tribes have made their service conditional on guarantees of greater autonomy and the removal of Shi’i militia forces. Yet the intake for training programs remains slow and drop-out rates high. On the one hand, tribes continue to resent the central government. On the other hand, they fear retribution should the Islamic State return.

Abadi’s visit to Washington in April 2015 focused on expanding and enhancing security cooperation with the United States. The United States has insisted that the PMF be brought more fully under the control of the Iraqi Security Forces and that PMF units reflect the demographics of the provinces and districts in which they operate. This would mean that in ethnically-mixed areas, such as in Nineveh or Babil, each ethnic group would have its own militia proportional to its size in the locality. The Iraq Train and Equip Program (ITEP) is slowly coming online, funneling American money and weapons to various local militia forces as well as ISF.

Cooperating with the United States has been a delicate balancing act for Abadi. While Kurdish and Sunni leaders see U.S. military support as a means to their own ends, Abadi’s own Shi’i political camp—as well as his allies in Tehran—are far more wary. When the U.S. Congress passed a bill in May 2015 effectively mandating the Defense Department to bypass Baghdad and provide support for Sunni and Kurdish fighters directly, Abadi protested that this constituted a grave violation of Iraqi sovereignty.

Still, reliance on the ragtag PMF alone is not sustainable in the long term. Operating far from home and with limited training, these overwhelmingly Shi’i forces cannot be expected to become an army of occupation in Sunni areas like Tikrit or Fallujah. Ultimately, local partners will be necessary to build and maintain peace and stability. The national guard, then, may well re-emerge as a more sustainable structure for administrative and security devolution.

While analysts and policymakers naturally focus on cases of success, there are important lessons to be learned from Iraq’s failures. For countries like Iraq where central armies have more or less broken down and a bevy of militias has emerged in its stead, as in Libya, Yemen, and Syria, the national guard could represent a path to reconstituting fragile state authority.

But for this to happen, several broad principles need to be heeded:

• National guards cannot simply be conceived as short-term, improvised solutions to immediate security crises. Rather, the creation of national guards is part of the impetus of security-sector reform (SSR) and post-conflict demobilization, disarmament, and reintegration (DDR) of armed groups.
• National guards must overcome the legacies of past authoritarian experiences where pro-government militias were often seen as mere thugs for the regime, not a disciplined professional fighting force. In particular, the older officer class of regular forces may see them as competitors. To build trust among the population and other military institutions, national guards should be accompanied by revisions to chain of command establishing clear relationships of authority between the guards, the police, the army, and other security agencies, and subordinating all security services to civilian authorities.
• National guard initiatives must also be accompanied by moves toward political power-sharing arrangements. The success of national guards ultimately depends not just on their short-term tactical effectiveness but on the degree of local buy-in. Constitutions can provide a structure for bolstering confidence between a central government and subnational militia forces. Since militia membership and cohesion is often based on geographic linkages—to town, municipality or province—national guards may well be a part of federalist power devolution, especially in countries with overlapping ethno-sectarian and regional cleavages.
• Western governments can assist in setting up and training national guards, but they must ensure that proper political and institutional reforms are also undertaken. In many cases, Western states provide models for how decentralized, federally-organized military forces can complement national armies and local police. The United States, for instance, has a great deal of experience with its own federalized national guard structure and can draw on this example in its train-and-equip programs. There are other potentially useful models as well, including the British Territorial Army, a part-time, volunteer force that was integrated into the British Army in the early twentieth century; the Danish Home Guard, which incorporated anti-Nazi resistance militias into a national command structure after World War II; or the Italian Carabineri, which is often discussed as a potential model for dealing with Libya’s unique security challenges.

Outside assistance to national guards must avoid exacerbating existing communal and political fault lines. Helping peripheral and minority groups set up their own armed forces can, on one hand, embolden these groups to resist the central government and, on the other hand, spur resentment from the central government and fear of future disloyalty or rebellion. These concerns become even more acute when national guards are seen as proxies for outside powers. With this in mind, the U.S. and outside powers should calibrate their assistance to both regionally-based national guards and central government forces to ensure rough parity between the two. This could entail making funding, equipment and training for the central security services contingent on a proportional commitment to strengthen the guards.

National guards are political institutions, not just military instruments. They can have far-ranging consequences for political stability and cohesion. While no panacea for the challenge of building effective states, they can play an important role in addressing security concerns and moving toward more meaningful power sharing.

Abstract

Scaling up of development interventions is much debated today as a way to improve their impact and effectiveness. Based on a review of scaling up literature and practice, this paper develops a framework for the key dynamics that allow the scaling up process to happen. The authors explore the possible approaches and paths to scaling up, the drivers of expansion and of replication, the space that has to be created for interventions to grow, and the role of evaluation and of careful planning and implementation. They draw a number of lessons for the development analyst and practitioner. More than anything else, scaling up is about political and organizational leadership, about vision, values and mindset, and about incentives and accountability—all oriented to make scaling up a central element of individual, institutional, national and international development efforts. The paper concludes by highlighting some implications for aid and aid donors.

An annotated bibliography of the literature on scaling up and development aid effectiveness was created by Oksana Pidufala to supplement this working paper. Read more »

Downloads

• Download

Editor's Note: The World Bank’s Independent Evaluation Group (IEG) recently published a self-evaluation of its activities. Besides representing current thinking among evaluation experts at the World Bank, it also more broadly reflects some of the strengths and gaps in the approaches that evaluators use to assess and learn from the performance of the international institutions with which they work. The old question “Quis custodet ipsos custodes?” – loosely translated as “Who evaluates the evaluators?” – remains as relevant as ever. Johannes Linn served as an external peer reviewer of the self-evaluation and provides a bird’s-eye view on the lessons learned.

An Overview of the World Bank’s IEG Self-Evaluation Report

In 2011 the World Bank’s Independent Evaluation Group (IEG) carried out and published a self-evaluation of its activities. The self-evaluation team was led by an internal manager, but involved a respected external evaluation expert as the principal author and also an external peer reviewer.

The IEG self-evaluation follows best professional practices as codified by the Evaluation Cooperation Group (ECG). This group brings together the evaluation offices of seven major multilateral financial institutions in joint efforts designed to enhance evaluation performance and cooperation among their evaluators. One can therefore infer that the approach and focus of the IEG self-evaluation is representative of a broader set of practices that are currently used by the evaluation community of international financial organizations.

At the outset the IEG report states that “IEG is the largest evaluation department among Evaluation Capacity Group (ECG) members and is held in high regard by the international evaluation community. Independent assessments of IEG’s role as an independent evaluation function for the Bank and IFC rated it above the evaluation functions in most other ECG members, international nongovernmental organizations, and transnational corporations and found that IEG follows good practice evaluation principles.”

The self-evaluation report generally confirms this positive assessment. For four out of six areas of its mandate IEG gives itself the second highest rating (“good”) out of six possible rating categories. This includes (a) the professional quality of its evaluations, (b) its reports on how the World Bank’s management follows up on IEG recommendations, (c) cooperation with other evaluation offices, and (d) assistance to borrowing countries in improving their own evaluation capacity. In the area of appraising the World Bank’s self-evaluation and risk management practices, the report offers the third highest rating (“satisfactory”), while it gives the third lowest rating (“modest”) for IEG’s impact on the Bank’s policies, strategies and operations. In addition the self-evaluation concludes that overall the performance of IEG has been “good” and that it operates independently, effectively and efficiently.

The report makes a number of recommendations for improvement, which are likely to be helpful, but have limited impact on its activities. They cover measures to further enhance the independence of IEG and the consistency of evaluation practices as applied across the World Bank Group’s branches – the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) –; to improve the design of evaluations and the engagement with Bank management upstream for greater impact; and monitoring the impact of recent organizational changes in IEG in terms of results achieved. The report also recommends that more be done to evaluate the Bank’s analytical work and that evaluations draw on comparative evidence.

Assessment

In terms of the parameters of self-evaluation set by the prevailing practice among the evaluators on international financial agencies, the IEG self-evaluation is accurate and helpful. From my own experience as an operational manager in the Bank whose activities were evaluated by IEG in years past, and as a user of IEG evaluations (and of evaluations of other international aid organizations) for my research on aid effectiveness, I concur that IEG is independent and effective in meeting its mandate as defined. Moreover, the self-evaluation produces useful quantitative evidence (including survey results, budget analysis, etc.) to corroborate qualitative judgments.

However, the self-evaluation suffers from a number of limitations in approach and gaps in focus, which are broadly representative of the practices prevalent among many of the evaluation offices of international aid agencies.

Approach of the IEG self-evaluation

The core of the self-evaluation report is about the evaluation process followed by IEG, with very little said about the substance of IEG’s evaluations. The following questions could have usefully been raised, but were not: do evaluations cover the right issues with the right intensity, such as growth and poverty; environmental, governance, and gender impacts; regional dimensions versus exclusive country or project focus; effectiveness in addressing the problems of fragile and conflict states; effectiveness in dealing with global public goods; sustainability and scaling up; etc. Therefore the report does not deal with the question of whether IEG effectively responds in its evaluations to the many important strategic debates and issues with which the development community is grappling.

Related to this limitation is the fact that the report assessed the quality of IEG’s mostly in terms of (a) whether its approach and processes meet certain standards established by the Evaluation Cooperation Group; and (b) how it is judged by stakeholders in response to a survey commissioned for this evaluation. Both these approaches are useful, but they do not have any basis in professional assessments of the quality of individual products. This is equivalent to IEG evaluating the World Bank’s projects on the quality of its processes (e.g., appraisal and supervision processes) and on the basis of stakeholder surveys, without evaluating individual products and their impacts.

Gaps in the Self-Evaluation and in Evaluation Practice

Careful reading of the report reveals six important gaps in the IEG self-evaluation, in the prevailing evaluation practice in the World Bank, and more generally in the way international financial organizations evaluate their own performance. The first three gaps relate to aspects of the evaluation approach used and the second three gaps relate to lack of focus in the self-evaluation on key internal organizational issues:

1. Impact Evaluations: The report notes that IEG carries out two to three impact evaluations per year, but it sidesteps the debate in the current evaluation literature and practice as to what extent the “gold standard” of randomized impact evaluation should occupy a much more central role. Given the importance of this debate and divergence of views, it would have been appropriate for the self-evaluation to assess IEG’s current practice of very limited use of randomized evaluations.

2. Evaluation of Scaling Up: The report does not address the question of to what extent current IEG practice not only assesses the performance of individual projects in terms of their outcomes and sustainability, but also in terms of whether the Bank has systematically built on its experience in specific projects to help scale up their impact through support for expansion or replication in follow-up operations or through effective hand-off to the government or other partners. In fact, currently IEG does not explicitly and systematically consider scaling up in its project and program evaluations. For example, in a recent IEG evaluation of World Bank funded municipal development projects (MDPs) , IEG found that the Bank has supported multiple MDPs in many countries over the years, but the evaluation did not address the obvious question whether the Bank systematically planned for the project sequence or built on its experience from prior projects in subsequent operations. While most other evaluation offices like IEG do not consider scaling up, some (in particular those of the International Fund for Agricultural Development and the United Nations Development Program) have started doing so in recent years.

3. Drawing on the Experience of and Benchmarking Against Other Institutions: The self-evaluation report does a good job in benchmarking IEG performance in a number of respects against that of other multilateral institutions. In the main text of the report it states that “IEG plans to develop guidelines for approach papers to ensure greater quality, in particular in drawing on comparative information from other sources and benchmarking against other institutions.” This is a welcome intention, but it is inadequately motivated in the rest of the report and not reflected in the Executive Summary. The reality is that IEG, like most multilateral evaluation offices, so far has not systematically drawn on the evaluations and relevant experience of other aid agencies in its evaluations of World Bank performance. This has severely limited the learning impact of the evaluations.

4. Bank Internal Policies, Management Processes and Incentives: IEG evaluations traditionally do not focus on how the Bank’s internal policies, management and incentives affect the quality of Bank engagement in countries. Therefore evaluations cannot offer any insights into whether and how Bank-internal operating modalities contribute to results. Two recent exceptions are notable exceptions. First, the IEG evaluation of the Bank’s approach to harmonization with other donors and alignment with country priorities assesses the incentives for staff to support harmonization and alignment. The evaluation concludes that there are insufficient incentives, a finding disputed by management. Second, is the evaluation of the Bank’s internal matrix management arrangements, which is currently under way. The self-evaluation notes that Bank management tried to quash the matrix evaluation on the grounds that it did not fall under the mandate of IEG. This is an unfortunate argument, since an assessment of the institutional reasons for the Bank’s performance is an essential component of any meaningful evaluation of Bank-supported programs. While making a good case for the specific instance of the matrix evaluation, the self-evaluation report shies away from a more general statement in support of engaging IEG on issues of Bank-internal policies, management processes and incentives. It is notable that IFAD’s Independent Office of Evaluation appears to be more aggressive in this regard: It currently is carrying out a full evaluation of IFAD’s internal efficiency and previous evaluations (e.g., an evaluation of innovation and scaling up) did not shy away from assessing internal institutional dimensions.

5. World Bank Governance: The IEG self-evaluation is even more restrictive in how it interprets its mandate regarding the evaluation of the World Bank’s governance structures and processes (including its approach to members’ voice and vote, the functioning of its board of directors, the selection of its senior management, etc.). It considers these topics beyond IEG’s mandate. This is unfortunate, since the way the Bank’s governance evolves will substantially affect its long-term legitimacy, effectiveness and viability as an international financial institution. Since IEG reports to the Bank’s board of directors, and many of the governance issues involve questions of the board’s composition, role and functioning, there is a valid question of how effectively IEG could carry out such an evaluation. However, it is notable that the IMF’s Independent Evaluation Office, which similarly reports to the IMF board of directors, published a full evaluation of the IMF’s governance in 2008, which effectively addressed many of the right questions.

6. Synergies between World Bank, IFC and MIGA: The self-evaluation report points out that the recent internal reorganization of IEG aimed to assure more effective and consistent evaluations across the three member branches of the World Bank Group. This is welcome, but the report does not assess how past evaluations addressed the question of whether the World Bank, IFC and MIGA effectively capitalized on the potential synergies among the three organizations. The recent evaluation of the World Bank Group’s response to the global economic crisis of 2008/9 provided parallel assessments of each agency’s performance, but did not address whether they work together effectively in maximizing their synergies. The reality is that the three organizations have deeply engrained institutional cultures and generally go their own ways rather than closely coordinating their activities on the ground. Future evaluations should explicitly consider whether the three effectively cooperate or not. While the World Bank is unique in the way it has organizationally separated its private sector and guarantee operations, other aid organizations also have problems of a lack of cooperation, coordination and synergy among different units within the agency. Therefore, the same comment also applies to their evaluation approaches.

Conclusions

Self-evaluations are valuable tools for performance assessment and IEG is to be congratulated for carrying out and publishing such an evaluation of its own activities. As for all self-evaluations, it should be seen as an input to an independent external evaluation, a decision that, for now, has apparently been postponed by the Bank’s board of directors.

IEG’s self-evaluation has many strengths and provides an overall positive assessment of IEG’s work. However, it does reflect some important limitations of analysis and of certain gaps in approach and coverage, which an independent external review should consider explicitly, and which IEG’s management should address. Since many of these issues also likely apply to most of the other evaluation approaches by other evaluation offices, the lessons have relevance beyond IEG and the World Bank.

Key lessons include:

• An evaluation of evaluations should focus not only on process, but also on the substantive issues that the institution is grappling with.
• An evaluation of the effectiveness of evaluations should include a professional assessment of the quality of evaluation products.
• An evaluation of evaluations should assess:
  o How effectively impact evaluations are used;
  o How scaling up of successful interventions is treated;
  o How the experience of other comparable institutions is utilized;
  o Whether and how the internal policies, management practices and incentives of the institution are effectively assessed;
  o Whether and how the governance of the institution is evaluated; and
  o Whether and how internal coordination, cooperation and synergy among units within the organizations are assessed.

Evaluations play an essential role in the accountability and learning of international aid organizations. Hence it is critical that evaluations address the right issues and use appropriate techniques. If the lessons above were reflected in the evaluation practices of the aid institutions, this would represent a significant step forward in the quality, relevance and likely impact of evaluations.

Event Information

Australia's mandatory Superannuation Guarantee requires its citizens to save at least 9 percent of their income towards retirement. In many Asian nations, economic growth has spurred reexamination of pension systems to meet the needs of rapidly evolving societies. Would a mandatory savings plan be more effective than the current U.S. voluntary system? How have Asian nations have restructured their pension systems to deal with legacy costs? And what can Americans learn from the way Australia uses both employer and employee representatives to shape investment choices?

On September 17, the Retirement Security Project at Brookings and the AARP Public Policy Institute hosted a discussion of what the United States might learn from retirement savings systems in Australia and Asia. Opening speakers included Nick Sherry, who helped shape the Australian system as a cabinet minister and ran a Superannuation fund in the private sector, and Josef Pilger, an advisor on pension reform to both the Malaysian and Hong Kong governments and many industry providers. Steve Utkus, David Harris and Benjamin Harris, retirement experts from both the United States and the United Kingdom, considered how reforms in Australia and Asia can shape the American debate and whether this country should adopt key features from those foreign systems.

Audio

• Retirement Savings in Australia, Asia and Beyond: What are the Lessons for the United States?

Transcript

• Uncorrected Transcript (.pdf)

Event Materials

• 20130917_retirement_savings_transcript

Event Information

Seven years ago, New Zealand recognized that if its people did not have sufficient assets as they aged, they would either face economic stress in retirement or place pressure on the government for costly additional benefits, and thus the KiwiSaver program was born. Designed to help citizens build retirement security, it guides individuals with limited financial experience while also giving them complete control of their finances. Benefits of this national automatic enrollment retirement savings plan include a $1,000 kick-start, employer contributions, and an annual tax credit. New Zealand Since its inception in July 2007, KiwiSaver has been deemed a great success, with over half of the eligible population as members, and over 70 percent of 18-24 year olds participating. Although membership continues to grow, it is at a slower rate than that seen in previous years.

Could the success of KiwiSaver mean that a similar program – at either the national or state level – might work here? On July 8^th, Diana Crossan, former Retirement Commissioner for New Zealand, will offer her insights into the KiwiSaver program and its impact on New Zealand saving, retirement security, and financial literacy. Ben Harris and David John, deputy directors of the Retirement Security Project at Brookings, will reflect on the role such a program might play in the U.S.

Email international@aarp.org to RSVP » 

American experience strongly suggests that the coming UK pension freedoms sound better in theory than they will work in practice. After nearly a decade where the UK has been the gold standard for retirement savings policy, it is about to take a step that it may regret.

As annuity purchases are not required, very few Americans buy them, feeling that they are spending a great deal of money for a comparatively small monthly income. Even those in traditional DB pension plans usually take a lump sum if they are allowed to do so. As a result, many US retirees spend unwisely, trust the wrong financial advisor, or make other financial mistakes.

Many people greatly overestimate how long their savings will last. Most others assume (often wrongly) that they can manage their own money as well as anyone else or that they can live comfortably on Social Security alone. U.S. Social Security pays a benefit that depends on the retirees’ individual income history. The average annual amount is about $13,000 (GBP 8,700).

One survey found that in West Virginia, a state with a relatively low average income, 78% of those near retirement and 67% of those at retirement would likely outlive their financial assets. Workers with lower incomes are most at risk. A recent national study found that by the 20th year of retirement, more than 81% of Americans with incomes up to $27,000 would run short of money, as would 38% of those earning up to $42,000, and 19% of those with incomes up to $65,000.  Even 8% of those with the highest incomes could not meet their expenses.

Advice alone is not likely to help. US experience shows that literally every minute that passes after general advice is given reduces the chance that the consumer will act on it – even when they have decided to do so. And even a significant number of those who consult with a financial planner fail to act on that guidance.

What does show promise is income illustration. In a 2014 U.S. survey, 85% of plan participants found estimates of the income they could anticipate from their retirement savings useful, and 35% said that they would save more. Income illustrations change the framing of retirement saving from gross amounts saved to retirement income.  Annuity-like products become insurance against running out of money, something Americans are increasingly concerned about.

Two other potential developments may help. One is longevity insurance, an annuity that provides income only after a set age. Purchasing a policy defines how long one must make retirement savings last, and the retiree is protected against running out of money. Because longevity insurance is deferred, one can receive higher amounts of monthly income for a lower cost.  In 2014, $50,000 would buy $275 a month at age 65 or $1200 a month starting at age 80.

Another idea is an automatic enrollment trial annuity. As developed by several Brookings Institution colleagues and me, new retirees would automatically use part of their savings for a two year annuity unless the retiree refused it. The rest of their savings would be available as a lump sum. After the trial period, the annuity would become permanent if they did nothing or they could cancel it and take the rest of their money as a lump sum.

The many annuity horror stories from the UK show a definite need for change, but the coming reforms go too far. US experience suggests that too many UK retirees are likely to see their savings exhausted all too quickly. There are alternatives that could do a better job of protecting retirees.

British politics is starting to resemble a bowling alley. One after another, political figures are tumbling–including the leading lights of the Brexit campaign. They sowed the wind and now are reaping the whirlwind.

First to topple was the prime minister. After the referendum, David Cameron announced that he would step down. Last week fellow Conservative Boris Johnson, the leading light of the Brexit campaign, said he would not run to succeed Mr. Cameron after his ally Michael Gove, the justice secretary, concluded, in quintessentially British style, that Mr. Johnson lacked “the team captaincy” required. Then Nigel Farage stepped down as leader of the UK Independence Party, saying “I want my life back.” Labour Party leader Jeremy Corbyn has lost the support of his parliamentary colleagues and may be next to fall.

The exit of the leading Brexiteers is a relief. The skills required to run a populist, fact-averse campaign are not the same skills needed to lead a nation. For all his mercurial talents, on full display during his colorful stint as mayor of London, Boris Johnson would have been a disastrous prime minister. The alternatives–especially Mr. Gove and Home Secretary Theresa May–are steadier souls. Both are also better positioned to unite Conservative members of Parliament and hold on until the next scheduled general election, in 2020.

Mr. Corbyn is likely to go; the question really is when. It he doesn’t, the Labour Party will break apart. In his case the departure will be only slightly about the vote to remain in or leave the European Union. Broadly, his fellow Labour MPs didn’t want him as their leader in the first place; it was the votes of more left-wing party members that propelled him to the leadership, and many see him as an electoral liability. (He is.)

There is no direct connection between Brexit and Donald Trump. But a few things can still be deduced on this side of the pond. First, Mr. Trump may succeed in making the connection tighter. His immediate announcement that the vote was about “declaring independence” reflected his sharpening political instincts. The day after the vote, Mr. Trump said: “The people of the United Kingdom have exercised the sacred right of all free peoples. They have declared their independence from the European Union. … Come November, the American people will have the chance to re-declare their independence. Americans will have a chance to vote for trade, immigration and foreign policies that put our citizens first.”

Independence is a powerful populist theme, one Mr. Trump is likely to exploit it to its fullest.

Brexit and the economic and political chaos it has already sparked are proof that no matter how crazy or far-fetched an electoral outcome appears, it can happen. Right up to the last minute, many believed that even if the vote were close, it would be to remain in the EU. At some level we just couldn’t imagine the alternative. Maybe Mr. Cameron and Mr. Corybn felt the same, which is why they were so complacent. Not so, the other side.

All this suggests the wisdom of treating every poll with a fistful of salt. Electorates are becoming more volatile and more visceral. Pollsters are getting it wrong as often as they get it right. The last general election in the U.K. is another case in point. Populist sentiment wrecks standard political models. When people are angry, they don’t weigh the costs and benefits of their actions in the usual way; that’s true in life and it’s true in voting.

It’s also why it’s risky to allow populist campaigners near the levers of power. I’ve written in this space before about the dangers of injecting direct democracy in a parliamentary political system. Think of referendums as akin to Ming vases: something rare, to be handled with great care. The British Parliament is now acting as a firebreak. The leading populists will not get the keys to 10 Downing Street.

But the United States holds direct elections for president. If Donald Trump wins in November, he will assume the most powerful office in the world. There is no firebreak, no buffer, no second chance.

Editor's note: This piece originally appeared on the Wall Street Journal's Washington Wire blog.

Event Information

Despite being a developed and prosperous country, Japan faces a host of basic challenges today and going forward—some of its own creation and others beyond the country’s control. For example, Japan lacks essential natural resources, while also facing overcrowding in cities and depopulation in rural areas. As a result, food and energy self-sufficiency is low. Also, while the dual phenomena of a low birthrate and an ageing population have long been deemed problematic, these issues are rapidly growing more serious. The problems Japan faces today are potentially the same problems the rest of the world will face in the near future. Japan, therefore, may serve as a bellwether for the global community as many nations anticipate similar challenges in the future.  

On October 26, the Center for East Asia Policy Studies at Brookings and the U.S.-Japan Research Institute co-hosted Hiroshi Komiyama, chairman of the Mitsubishi Research Institute and president emeritus of the University of Tokyo, for a discussion of his recent book, “Beyond the Limits to Growth: New Ideas for Sustainability from Japan.” In this book, Komiyama examines the issues facing Japan—and the world—presenting a number of potential viable solutions and offering insights into Japan’s experiences and the lessons it can provide for a more sustainable future.

Audio

• Overcoming the limits to growth: Sustainability lessons from Japan

Transcript

• Uncorrected Transcript (.pdf)

Event Materials

• 20151026_japan_sustainability_transcript
• hiroshi komiyama presentation

Event Information

Despite the notable success in enrolling children in primary school over the past decade, the education agenda is unfinished as millions of children are still excluded from learning opportunities and millions more leave school without having acquired the essential knowledge and skills needed to participate in society.

On January 27, the Center for Universal Education at Brookings hosted a half-day conference that focused on the research examining “what works in education” to achieve improved learning opportunities and outcomes. In addition to hearing from researchers studying the effectiveness of various education strategies, participants discussed how to facilitate a future research agenda that could have the most meaningful impact on learning. Senior Fellow Jacques van der Gaag moderated the discussion.

View the full event summary »

Transcript

• Improving Education in the Developing World: Evidence from Randomized Evaluations Presentation (.pdf)
• Sociocultural Gradients for Language Scores by Country Slide (.pdf)
• School Resources and Educational Outcomes in Developing Countries Presentation (.pdf)
• Making Learning for All a Reality Presentation (.pdf)
• Addressing the Global Learning Crisis Presentation (.pdf)

Event Materials

• 0127_improving_education_in_developing_world
• 0127_slide_vandergaag
• 0127_edinptoh
• 0127_king
• 0127_wagner

It has been hailed as the most significant discovery in biology since polymerase chain reaction allowed for the mass replication of DNA samples. CRISPR-Cas9 is an inexpensive and easy-to-use gene-editing method that promises applications ranging from medicine to industrial agriculture to biofuels. Currently, applications to treat leukemia, HIV, and cancer are under experimental development.¹ However, new technical solutions tend to be fraught with old problems, and in this case, ethical and legal questions loom large over the future.

Disagreements on ethics

The uptake of this method has been so fast that many scientists have started to worry about inadequate regulation of research and its unanticipated consequences.² Consider, for instance, the disagreement on research on human germ cells (eggs, sperm, or embryos) where an edited gene is passed onto offspring. Since the emergence of bioengineering applications in the 1970s, the scientific community has eschewed experiments to alter human germline and some governments have even banned them.³ The regulation regimes are expectedly not uniform: for instance, China bans the implantation of genetically modified embryos in women but not the research with embryos.

Last year, a group of Chinese researchers conducted gene-editing experiments on non-viable human zygotes (fertilized eggs) using CRISPR.⁴ News that these experiments were underway prompted a group of leading U.S. geneticists to meet in March 2015 in Napa, California, to begin a serious consideration of ethical and legal dimensions of CRISPR and called for a moratorium on research editing genes in human germline.⁵ Disregarding that call, the Chinese researchers published their results later in the year largely reporting a failure to precisely edit targeted genes without accidentally editing non-targets. CRISPR is not yet sufficiently precise.

CRISPR reignited an old debate on human germline research that is one of the central motivations (but surely not the only one) for an international summit on gene editing hosted by the U.S. National Academies of Sciences, the Chinese Academy of Sciences, and the U.K.'s Royal Society in December 2015. About 500 scientists as well as experts in the legal and ethical aspects of bioengineering attended.⁶ Rather than consensus, the meeting highlighted the significant contrasts among participants about the ethics of inquiry, and more generally, about the governance of science. Illustrative of these contrasts are the views of prominent geneticists Francis Collins, Director of the National Institutes of Health, and George Church, professor of genetics at Harvard. Collins argues that the “balance of the debate leans overwhelmingly against human germline engineering.” In turn, Church, while a signatory of the moratorium called by the Napa group, has nevertheless suggested reasons why CRISPR is shifting the balance in favor of lifting the ban on human germline experiments.⁷

The desire to speed up discovery of cures for heritable diseases is laudable. But tinkering with human germline is truly a human concern and cannot be presumed to be the exclusive jurisdictions of scientists, clinicians, or patients. All members of society have a stake in the evolution of CRISPR and must be part of the conversation about what kind of research should be permitted, what should be discouraged, and what disallowed. To relegate lay citizens to react to CRISPR applications—i.e. to vote with their wallets once applications hit the market—is to reduce their citizenship to consumer rights, and public participation to purchasing power.⁸ Yet, neither the NAS summit nor the earlier Napa meeting sought to solicit the perspectives of citizens, groups, and associations other than those already tuned in the CRISPR debates.⁹

The scientific community has a bond to the larger society in which it operates that in its most basic form is the bond of the scientist to her national community, is the notion that the scientist is a citizen of society before she is a denizen of science. This bond entails liberties and responsibilities that transcend the ethos and telos of science and, consequently, subordinates science to the social compact. It is worth recalling this old lesson from the history of science as we continue the public debate on gene editing. Scientists are free to hold specific moral views and prescriptions about the proper conduct of research and the ethical limits of that conduct, but they are not free to exclude the rest of society from weighing in on the debate with their own values and moral imaginations about what should be permitted and what should be banned in research. The governance of CRISPR is a question of collective choice that must be answered by means of democratic deliberation and, when irreconcilable differences arise, by the due process of democratic institutions.

Patent disputes

More heated than the ethical debate is the legal battle for key CRISPR patents that has embroiled prominent scientists involved in perfecting this method. The U.S. Patent and Trademark Office initiated a formal contestation process, called interference, in March 2016 to adjudicate the dispute. The process is likely to take years and appeals are expected to extend further in time. Challenges are also expected to patents filed internationally, including those filed with the European Patent Office.

To put this dispute in perspective, it is instructive to consider the history of CRISPR authored by one of the celebrities in gene science, Eric Lander.¹⁰ This article ignited a controversy because it understated the role of one of the parties to the patent dispute (Jennifer Doudna and Emmanuelle Charpentier), while casting the other party as truly culminating the development of this technology (Feng Zhang, who is affiliated to Lander’s Broad Institute). Some gene scientists accused Lander of tendentious inaccuracies and of trying to spin a story in a manner that favors the legal argument (and economic interest) of Zhang.

Ironically, the contentious article could be read as an argument against any particular claim to the CRISPR patents as it implicitly questions the fairness of granting exclusive rights to an invention. Lander tells the genesis of CRISPR that extends through a period of two decades and over various countries, where the protagonists are the many researchers who contributed to the cumulative knowledge in the ongoing development of the method. The very title of Lander’s piece, “The Heroes of CRISPR” highlights that the technology has not one but a plurality of authors.

A patent is a legal instrument that recognizes certain rights of the patent holder (individual, group, or organization) and at the same time denies those rights to everyone else, including those other contributors to the invention. Patent rights are thus arbitrary under the candle of history. I am not suggesting that the bureaucratic rules to grant a patent or to determine its validity are arbitrary; they have logical rationales anchored in practice and precedent. I am suggesting that in principle any exclusive assignation of rights that does not include the entire community responsible for the invention is arbitrary and thus unfair. The history of CRISPR highlights this old lesson from the history of technology: an invention does not belong to its patent holder, except in a court of law.

Some scientists may be willing to accept with resignation the unfair distribution of recognition granted by patents (or prizes like the Nobel) and find consolation in the fact that their contribution to science has real effects on people’s lives as it materializes in things like new therapies and drugs. Yet patents are also instrumental in distributing those real effects quite unevenly. Patents create monopolies that, selling their innovation at high prices, benefit only those who can afford them. The regular refrain to this charge is that without the promise of high profits, there would be no investments in innovation and no advances in life-saving medicine. What’s more, the biotech industry reminds us that start-ups will secure capital injections only if they have exclusive rights to the technologies they are developing. Yet, Editas Medicine, a biotech start-up that seeks to exploit commercial applications of CRISPR (Zhang is a stakeholder), was able to raise $94 million in its February 2016 initial public offering. That some of Editas’ key patents are disputed and were entering interference at USPTO was patently not a deterrent for those investors.

Towards a CRISPR democratic debate

Neither the governance of gene-editing research nor the management of CRISPR patents should be the exclusive responsibility of scientists. Yet, they do enjoy an advantage in public deliberations on gene editing that is derived from their technical competence and from the authority ascribed to them by society. They can use this advantage to close the public debate and monopolize its terms, or they could turn it into stewardship of a truly democratic debate about CRISPR.

The latter choice can benefit from three steps. A first step would be openness: a public willingness to consider and internalize public values that are not easily reconciled with research values. A second step would be self-restraint: publicly affirming a self-imposed ban on research with human germline and discouraging research practices that are contrary to received norms of prudence. A third useful step would be a public service orientation in the use of patents: scientists should pressure their universities, who hold title to their inventions, to preserve some degree of influence over research commercialization so that the dissemination and access to innovations is consonant with the noble aspirations of science and the public service mission of the university. Openness, self-restraint, and an orientation to service from scientists will go a long way to make of CRISPR a true servant of society and an instrument of democracy.

Other reading: See media coverage compiled by the National Academies of Sciences.

¹Nature: an authoritative and accessible primer. A more technical description of applications in Hsu, P. D. et al. 2014. Cell, 157(6): 1262–1278.

³More about ethical concerns on gene editing here: http://www.geneticsandsociety.org/article.php?id=8711

⁴Liang, P. et al. 2015. Protein & Cell, 6, 363–372

⁵Science: A prudent path forward for genomic engineering and germline gene modification.

⁶Nature: NAS Gene Editing Summit.

⁷While Collins and Church participated in the summit, their views quoted here are from StatNews.com: A debate: Should we edit the human germline. See also Sciencenews.org: Editing human germline cells sparks ethics debate.

⁸Hurlbut, J. B. 2015. Limits of Responsibility, Hastings Center Report, 45(5): 11-14.

⁹This point is forcefully made by Sheila Jasanoff and colleagues: CRISPR Democracy, 2015 Issues in S&T, 22(1).

¹⁰Lander, E. 2016. The Heroes of CRISPR. Cell, 164(1-2): 18-28.

Editor's Note: The World Bank’s Independent Evaluation Group (IEG) recently published a self-evaluation of its activities. Besides representing current thinking among evaluation experts at the World Bank, it also more broadly reflects some of the strengths and gaps in the approaches that evaluators use to assess and learn from the performance of the international institutions with which they work. The old question “Quis custodet ipsos custodes?” – loosely translated as “Who evaluates the evaluators?” – remains as relevant as ever. Johannes Linn served as an external peer reviewer of the self-evaluation and provides a bird’s-eye view on the lessons learned.

An Overview of the World Bank’s IEG Self-Evaluation Report

In 2011 the World Bank’s Independent Evaluation Group (IEG) carried out and published a self-evaluation of its activities. The self-evaluation team was led by an internal manager, but involved a respected external evaluation expert as the principal author and also an external peer reviewer.

The IEG self-evaluation follows best professional practices as codified by the Evaluation Cooperation Group (ECG). This group brings together the evaluation offices of seven major multilateral financial institutions in joint efforts designed to enhance evaluation performance and cooperation among their evaluators. One can therefore infer that the approach and focus of the IEG self-evaluation is representative of a broader set of practices that are currently used by the evaluation community of international financial organizations.

At the outset the IEG report states that “IEG is the largest evaluation department among Evaluation Capacity Group (ECG) members and is held in high regard by the international evaluation community. Independent assessments of IEG’s role as an independent evaluation function for the Bank and IFC rated it above the evaluation functions in most other ECG members, international nongovernmental organizations, and transnational corporations and found that IEG follows good practice evaluation principles.”

The self-evaluation report generally confirms this positive assessment. For four out of six areas of its mandate IEG gives itself the second highest rating (“good”) out of six possible rating categories. This includes (a) the professional quality of its evaluations, (b) its reports on how the World Bank’s management follows up on IEG recommendations, (c) cooperation with other evaluation offices, and (d) assistance to borrowing countries in improving their own evaluation capacity. In the area of appraising the World Bank’s self-evaluation and risk management practices, the report offers the third highest rating (“satisfactory”), while it gives the third lowest rating (“modest”) for IEG’s impact on the Bank’s policies, strategies and operations. In addition the self-evaluation concludes that overall the performance of IEG has been “good” and that it operates independently, effectively and efficiently.

The report makes a number of recommendations for improvement, which are likely to be helpful, but have limited impact on its activities. They cover measures to further enhance the independence of IEG and the consistency of evaluation practices as applied across the World Bank Group’s branches – the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) –; to improve the design of evaluations and the engagement with Bank management upstream for greater impact; and monitoring the impact of recent organizational changes in IEG in terms of results achieved. The report also recommends that more be done to evaluate the Bank’s analytical work and that evaluations draw on comparative evidence.

Assessment

In terms of the parameters of self-evaluation set by the prevailing practice among the evaluators on international financial agencies, the IEG self-evaluation is accurate and helpful. From my own experience as an operational manager in the Bank whose activities were evaluated by IEG in years past, and as a user of IEG evaluations (and of evaluations of other international aid organizations) for my research on aid effectiveness, I concur that IEG is independent and effective in meeting its mandate as defined. Moreover, the self-evaluation produces useful quantitative evidence (including survey results, budget analysis, etc.) to corroborate qualitative judgments.

However, the self-evaluation suffers from a number of limitations in approach and gaps in focus, which are broadly representative of the practices prevalent among many of the evaluation offices of international aid agencies.

Approach of the IEG self-evaluation

The core of the self-evaluation report is about the evaluation process followed by IEG, with very little said about the substance of IEG’s evaluations. The following questions could have usefully been raised, but were not: do evaluations cover the right issues with the right intensity, such as growth and poverty; environmental, governance, and gender impacts; regional dimensions versus exclusive country or project focus; effectiveness in addressing the problems of fragile and conflict states; effectiveness in dealing with global public goods; sustainability and scaling up; etc. Therefore the report does not deal with the question of whether IEG effectively responds in its evaluations to the many important strategic debates and issues with which the development community is grappling.

Related to this limitation is the fact that the report assessed the quality of IEG’s mostly in terms of (a) whether its approach and processes meet certain standards established by the Evaluation Cooperation Group; and (b) how it is judged by stakeholders in response to a survey commissioned for this evaluation. Both these approaches are useful, but they do not have any basis in professional assessments of the quality of individual products. This is equivalent to IEG evaluating the World Bank’s projects on the quality of its processes (e.g., appraisal and supervision processes) and on the basis of stakeholder surveys, without evaluating individual products and their impacts.

Gaps in the Self-Evaluation and in Evaluation Practice

Careful reading of the report reveals six important gaps in the IEG self-evaluation, in the prevailing evaluation practice in the World Bank, and more generally in the way international financial organizations evaluate their own performance. The first three gaps relate to aspects of the evaluation approach used and the second three gaps relate to lack of focus in the self-evaluation on key internal organizational issues:

1. Impact Evaluations: The report notes that IEG carries out two to three impact evaluations per year, but it sidesteps the debate in the current evaluation literature and practice as to what extent the “gold standard” of randomized impact evaluation should occupy a much more central role. Given the importance of this debate and divergence of views, it would have been appropriate for the self-evaluation to assess IEG’s current practice of very limited use of randomized evaluations.

2. Evaluation of Scaling Up: The report does not address the question of to what extent current IEG practice not only assesses the performance of individual projects in terms of their outcomes and sustainability, but also in terms of whether the Bank has systematically built on its experience in specific projects to help scale up their impact through support for expansion or replication in follow-up operations or through effective hand-off to the government or other partners. In fact, currently IEG does not explicitly and systematically consider scaling up in its project and program evaluations. For example, in a recent IEG evaluation of World Bank funded municipal development projects (MDPs) , IEG found that the Bank has supported multiple MDPs in many countries over the years, but the evaluation did not address the obvious question whether the Bank systematically planned for the project sequence or built on its experience from prior projects in subsequent operations. While most other evaluation offices like IEG do not consider scaling up, some (in particular those of the International Fund for Agricultural Development and the United Nations Development Program) have started doing so in recent years.

3. Drawing on the Experience of and Benchmarking Against Other Institutions: The self-evaluation report does a good job in benchmarking IEG performance in a number of respects against that of other multilateral institutions. In the main text of the report it states that “IEG plans to develop guidelines for approach papers to ensure greater quality, in particular in drawing on comparative information from other sources and benchmarking against other institutions.” This is a welcome intention, but it is inadequately motivated in the rest of the report and not reflected in the Executive Summary. The reality is that IEG, like most multilateral evaluation offices, so far has not systematically drawn on the evaluations and relevant experience of other aid agencies in its evaluations of World Bank performance. This has severely limited the learning impact of the evaluations.

4. Bank Internal Policies, Management Processes and Incentives: IEG evaluations traditionally do not focus on how the Bank’s internal policies, management and incentives affect the quality of Bank engagement in countries. Therefore evaluations cannot offer any insights into whether and how Bank-internal operating modalities contribute to results. Two recent exceptions are notable exceptions. First, the IEG evaluation of the Bank’s approach to harmonization with other donors and alignment with country priorities assesses the incentives for staff to support harmonization and alignment. The evaluation concludes that there are insufficient incentives, a finding disputed by management. Second, is the evaluation of the Bank’s internal matrix management arrangements, which is currently under way. The self-evaluation notes that Bank management tried to quash the matrix evaluation on the grounds that it did not fall under the mandate of IEG. This is an unfortunate argument, since an assessment of the institutional reasons for the Bank’s performance is an essential component of any meaningful evaluation of Bank-supported programs. While making a good case for the specific instance of the matrix evaluation, the self-evaluation report shies away from a more general statement in support of engaging IEG on issues of Bank-internal policies, management processes and incentives. It is notable that IFAD’s Independent Office of Evaluation appears to be more aggressive in this regard: It currently is carrying out a full evaluation of IFAD’s internal efficiency and previous evaluations (e.g., an evaluation of innovation and scaling up) did not shy away from assessing internal institutional dimensions.

5. World Bank Governance: The IEG self-evaluation is even more restrictive in how it interprets its mandate regarding the evaluation of the World Bank’s governance structures and processes (including its approach to members’ voice and vote, the functioning of its board of directors, the selection of its senior management, etc.). It considers these topics beyond IEG’s mandate. This is unfortunate, since the way the Bank’s governance evolves will substantially affect its long-term legitimacy, effectiveness and viability as an international financial institution. Since IEG reports to the Bank’s board of directors, and many of the governance issues involve questions of the board’s composition, role and functioning, there is a valid question of how effectively IEG could carry out such an evaluation. However, it is notable that the IMF’s Independent Evaluation Office, which similarly reports to the IMF board of directors, published a full evaluation of the IMF’s governance in 2008, which effectively addressed many of the right questions.

6. Synergies between World Bank, IFC and MIGA: The self-evaluation report points out that the recent internal reorganization of IEG aimed to assure more effective and consistent evaluations across the three member branches of the World Bank Group. This is welcome, but the report does not assess how past evaluations addressed the question of whether the World Bank, IFC and MIGA effectively capitalized on the potential synergies among the three organizations. The recent evaluation of the World Bank Group’s response to the global economic crisis of 2008/9 provided parallel assessments of each agency’s performance, but did not address whether they work together effectively in maximizing their synergies. The reality is that the three organizations have deeply engrained institutional cultures and generally go their own ways rather than closely coordinating their activities on the ground. Future evaluations should explicitly consider whether the three effectively cooperate or not. While the World Bank is unique in the way it has organizationally separated its private sector and guarantee operations, other aid organizations also have problems of a lack of cooperation, coordination and synergy among different units within the agency. Therefore, the same comment also applies to their evaluation approaches.

Conclusions

Self-evaluations are valuable tools for performance assessment and IEG is to be congratulated for carrying out and publishing such an evaluation of its own activities. As for all self-evaluations, it should be seen as an input to an independent external evaluation, a decision that, for now, has apparently been postponed by the Bank’s board of directors.

IEG’s self-evaluation has many strengths and provides an overall positive assessment of IEG’s work. However, it does reflect some important limitations of analysis and of certain gaps in approach and coverage, which an independent external review should consider explicitly, and which IEG’s management should address. Since many of these issues also likely apply to most of the other evaluation approaches by other evaluation offices, the lessons have relevance beyond IEG and the World Bank.

Key lessons include:

• An evaluation of evaluations should focus not only on process, but also on the substantive issues that the institution is grappling with.
• An evaluation of the effectiveness of evaluations should include a professional assessment of the quality of evaluation products.
• An evaluation of evaluations should assess:
  o How effectively impact evaluations are used;
  o How scaling up of successful interventions is treated;
  o How the experience of other comparable institutions is utilized;
  o Whether and how the internal policies, management practices and incentives of the institution are effectively assessed;
  o Whether and how the governance of the institution is evaluated; and
  o Whether and how internal coordination, cooperation and synergy among units within the organizations are assessed.

Evaluations play an essential role in the accountability and learning of international aid organizations. Hence it is critical that evaluations address the right issues and use appropriate techniques. If the lessons above were reflected in the evaluation practices of the aid institutions, this would represent a significant step forward in the quality, relevance and likely impact of evaluations.

The challenge of rural poverty and food insecurity in the developing world remains daunting. Recent estimates show that “there are still about 1.2 billion extremely poor people in the world. In addition, about 870 million people are undernourished, and about 2 billion people suffer from micronutrient deficiency. About 70 percent of the world’s poor live in rural areas, and many have some dependency on agriculture,” (Cleaver 2012). Addressing this challenge by assisting rural small-holder farmers in developing countries is the mandate of the International Fund for Agricultural Development (IFAD), an international financial institution based in Rome.

The International Fund for Agricultural Development is a relatively small donor in the global aid architecture, accounting for approximately one-half of 1 percent of all aid paid directly to developing countries in 2010. Although more significant in its core area of agricultural and rural development, IFAD still accounts for less than 5 percent of total official development assistance in that sector.1 Confronted with the gap between its small size and the large scale of the problem it has been mandated to address, IFAD seeks ways to increase its impact for every dollar it invests in agriculture and rural development on behalf of its member states. One indicator of this intention to scale up is that it has set a goal to reach 90 million rural poor between 2012 and 2015 and lift 80 million out of poverty during that time. These numbers are roughly three times the number of poor IFAD has reached previously during a similar time span. More generally, IFAD has declared that scaling up is “mission critical,” and this scaling-up objective is now firmly embedded in its corporate strategy and planning statements. Also, increasingly, IFAD’s operational practices are geared towards helping its clients achieve scaling up on the ground with the support of its loans and grants.

This was not always the case. For many years, IFAD stressed innovation as the key to success, giving little attention to systematically replicating and building on successful innovations. In this regard, IFAD was not alone. In fact, few aid agencies have systematically pursued the scaling up of successful projects. However, in 2009, IFAD management decided to explore how it could increase its focus on scaling up. It gave a grant to the Brookings Institution to review IFAD’s experience with scaling up and to assess its operational strategies, policies and processes with a view to strengthening its approach to scaling up. Based on an extensive review of IFAD documentation, two country case studies and intensive interactions with IFAD staff and managers, the Brookings team prepared a report that it submitted to IFAD management in June 2010 and published as a Brookings Global Working Paper in early 2011 (Linn et al. 2011).

Download the paper (PDF) »

Downloads

• Download the paper

To achieve the desired outcomes of the Sustainable Development Goals as well as the global targets from the Paris COP21 Climate Summit by 2030, governments will have to find ways to meet the top-down objectives with bottom-up approaches. A systematic focus on scaling up successful development interventions could serve to bridge this gap, or what’s been called the “missing middle.” However, the question remains how to actually address the challenge of scaling up.

When Arna Hartmann, adjunct professor of international development, and I first looked at the scaling up agenda in development work in the mid-2000s, we concluded that development agencies were insufficiently focused on supporting the scaling up of successful development interventions. The pervasive focus on one-off projects all too often resulted in what I’ve come to refer to as “pilots to nowhere.” As a first step to fix this, we recommended that each aid organization carry out a review to be sure to focus effectively on scaling up. 

The institutional dimension is critical, given their role in developing and implementing scaling up pathways. Of course, individuals serve as champions, designers, and implementers, but experience illustrates that if individuals lack a strong link to a supportive institution, scaling up is most likely to be short-lived and unsustainable. “Institutions” include many different types of organizations, such as government ministries and departments, private firms and social enterprises, civil society organizations, and both public and private external donors and financiers.

The Brookings book “Getting to Scale: How to Bring Development Solutions to Millions of Poor People” explores the opportunities and challenges that such organizations face, on their own or, better yet, partnering with each other, in scaling up the development impact of their successful interventions.

Over the past decade I have worked with 10 foreign aid institutions—multilateral and bilateral agencies, as well as big global non-governmental organizations—helping them to focus systematically on scaling up operational work and developing approaches to do so. There are common lessons that apply across the board to these agencies, with one salutary example being the International Fund for Agricultural Development (IFAD) which has tackled the scaling up agenda systematically and persistently.

Following are eight takeaway lessons I gleaned from my work with IFAD:

1. Look into the “black box” of institutions. It is not enough to decide that an institution should focus on and support scaling up of successful development interventions. You actually need to look at how institutions function in terms of their mission statement and corporate strategy, their policies and processes, their operational instruments, their budgets, management and staff incentives, and their monitoring and evaluation practices. Check out the Brookings working paper that summarizes the results of a scaling up review of the IFAD.
2. Scaling needs to be pursued institution-wide. Tasking one unit in an organization with innovation and scaling up, or creating special outside entities (like the Global Innovation Fund set up jointly by a number of donor agencies) is a good first step. But ultimately, a comprehensive approach must be mainstreamed so that all operational activities are geared toward scaling up.
3. Scaling up must be championed from the top. The governing boards and leadership of the institutions need to commit to scaling up and persistently stay on message, since, like any fundamental institutional change, effectively scaling up takes time, perhaps a decade or more as with IFAD.
4. The scaling up process must be grown within the institution. External analysis and advice from consultants can play an important role in institutional reviews. But for lasting institutional change, the leadership must come from within and involve broad participation from managers and staff in developing operational policies and processes that are tailored to an institution’s specific culture, tasks, and organizational structure.
5. A well-articulated operational approach for scaling up needs to be put in place. For more on this, take a look at a recent paper by Larry Cooley and I that reviews two helpful operational approaches, which are also covered in Cooley’s blog. For the education sector, the Center for Universal Education at Brookings just published its report “Millions Learning,” which provides a useful scaling up approach specifically tailored to the education sector.
6. Operational staffs need to receive practical guidance and training. It is not enough to tell staff that they have to focus on scaling up and then give them a general framework. They also need practical guidance and training, ideally tailored to the specific business lines they are engaged in. IFAD, for example, developed overall operational guidelines for scaling up, as well as guidance notes for specific area of engagement, including livestock development, agricultural value chains, land tenure security, etc.  This guidance and training ideally should also be extended to consultants working with the agency on project preparation, implementation, and evaluation, as well as to the agency’s local counterpart organizations.
7. New approaches to monitoring and evaluation (M&E) have to be crafted. Typically the M&E for development projects is backward looking and focused on accountability, narrow issues of implementation, and short-term results. Scaling up requires continuous learning, structured experimentation, and innovation based on evidence, including whether the enabling conditions for scaling up are being established. And it is important to monitor and evaluate the institutional mainstreaming process of scaling up to ensure that it is effectively pursued. I’d recommend looking at how the German Agency for International Development (GIZ) carried out a corporate-wide evaluation of its scaling up experience.
8. Scaling up helps aid organizations mobilize financial resources. Scaling up leverages limited institutional resources in two ways: First, an organization can multiply the impact of its own financial capacity by linking up with public and private agencies and building multi-stakeholder coalitions in support of scaling up. Second, when an organization demonstrates that it is pursuing not only one-off results but also scaled up impact, funders or shareholders of the organization tend to be more motivated to support the organization. This certainly was one of the drivers of IFAD’s successful financial replenishment consultation rounds over the last decade.

By adopting these lessons, development organizations can actually begin to scale up to the level necessary to bridge the missing middle. The key will be to assure that a focus on scaling up is not the exception but instead becomes ingrained in the institutional DNA. Simply put, in designing and implementing development programs and projects, the question needs to be answered, “What’s next, if this intervention works?”

In 2015 the international community agreed on a set of ambitious sustainable development goals (SDGs) for the global society, to be achieved by 2030. One of the lessons that the implementation of the Millennium Development Goals (MDG s) has highlighted is the importance of a systematic approach to identify and sequence development interventions—policies, programs, and projects—to achieve such goals at a meaningful scale. The Chinese approach to development, which consists of identifying a problem and long-term goal, testing alternative solutions, and then implementing those that are promising in a sustained manner, learning and adapting as one proceeds—Deng Xiaoping’s “crossing the river by feeling the stones”—is an approach that holds promise for successful achievement of the SDGs.

Having observed the Chinese way, then World Bank Group President James Wolfensohn in 2004, together with the Chinese government, convened a major international conference in Shanghai on scaling up successful development interventions, and in 2005 the World Bank Group (WBG ) published the results of the conference, including an assessment of the Chinese approach. (Moreno-Dodson 2005). Some ten years later, the WBG once again is addressing the question of how to support scaling up of successful development interventions, at a time when the challenge and opportunity of scaling up have become a widely recognized issue for many development institutions and experts.

Since traditional private and public service providers frequently do not reach the poorest people in developing countries, social enterprises can play an important role in providing key services to those at the “base of the pyramid.”

In parallel with the recognition that scaling up matters, the development community is now also focusing on social enterprises (SEs), a new set of actors falling between the traditionally recognized public and private sectors. We adopt here the World Bank’s definition of “social enterprises” as a social-mission-led organization that provides sustainable services to Base of the Pyramid (BoP) populations. This is broadly in line with other existing definitions for the sector and reflects the World Bank’s primary interest in social enterprises as a mechanism for supporting service delivery for the poor. Although social enterprises can adopt various organizational forms—business, nongovernmental organizations (NGOs), and community-based organizations are all forms commonly adopted by social enterprises—they differ from private providers principally by combining three features: operating with a social purpose, adhering to business principles, and aiming for financial sustainability. Since traditional private and public service providers frequently do not reach the poorest people in developing countries, social enterprises can play an important role in providing key services to those at the “base of the pyramid.” (Figure 1)

Figure 1. Role of SE sector in public service provision

Social enterprises often start at the initiative of a visionary entrepreneur who sees a significant social need, whether in education, health, sanitation, or microfinance, and who responds by developing an innovative way to address the perceived need, usually by setting up an NGO, or a for-profit enterprise. Social enterprises and their innovations generally start small. When successful, they face an important challenge: how to expand their operations and innovations to meet the social need at a larger scale. 

Development partner organizations—donors, for short—have recognized the contribution that social enterprises can make to find and implement innovative ways to meet the social service needs of people at the base of the pyramid, and they have started to explore how they can support social enterprises in responding to these needs at a meaningful scale. 

The purpose of this paper is to present a menu of approaches for addressing the challenge of scaling up social enterprise innovations, based on a review of the literature on scaling up and on social enterprises. The paper does not aim to offer specific recommendations for entrepreneurs or blueprints and guidelines for the development agencies. The range of settings, problems, and solutions is too wide to permit that. Rather, the paper provides an overview of ways to think about and approach the scaling up of social enterprise innovations. Where possible, the paper also refers to specific tools that can be helpful in implementing the proposed approaches. 

Note that we talk about scaling up social enterprise innovations, not about social enterprises. This is because it is the innovations and how they are scaled up that matter. An innovation may be scaled up by the social enterprise where it originated, by handoff to a public agency for implementation at a larger scale, or by other private enterprises, small or large. 

This paper is structured in three parts: Part I presents a general approach to scaling up development interventions. This helps establish basic definitions and concepts. Part II considers approaches for the scaling up of social enterprise innovations. Part III provides a summary of the main conclusions and lessons from experience. A postscript draws out implications for external aid donors. Examples from actual practice are used to exemplify the approaches and are summarized in Annex boxes.

Downloads

• Download the full paper (PDF)

On June 20, I moderated a conversation on the future of the Department of Veterans Affairs with Secretary Robert McDonald. When he took office almost two years ago, Secretary McDonald inherited an organization in crisis: too many veterans faced shockingly long wait-times before they received care, VA officials had allegedly falsified records, and other allegations of mismanagement abounded.

Photo: Paul Morigi

Since he was sworn into office, Secretary McDonald has led the VA through a period of ambitious reform, anchored by the MyVA program. He and his team have embraced three core strategies that are securing meaningful change. They are important insights for all government leaders, and private sector ones as well.

1. Set bold goals

Secretary McDonald’s vision is for the VA to become the number one customer-service agency in the federal government. But he and his team know that words alone won’t make this happen. They developed twelve breakthrough priorities for 2016 that will directly improve service to veterans. These actionable short-term objectives support the VA’s longer term aim to deliver an exceptional experience for our veterans. By aiming high, and also drafting a concrete roadmap, the VA has put itself on a path to success.

2. Hybridize the best of public and private sectors

To accomplish their ambitious goal, VA leadership is applying the best practices of customer-service businesses around the nation. The Secretary and his colleagues are leveraging the goodwill, resources, and expertise of both the private and public sector. To do that, the VA has brought together diverse groups of business leaders, medical professionals, government executives, and veteran advocates under their umbrella MyVA Advisory Committee. Following the examples set by private sector leaders in service provision and innovation, the VA is developing user-friendly mobile apps for veterans, modernizing its website, and seeking to make hiring practices faster, more competitive, and more efficient. And so that no good idea is left unheard, the VA has created a "shark tank” to capture and enact suggestions and recommendations for improvement from the folks who best understand daily VA operations—VA employees themselves.

3. Data, data, data

The benefits of data-driven decision making in government are well known. As led by Secretary McDonald, the VA has continued to embrace the use of data to inform its policies and improve its performance. Already a leader in the collection and publication of data, the VA has recently taken even greater strides in sharing information between its healthcare delivery agencies. In addition to collecting administrative and health-outcomes information, the VA is gathering data from veterans about what they think . Automated kiosks allow veterans to check in for appointments, and to record their level of satisfaction with the services provided.

The results that the Secretary and his team have achieved speak for themselves:

• 5 million more appointments completed last fiscal year over the previous fiscal year
• 7 million additional hours of care for veterans in the last two years (based on an increase in the clinical workload of 11 percent over the last two years)
• 97 percent of appointments completed within 30 days of the veteran’s preferred date; 86 percent within 7 days; 22 percent the same day
• Average wait times of 5 days for primary care, 6 days for specialty care, and 2 days for mental health are
• 90 percent of veterans say they are satisfied or completely satisfied with when they got their appointment (less than 3 percent said they were dissatisfied or completely dissatisfied).
• The backlog for disability claims—once over 600,000 claims that were more than 125 days old—is down almost 90 percent.

Thanks to Secretary McDonald’s continued commitment to modernization, the VA has made significant progress. Problems, of course, remain at the VA and the Secretary has more work to do to ensure America honors the debt it owes its veterans, but the past two years of reform have moved the Department in the right direction. His strategies are instructive for managers of change everywhere.

Fred Dews and Andrew Kenealy contributed to this post.