Egypt needs a new inclusive and equitable economic growth model. Unemployment has spiked since the 2011 revolution, clearing over 12 percent, a figure which is not expected to decrease for several years at least and the situation is even more dire for the country’s youth. While the likely IMF program will offer the macroeconomy a measure of relief, it cannot reverse decades of mismanagement. Egypt’s private sector may therefore not experience a recovery in the near future. The government’s situation looks similarly stressed as its gross debt is projected to rise from 73 percent of GDP in 2010 to 79 percent this year. Combined with the confusion surrounding the government’s structure and organization, it is unlikely that the public sector can fill the jobs gap or provide the needed high quality and affordable goods and services. However, the legal limbo surrounding inclusive business models (IBs) as well as intermediary support organizations (ISOs), which are supposed to provide the needed support to IBs, has unnecessarily shrunk this sector of the economy and disabled it from playing its necessary role.

In his inaugural speech, Egyptian President Mohamed Morsi portrayed himself as a president for all Egyptians, including the menial and underprivileged rickshaw drivers. The Muslim Brotherhood’s Al-Nahda Program emphasizes social justice and a consensus vision across all groups in society. The new leadership is committed to social innovation with “a national strategy to develop mechanisms to support innovation dealing with community issues.”

Although the constitution has not yet been drafted and there is currently no parliament, this moment in time contains a golden opportunity for the government of Egypt to capture the energy, civic engagement and entrepreneurial spirit in the country. Under Mubarak, Egypt’s economic growth and business policy reforms helped foster the private sector, but 85 percent of the population continued to live under $5/day and this ratio did not change during the decade of growth prior to 2008. Safeguards against abuse and incentives for inclusiveness were missing, and the economy became dominated by crony capitalism with wealth concentrated in the hands of a few. People’s perception of inequity and dissatisfaction with public services increased. The governance indicators of “Voice & Accountability” and “Control of Corruption” deteriorated from 2000 to 2010, even though there was a steady improvement in “Regulatory Quality.”

Egypt needs an enabling legal framework to promote a more equitable growth model. Such a framework should encourage forms of inclusive businesses (such as cooperatives) and ISOs that could help micro and small enterprises. These firms (with less than 50 employees) represent nearly 99 percent of all non-public sector, non-agricultural firms and provide about 80 percent of employment in Egypt. But their expansion has been restricted because of the weakness of the ecosystem of incubators, angel investor networks, microfinance institutions (MFIs) and impact investors necessary to allow young entrepreneurs to start up and grow. This policy paper argues that legal and regulatory reforms that encourage ISOs and allow new forms of inclusive business to register and operate are a necessary first step towards a new inclusive growth model.

Downloads

• Download the full paper

      

       

       

      

      

Event Information

Register for the Event

As President Barack Obama and NATO leaders join on April 3 and 4 to celebrate the Alliance’s 60th anniversary, they also must confront the daunting challenges facing NATO today. How should the Alliance proceed in Afghanistan, its largest ever military operation? How can NATO broaden its restored relationship with Russia while continuing to deepen its links with Ukraine and Georgia? As the Alliance begins to devise a new strategic concept, how should it balance its focus between preparing for expeditionary operations and meeting its collective defense obligations? How will France’s full return to NATO’s integrated military structure add to Alliance capabilities?

On March 30, the Center on the United States and Europe (CUSE) at Brookings held a public event to preview President Obama’s first NATO summit. Daniel Hamilton, professor at the Johns Hopkins Paul H. Nitze School of Advanced International Studies, and Brookings experts Steven Pifer, Jeremy Shapiro and Justin Vaisse described the challenges facing the president and NATO. Brookings Vice President and Director of Foreign Policy Carlos Pascual gave introductory remarks and moderated the discussion.

Video

• Carlos Pascual
• Steven Pifer
• Justin Vaisse
• Jeremy Shapiro
• Daniel Hamilton

Audio

• Tackling NATO's Challenges

Transcript

• Transcript (.pdf)

Event Materials

• 20090330_nato

      

Today, the Center on the United States and Europe (CUSE) at Brookings hosted Georgian Defense Minister Irakli Alasania for an address on Georgia's vision for Euro-Atlantic integration during a period of increased insecurity in the region. In his remarks, Minister Alasania shared his insights on the upcoming NATO summit and Georgia's approach to enhancing its relations with the West while attempting to normalize relations with Russia to lower tensions still simmering from the war six years ago.

Minister Alasania said that his country's "path toward NATO and European integration is unchanged" and offered next steps on "how we're going to make sure that the credibility of the west, the credibility of NATO as an organization will continue to be relevant to safeguard the values that we all cherish: freedom, democracy, and a Europe whole and free."

"We are acting like a NATO country," he said. Continuing:

We are acting like a European country, because we believe that our future is within Europe. And we regard ourselves as a future member. And this is why we are preparing ourselves institution-wise, in terms of freedom, in terms of democracy, and the military capabilities when ... the historical opportunity will open up to Georgia to join NATO and the EU.

The defense minister added that "We are looking at the future." We:

cannot be dragged back to the confrontation of the early 1990s. And we want to make sure that our policies, our economic policies, our foreign policy, [are] specifically working to make sure that the Georgian people who elected us are now moving closer and closer to the European way of living standards. And this only can be done if the efforts that Georgia is making will be validated, will be appreciated by the NATO and the European countries.

One of the things we are looking forward to is the signing of the association agreement. The next step obviously is the NATO summit. And what the NATO summit will decide is how effectively they can assure the allies, but also the partners, like Georgia.

On Russia, Minister Alasania spoke in both hopeful and realistic terms, saying that:

We are now approaching foreign policy and specifically the issue with Russia with a rather mature approach. We don't have any illusions that Russia will change its behavior or policies toward Georgia's territorial integrity or NATO aspirations. But we do hope the diffusion of tensions, the decrease of the military rhetoric between the two countries, will serve Georgia's interests best.

And it will give us more space to develop ourselves, to develop our relationship with the Abkhazia and South Ossetian areas. This is the cornerstone of our policy actually. Be uncompromising on the territorial integrity. Be uncompromising on NATO aspiration, membership in NATO and the EU. But at the same time be sure that we are not going give a pretext to anybody in the region, specifically to Russians, to attack us politically or otherwise.

---

Listen to audio of the event below or on the event's web page to get the full conversation, which was moderated by CUSE Director Fiona Hill.

Audio

• Georgia's Euro-Atlantic Aspirations and Regional Security

       

       

       

Policy Brief #175

As the nation strives to recover from the “Great Recession,” job creation remains one of the biggest challenges to renewed prosperity. Small businesses have been among the most powerful generators of new jobs historically, suggesting the value of a stronger focus on supporting small businesses—especially high-growth firms—and encouraging entrepreneurship. Choosing the right policies will require public and private decision-makers to establish clear goals, such as increasing employment, raising the overall return on investment, and generating innovations with broader benefits for society. Good mechanisms will also be needed for gauging their progress and ultimate success. This brief examines policy recommendations to strengthen the small business sector and provide a platform for effective programs. These recommendations draw heavily from ideas discussed at a conference held at the Brookings Institution with academic experts, successful private-sector entrepreneurs, and government policymakers, including leaders from the Small Business Administration. The gathering was intended to spur the development of creative solutions in the private and public sectors to foster lasting economic growth.

	RECOMMENDATIONS
	What incentives and assistance could be made available to “gazelles” and to small business more generally? What policies are likely to work most effectively? In the near term, government policies aimed at bolstering the recovery and further strengthening the financial system will help small businesses that have been hard hit by the economic downturn. Spurred by the interchange of ideas at a Brookings forum on small businesses, we have identified the following more targeted ideas for fostering the health and growth of small businesses (and, in many cases, larger businesses) over the longer run: Improve access to public and private capital. Reexamine corporate tax policy with an eye toward whether provisions of our tax code are discouraging small business development. Promote education to help businesses struggling with shortages of workers with particular skills, and promote research to spur innovation. Rethink immigration policy, as current policy may be contributing to shortages of key workers and deterring entrepreneurs who wish to start promising businesses in our country. Explore ways to foster “innovation-friendly” environments, such as regional cluster initiatives. Strengthen government counseling programs.

The term “small business” applies to many different types of firms. To begin, the small business community encompasses an enormous range of “Main Street” stores and services we use every day, such as restaurants, dry cleaners, card shops and lawn care providers. When such a business fails, it is often replaced by a similar firm. The small business community also includes somewhat bigger firms—in industries such as manufacturing, consulting, advertising and auto sales—that may have more staying power than Main Street businesses, but still tend to stay relatively small, with under 250 employees. While these two kinds of small businesses contribute relatively little to overall employment growth, they are a steady source of mainstream employment. If economic conditions do not support the formation of new businesses to replace the ones that fail, there would be a significant net destruction of jobs and harm to local communities.

Yet another type of small business has an explicit ambition for rapid growth. These high-growth companies are sometimes known as “gazelles.” According to the Small Business Administration, small businesses account for two-thirds of new jobs, and the gazelles account for much of this job creation. The most striking examples—such as Google and eBay—have tended to be in high-tech industries and were gazelles for a significant time before they graduated to be very large businesses. However, gazelles exist in all industry types and in all regions of the country, and the large majority are not grazing in the nation’s technology-dominated Silicon Valleys. According to one expert, the three largest industry categories for high-growth companies are restaurant chains, administrative services and health care companies. One non-high-tech example is Potbelly Sandwiches, a restaurant chain that began in Chicago. Another is the San Francisco-based Gymboree Corporation, a provider of child development programs and children’s clothing.

 

Fostering the Development of High-Growth Companies

High-growth small businesses represent only about 5 percent of total startups, making it important to determine how to spot and foster them. A key common characteristic is that growth is critically dependent on the entrepreneurs who start these companies; they are people on a mission, charismatic leaders who can inspire creativity and commitment from their staffs.

The age of these firms is highly correlated with when their growth is highest. Generally, the most dramatic growth occurs after at least four years of existence—and coincidentally lasts about four years—before it slows again to a more typical pace for small businesses. Of course, some firms such as Google defy this pattern and continue to experience high growth for many years.

Although dynamic small businesses can be found nearly everywhere and in many industries, some regions spawn more of them than others. These regions may have especially supportive features, such as a critical mass of potential workers with relevant skills, a social climate and network that encourage idea generation, locally available venture capital, or some combination of these factors.

Unfortunately, attempts to anticipate which companies or even industries are likely to produce gazelles are prone to error. Thus, excessive emphasis on national industrial policies that favor specific industries are likely misplaced. Without knowing how to target assistance precisely, broad strategies, such as assistance with funding, knowledge, contacts and other essential resources, may be the best approach to fostering high-growth businesses. Such support has the added value of also aiding Main Street businesses.

Many of the most promising policies focus on removing obstacles that hinder entrepreneurs with solid business plans from launching and expanding their businesses.

Funding

As a result of the burst of the dot.com bubble in early 2000 and the recent financial crisis, small businesses have found the availability of venture capital funds drastically diminished. The crisis has also made it more difficult to obtain funding from banks and other conventional means. These trends particularly affect the “missing middle” of small businesses—roughly, those with between 10 and 100 employees.

The venture capital market. Historically, venture capital has financed only a relatively small portion of small businesses, but those financed have tended to be the ones with the greatest growth potential. In recent years, firms that eventually grew to where they could issue initial public stock offerings generally relied more heavily on venture capital financing than the average small business.

The dollar value of venture capital deals funded today is only about one-fifth the size it reached at its peak. While the peak amount may have been too large, today’s value is probably too small. With their capital heavily invested in a small range of industries and locales, it seems likely that venture capital firms have missed a high proportion of potential investment opportunities. Further, “once burned, twice shy” funders have increasingly focused on larger, later-stage ventures. Consequently, mezzanine financing, which new companies need to survive and thrive in the critical early stages, is scarce.

The funding problems partly stem from venture capital firms today having less money to invest. Some investors who formerly contributed to such firms have become more risk-averse, and worse performance figures have discouraged new investors. Lack of venture capital affects some industries more than others, and even some green energy companies—viewed by some as one of the nation’s more promising industry sectors—have moved to China, where financial support is more readily available.

Bank lending. In contrast to large businesses, which can turn to capital markets for funding, many small businesses are dependent on banks for financing. Although the worst of the 2008–09 credit crunch is behind us, many small businesses still find it difficult to obtain bank loans. Community banks, a key source of small business financing, have been hard hit by losses in commercial real estate, which have limited their lending capacity. Further, many small business owners who historically would have used real estate assets as collateral for expansion loans can no longer do so because of declines in real estate prices. In addition, small businesses that have, in the past, used credit cards to purchase equipment and supplies have been hindered by reductions in credit limits.

Overall economic conditions

The high degree of uncertainty currently surrounding the economic and financing climate may have prompted many entrepreneurs and would-be entrepreneurs to hold off on growth plans. Despite their reputation as high-flying risk-takers, good entrepreneurs take only calculated risks, where the benefits outweigh the dangers. Uncertainties about the future trajectory of the economy merely increase risk without raising potential rewards.

Government policies

Government policies affect the climate for small businesses in many ways. For example, small businesses face substantial hurdles when entering the complicated world of federal grants and contracts. At the state level, severe budget shortfalls mean that even well-designed initiatives to boost small businesses may founder.

The Small Business Administration (SBA) assists the full continuum of small businesses through a variety of means. These include: an $80 billion loan guarantee portfolio; specialized counseling and training centers; specialized business development programs targeting the socially and economically disadvantaged; oversight to ensure that at least 23 percent of federal government contracts go to small businesses (with certain preferences for minority and women-owned businesses); and the Small Business Innovation Research and Small Business Investment Companies programs.

The Obama administration is attempting to broaden support for small businesses by bringing the SBA into multi-agency initiatives that tackle common problems. For example, the Departments of Energy, Commerce, Housing and Urban Development, Education, and Labor, along with the National Science Foundation and the SBA, are supporting a five-year, nearly $130 million Energy Regional Innovation Cluster.

Strength of “social capital”

Through the 1990s, the United States was a worldwide leader in fostering innovation and entrepreneurship and reaped the reward of employment growth. Current international comparisons suggest that we are now closer to tenth place among some 70 nations in our ability to support innovation. Much of what has kept our nation from remaining in the top spot appears to relate to insufficient cultural support for entrepreneurship.

Strong social networks in specific geographic regions appear to substantially bolster the growth of innovative businesses. These networks are built around entrepreneurial dealmakers who serve as the nodes of the network, forming connections among researchers, entrepreneurs and investors. Unfortunately, many regions and industries lack strong networks.

Access to decision-making information. Entrepreneurs need an array of information and advice about how to tackle the problems that arise at different stages in business development. The SBA reports that companies that have taken advantage of their long-term counseling programs, for example, have higher growth than companies that have not.

Opportunity for all. Social networks are self-selecting, and some people have to work extra hard to gain entry to a region’s network of entrepreneurs. While various organizations exist to help women and people of color access entrepreneurial skills and information, these efforts may not suffice. Under-representation of any group presumably would filter out a number of potential high-growth companies.

Workforce issues

A long-time strength of the American workforce, worker mobility has declined. This trend has been attributed in part to an aging population and in part to the current difficulty people have in selling their homes. Businesses report difficulty finding employees with the right training, especially at the technician level, where straightforward vocational training could help.

Global competition

Increasing global competition for good projects, entrepreneurs and capital is a positive trend from an international perspective, but runs counter to the national goal of promoting rapid growth in U.S. industry and employment. Today, many entrepreneurs can choose among starting a business here, in their home country, or even in a third, more hospitable nation. At the same time, current U.S. immigration policy hinders entrepreneurs from coming here to launch their companies. A recent report from The Brookings- Duke Immigration Policy Roundtable concluded that “educated workers with the knowledge and skills to innovate are critical” to the United States and recommended increasing the annual number of skilled visas.

 

Policy Goals for Small Business

Measuring Results

More work is needed to identify key policy goals and priorities related to small business success. Critically, what would constitute “improvement” in public policy regarding small business employment, and how would we measure it? Clearly, increasing the total number of jobs created each year (by both small and large businesses, net of job destruction) would be a positive outcome, all else being equal. Another potential goal would be improving the “quality” of the jobs created, as measured by average compensation or by job creation in new industries or geographic areas where unemployment is high. Creating “good jobs” that bring generous compensation would seem to be always desirable, but this outcome could conflict with other social goals, for example, if the jobs created required skills out of the reach of groups that are traditionally difficult to employ.

Slowing job destruction could be as important as increasing the creation of new jobs, but discouraging layoffs without increasing performance would do more harm than good. The trick is to raise the quality of marginal firms so that their improved performance allows them to retain employees they would otherwise have to let go.

A final key factor in setting policy goals that would support small businesses is measuring the cost to taxpayers of the initiatives that flow from the goals. This includes the subsidy cost contained in the federal budget, as well as costs and tradeoffs in society at large.

Changing Key Policies

Small businesses face both short-run and long-run challenges. With regard to the former, many small businesses have been hard hit by the recession and appear to be lagging behind larger businesses in their recovery. The cyclical struggles of this sector in part reflect the dependence of many small firms on the still-strained banking system for their financing; they also reflect the high toll that our extremely soft labor markets have taken on demand for Main Street goods and services. Thus, government policies aimed at broadly bolstering the recovery and further strengthening the financial system will yield important benefits to small businesses.

The government, in conjunction with the private sector, can also take steps that will foster an economic environment that is supportive of entrepreneurship and economic growth over the long run. Specific policy steps that might help small businesses (and, in many cases, large businesses) include:

Improve access to public and private capital. Implementing serious financial reform will reduce the likelihood that we will see a repeat of the recent credit cycle that has been so problematic for the small business sector. When credit market disruptions do occur, policymakers should be attentive to whether temporary expansions of the SBA loan guarantee program are needed to sustain lending to creditworthy borrowers. The SBA should also consider expanding the points of access to its loan programs through an expansion of its lending partners. Finally, the SBA (or a similar entity) might encourage venture capital funds to broaden their investments beyond familiar areas by systematically bringing these investors together with entrepreneurs from neglected geographic regions and business sectors.

Reexamine corporate tax policy. More thinking is needed about whether provisions in our tax code discourage small business development in a way that is harmful to the broader economy and that places the United States at a relative disadvantage internationally. For example, Congress might consider whether it would be beneficial, on net, to lower employment taxes as a way of spurring hiring at businesses with high-growth potential. In addition, some analysts believe there would be gains from increasing tax credits for research and development and further lowering taxes on capital equipment. A design priority in all cases should be simplicity, as complicated rules can limit take-up among smaller firms that do not have extensive accounting or legal expertise.

Promote education and research. Entrepreneurs report difficulty in finding workers with the skills they need for manufacturing, technology and other jobs that do not require four-year college degrees. Access to such educational opportunities, including tailored vocational training, should be affordable and ubiquitous.

At the university level, improvements are needed in the way academic research is brought to the commercial market. Continued public and private support for basic research might be wise, particularly if we are in a trough between waves of innovation, as some analysts believe. The large investments by the National Science Foundation, National Institutes of Health, Defense Advanced Research Projects Agency, and other ambitious public and private programs laid the groundwork for many of the high-growth businesses of today. It may be worth exploring whether support for research in “softer” areas than the sciences might do an equal or better job of inspiring innovations.

Rethink immigration policy. A reconsideration of limits on H1-B visas might help entrepreneurs struggling with shortages of workers with particular skills. In addition, current immigration policy discourages immigrants who want to establish entrepreneurial businesses in America. Any efforts to expand immigration are frequently perceived as “taking jobs away from Americans,” but studies have shown that new businesses create jobs for Americans.

Explore ways to foster “innovation-friendly” environments. Some regions of the United States clearly do a better job of encouraging innovation. Silicon Valley is the classic example, but there may be as many as 40 such clusters scattered around the country. While clusters often arise organically, typically near major universities, some states have made an explicit commitment to innovation and entrepreneurship. Examples include the Massachusetts Technology Collaborative and California’s Biological Technologies Initiative, involving community colleges statewide. Federal, state and local policymakers should keep a keen eye on ways of adapting best practices from these initiatives as information becomes available about which elements are most effective.

Strengthen government counseling programs. The SBA might do more to expand and tailor its already successful growth counseling programs to better meet the needs of both Main Street and potential high-growth businesses, as well as firms at different developmental stages. Any effort to expand small businesses’ opportunities for federal grants and contracts should be accompanied by significant streamlining of the application process.

Downloads

• Download Policy Brief

      

       

       

Editor's Note: In an interview with La Repubblica's Rosalba Castelletti, Jonathan Laurence discussed the significance of the revelations that the United States has continued to spy on Germany, and what they mean for the future of the transatlantic relationship.

"È un momento delicato, ma non penso che la Germania abbia interesse ad esagerare le tensioni con gli Stati Uniti". A sostenerlo è Jonathan Laurence, professore di Scienze politiche al Boston College ed esperto di Relazioni transatlantiche presso il think tank Brookings Institution di Washington.

Professor Laurence, quest'episodio come inciderà sulle relazioni tra i due Paesi?

"La situazione è tesa. Berlino stavolta non ha espresso solo la consueta indignazione, ma ha compiuto un atto formale con l'espulsione del capo dei servizi segreti, perché è la terza volta che il popolo tedesco apprende di essere spiato dagli americani. La prima volta è successo con il Datagate, la seconda con l'intercettazione del cellulare della cancelliera e ora con due spie tedesche al soldo degli americani".

In cosa differisce quest'ultimo caso dai precedenti?

"Non si tratta di programmi d'alta tecnologia, ma di spionaggio più "vecchia maniera": documenti in cambio di soldi. Stavolta poi non c'è in ballo un problema di sicurezza internazionale. È un nuovo colpo per la reputazione Usa perché ancora una volta si dimostra indifferente alla sensibilità europea riguardo alla raccolta di dati".

E i tedeschi sono forse i più sensibili, visto che hanno sperimentato lo spionaggio della Gestapo e della Stasi...

"Di fatti. L'attuale cancelliera ha fatto il suo debutto in politica proprio dopo il crollo della Stasi. Ecco perché dobbiamo aspettarci che la Germania dichiari a gran voce la sua collera".

Cosa può fare l'amministrazione Usa per riparare?

"Qualcosa di più che cercare infruttuosi colloqui bilaterali o accordi di non spionaggio reciproco. La Germania non è ingenua, sa che i servizi americani hanno bisogno di operare soprattutto dopo il 2001, ma vuole che si lavori insieme. Non credo però che cerchi il conflitto. Berlino e Washington hanno bisogno l'una dell'altra sia sulle sanzioni contro la Russia in merito alla crisi Ucraina sia sull'accordo di libero scambio".

      

Editor’s Note: This post is part of a series on the 2015 Brookings Financial and Digital Inclusion Project (FDIP) Report and Scorecard, which were launched at a Brookings public event in August. Previous posts have highlighted regional findings from Southeast and Central Asia, the Middle East, and Africa, as well as selected financial inclusion milestones from FDIP countries. This post focuses on key financial inclusion achievements and challenges regarding the five Latin American FDIP countries: Brazil, Chile, Colombia, Mexico, and Peru.

Financial inclusion growth and opportunities in Latin America

With its well-developed banking infrastructure and growing mobile ecosystem, Latin America presents a unique set of opportunities and obstacles with respect to promoting greater financial inclusion. From 2011 to 2014, there was a 12 percentage point increase in the number of adults in Latin America and the Caribbean with formal financial accounts, according to the World Bank’s Global Financial Inclusion (Global Findex) database. As noted in the 2015 GSMA report “Mobile financial services in Latin America & the Caribbean,” in 2014 Latin America and the Caribbean saw the fastest growth of any region in terms of new registered mobile money accounts.

Moreover, these accounts are often used for more advanced transactions that go beyond simple transfers: As stated in a 2015 post published by the GSMA, “ecosystem transactions (transactions that involve third parties, e.g. bill payment, merchant payment or bulk payment) already make up 27% of transaction volumes in Latin America & the Caribbean.” In contrast, only 6 percent of transaction volumes over the same period were considered ecosystem transactions in East Africa, where mobile money has been most widely adopted and used.

Moving forward, facilitating greater adoption of a suite of digital financial services (e.g., savings) will be a vital component of promoting sustainable financial inclusion in the region. Recent regulatory changes in several Latin American countries designed to promote a greater diversity of service providers should propel financial inclusion growth, although a need for regulatory clarity persists in some places. Financial inclusion strengths and challenges germane to our five Latin American FDIP countries are explored below.

Brazil: Branchless banking leadership combined with dynamic mobile market

Brazil achieved the highest ranking of any Latin American country on the Brookings 2015 FDIP Scorecard, ranking 3^rd overall with a score of 78 percent. Brazil’s economy is the largest in Latin America, with a GDP (in current US dollars) of about $2.3 trillion as of 2014; for comparison, Mexico, the Latin American country with the second largest economy, had a GDP of about $1.3 trillion within that same period.

Brazil received strong country commitment and mobile capacity scores (89 and 83 percent, respectively) in the 2015 FDIP Scorecard and earned the highest regulatory environment score among the Latin American FDIP countries, which also included Chile, Colombia, Mexico, and Peru. As noted in the 2015 FDIP Report, Brazil launched a National Partnership for Financial Inclusion in November 2011, which has supported the development of a number of enabling financial inclusion initiatives. In 2013, Law 12865 and associated regulations permitted non-banks to issue e-money as payments institutions. Brazil boasted the largest mobile market in Latin America as of 2014, with a unique subscribership rate of about 57 percent in 2015 (a lower unique subscribership rate than Chile’s by about 7 percentage points, but otherwise higher than that of any of the other Latin American FDIP countries).

Brazil received 4^th place on the 2015 FDIP Scorecard for adoption of selected traditional and digital financial services. As with many other countries in Latin America, branchless banking (i.e., access to formal financial services beyond a traditional brick-and-mortar bank) through “agents” is popular in Brazil — as of 2014, Brazilian banks’ agent networks had a presence in all of the country’s approximately 6,000 municipalities, contributing to formal account growth. Chile was the only Latin American country that received a higher ranking for the adoption dimension, placing 2^nd. In terms of account usage, government-to-person payments comprise a significant source of activity for formal accounts: The 2014 Global Findex report noted that among recipients of government payments in Brazil, 88 percent received their transfers directly into an account.

Yet according to the Global Findex, about 32 percent of Brazilian adults age 15 and older still do not have accounts with a formal financial institution or mobile money provider. As with the other Latin American countries in the FDIP sample, mobile money adoption in Brazil has remained low: Brazil received the lowest score (one out of three possible points) for all six mobile money indicators included in the 2015 FDIP Scorecard. However, given that as of 2014 Brazil had the fifth-largest global smartphone market in the world in terms of subscribers, a combination of growing smartphone penetration and an increasingly enabling regulatory environment should drive greater adoption of digital financial services in the future.

Chile: Opportunities for enhanced e-money regulatory clarity

Chile tied with Colombia and Turkey for 6^th place on the overall 2015 FDIP Scorecard. Chile’s financial inclusion environment is characterized by a firm national commitment to financial inclusion (earning a country commitment score of 89 percent) but a less developed mobile money environment than the other Latin American FDIP countries. While Chile’s unique mobile subscribership rate and 3G network coverage rate by population are higher than and on par with other countries in the region, respectively, Chile’s mobile money offerings are limited. The lack of a robust mobile money market contributed to Chile’s mobile capacity score of 72 percent, the lowest score among the FDIP Latin American countries.

Chile’s regulatory environment score (67 percent) was also the lowest of the Latin American FDIP countries, primarily due to a lack of regulatory clarity surrounding digital financial services. Developing or clarifying regulations pertaining to electronic money in particular could potentially drive more engagement with the sector and advance the diversity of mobile money providers and offerings. Further, supporting the interoperability of digital and traditional financial services could enhance the utility of these products for customers.

Given that 37 percent of adults in Chile did not have an account with a formal financial provider as of 2014, there is also room for growth in terms of expanding financial inclusion. However, it should be noted that Chile earned the highest adoption ranking of any Latin American country featured in the 2015 FDIP Scorecard. While Chile’s adoption levels with respect to mobile money services were limited, adoption rates of other formal financial services were among the highest of the FDIP countries. Chile received three out of three possible points for all but one indicator (savings at a formal financial institution) related to traditional financial services. Chile’s performance on the adoption dimension of the scorecard contributed to its 6^th place ranking overall.

While Chile’s mobile money adoption rates are low, use of other digital financial services is increasingly popular. For example, as noted in the “2015 Maya Declaration Progress Report,” since 2012 the number of CuentaRUT accounts (accounts that feature debit cards associated with a savings account provided by Chile’s BancoEstado) has increased by about 47 percent. As of 2014, there were over 7 million active CuentaRUT cards in Chile.

Colombia: Regulatory advancements coupled with sustained country commitment

As noted above, Colombia tied with Chile for 6^th place on the overall 2015 FDIP Scorecard. Colombia has demonstrated strong commitment to financial inclusion, including through involvement in multinational organizations such as the Alliance for Financial Inclusion (AFI). An example of Colombia’s national-level financial inclusion commitment is the 2006 establishment of Banca de las Oportunidades, an entity charged with fostering regulatory reforms conducive to financial inclusion. Another key player in the financial inclusion space is the Intersectoral Economic and Financial Education Committee, created in February 2014 under Decree 457.

In terms of the country’s regulatory environment, Law 1735 of 2014 permitted new institutions, called Sociedades Especializadas en Depósitos y Pagos Electrónicos, to offer mobile financial services. As part of the law, proportionate “know-your-customer” (KYC) requirements were also instituted for under-resourced customers in order to facilitate greater access to financial services among low-risk populations. In July 2015, Decree 1491 implemented Colombia’s financial inclusion law and highlighted the regulatory regime for the mobile money market. Colombia’s regulatory environment earned a score of 89 percent, ranking it 2^nd among the Latin American FDIP countries in this dimension.

On the supply side, banking correspondents (also known as agents) have been utilized to extend financial access to underserved populations.  As of 2015, all of Colombia’s 1,102 municipalities had at least one financial access point, defined as bank branches, banking correspondents, and ATMs. Another innovative approach to branchless banking in Colombia is bank Davivienda’s initiative to use DaviPlata mobile wallet accounts to distribute government transfers to more than 900,000 recipients of welfare program “Familias en Accion.”

With respect to demand side figures, Colombia tied with Mexico for 7^th place on the adoption dimension. As of 2014, about 38 percent of adults in Colombia had an account with a formal financial institution, and about 2 percent of adults were mobile money account holders. In terms of advancing future mobile money use, Colombia received the highest score of the Latin American countries on the mobile capacity dimension; thus, Colombia is well-positioned to advance access to and use of mobile money services in the future. Promoting usage of appropriate, quality financial services is critical, as dormancy rates have been identified as an obstacle to financial inclusion; about half of accounts in Colombia (including savings accounts, simplified accounts, and electronic deposits) were identified as dormant in 2014.

Mexico: Recent reforms may enhance competition and drive digital takeup

Mexico ranked 9^th on the overall 2015 FDIP Scorecard, with adoption of traditional and digital financial services as its highest-ranked dimension. Among the Latin American FDIP countries, Mexico features the greatest parity in terms of formal financial account ownership rates among men and women, at about 39 percent each.  In terms of national-level commitment to financial inclusion, Mexico tied with Peru for the highest ranking among the Latin American countries. AFI’s Maya Declaration was signed at the 2011 Global Policy Forum held in Riviera Maya, Mexico, signaling Mexico’s public commitment to financial inclusion.

With respect to mobile capacity, as of the first quarter of 2015 Mexico’s unique subscribership rates were the lowest of the Latin American countries. Mexico tied with Chile and Brazil for 3G network coverage by population. In terms of mobile money, Mexico’s market is still developing; several providers were available as of May 2015, but the extent of offerings was somewhat limited. As noted in the GSMA’s “Mobile Economy: Latin America 2014” report, new telecommunications reforms recently passed in Mexico are expected to affect the mobile market and potentially increase competition among the telecommunications sector. This increased competition could in turn drive the development of a greater array of innovative, affordable mobile money products.

Regarding Mexico’s regulatory environment, the country has been lauded for its risk-based KYC requirements that enable underserved individuals to access low-value accounts without fulfilling the full array of traditional identification processes, which can sometimes be burdensome for under-resourced groups. Under Mexico’s four-tiered KYC system (introduced in 2011), “level one” (very low-risk) accounts feature monthly deposit limits and a maximum balance limit of about 400 dollars; accounts can be opened at a bank branch, banking agent, over the internet, or by telephone. Higher-tier accounts have more stringent KYC requirements. A 2015 AFI article noted that Mexico's banking and securities regulator, the Comisión Nacional Bancaria y de Valores, indicated about 7.5 million new accounts were opened between August 2011 and September 2012, including over 4 million “level one” accounts.

Mexico tied with Colombia for 7^th place on the adoption dimension of the 2015 FDIP Scorecard. About 39 percent of adults in Mexico held accounts with a formal financial institution as of 2014, while about 3 percent of adults held mobile money accounts. As with other countries in Latin America, debit card and credit card use were much higher than mobile money use as of 2014, although usage of both kinds of cards was lower in Mexico than in several other Latin American FDIP countries such as Brazil and Chile. Initiatives such as the Saldazo debit card, which enables customers to use a debit card associated with a savings account and does not require a minimum balance, have helped drive adoption of digital financial services in Mexico.

Peru: Enabling regulatory environment, but constrained adoption of financial services

Peru presents perhaps one of the most interesting paradoxes among the FDIP countries. While Peru’s regulatory environment has been consistently recognized as among the best in the world for enabling financial inclusion, adoption of formal financial services remains quite low. Peru received 17^th place overall on the 2015 FDIP Scorecard, which can primarily be attributed to its low adoption score: Peru received a 15^th place ranking on the adoption dimension, the lowest score among the Latin American FDIP countries. However, we anticipate that recent regulatory changes in Peru, coupled with increasing smartphone penetration rates (Peru’s 2014 adoption rates were about 12 percentage points below the Latin American average), will facilitate adoption of digital financial services and drive greater financial inclusion in the future.

With respect to the supply side aspect of financial inclusion, as of 2014 about 92 percent of Peru’s population lived in a district with access to financial services, according to the Superintendencia de Banca, Seguros y AFP (SBS) del Peru. Nonetheless, demand side figures lag behind: The Global Findex found that only about 29 percent of adults had an account with a formal financial provider as of 2014. Peru received a “1” for two-thirds of the non-mobile money indicators on the adoption dimension of the 2015 FDIP Scorecard, and mobile money adoption was negligible. Moreover, as of 2014 there was a 14 percentage point disparity in financial account ownership between men and women, the highest financial inclusion “gender gap” among the Latin American FDIP countries.

However, given Peru’s strong national commitment to financial inclusion (reflected in Peru’s country commitment score of 94 percent) and legislative initiatives designed to promote an enabling regulatory environment, we fully anticipate that financial inclusion growth will accelerate in the future. For example, Peru recently finalized its national financial inclusion strategy, as discussed in our earlier post. Moreover, Peru has adopted laws and regulations that permit a greater diversity of players to enter the financial services market. Law 2998 of January 2013 allowed both banks and non-banks to issue e-money, and October 2013 regulations issued by the SBS enabled e-money issuers to follow a simplified account opening process. These initiatives should facilitate greater access to and usage of formal financial accounts in the future.

In terms of electronic payments specifically, diversifying the mobile money market and increasing unique subscribership could help facilitate greater adoption of mobile money services. Demand side factors, such as ensuring that services are a good fit for customers, are also critical — as evidenced by the fact that Mexico, which had comparable smartphone adoption rates to Peru and lower unique subscribership rates as of 2014, features significantly higher rates of mobile money adoption across all demographics than Peru. Peru is making a concerted effort to develop innovative electronic platforms — for example, the Peruvian Association of Banks (ASBANC) is working on the creation of an electronic money platform accessible by both financial institutions and telecommunications companies. Implementation of this interoperable platform is expected to promote further adoption of digital financial services.

       

      

As noted in our recent Brookings Institution report, unmanned aerial vehicles (UAVs), commonly referred to as drones, are an emerging technology that requires the attention of local governments. Unfortunately, regulations governing their usage are significantly lagging the pace of innovation. Individual citizens who do not want these devices flying over (or even near) their property due to privacy or safety concerns have limited options. You can stay in your home and turn the music up until it goes away. Or you can go about your business and ignore the possibility that the drone has a camera to see inside your home. Others might prefer a more active response. In fact, there have been several recent instances where residents have taken it upon themselves to remove these drones from the skies…by force.

Misuses of drones

The usage of UAVs and the lack of a functional regulatory environment have not been without incident. Fire personnel in southern San Bernardino County were fighting the first major fire of the season and had to abort their tanker flights due to someone flying a drone at approximately 12,000 feet and interfering with the safety of the pilots. Just two weeks later, firefighters in Southern California were using several manned aircraft to help put out 20 car fires on an interstate highway that were caused when a wildfire jumped the highway unexpectedly.  Pilots had to ground the planes when it was reported that five drones were flying around the area to get a good look at the fires (two of which were witnessed actually chasing the tanker planes!).

In addition to the general lack of common sense by a few users interfering with life-saving aircraft around the U.S., Britain, Poland, and elsewhere, there have been an increasing number of incidents involving drones accused of serving as remote “peeping toms.” UAVs have also crashed into cars and homes; they have even been used to smuggle drugs across the U.S.-Mexico border in addition to smuggling marijuana into prisons in South Carolina and in Ohio.

Uneven regulations

When it comes to regulations around drones, we are living in the proverbial wild-west. A few states, like Nevada and Wisconsin, have passed legislation to prevent the weaponization of drones. But in July, a YouTube video went viral of a teenager in Connecticut who modified his drone to fire a semi-automatic handgun successfully. When confronted by law enforcement officials, they determined that no laws had actually been broken. Virginia was the first state legislature to put in place a two-year moratorium on drone usage by state or law enforcement agencies. That moratorium expired July 1^st. By the end of 2014, 36 states had introduced legislation aimed at protecting individual privacy in some manner. Only four of those passed last year. Currently, there are 17 states with some form of drone regulation on their books, and several other states still have legislation pending. Most of the laws that have passed, such as those in Idaho and Florida, focus on limiting police usage of drones by requiring probable cause warrants.

Nevada has been one of the more active states in the drone legislation arena. In addition to their legislation prohibiting the weaponization of civilian drones, the state also has passed legislation to provide homeowners rights to sue drone owners who fly their drones over personal property in certain circumstances. Furthermore, Nevada now requires law enforcement agencies to get warrants when using drones near any home “where there is an expectation of privacy.”

Potential benefits and rulemaking challenges

We do acknowledge and are excited about the positive benefits that drone technology is poised to provide. Amazon has been testing their commercial “Prime Air” package delivery system under an experimental testing agreement with the FAA since early 2015, which will likely impact the nature of their almost two year old partnership with the U.S. Postal Service. Drone startup company Flirtey successfully demonstrated their ability to deliver medicine to a rural medical facility in Virginia as part of their proof of concept efforts this July. Drones may even represent the future of pizza delivery.

The challenge this rapidly developing technology is creating is well ahead of local government efforts to rein in excessive activities. State and local governments need to engage on this policy issue more proactively. To do so, however, requires a delicate balancing act of the multiple competing interests of legitimate commercial uses, policing, public safety, privacy, and private property concerns. And this balancing has to take place in an environment where federal law remains unsettled too.

One thing we would definitely caution against is ‘regulation by default.’ To date, the efforts to regulate drone policy has focused on the drones themselves. As is commonly the case with new technology, governments typically engaged with a heavy hand that sometimes misses the opportunities afforded by the new technologies to improve city services and quality of life. Examples of this possible overreaction is Iowa City, Iowa and Charlottesville, Virginia, both of which were early adopters of complete bans on all surveillance drones within city limits back in 2013.

Local governments need to accept that drone technology is here for the near future. They must recognize that technology is not the problem, but how it is used can be a potential problem. Given the potential drawbacks and benefits, there is justification for reasoned regulation of drone technology.

       

Editor's Note: The World Bank’s Independent Evaluation Group (IEG) recently published a self-evaluation of its activities. Besides representing current thinking among evaluation experts at the World Bank, it also more broadly reflects some of the strengths and gaps in the approaches that evaluators use to assess and learn from the performance of the international institutions with which they work. The old question “Quis custodet ipsos custodes?” – loosely translated as “Who evaluates the evaluators?” – remains as relevant as ever. Johannes Linn served as an external peer reviewer of the self-evaluation and provides a bird’s-eye view on the lessons learned.

An Overview of the World Bank’s IEG Self-Evaluation Report

In 2011 the World Bank’s Independent Evaluation Group (IEG) carried out and published a self-evaluation of its activities. The self-evaluation team was led by an internal manager, but involved a respected external evaluation expert as the principal author and also an external peer reviewer.

The IEG self-evaluation follows best professional practices as codified by the Evaluation Cooperation Group (ECG). This group brings together the evaluation offices of seven major multilateral financial institutions in joint efforts designed to enhance evaluation performance and cooperation among their evaluators. One can therefore infer that the approach and focus of the IEG self-evaluation is representative of a broader set of practices that are currently used by the evaluation community of international financial organizations.

At the outset the IEG report states that “IEG is the largest evaluation department among Evaluation Capacity Group (ECG) members and is held in high regard by the international evaluation community. Independent assessments of IEG’s role as an independent evaluation function for the Bank and IFC rated it above the evaluation functions in most other ECG members, international nongovernmental organizations, and transnational corporations and found that IEG follows good practice evaluation principles.”

The self-evaluation report generally confirms this positive assessment. For four out of six areas of its mandate IEG gives itself the second highest rating (“good”) out of six possible rating categories. This includes (a) the professional quality of its evaluations, (b) its reports on how the World Bank’s management follows up on IEG recommendations, (c) cooperation with other evaluation offices, and (d) assistance to borrowing countries in improving their own evaluation capacity. In the area of appraising the World Bank’s self-evaluation and risk management practices, the report offers the third highest rating (“satisfactory”), while it gives the third lowest rating (“modest”) for IEG’s impact on the Bank’s policies, strategies and operations. In addition the self-evaluation concludes that overall the performance of IEG has been “good” and that it operates independently, effectively and efficiently.

The report makes a number of recommendations for improvement, which are likely to be helpful, but have limited impact on its activities. They cover measures to further enhance the independence of IEG and the consistency of evaluation practices as applied across the World Bank Group’s branches – the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) –; to improve the design of evaluations and the engagement with Bank management upstream for greater impact; and monitoring the impact of recent organizational changes in IEG in terms of results achieved. The report also recommends that more be done to evaluate the Bank’s analytical work and that evaluations draw on comparative evidence.

Assessment

In terms of the parameters of self-evaluation set by the prevailing practice among the evaluators on international financial agencies, the IEG self-evaluation is accurate and helpful. From my own experience as an operational manager in the Bank whose activities were evaluated by IEG in years past, and as a user of IEG evaluations (and of evaluations of other international aid organizations) for my research on aid effectiveness, I concur that IEG is independent and effective in meeting its mandate as defined. Moreover, the self-evaluation produces useful quantitative evidence (including survey results, budget analysis, etc.) to corroborate qualitative judgments.

However, the self-evaluation suffers from a number of limitations in approach and gaps in focus, which are broadly representative of the practices prevalent among many of the evaluation offices of international aid agencies.

Approach of the IEG self-evaluation

The core of the self-evaluation report is about the evaluation process followed by IEG, with very little said about the substance of IEG’s evaluations. The following questions could have usefully been raised, but were not: do evaluations cover the right issues with the right intensity, such as growth and poverty; environmental, governance, and gender impacts; regional dimensions versus exclusive country or project focus; effectiveness in addressing the problems of fragile and conflict states; effectiveness in dealing with global public goods; sustainability and scaling up; etc. Therefore the report does not deal with the question of whether IEG effectively responds in its evaluations to the many important strategic debates and issues with which the development community is grappling.

Related to this limitation is the fact that the report assessed the quality of IEG’s mostly in terms of (a) whether its approach and processes meet certain standards established by the Evaluation Cooperation Group; and (b) how it is judged by stakeholders in response to a survey commissioned for this evaluation. Both these approaches are useful, but they do not have any basis in professional assessments of the quality of individual products. This is equivalent to IEG evaluating the World Bank’s projects on the quality of its processes (e.g., appraisal and supervision processes) and on the basis of stakeholder surveys, without evaluating individual products and their impacts.

Gaps in the Self-Evaluation and in Evaluation Practice

Careful reading of the report reveals six important gaps in the IEG self-evaluation, in the prevailing evaluation practice in the World Bank, and more generally in the way international financial organizations evaluate their own performance. The first three gaps relate to aspects of the evaluation approach used and the second three gaps relate to lack of focus in the self-evaluation on key internal organizational issues:

1. Impact Evaluations: The report notes that IEG carries out two to three impact evaluations per year, but it sidesteps the debate in the current evaluation literature and practice as to what extent the “gold standard” of randomized impact evaluation should occupy a much more central role. Given the importance of this debate and divergence of views, it would have been appropriate for the self-evaluation to assess IEG’s current practice of very limited use of randomized evaluations.

2. Evaluation of Scaling Up: The report does not address the question of to what extent current IEG practice not only assesses the performance of individual projects in terms of their outcomes and sustainability, but also in terms of whether the Bank has systematically built on its experience in specific projects to help scale up their impact through support for expansion or replication in follow-up operations or through effective hand-off to the government or other partners. In fact, currently IEG does not explicitly and systematically consider scaling up in its project and program evaluations. For example, in a recent IEG evaluation of World Bank funded municipal development projects (MDPs) , IEG found that the Bank has supported multiple MDPs in many countries over the years, but the evaluation did not address the obvious question whether the Bank systematically planned for the project sequence or built on its experience from prior projects in subsequent operations. While most other evaluation offices like IEG do not consider scaling up, some (in particular those of the International Fund for Agricultural Development and the United Nations Development Program) have started doing so in recent years.

3. Drawing on the Experience of and Benchmarking Against Other Institutions: The self-evaluation report does a good job in benchmarking IEG performance in a number of respects against that of other multilateral institutions. In the main text of the report it states that “IEG plans to develop guidelines for approach papers to ensure greater quality, in particular in drawing on comparative information from other sources and benchmarking against other institutions.” This is a welcome intention, but it is inadequately motivated in the rest of the report and not reflected in the Executive Summary. The reality is that IEG, like most multilateral evaluation offices, so far has not systematically drawn on the evaluations and relevant experience of other aid agencies in its evaluations of World Bank performance. This has severely limited the learning impact of the evaluations.

4. Bank Internal Policies, Management Processes and Incentives: IEG evaluations traditionally do not focus on how the Bank’s internal policies, management and incentives affect the quality of Bank engagement in countries. Therefore evaluations cannot offer any insights into whether and how Bank-internal operating modalities contribute to results. Two recent exceptions are notable exceptions. First, the IEG evaluation of the Bank’s approach to harmonization with other donors and alignment with country priorities assesses the incentives for staff to support harmonization and alignment. The evaluation concludes that there are insufficient incentives, a finding disputed by management. Second, is the evaluation of the Bank’s internal matrix management arrangements, which is currently under way. The self-evaluation notes that Bank management tried to quash the matrix evaluation on the grounds that it did not fall under the mandate of IEG. This is an unfortunate argument, since an assessment of the institutional reasons for the Bank’s performance is an essential component of any meaningful evaluation of Bank-supported programs. While making a good case for the specific instance of the matrix evaluation, the self-evaluation report shies away from a more general statement in support of engaging IEG on issues of Bank-internal policies, management processes and incentives. It is notable that IFAD’s Independent Office of Evaluation appears to be more aggressive in this regard: It currently is carrying out a full evaluation of IFAD’s internal efficiency and previous evaluations (e.g., an evaluation of innovation and scaling up) did not shy away from assessing internal institutional dimensions.

5. World Bank Governance: The IEG self-evaluation is even more restrictive in how it interprets its mandate regarding the evaluation of the World Bank’s governance structures and processes (including its approach to members’ voice and vote, the functioning of its board of directors, the selection of its senior management, etc.). It considers these topics beyond IEG’s mandate. This is unfortunate, since the way the Bank’s governance evolves will substantially affect its long-term legitimacy, effectiveness and viability as an international financial institution. Since IEG reports to the Bank’s board of directors, and many of the governance issues involve questions of the board’s composition, role and functioning, there is a valid question of how effectively IEG could carry out such an evaluation. However, it is notable that the IMF’s Independent Evaluation Office, which similarly reports to the IMF board of directors, published a full evaluation of the IMF’s governance in 2008, which effectively addressed many of the right questions.

6. Synergies between World Bank, IFC and MIGA: The self-evaluation report points out that the recent internal reorganization of IEG aimed to assure more effective and consistent evaluations across the three member branches of the World Bank Group. This is welcome, but the report does not assess how past evaluations addressed the question of whether the World Bank, IFC and MIGA effectively capitalized on the potential synergies among the three organizations. The recent evaluation of the World Bank Group’s response to the global economic crisis of 2008/9 provided parallel assessments of each agency’s performance, but did not address whether they work together effectively in maximizing their synergies. The reality is that the three organizations have deeply engrained institutional cultures and generally go their own ways rather than closely coordinating their activities on the ground. Future evaluations should explicitly consider whether the three effectively cooperate or not. While the World Bank is unique in the way it has organizationally separated its private sector and guarantee operations, other aid organizations also have problems of a lack of cooperation, coordination and synergy among different units within the agency. Therefore, the same comment also applies to their evaluation approaches.

Conclusions

Self-evaluations are valuable tools for performance assessment and IEG is to be congratulated for carrying out and publishing such an evaluation of its own activities. As for all self-evaluations, it should be seen as an input to an independent external evaluation, a decision that, for now, has apparently been postponed by the Bank’s board of directors.

IEG’s self-evaluation has many strengths and provides an overall positive assessment of IEG’s work. However, it does reflect some important limitations of analysis and of certain gaps in approach and coverage, which an independent external review should consider explicitly, and which IEG’s management should address. Since many of these issues also likely apply to most of the other evaluation approaches by other evaluation offices, the lessons have relevance beyond IEG and the World Bank.

Key lessons include:

• An evaluation of evaluations should focus not only on process, but also on the substantive issues that the institution is grappling with.
• An evaluation of the effectiveness of evaluations should include a professional assessment of the quality of evaluation products.
• An evaluation of evaluations should assess:
  o How effectively impact evaluations are used;
  o How scaling up of successful interventions is treated;
  o How the experience of other comparable institutions is utilized;
  o Whether and how the internal policies, management practices and incentives of the institution are effectively assessed;
  o Whether and how the governance of the institution is evaluated; and
  o Whether and how internal coordination, cooperation and synergy among units within the organizations are assessed.

Evaluations play an essential role in the accountability and learning of international aid organizations. Hence it is critical that evaluations address the right issues and use appropriate techniques. If the lessons above were reflected in the evaluation practices of the aid institutions, this would represent a significant step forward in the quality, relevance and likely impact of evaluations.

       

       

Please note that this Engelberg Center for Health Care Reform Health Policy Issue Brief first appeared in the Health Affairs Blog on July 31, 2014. Click here for the Health Affairs Blog version.

Abstract

Individual patient expanded access is a process by which patients can obtain investigational drugs that have not been approved by the Food and Drug Administration (FDA) outside of a clinical trial setting from biopharmaceutical companies when no other alternative therapy is available. Currently, no industry-wide structural principles exist to help companies navigate this process while balancing the needs of getting a drug to the market as quickly as possible with providing potentially life-saving treatment to individual patients. The Engelberg Center convened a stakeholder group to identify common themes and identify common principles related to expanded access, as none currently exist. The result was 4 A’s - Anticipation, Accessibility, Accountability, and Analysis – to help assist patients, providers, and companies with expanded access. Process and capacity building recommendations for the FDA also were proposed to assist companies with sustaining expanded access programs.

Call to Action: The Importance of Expanded Access Programs

Individual patient expanded access, sometimes termed “compassionate use,” refers to situations where access to a drug still in the development process is granted to patients on a case-by-case basis outside of a clinical trial, prior to completion of mandated clinical trials and approval by the Food and Drug Administration (FDA). This typically involves filing a single patient or emergency investigational new drug (IND) request with the Food and Drug Administration and voluntary release of the drug by the manufacturer. Generally, the following criteria must be met: there is reasonable expectation of meaningful benefit despite the absence of definitive clinical trial data, the patient has a serious or life-threatening condition, there are no comparable or satisfactory treatment alternatives, and there are no suitable clinical trials for the drug available to the patient. This form of expanded access, which is the focus of this paper, is different from the situation in which a drug is discharged to a large group of needy patients in the interval between successful phase 3 trials and presumed FDA approval, a strategy often termed a “treatment” IND or protocol, which was initially used in the 1980s for releasing zidovudine to patients with acquired immune deficiency syndrome.

The Engelberg Center for Health Care Reform at the Brookings Institution recently invited senior leaders from several pharmaceutical companies, two bioethicists, a senior FDA representative, and a patient advocate to share experiences and discuss organizational strategies related to expanded access (see acknowledgements). A driving factor for this meeting was a recent flurry of highly public cases of desperate patients seeking access to experimental drugs, which lead to social media campaigns and media coverage. Such cases included 7-year-old Josh Hardy (brincidofovir from Chimerix for disseminated adenovirus infection), 45-year-old Andrea Sloan (BMN673 from BioMarin for ovarian cancer), 41-year-old Nick Auden (pembrolizumab from Merck for melanoma), and 6-year-old Jack Fowler (intrathecal idursulfase from Shire for Hunter Syndrome). Expanded access requests to the FDA for new patients are increasing, from 1,000 patients nationwide in 2010 to more than 1,200 in 2012.[i] (This is likely an underestimate, since it does not include appeals made directly to companies.)

In the wake of these events, it became clear that many biopharmaceutical companies had varying experiences and policies related to such access. From the domestic regulatory standpoint, the FDA revised its expanded access regulations in 2009, which define criteria that must be met to authorize expanded access, list requirements for expanded access submissions, describe safeguards that will protect patients, and preserve the ability to develop meaningful data about the use of the drug. Biopharmaceutical companies typically face a complex global environment in which legal and regulatory frameworks can differ substantially. At the meeting, a senior FDA representative indicated the agency has approved over 99 percent of expanded access requests submitted via single patient or emergency INDs since 2009, suggesting the regulatory agency is not a major barrier to expanded access. As such, provided the access request is reasonably related to the potential benefits of the drug, the biopharmaceutical company is almost solely responsible for the decision and liability regarding whether to grant expanded access to an individual. Still, the public belief persists that the FDA is the main bottleneck that restricts access. In April 2014, Representative Morgan Griffith (R-VA) proposed H.R. 4475, The Compassionate Freedom of Choice Act of 2014, designed to restrict the FDA’s ability to prevent the use of investigational drugs in terminally ill patients. Similarly, some states have passed “Right to Try” legislation to reduce FDA oversight, but contains no requirement that companies must make drugs available.[ii]

The goal of our meeting was to identify common themes and possibly broad outlines to suggest industry-wide policies related to expanded access, as none currently exist. The group first discussed background issues related to expanded access and agreed on definitions. The meeting then focused on three topics. First, the group participants who play key roles in evaluating expanded access requests were invited to share narrative experiences in specific clinical cases, in an effort to lay the groundwork for trust and open discussion. Second, the group was asked to identify internal industry-specific structural barriers, such as the existence of clear procedures or tracking mechanisms within companies to handle requests. Finally, the participants reflected on situations in which expanded access may not be appropriate, or where regulatory barriers or liability concerns may hinder expanded access. This paper reflects the authors’ observations and assessment of the internal and external landscape, based upon information provided by the meeting participants.

Laying the Groundwork with Shared Experiences

The FDA allows companies to provide drugs and charge individual patients that do not meet the enrollment criteria for clinical trials geared towards regulatory approval through expanded access programs.[iii] These programs are meant to provide the drug directly to treat the patient’s condition, rather than having the primary goal of collecting efficacy or detailed safety data in support of approval. Before 1987, the FDA lacked formal recognition of expanded access, although investigational drugs were provided informally.[iv] Since then, the FDA has instituted novel classes of individual INDs so that a company sponsor or licensed physician can legally obtain treatment access from the FDA to provide a drug while it is still in the approval process.[v] Essentially, this provides companies a legal exception from the law to ship unapproved drugs across state lines, and if they desire, to charge for them. These INDs are designed solely for the potential benefit of desperate patients and not intended to formally collect safety or efficacy data that could potentially inform a regulatory decision, but can have regulatory impact, nonetheless.

At the outset, several participants objected to the term “compassionate use,” since it introduces inherent value decisions, can emotionally charge discussions, and does not recognize that there may be valid and ethically appropriate reasons for denial. The generally agreed upon term “expanded access,” is used throughout this paper. (One participant suggested the term “early access.”) Ideally, the term would make it obvious that this is access to an unapproved drug, in order to temper expectations of favorable results. Somewhat confusingly, the FDA uses the terms “expanded access,” “access,” and “treatment use” interchangeably to refer to the use of a drug, and of which none clearly identify the stage of development.[vi]

Participants shared numerous examples of requests for expanded access and explained that their companies handle anywhere from a handful to several hundred requests per year. The following selected stories illustrate the wide range of experiences and situations that companies encounter when navigating the complex decisions involved in administering an expanded access program. Several other examples were discussed and the specific participants expressed that they would be willing to share these particular examples publicly.

Chimerix, a 54-employee company based in Durham, North Carolina, is developing the drug brincidofovir and previously had created an intermediate expanded access protocol for the drug (CMX001-350) as encouraged by the FDA following over 200 emergency INDs granted for access to brincidofovir.[vii] One such case was for an armed services member with previously undiagnosed acute myelogenous leukemia who developed life-threatening vaccinia infection following smallpox vaccination in 2009.[viii] The patient received the drug from Chimerix through an emergency IND. After two years, the company had not secured FDA approval for the drug and eliminated expanded access in February 2012 in order to focus on studies which would inform a regulatory decision. In March 2014, Chimerix originally rejected an emergency IND request for 7-year old, Josh Hardy, who was critically ill from disseminated adenovirus infection after bone marrow transplantation. A highly public social media campaign targeted the company in the wake of this decision, and the experience was traumatizing for many of the employees. Following discussion with the FDA, Chimerix initiated a new clinical trial for the treatment of adenovirus infection in order to collect safety and efficacy data to support an NDA submission. Hardy was the first patient enrolled in the clinical trial, and his family reported through several media outlets that he recovered from the adenovirus infection and was discharged home.

One biopharmaceutical company representative described receiving a middle-of-the-night telephone call directly at home, with an emergent, time-sensitive request for an experimental therapy for a critically ill child with a rare acute disease in a foreign pediatric intensive care unit, where regulatory standards were different from those in the U.S. The ideal pediatric dosage was unknown, and only limited safety data and clinical details were available. Urgent efforts were made to gather more information and the request was approved, but despite these efforts the patient did not survive.

Bristol-Myers Squibb began a clinical trial for a cancer drug several years ago.[ix] A woman with pancreatic cancer enrolled in the trial and saw that her tumor was no longer growing. After the 3.5 year trial, the study closed because the drug was deemed ineffective for all other patients and was not approved for further development. However, the company continued to provide the drug for the one woman for whom the drug was effective through a single patient IND for an additional 9 years.

To demonstrate the volume of expanded access requests, one participant showed several messages on his mobile device during the half-day discussion, directly from patients who had located his email addresses through on-line searches, to plead for expanded access to an anticancer therapy.

Development of Structural Principles: The Four A's 

Broadly, no specific industry-wide consensus on expanded access procedures exists. As a result, there is significant variation in company policies and procedures. During this phase of discussion, participants shared their own company strategies and suggested possible areas of consensus that might form the basis for shared principles and industry-wide practices. These suggestions fell into four categories, which we termed the 4 “A’s”: Anticipation, Accessibility, Accountability, and Analysis (see Figure 1).

---

First, the group agreed that large and small companies should anticipate the need for and creation of expanded access programs when developing drugs expected to generate expanded access requests and as part of the drug development plan. This is particularly important for drugs that might be considered for priority or breakthrough designation during FDA approval. In these cases, companies should strongly consider developing a written expanded use policy with clear guidelines for inclusion and exclusion, which would also feature a defined review process, clear decision making criteria, and a defined time frame for response to requests. This also allows companies to plan for the demands that may be placed on their supply chain and staff resources to ensure sufficient supply for investigational and expanded use purposes. Identifying a decision maker within each company and for each disease area/product will also help patients or physicians reach the appropriate contact when requesting a drug, as well as assist the company in gaining expertise in responding to these requests. For example, one large company identifies one point of contact for all expanded access requests regarding each product and posts that individual’s contact information on the website.

In the early stages of drug development, supplies of investigational drugs are extremely limited. This is often because the technically-challenging process of optimizing drug product manufacture takes a considerable amount of time. Low yielding manufacture batches are not uncommon at the early phases of research. Some companies do not approve expanded access requests because they do not have enough of the drug in stock to supply these external requests and meet the needs of investigational study patients and individuals participating in clinical trials, an issue which may be particularly acute for biologics. Smaller companies may have more resource constraints, such as inadequate staff to manage requests or supply chain and logistics issues. One representative suggested that if a company had early transparency from regulators about the final numbers of subjects they would be willing to accept to achieve drug development milestones, it would make it much easier for the company to feel less reservation about its drug supply. (It may be beneficial for companies to analyze their financial ability to provide drugs potentially at no cost or when there is not a large enough supply, ideally in a transparent manner.)

---

Once an expanded access policy is anticipated and developed, the second key principle the group identified was making the policy accessible to all individuals who may qualify. First, for patients, with guidance from their treating physician, the company making the drug should always steer the patient to enter a clinical trial (if they meet eligibility criteria). If the contacted company cannot accommodate the patient, they should steer them to other open trials if possible, even if sponsored by another company. Many of our participants noted that this already occurs.

The group was particularly cognizant of the disparity in access to drug companies and their expanded access programs: patients with savvy social media strategies are more likely to succeed in navigating across organizational constraints than without similar sophistication. The group believes that increased accessibility would assist in making opportunities for expanded access more equitable. In addition, these policies could help educate patients and physicians about submitting legitimate expanded access requests and help decrease the costs of reviewing inappropriate requests on the company (for example, if there are other proven therapies or the situation is not life threatening).

If the patient is ineligible for a trial, the patient should be able to easily access the written expanded access policy online. For example, both large and small companies like Pfizer, Bristol-Myers Squibb, Shire, and Merck post their expanded access policies on their websites, though the terminology may in some cases be complex. In addition, Janssen has developed a video explaining their policies in non-technical terms. Ideally, such policies should be available in some web based or public facing platform to both patients and physicians and written in a clear manner that is jargon free and accessible to individuals at various education levels. Most participants felt strongly that requests for expanded access should originate from a medical provider, not from a patient, since expertise is needed to first screen appropriate candidates. This is consistent with current FDA regulations for an IND, in which a physician or qualified medical expert must sponsor an IND or serve as an investigator under an existing IND for expanded access.

---

Third, companies should have accountability to the requesting party for expanded use requests that they receive and review them within a specified, transparent amount of time. If the request could not be approved, the company should consider clear communication and provide an explanation of why the request was turned down. In these cases, some participants suggested that the company might also consider instituting an appeals process by which a patient can receive an additional review if not approved, potentially from a non-binding third party such as an independent, multidisciplinary body or a regulatory agency like the FDA. (Two participants, however, were uncomfortable with any third party review.)

Companies can track expanded access requests in order to guarantee that the patient has received follow-up and that the communication loop has been closed. One large pharmaceutical company conducted an internal audit of its expanded access procedures and found that the largest problem was that employees did not know where to find information. Another representative noted that it is important to maintain consistency across patients and the process of requesting a drug.

---

The final principle would encourage companies to release timely analysis of data from expanded access patients. In addition to tracking communication, companies should keep a database of the number of requests and outcomes, in a manner that doesn’t slow getting drugs to needy patients rapidly. One company refined its internal tracking tools to determine who was requesting drugs, for what conditions, and where they lived. Where possible, companies might be encouraged to share anecdotal or preliminary safety or efficacy data from expanded access in peer-reviewed or other refereed venues in a prudent time frame following collections, if this is available or known. This is not always possible, because emergency INDs do not require provision of safety or outcome data to the company.

There are several challenges associated with operationalizing this in the current model, namely the appropriateness of anecdotal data, the level of detailed safety and efficacy data currently available through expanded access, suitability for publication, and funding for these activities in the current budget climate. One potential approach to address this is funding from federal or state regulatory agencies or payers for the reasonable costs of follow-up and reporting outcomes.

---

Regulatory Considerations

The participants then discussed the types of risks, including regulatory and financial, that may affect companies’ expanded access policies. When a company is considering expanded access requests, they consider the risks-benefits of providing the drug outside of a clinical trial as well as the potential for any regulatory issues in an era of litigation and an increased threshold for demonstration of safety. While a company’s provision of a drug for expanded access is voluntary, the FDA does require the company to collect and report safety data. Notably, none of the representatives felt that the FDA is a major regulatory barrier to processing and approving expanded access requests once the sponsor has reviewed the request, assessed the benefit-risk, and determined the request meets FDA requirements and evidentiary standards. In addition, the attendees felt that adverse effects and related liability risk were not of particular concern given that the drugs are assessed on a risk-benefit analysis.

However, companies that make drugs in particularly limited markets with small numbers of patients (for example, for unusual diseases with less than 200,000 patients nationwide which may justify a special designation called “orphan status”) may be more concerned about restrictive labeling if an unusual adverse event occurred even in one or two patients during expanded access of an orphan or small market therapy. However, there is no data of which participants were aware and no public reports that an adverse event during expanded access has harmed regulatory approval.[x] The group opinion was that that safety data would be available eventually in any event and an FDA “safe harbor” provision would not necessarily affect companies’ willingness to accept more requests for expanded access. A final concern was that there is no regulatory mechanism to consider data from expanded access in the evidence generation process for approval.

An Expanded Role for the FDA

While the FDA may not serve as a strong barrier to expanded access, the group considered strategies to promote equitable and fair access. For example, some argued that the breakthrough or priority review categories for FDA review might identify products that could have high potential for expanded access requests. This designation expedites “the development and review of drugs for serious or life-threatening conditions.”[xi] As of mid-April 2014, the FDA had received nearly 180 requests for breakthrough designation, with 44 requests granted.[xii] By hastening the drug development process, the FDA has already begun to bring drugs that have a reasonable expectation of benefit to the market faster. In order to receive breakthrough therapy designation, current legislation might be amended so companies could be asked to provide evidence that the 4 A’s are being followed in some capacity.

The FDA might also assist companies in establishing expanded access programs during open clinical trials in two main areas: process and capacity building. First, in terms of process, the FDA could be asked to create a defined path for regulatory approval with provisions that would encourage companies, both large and small, to include plans for expanded access programs when developing a drug. While FDA’s draft guidance related to INDs notes that larger expanded access programs could threaten enrollment in clinical trials,[xiii] and some participants agreed that this was a significant issue, not all companies have had difficulties enrolling patients in both clinical trials and expanded access programs. For example, one large pharmaceutical company left a Phase 1 clinical trial open for a promising therapy while concurrently enrolling individuals who didn’t qualify for open clinical trials into an expanded access program, without appreciable leakage of enrollees in their advanced phase trials that might affect the key development pathway.

Second, the FDA could support convening around capacity building and sharing best practices with companies. With the understanding that there are many small biotechnology or pharmaceutical companies with limited budgets and staff, the FDA could foster a partnership of large and small companies. This partnership could be achieved by convening meetings where companies share their experiences in creating and sustaining expanded access programs. This could be supported by creating a database for these shared ideas, as well as any expanded access data that can be made legally available, such as how many requests are granted or patient outcomes.

To ensure equitable, consistent, and transparent review of requests, some companies suggested the use of an impartial external advisory board. Similar to an unbiased review from an institutional review board (IRB), this committee could have an advisory or decision making function. Companies with supply constraints may feel that if they cannot give the drug to everyone who requests it, then they should give it to no one. This committee could help the company triage the patients who would benefit the most, and would be protected from liability.

Next Steps

The most efficient and equitable way to make new effective treatments to the largest number of needy patients is regulatory approval, accelerated or otherwise, following successful demonstration of efficacy and safety for a given indication in a specific population. Until that process is complete, access to an experimental therapy is by definition an additional risk, as the agreed necessary safety and efficacy have not yet been demonstrated. True informed consent in this setting is difficult to obtain (i.e. studies have shown that severely ill patients, such as those with life-threating circumstances requesting expanded access, had less retention of information discussed in the informed-consent process and less-clear understanding of the risks of therapy compared to healthier patients[xiv]).

One position companies and regulators can consider is that the default answer to expanded access requests should be affirmative, unless there are compelling reasons for not approving requests to patients with life-threatening illnesses. (Such reasons, for example, might include limited treatment supply or lack of reasonable expectation of benefits versus risks.) Such a position would require, however, that there be broader industry, clinician, regulatory, and patient advocacy agreement of shared principles. This paper outlines the experiences, structural principles, and regulatory considerations of a small group, but further meetings may convene a broader group of stakeholders to build upon these concepts. Such consensus-based approaches might lead to durable systems that meet the needs of desperate patients who have run out of options—while allowing innovation to continue to benefit those who may come afterwards.

---

Acknowledgements: We are grateful for the participation of the following representatives in the roundtable: Jeff Allen (Friends of Cancer Research), Michelle Berrey (Chimerix), Renzo Canetta (Bristol-Myers Squibb), Anne Cropp (Pfizer), Joseph Eid (Merck), Aaron Kesselheim (Harvard Medical School), Howard Mayer (Shire), Jeffrey Murray (FDA), Lilli Petruzzelli (Novartis), Amrit Ray (Janssen), and Robert Truog (Harvard Medical School). We thank Mark McClellan (Brookings Institution) for helpful discussions of this topic and comments on the manuscript, and to the Richard Merkin Foundation for support. The views and opinions expressed in this article were interpreted and organized by the staff of the Brookings Institution. They do not necessarily reflect the official policy or position of any individual roundtable representative, their companies, or their employers.

---

References

Downloads

• The Four A's of Expanded Patient Access

       

       

2014 was a relatively friendless year for the Democratic Republic of Korea (DPRK or North Korea). It publicly lost its best friend and patron, China, to its erstwhile nemesis, the Republic of Korea (ROK or South Korea), when Presidents Park Geun-hye and Xi Jinping celebrated their growing friendship at the July summit in Seoul. Recently, retired PLA General Wang Hongguang wrote in the Chinese language site of Global Times, which is closely linked to the Chinese Communist Party, that China tired of cleaning up North Korea’s “mess” and would not step in to “save” North Korea if it collapses or starts a war.[1] And there is a vigorous debate in Beijing on whether the DPRK should be treated on a “normal” basis with China’s interests as the sole guide and purpose or be treated as a special case needing China’s indulgence and protection.[2] Since the Sony hack of November, North Korea has been under tighter scrutiny, both real and virtual, by Seoul, Beijing and Washington, accompanied by tighter sanctions in the new year. Bludgeoned by global condemnation of its atrocious human rights record, Pyongyang’s pariah status has intensified. Only Russia has been warming up to North Korea out of its own economic and political self-interest.

Is there any sizable country with good intentions for the region that is not giving up or beating up on North Korea? Is there any country Pyongyang likes and possibly even trusts? Mongolia stands out as the sole candidate, and it is friendly with both the East and the West.

Since the 2000s, Mongolia has played an increasingly constructive and steady role in in its bilateral ties with the DPRK and in its promotion of peace and cooperation in Northeast Asia. President Tsakhiagiin Elbegdorj, who visited Pyongyang in 2013, was the first head of state to reach out to the DPRK since Kim Jung Un assumed power and helped author the “Ulaanbaatar Dialogue on Northeast Asia Security,” which held its first meeting in June, 2014. It is a unique forum that combines official (track one) and unofficial academic/think tank/NGO (track two) participants, on a variety of important regional issues. The goals are to decrease distrust among nations and increase cooperation and peace. Both the DPRK and the ROK (Republic of Korea or South Korea) were represented at the inaugural meeting, as were the United States, China, Russia, Japan, and some European nations.

The UB Dialogue, as a consultative mechanism, has the potential to bring together policymakers, international organizations such as the United Nations Economic and Social Commission for Asia and the Pacific (ESCAP), and civil society entities and facilitate a range of initiatives related to economic cooperation; military transparency; environmental issues; non-traditional security threats; regional stability, cultural and educational exchange among the participants, including the two Koreas. These are official agenda items and goals of the UB Dialogue. With the Six-Party Talks nearly defunct and inter-Korean relations unable to address regional issues that affect the peninsula, Mongolia may be able to serve as a “Geneva or Helsinki of the East” as some observers have suggested.

Mongolia’s expanding global presence

Mongolia is uniquely positioned as the only country in Northeast Asia that enjoys good relations not only with North Korea but also South Korea, the United States, China, Russia, and Japan.

Mongolia’ relations with the United States, Canada, and Western Europe have steadily improved and deepened since the late 1980s. In recent decades, both Democratic and Republication administrations in Washington have enjoyed mutually warm and collaborative relations with Mongolia. President George W. Bush was the first sitting U.S. president to visit the country in 2005; he thanked the Mongolians for sending troops to join U.S.-led forces in Iraq and Afghanistan and for supporting anti-terrorism initiatives. Former Secretary of Defense Donald Rumsfeld also visited in the same year. In 2007, President Nambaryn Enkhbayar visited Washington to co-sign the Millennium Challenge Corporation Compact with President Bush. The next (and current) leader, President Elbegdorj, met U.S. President Barack Obama at the White House in 2011, as did the first civilian Minister of Defense, L. Bold. Vice President Joe Biden included Mongolia on a three-country Asia visit in August, 2011; China and Japan were the other two. A year later, Secretary of State Hilary Clinton took her turn in Ulaanbaatar. The most recent visit by top-level U.S. officials to Mongolia was by Secretary of Defense Chuck Hagel in April 2014.

Mongolia’s pursuit of the “third neighbor” policy allows the country to develop cooperative relations with the United States, Western Europe, ASEAN nations and others partly as “an air pocket” from its economic and security reliance on Beijing and Moscow. The softer side of this diplomatic push has been demonstrated by Ulaanbaatar’s membership in the Organization for Security and Cooperation in Europe and its previous chairmanship of the Community on Democracies.”[3]

Western experts on Mongolia applaud the way the country has developed a unique “peacekeeping niche” that facilitates participation in UN peacekeeping activities, international anti-terrorism measures, and humanitarian actions. For its small population of about three million, Mongolia takes on a heavy load of peacekeeping activities, ranking 26^th on the UN’s list of contributing nations.[4]

Since 2003, Mongolia annually hosts the “Khaan Quest” peacekeeping exercises for the purpose of tactical advancement and capacity building for its Mongolian Armed Forces (MAF) and for the improvement of regional confidence building. Although the United States and NATO play prominent roles, the Quest has attracted more diverse participants over the years so that by 2012, the number of interested parties expanded to include representatives from China and India as well as an array of developing nations such as Vietnam and Cambodia. These exercises are acknowledged as gatherings devoted to strengthening international cooperation and interoperability on peacekeeping initiatives around the world.[5]

On the economic side, Mongolia has been diversifying its external relations, with the maintenance of sovereignty and the related desire to reduce its overwhelming dependence on China as important goals. Expansion of economic relations is driven in part by a desire to participate in and benefit from global standards investment funds, and market access is a national priority. In that context, Mongolia’s relations with the West have been constructive and collaborative. For example, in 2013, the United States Trade Representative Michael Froman and Mongolia's Minister for Foreign Affairs, Luvsanvandan Bold, signed the Agreement on Transparency in Matters Related to International Trade and Investment between the United States of America and Mongolia. The Agreement commits the parties to provide opportunities for public comment on proposed laws and regulations and to publish final laws and regulations in Mongolian and English in order to facilitate access, openness, fairness, and procedural coherence in international trade and investment between Mongolia and other countries. “Additional commitments address the application of disciplines on bribery and corruption.” This type of administrative and legal modernization and the incorporation of measures to prevent and correct corruption are exemplary measures that could be helpful to the DPRK and other countries that are unfamiliar with or lagging in appropriate frameworks for doing business with diverse international actors.

Maintaining sovereignty between giants

China and Russia have vied for influence over Mongolia for many decades, from the time when Mongolia was in the Soviet sphere in influence to the present. Although 89 percent of foreign trade in 2013 was with China and Russia provides about 75 percent of Mongolia’s gasoline and diesel fuel and much of its electricity, Ulaanbaatar is assertively broadening and deepening its economic interests with the two big neighbors, especially greater transportation access and cheaper costs (vital to the landlocked nation), participation in the development of the New Silk Road corridor, and the construction of a Russian oil and gas pipeline through Mongolia that reaches China. All three countries have mutual interests and investments in developing Mongolia’s well-endowed mining industry.

But being sandwiched between two giants means Mongolia has to be prudent in preserving its sovereignty and independence, and Ulaanbaatar has done so in practical ways, balancing the two large powers’ interests with its own. The 2010 National Security Concept’s “One-Third Clause” sets a clear limit on the proportion of foreign direct investment from any one country: one-third. Legislation limits (foreign) state-owned companies from gaining control of strategic assets. And as numerous bilateral security and military cooperation agreements link Mongolia with China and Russia, UB has strategically and legally created elbow room for its autonomy. The government’s National Security and Foreign Policy Concepts outline a specific policy of not allowing foreign troops the use of its territory. Such preservationist measures to maintain sovereignty and independence in economic and security terms would be welcome examples to a North Korea which zealously prioritizes national sovereignty.

Mongolia and the Korean peninsula

Mongolia’s potential role as a non-nuclear peace broker in the region was further evidenced by its successful hosting of DPRK-Japan negotiations since 2012, which have yielded bilateral progress on longstanding abduction issues. In March 2014, Ulaanbaatar hosted the first-ever reunion between the parents of one of the abductees, Megumi Yokota (whom North Korea claims is dead), and her daughter, son-in-law, and their child who live in North Korea. Mongolia also served as a neutral venue for high-level talks on normalizing Japan-DPRK relations back in September 2007 as part of the Six-Party Talks framework. Asia Times reported that “arranging this recent meeting reflected Ulaanbaatar's ‘contribution to satisfy regional stability in Northeast Asia’ and how it could play a role in deepening understanding and normalizing DPRK-Japan relations.” President Elbegdorj's administration took particular care in staging the negotiations, including the use of the official state compound in Ikh Tenger as the meeting place. According to Alicia Campi, an American expert on Mongolia and the author of the AT article, Ikh Tenger was requested by the North Koreans.[6]

Mongolian President Elbegdorj is often described as an activist head of state, both for his focused efforts on developing Mongolia internally and advancing the country’s role and contributions internationally. One of his main foreign policy priorities is to promote regional economic integration and cooperation and peace and security. Dialogue and trust-building, two key components of his approach, coincide with ROK President Park Geun-hye’s emphasis on trustpolitik and the proposed Northeast Asia Peace and Cooperation Initiative (NAPCI). Both NAPCI and the UB Dialogue seek to chip away at distrust among Northeast Asian countries and increase collaboration and cooperation through multi-layered activities, including mutually reinforcing Track 1, 1.5 and 2 gatherings. Both emphasize multilateral cooperation on non-traditional security issues and people-to-people exchanges as ways to help build trust and resolve regional problems step by step. NAPCI held a track 1.5 forum in October 2014 in Seoul. In sharp contrast to its reaction to the first UB Dialogue of June that year, the DPRK flatly rejected the invitation to participate in the Seoul dialogue and criticized NAPCI as a cover for pressuring Pyongyang to relinquish its nuclear program and for reunification by absorption.[7]

There is no reason why the Ulaanbaatar Dialogue and NAPCI cannot be complementary and mutually reinforcing. Given that trust in inter-Korean relations is non-existent while Mongolia has gained deeper trust with both Koreas over the past two decades, NAPCI activities could benefit from Mongolia’s unique position in its relations with the DPRK. Ulaanbaatar potentially can serve as a neutral meeting ground, literally and metaphorically, for Pyongyang and Seoul. Moreover, given that the NAPCI seeks to maintain a cooperative relationship with other multilateral bodies and places emphasis on complementarity and inclusiveness, working with and supporting successful rounds of the UB Dialogues would be a principled move on the part of South Koreans. Moreover, engagement with North Korea through the UB Dialogue most likely represents an easier path to increasing inter-Korean trust than bilateral efforts and even easier than the NAPCI. South Korea’s domestic divisions and bitter left-right infighting tend to weaken the government’s position in approaches to the North. Seoul’s military standoff and competition with the North, its alliance with the United States, and participation in international sanctions regimes all cause suspicion in Pyongyang. In short, Seoul’s complex list of concerns and goals, some of which are contradictory to the spirit and practice of trust-building and cooperation with North Korea, create difficult conditions for progress through NAPCI alone.

In addition to lacking this baggage, Mongolia has unique standing with both North and South. It is a former Soviet satellite state that asserted full independence in 1990, and it is notable for successfully transitioning from a communist state to a vibrant democracy without civil war or bloodshed. President Elbegdorj’s 2013 speech in Pyongyang contained strong enunciation of the tenets of liberty. At the elite Kim Il Sung University in Pyongyang, he addressed students with these bold words: "No tyranny lasts forever. It is the desire of the people to live free that is the eternal power." And the Mongolian government has been keeping its border open to North Koreans who risk the arduous journey out of the DPRK and has permitted its airlines to transport them to South Korea.

Additionally, Mongolia has become a model of economic modernization and prosperous participation in the global economy. Although it faces some economic imbalances, its GDP rate was sky-high at 11.7 percent in 2013. There are good lessons to share with North Korea, and President Elberdorgj has made it clear that Mongolia would be very willing to work with the DPRK on economic development, IT, infrastructure, the management of mining precious earth resources and refineries. The two countries also engage in a worker exchange program, affording DPRK citizens the opportunity to breathe the air of freedom and to be exposed to South Korean television programming while they reside in Mongolia.

In recent years, Mongolia has pursued multiple types of people-to-people activities involving North Koreans, including academic exchanges, northeast Asian mayoral forums, and women’s parliamentary exchanges including female leaders from both Koreas. In June 2015, the second Track 2 conference of the UB Dialogue will convene in Ulaanbaatar with scholars from across the region and the United States with the theme of “Energy, Infrastructure, and Regional Connectivity.”

Sports and cultural initiatives in the past years have included international boxing matches in Ulaanbaatar with boxers from the DPRK, ROK, Mongolia, Russia and China. In 2013, Mongolia established an International Cooperation Fund which has supported children’s summer camps, basketball training and other exchanges with the DPRK in order to promote positive peace and people-to-people development in the region.

In the humanitarian arena, food aid to the DPRK has been channeled through international organizations, and the two countries have cooperated on physician exchanges. Prior research by Caprara and Ballen, conducted in cooperation with United Nations Special Envoy for Financing the Health Millennium Development Goals and for Malaria, has noted the additional soft power benefits of cooperative service development projects.

A recent global development forum hosted at the United Nations Asia-Pacific headquarters in Bangkok launched an Asia Pacific Peace Service Alliance which could build on these bilateral and regional exchanges in the critical area of humanitarian action and development in North Korea. An International Youth Leaders Assembly has been proposed in Ulaanbaatar for June, 2015, which would further the role of youth in fostering track two initiatives of service and dialogue.

Dr. Tsedendamba Batbayar, Mongolia’s Director of Policy Planning in the Ministry of Foreign Affairs and Trade, visited Washington in November, 2014 and noted the broad range of Mongolia-DPRK exchanges. Together with Mongolia Ambassador Bulgaa Altangerel, he emphasized his country’s desire to serve as a fair broker and mediator for the Northeast Asia region and to pursue prudent and practical measures to help build bridges of understanding between the people of North Korea and other parties.

But despite its uniquely constructive approach to dealing with the DPRK and other regional neighbors, Mongolia faces unique challenges in the mediator role it seeks to achieve. First, Ulaanbaatar has been able to gain Pyongyang’s trust because of the quiet diplomacy it has pursued, staying behind the scenes and out of the limelight. This has enabled a steady channel to the Pyongyang elite, and a focus on bilateral interests has been maintained. In short, drama has been avoided. But if Mongolia plays a more high-profile role with North Korea and multilateral actors, it will most likely be difficult to avoid some drama—posturing, rhetoric, and standoffs—emanating from various parties. Second, any increased or intensified involvement of China, Russia, and the United States in UB-led dialogue could come with the headache of big power arrogance and competition over leadership. The value of Mongolia’s role and activities for regional cooperation and peace stems from the fact that Ulaanbaatar does not assume airs or seek to dominate others. Whether China, Russia, and the United States would be able to refrain from seeking leadership and disproportionate influence in UB-led initiatives is highly questionable. Third, with respect to peninsular issues, for the UB Dialogues to gain more acceptance and credibility regionally and internationally requires that the DPRK become a consistent and collaborative presence at gatherings. Whether any nation or actor has the capacity to deliver consistent and collaborative participation by Pyongyang is an open question.

In addition, some observers believe that the impasse between North Korea and other nations is not simply the result of a trust deficit, but reflects mutually exclusive goals. While Mongolian mediation may not be able to solve the nuclear issue, it can be an effective channel – among others – for increasing communication, finding common ground, and beginning to ease tension.

Mongolia is the one Northeast Asian country that has kept its emotional cool and balanced policy interests with North Korea and other regional actors. It has not tripped over its own feet by politicizing historical grievances with its neighbors. Rather, it has exercised a calm can-do approach while its neighbors have engulfed themselves in hyper-nationalistic and ideological mire. And it has smartly used diplomacy and entrepreneurship to make friends and develop its own economy and people. These are significant assets that can be of benefit not only to UB but also to the region.

Recommendations

1. The Obama administration should actively support the Ulaanbaatar Dialogue process and encourage Seoul to find common cause in advancing greater regional dialogue and collaboration with the Mongolians through Track 2 and 1.5 processes. A precedent for this can be found in the case of Oman, which the current administration effectively tapped for back channel dialogue with Iran, kick-starting the present nuclear talks. Also, support by Washington would build on a prior exchange with Mongolia hosted by the Korea Institute for National Unification (KINU), where scholars noted potential benefits from three-way economic cooperation and the possibility of providing the North Koreans with a proven model of transformation from a closed statist system to a prosperous and more open system.

2. ROK President Park’s proposed regional cooperation mechanism should receive serious attention together with the Ulaanbaatar initiative. The two parallel efforts could benefit from being part of inter-connected strategies to defuse regional tension and forge greater trustpolitik.

3. The UN ESCAP headquarters can serve as an important multilateral bridge for humanitarian aid together with the multi-stakeholder Asia Pacific Peace Service Alliance (APPSA), which was launched at the UN headquarters in Bangkok last October. The U.S. Agency for International Development (USAID) could partner with UN ESCAP and the World Food Program to establish a verifiable humanitarian aid regime, building on prior food aid oversight protocols developed during the Bush administration. Mongolia also would be an excellent candidate for the training of an international volunteer corps for potential disaster and humanitarian relief and economic development projects concerning the DPRK and the broader Northeast Asia region. Mongolia has excellent working relations with the U.S. Peace Corps, which also helped facilitate the recent launch of the APPSA.

4. In the context of peninsula unification planning, regional economic cooperation on private and multi-stakeholder investment projects and the enabling of market-friendly policies could be further explored with Mongolia and other Northeast Asian partners in areas such as infrastructure, energy, and technology.5. Cultural and educational exchanges between Mongolia and the DPRK could be expanded on a multilateral basis over time to include the ROK, China, Russia, Japan and ASEAN nations together with UNESCO to further cultural bases and norms of peace.

       

Abstract

Progress in closing differences in many objective outcomes for blacks relative to whites has slowed, and even worsened, over the past three decades. However, over this period the racial gap in wellbeing has shrunk. In the early 1970s data revealed much lower levels of subjective well-being among blacks relative to whites. Investigating various measures of well-being, we find that the well-being of blacks has increased both absolutely and relative to that of whites. While a racial gap in well-being remains, two-fifths of the gap has closed and these gains have occurred despite little progress in closing other racial gaps such as those in income, employment, and education. Much of the current racial gap in well-being can be explained by differences in the objective conditions of the lives of black and white Americans. Thus making further progress will likely require progress in closing racial gaps in objective circumstances.

Downloads

• Subjective and Objective Indicators of Racial Progress

      

Event Information

This event has been postponed, and will be rescheduled for a later date.

With ongoing crises in Ukraine, Syria, and other regions of the world, U.S. global leadership is arguably as critical now as it has ever been. However, many question how the United States should exercise its leadership, what foreign policy agenda it should pursue, and how it should configure its military and security agencies going forward. In a recent speech at West Point, President Obama laid out his foreign policy agenda for the remainder of his presidency. While the Obama Administration will pursue the president’s agenda as laid out at West Point, others in Washington have different views on how best to manage U.S. foreign policy going forward.

On June 11, the Foreign Policy Program at Brookings will host Senator John McCain (R-AZ), former presidential candidate and member of the Senate Committee on Foreign Relations, for an address on the future of U.S. foreign and security policy. The address will be introduced by Brookings Senior Fellow and Director of Research for Foreign Policy Michael O’Hanlon, and the discussion following the Senator’s address will be moderated by Senior Fellow Robert Kagan.

After the program, Senator McCain will take audience questions.

Join the conversation on Twitter using #McCain

      

       

Earlier this summer, we examined how the Big Four banks – Bank of America, Citigroup, JPMorgan Chase, and Wells Fargo – performed before, during, and after the 2007-09 financial crisis.  We also blogged about the lending trends within these large banks, expressing concern about the growing gap between deposits taken and loans made by the Big Four, and calling on policymakers to explore the issue further.  We have conducted a similar analysis on the regional banks - The regional banks: The evolution of the financial sector, Part II - and find that these smaller banks are actually faring somewhat better than their bigger counterparts.

Despite the mergers and acquisitions that happened during the crisis, the Big Four banks are a smaller share of banking today than they were in 2007.  The 15 regionals we evaluated, on the other hand, are thriving in the post-crisis environment and have a slightly larger share of total bank assets than they had in 2007.  The Big Four experienced rapid growth in the years leading up to the crisis but much slower growth in the years since.  The regionals, however, have been chugging along: with the exception of a small downward trend during the crisis, they have enjoyed slow but steady growth since 2003.

There is a gap between deposits and loans among the regionals, but it is smaller than the Big Four’s gap.  Tellingly, the regionals’ gap has remained basically constant in size during the recovery, unlike the Big Four’s gap, which is growing.  Bank loans are important to economic growth, and the regional banks are growing their loan portfolios faster than the biggest banks.  That may be a good sign for the future if the regional banks provide more competition for the big banks and a more competitive banking sector overall.

      

       

       

Editors’ Note: Though our allies could do more, the United States benefits not just from military spending, write Kathleen Hicks and Michael O’Hanlon. U.S. allies are not just an advantage for America; they also are the central characteristic of its global leadership. This post originally appeared on USA Today.

Donald Trump questions the value of America's alliances, and at one level it must be acknowledged that he has a point. There is not enough burden-sharing.

Most starkly, the United States spends 3 percent of gross domestic product on its armed forces, while the rest of NATO averages 1.4 percent of GDP even after agreeing formally to a 2 percent target. And the consequences are natural—for example, at the peak of the Afghanistan war, the U.S. provided 100,000 troops to the mission while the rest of NATO managed only about 35,000.

[W]hatever the imperfections of America's allies, it is unconvincing to view them as a drain on the country.

But whatever the imperfections of America's allies, it is unconvincing to view them as a drain on the country. On balance, they are not just an advantage for America; they also are the central characteristic of its global leadership. Russia and China are blessed with the likes of Belarus and North Korea as their formal security partners. Meanwhile, America has nearly 60, most of them among the world's dominant technological and economic powers. Two-thirds of global GDP and two-thirds of global military spending is found within the broad, U.S.-led Western security coalition. This is unprecedented in the history of the world, especially in the absence of a clear agreed threat that would motivate such an alignment.

The stakes are very high. Trump is apparently willing to disband NATO as well as our key Asian alliances, and to withdraw from the Middle East as well—a "Trexit." At risk is a core principle of America's post-World War II strategy—that trying to stay out of others’ business did not work and, in fact, helped lead to the world wars. Trump in particular seems to reject the core elements of America’s strengths in the world market and international security system. With the tumult roiling Europe today, it is worth reviewing the basics on why our unparalleled alliance structure pays off.

Allies are not all alike. Although our Western European allies generally under-invest in common defense objectives, our Middle East partners and several in Asia do far better. The Middle East allies spend considerably more than the NATO members; South Korea comes in at about 2.5 percent, with Australia at 2 percent. Moreover, contrary to conventional wisdom, the U.S. does not squander huge sums of money basing troops on the territories of its wealthy allies. According to RAND analyses and our own estimates, it costs less than $10 billion a year to keep U.S. forces in key industrial nations. That amounts to less than 2 percent of the defense budget. And of that nearly $10 billion, half or more is paid by the host nations themselves.

Moreover, basing those U.S. forces abroad can be more efficient than keeping them at home, allowing a modestly smaller force to do the same job in some cases (as with the aircraft carrier based in Japan, for example)—saving substantially more than $10 billion annually, in fact.

Military might and spending are not the only measure of alliance contributions to security. It is worth remembering that many wealthy allies spend a lot more of their national wealth on development aid and refugee resettlement than does the United States—on average, in the range of 0.5 percent of GDP more than we do, which is real money even measured against the sizes of our military budgets. NATO allies also commit much more to United Nations peacekeeping missions than we do—roughly 10,000 troops, plus or minus, in recent decades. While they did not match us in Afghanistan, they collectively suffered more than 1,000 fatalities and have stuck with the mission for 15 long, hard years.

Those who question the basic value of our alliances or engagement overseas go too far.

Nor is it all about military power. European members of NATO are absorbing the greatest costs and risks in applying sanctions on Russia over its behavior in Ukraine. The U.S. tally is not even a 10th of the European Union's Russian trade. Europe was also collectively crucial in applying sanctions on Iran.

Perhaps this would have happened anyway, without NATO, in the world Trump wants to create. Perhaps not.

It would be a different matter if America had lots of allies that went around causing wars and then expecting U.S. GIs and taxpayers to bail their chestnuts out of the fire once they overreached.

But that is not the case. Since the creation of the U.S. post-World War II alliance system, those few interstate wars that have happened have largely been caused by neutral or adversarial states—North Korea attacking South Korea, Arab states attacking Israel (in the days before many of the former became security partners with the United States), Iraq attacking Iran and Kuwait, China and Vietnam throwing their weight around their neighborhoods in earlier Cold War decades, and Russian President Vladimir Putin redrawing borders in Europe (after decades of the Soviet Union drawing an Iron Curtain across Europe).

In those rare cases where U.S. allies did employ force, as with Pakistan attacking India in earlier eras, or the British and French acting in former colonial states, it was generally understood that America would not help them, and we didn’t.

There are valid critiques about alliance burden-sharing, many of which are universally shared by foreign policy practitioners. Even so, the verdict is simple: Those who question the basic value of our alliances or engagement overseas go too far. In so doing, they distort the big picture. On balance, America’s alliances help this country undergird a global security system that has dramatically reduced the prevalence of war between nations in modern times, while currently costing the country only 3 percent of GDP.

A Trexit would be a disaster for Americans. To paraphrase Trump himself, our international security posture is a very good investment—and one that the U.S. global system of alliances does much to make possible.

       

       

Editor's Note: The World Bank’s Independent Evaluation Group (IEG) recently published a self-evaluation of its activities. Besides representing current thinking among evaluation experts at the World Bank, it also more broadly reflects some of the strengths and gaps in the approaches that evaluators use to assess and learn from the performance of the international institutions with which they work. The old question “Quis custodet ipsos custodes?” – loosely translated as “Who evaluates the evaluators?” – remains as relevant as ever. Johannes Linn served as an external peer reviewer of the self-evaluation and provides a bird’s-eye view on the lessons learned.

An Overview of the World Bank’s IEG Self-Evaluation Report

In 2011 the World Bank’s Independent Evaluation Group (IEG) carried out and published a self-evaluation of its activities. The self-evaluation team was led by an internal manager, but involved a respected external evaluation expert as the principal author and also an external peer reviewer.

The IEG self-evaluation follows best professional practices as codified by the Evaluation Cooperation Group (ECG). This group brings together the evaluation offices of seven major multilateral financial institutions in joint efforts designed to enhance evaluation performance and cooperation among their evaluators. One can therefore infer that the approach and focus of the IEG self-evaluation is representative of a broader set of practices that are currently used by the evaluation community of international financial organizations.

At the outset the IEG report states that “IEG is the largest evaluation department among Evaluation Capacity Group (ECG) members and is held in high regard by the international evaluation community. Independent assessments of IEG’s role as an independent evaluation function for the Bank and IFC rated it above the evaluation functions in most other ECG members, international nongovernmental organizations, and transnational corporations and found that IEG follows good practice evaluation principles.”

The self-evaluation report generally confirms this positive assessment. For four out of six areas of its mandate IEG gives itself the second highest rating (“good”) out of six possible rating categories. This includes (a) the professional quality of its evaluations, (b) its reports on how the World Bank’s management follows up on IEG recommendations, (c) cooperation with other evaluation offices, and (d) assistance to borrowing countries in improving their own evaluation capacity. In the area of appraising the World Bank’s self-evaluation and risk management practices, the report offers the third highest rating (“satisfactory”), while it gives the third lowest rating (“modest”) for IEG’s impact on the Bank’s policies, strategies and operations. In addition the self-evaluation concludes that overall the performance of IEG has been “good” and that it operates independently, effectively and efficiently.

The report makes a number of recommendations for improvement, which are likely to be helpful, but have limited impact on its activities. They cover measures to further enhance the independence of IEG and the consistency of evaluation practices as applied across the World Bank Group’s branches – the World Bank, the International Finance Corporation (IFC), and the Multilateral Investment Guarantee Agency (MIGA) –; to improve the design of evaluations and the engagement with Bank management upstream for greater impact; and monitoring the impact of recent organizational changes in IEG in terms of results achieved. The report also recommends that more be done to evaluate the Bank’s analytical work and that evaluations draw on comparative evidence.

Assessment

In terms of the parameters of self-evaluation set by the prevailing practice among the evaluators on international financial agencies, the IEG self-evaluation is accurate and helpful. From my own experience as an operational manager in the Bank whose activities were evaluated by IEG in years past, and as a user of IEG evaluations (and of evaluations of other international aid organizations) for my research on aid effectiveness, I concur that IEG is independent and effective in meeting its mandate as defined. Moreover, the self-evaluation produces useful quantitative evidence (including survey results, budget analysis, etc.) to corroborate qualitative judgments.

However, the self-evaluation suffers from a number of limitations in approach and gaps in focus, which are broadly representative of the practices prevalent among many of the evaluation offices of international aid agencies.

Approach of the IEG self-evaluation

The core of the self-evaluation report is about the evaluation process followed by IEG, with very little said about the substance of IEG’s evaluations. The following questions could have usefully been raised, but were not: do evaluations cover the right issues with the right intensity, such as growth and poverty; environmental, governance, and gender impacts; regional dimensions versus exclusive country or project focus; effectiveness in addressing the problems of fragile and conflict states; effectiveness in dealing with global public goods; sustainability and scaling up; etc. Therefore the report does not deal with the question of whether IEG effectively responds in its evaluations to the many important strategic debates and issues with which the development community is grappling.

Related to this limitation is the fact that the report assessed the quality of IEG’s mostly in terms of (a) whether its approach and processes meet certain standards established by the Evaluation Cooperation Group; and (b) how it is judged by stakeholders in response to a survey commissioned for this evaluation. Both these approaches are useful, but they do not have any basis in professional assessments of the quality of individual products. This is equivalent to IEG evaluating the World Bank’s projects on the quality of its processes (e.g., appraisal and supervision processes) and on the basis of stakeholder surveys, without evaluating individual products and their impacts.

Gaps in the Self-Evaluation and in Evaluation Practice

Careful reading of the report reveals six important gaps in the IEG self-evaluation, in the prevailing evaluation practice in the World Bank, and more generally in the way international financial organizations evaluate their own performance. The first three gaps relate to aspects of the evaluation approach used and the second three gaps relate to lack of focus in the self-evaluation on key internal organizational issues:

1. Impact Evaluations: The report notes that IEG carries out two to three impact evaluations per year, but it sidesteps the debate in the current evaluation literature and practice as to what extent the “gold standard” of randomized impact evaluation should occupy a much more central role. Given the importance of this debate and divergence of views, it would have been appropriate for the self-evaluation to assess IEG’s current practice of very limited use of randomized evaluations.

2. Evaluation of Scaling Up: The report does not address the question of to what extent current IEG practice not only assesses the performance of individual projects in terms of their outcomes and sustainability, but also in terms of whether the Bank has systematically built on its experience in specific projects to help scale up their impact through support for expansion or replication in follow-up operations or through effective hand-off to the government or other partners. In fact, currently IEG does not explicitly and systematically consider scaling up in its project and program evaluations. For example, in a recent IEG evaluation of World Bank funded municipal development projects (MDPs) , IEG found that the Bank has supported multiple MDPs in many countries over the years, but the evaluation did not address the obvious question whether the Bank systematically planned for the project sequence or built on its experience from prior projects in subsequent operations. While most other evaluation offices like IEG do not consider scaling up, some (in particular those of the International Fund for Agricultural Development and the United Nations Development Program) have started doing so in recent years.

3. Drawing on the Experience of and Benchmarking Against Other Institutions: The self-evaluation report does a good job in benchmarking IEG performance in a number of respects against that of other multilateral institutions. In the main text of the report it states that “IEG plans to develop guidelines for approach papers to ensure greater quality, in particular in drawing on comparative information from other sources and benchmarking against other institutions.” This is a welcome intention, but it is inadequately motivated in the rest of the report and not reflected in the Executive Summary. The reality is that IEG, like most multilateral evaluation offices, so far has not systematically drawn on the evaluations and relevant experience of other aid agencies in its evaluations of World Bank performance. This has severely limited the learning impact of the evaluations.

4. Bank Internal Policies, Management Processes and Incentives: IEG evaluations traditionally do not focus on how the Bank’s internal policies, management and incentives affect the quality of Bank engagement in countries. Therefore evaluations cannot offer any insights into whether and how Bank-internal operating modalities contribute to results. Two recent exceptions are notable exceptions. First, the IEG evaluation of the Bank’s approach to harmonization with other donors and alignment with country priorities assesses the incentives for staff to support harmonization and alignment. The evaluation concludes that there are insufficient incentives, a finding disputed by management. Second, is the evaluation of the Bank’s internal matrix management arrangements, which is currently under way. The self-evaluation notes that Bank management tried to quash the matrix evaluation on the grounds that it did not fall under the mandate of IEG. This is an unfortunate argument, since an assessment of the institutional reasons for the Bank’s performance is an essential component of any meaningful evaluation of Bank-supported programs. While making a good case for the specific instance of the matrix evaluation, the self-evaluation report shies away from a more general statement in support of engaging IEG on issues of Bank-internal policies, management processes and incentives. It is notable that IFAD’s Independent Office of Evaluation appears to be more aggressive in this regard: It currently is carrying out a full evaluation of IFAD’s internal efficiency and previous evaluations (e.g., an evaluation of innovation and scaling up) did not shy away from assessing internal institutional dimensions.

5. World Bank Governance: The IEG self-evaluation is even more restrictive in how it interprets its mandate regarding the evaluation of the World Bank’s governance structures and processes (including its approach to members’ voice and vote, the functioning of its board of directors, the selection of its senior management, etc.). It considers these topics beyond IEG’s mandate. This is unfortunate, since the way the Bank’s governance evolves will substantially affect its long-term legitimacy, effectiveness and viability as an international financial institution. Since IEG reports to the Bank’s board of directors, and many of the governance issues involve questions of the board’s composition, role and functioning, there is a valid question of how effectively IEG could carry out such an evaluation. However, it is notable that the IMF’s Independent Evaluation Office, which similarly reports to the IMF board of directors, published a full evaluation of the IMF’s governance in 2008, which effectively addressed many of the right questions.

6. Synergies between World Bank, IFC and MIGA: The self-evaluation report points out that the recent internal reorganization of IEG aimed to assure more effective and consistent evaluations across the three member branches of the World Bank Group. This is welcome, but the report does not assess how past evaluations addressed the question of whether the World Bank, IFC and MIGA effectively capitalized on the potential synergies among the three organizations. The recent evaluation of the World Bank Group’s response to the global economic crisis of 2008/9 provided parallel assessments of each agency’s performance, but did not address whether they work together effectively in maximizing their synergies. The reality is that the three organizations have deeply engrained institutional cultures and generally go their own ways rather than closely coordinating their activities on the ground. Future evaluations should explicitly consider whether the three effectively cooperate or not. While the World Bank is unique in the way it has organizationally separated its private sector and guarantee operations, other aid organizations also have problems of a lack of cooperation, coordination and synergy among different units within the agency. Therefore, the same comment also applies to their evaluation approaches.

Conclusions

Self-evaluations are valuable tools for performance assessment and IEG is to be congratulated for carrying out and publishing such an evaluation of its own activities. As for all self-evaluations, it should be seen as an input to an independent external evaluation, a decision that, for now, has apparently been postponed by the Bank’s board of directors.

IEG’s self-evaluation has many strengths and provides an overall positive assessment of IEG’s work. However, it does reflect some important limitations of analysis and of certain gaps in approach and coverage, which an independent external review should consider explicitly, and which IEG’s management should address. Since many of these issues also likely apply to most of the other evaluation approaches by other evaluation offices, the lessons have relevance beyond IEG and the World Bank.

Key lessons include:

• An evaluation of evaluations should focus not only on process, but also on the substantive issues that the institution is grappling with.
• An evaluation of the effectiveness of evaluations should include a professional assessment of the quality of evaluation products.
• An evaluation of evaluations should assess:
  o How effectively impact evaluations are used;
  o How scaling up of successful interventions is treated;
  o How the experience of other comparable institutions is utilized;
  o Whether and how the internal policies, management practices and incentives of the institution are effectively assessed;
  o Whether and how the governance of the institution is evaluated; and
  o Whether and how internal coordination, cooperation and synergy among units within the organizations are assessed.

Evaluations play an essential role in the accountability and learning of international aid organizations. Hence it is critical that evaluations address the right issues and use appropriate techniques. If the lessons above were reflected in the evaluation practices of the aid institutions, this would represent a significant step forward in the quality, relevance and likely impact of evaluations.