Ninety-three years ago yesterday, the Boston police force went on strike, leaving the city unprotected while the state scrambled to find replacements. Governor Calvin Coolidge’s declaration of support for the city—he said that “There is no right to strike against the public safety, anywhere, anytime”—established his national reputation that ultimately led to the presidency.

Public outrage at labor actions that compromise public safety has historically been a bipartisan affair.  Coolidge was a Republican but his actions earned the respect of Democratic President Woodrow Wilson, who hailed his re-election as Massachusetts governor as “a victory for law and order.” Nearly 20 years later, President Franklin Roosevelt shared his view that a strike by public employees of any sort is “unthinkable and intolerable.”

The impacts of the Chicago teacher strike that began today may not be as immediately obvious as the looting and vandalism that descended on Boston in 1919, but they are just as serious. Research from a large, urban school district found that teacher absenteeism has a negative impact on student learning in math.

But a strike doesn't leave students with substitute teachers—it leaves them without any school at all. Research on summer learning loss shows that being out of school has a disproportionate effect on low-income students. One recent study found that “while all students lose some ground in mathematics over the summer, low-income students lose more ground in reading, while their higher-income peers may even gain.” In other words, the consequence of being out of school is to increase the already unacceptably large achievement gap between low-income students and their affluent peers.

The American labor movement has made important contributions in areas ranging from workplace safety to child labor to employment discrimination. There are good reasons to believe that the public ought to accept higher coal prices resulting from a strike to protect the lives of miners. But the public should not tolerate damage to the education of disadvantaged students resulting from a strike over disagreements about teachers’ salaries, benefits, job security, and method of evaluation.

The Chicago Teachers Union’s differences with the city over how the public schools ought to be run may well be legitimate. But those battles should be fought in the court of public opinion and ultimately at the ballot box, not through strikes that come largely at the expense of poor children.

Donald Trump has made waves during this year’s election cycle by taking a hard line on illegal immigration. This, however, builds on years of heated debate among policymakers. It is also an enduring hot-button issue in Arizona, which has passed several immigration laws over the years.  In 2010, the passage of SB 1070 brought national attention to this debate.  Deemed the strictest immigration law to date, SB 1070 sought to achieve “attrition [of illegal immigrants] through enforcement” by requiring law enforcement to detain any person whom they believed to be residing in the country illegally. Although SB 1070’s effects on individuals and families have been well documented, little is known about its impact on students and schools. To this end, we sought to estimate the relationship between the passage of SB 1070 and school-level student achievement.

We anticipated that anti-immigration policies would primarily affect children from the families of undocumented immigrants. Such effects could be observed in different ways. For instance, the emotional and psychological distress of these children could result in a decline in average test scores at the school-level. On the other hand, students might have left the country or the state under the threat of being deported in which case school-level test scores would rise (since these students often perform below their peers). To this end, we considered three scenarios: 

1. Immigrant children remain in the state but experience higher levels of stress.  As a result, average school-level test scores will drop while Hispanic enrollment remains the same.
2. Children of undocumented immigrants leave the state, which results in a drop in Hispanic enrollment accompanied by an increase in school-level test scores.
3. Or, the first two scenarios occur simultaneously and we do not observe any change in test scores since the two effects would cancel each other, but note a slight decrease in Hispanic enrollment.

In order to see which of these hypothetical scenarios is supported by the data, we first estimated the relationship between the passage of SB 1070 and average school-level reading test scores. We then attempted to unpack the mechanism through which such an effect might have taken place. To this end, we used publicly available data on school-level achievement and enrollment collected by the Arizona Department of Education (ADE). Given the targeted nature of the policy and the demographics of immigrants in Arizona, the majority of whom are of Hispanic or Mexican descent, we focused on schools that traditionally enroll large proportions of Hispanic students. We identified schools with high (more than 75 percent) shares of Hispanic students as those whose average achievement and student composition are most likely to be affected by immigration reform. We contrasted changes in school-level achievement and enrollment in those schools with schools that enroll less than 25 percent Hispanic students, as these schools are less likely to experience any changes as a result of tightening immigration laws.

Figures 1 and 2 show trends in the average percentage of students passing the state reading test and average Hispanic enrollment at these schools between 2006-2007 and 2011-2012.           

Figure 1. Average Percent of Students Passing AIMS Reading

Figure 2. Average Hispanic Student Enrollment

Clearly, the rate of growth in school-level reading scores was much higher for high Hispanic schools after the passage of SB 1070 in 2010 (Figure 1). At the same time, there was a significant decrease in Hispanic enrollment in these schools (Figure 2). Thus, it appears the second scenario is likely driving the patterns we observe.

The data also suggest that the trends for high Hispanic and low Hispanic schools started diverging before the passage of SB 1070 - after the 2007-2008 school year.  This happens to be the year that Arizona passed an even more restrictive, though less controversial, immigration law – the Legal Arizona Workers Act (LAWA). LAWA required Arizona business owners to verify the legal status of their employees using E-Verify, an online tool managed by the federal government. Although LAWA used a different mechanism, similar to SB 1070 it sought to achieve the attrition of undocumented immigrants from the state. 

We then would anticipate both laws to have similar effects on school-level achievement and Hispanic enrollment. Indeed, we estimated that LAWA likely led to an average increase of roughly 4 percent of students passing the state reading test at high Hispanic schools. This was accompanied by an average loss of 38 Hispanic students per school. Because the passage of SB 1070 was preceded by the passage LAWA as well as a language policy that would have affected treatment schools, disentangling the effects of these two policies is not straightforward. However, based on our analysis, we estimate that SB 1070 is associated with an average increase of between 1.5 percent and 4.5 percent of students passing the state reading test at the school-level accompanied by an average loss of between 14 and 40 Hispanic students. 

Despite the fact that we cannot pin down the exact magnitude of SB 1070’s effect on school-level achievement, our analysis shows that when Arizona passed restrictive immigration laws in 2008 and 2010, it looked as if the state’s lowest performing schools were improving rapidly. This, however, likely had more to do with the changing composition of schools as an indirect though anticipated effect of immigration policies than with policies aimed at improving student achievement. 

Despite this, the Arizona Department of Education took credit for these gains. Similarly, Arizona was recently recognized as one of the nation’s leaders in growth on the National Assessment of Educational Progress (NAEP) over the last ten years. Although wrongly attributing these gains may seem harmless at first glance, it is important to remember that Arizona is viewed by many as a model for controversial education reforms like school choice and high-stakes accountability. It is easy to imagine how policymakers might look at increasing test scores in Arizona and wrongly attribute them to these kinds of reforms. That’s not to say that these policies don’t have merit. However, if other states adopt education policy reforms under the assumption that they worked in Arizona, then they might find that these policies fail to deliver.

As every good teacher knows, education is not just about academics. It is about broadening horizons and discovering passions. (The root of education is the Latin e ducere, meaning “to draw out.”) From this perspective, extra-curricular activities count for a great deal. But as Robert Putnam highlights in his book Our Kids, there are growing class gaps in the availability of music, sports, and other non-classroom activities.

Fewer field trips?

Schools under pressure may also cut back on field trips outside the school walls to parks, zoos, theaters, or museums. In the 2008-09 school year, 9 percent of school administrators reported eliminating field trips, according to the annual surveys by the American Association of School Administrators (AASA). That figure rose through the recession:

Just 12 percent of the administrators surveyed about 2015-16 said they had brought back their field trips to pre-recession levels. Museums around the country report hosting fewer students, from Los Angeles and Sarasota, to Minneapolis, and Columbia, Missouri. None of this is definitive proof of a decline in field trips, since we are relying on a single survey question. But it suggests a downward trend in recent years.

Museums help with science tests

If some children are missing out on field trips, does it matter? They may be nice treats, but do they have any real impact, especially when they take time away from traditional learning? There is some evidence that they do.

Middle school children with the chance to go on a field trip score higher on science tests, according to a 2015 study by Emilyn Ruble Whitesell.

She studied New York City middle schools with teachers in Urban Advantage, a program that gives science teachers additional training and resources—as well as vouchers for visiting museums. In some schools, the Urban Advantage teachers used the field trip vouchers more than others. Whitesell exploits this difference in her study, and finds that attending a school with at least 0.25 trips per student increased 8^th grade scores by 0.026 standard deviations (SD). The odds of a student passing the exam improved by 1.2 percentage points. There were bigger effects for poor students, who saw a 0.043 SD improvement in test scores, and 1.9 percentage point increase in exam pass rates.

Art broadens young minds

Students visiting an art museum show statistically significant increases in critical thinking ability and more open-minded attitudes, according to a randomized evaluation of student visits to the Crystal Bridges Museum in northwest Arkansas. One example: those who visited the museum more often agreed with statements like: “I appreciate hearing views different from my own” and “I think people can have different opinions about the same thing.” The effects are modest. But the intervention (a single day at the museum) is, too. Again, there were larger effects for poor students:

All this needs to be put in perspective. In comparison with the challenge of closing academic gaps and quality teaching, field trips are small beer. But schools create citizens as well as undergraduates and employees. It matters, then, if we have allowed field trips to become a casualty of the great recession.

On July 5, the results from the first year of the world’s first development impact bond (DIB) for education in Rajasthan, India, were announced. The Center for Universal Education hosted a webinar in which three stakeholders in the DIB shared their perspective on the performance of the intervention, their learnings about the DIB process, and their thoughts for the future of DIBs and other results-based financing mechanisms.

Approximately 3 million girls ages 6 to 13 were out of school in India according to most recent data, 350,000 of which are in the state of Rajasthan. Child marriage is also a large issue in the state; no state-specific data exists, but nationwide 47 percent of girls ages 20 to 24 are married before age 18. According to Educate Girls, a non-governmental organization based in Rajasthan, girls’ exclusion is primarily a result of paternalistic societal mindsets and traditions. Given the evidence linking education and future life outcomes for girls, this data is greatly concerning.

The DIB finances a portion of the services provided by Educate Girls, which has been working to improve enrollment, retention, and learning outcomes for girls (and boys) in Rajasthan since 2007. The organization trains a team of community volunteers ages 18 to 30 to make door-to-door visits encouraging families to enroll their girls in school and to deliver curriculum enhancement in public school classrooms. Their volunteers are present in over 8,000 villages and 12,500 schools in Rajasthan. The DIB was launched in March of 2015 to finance services in 166 schools, which represents 5 percent of Educate Girls’ annual budget. The DIB is intended to be a “proof of concept” of the mechanism using this relatively small selection of beneficiaries.

The investor in the DIB is UBS Optimus Foundation, who has provided $238,000 in working capital to fund the service delivery. ID Insight, a non-profit evaluation firm, will evaluate the improvement in learning of girls and boys in the treatment schools in comparison to a control group and will validate the number of out of school girls enrolled. The Children’s Investment Fund Foundation serves as the outcome funder, and has agreed to pay UBS Optimus Foundation 43.16 Swiss francs ($44.37) for each unit of improved learning and 910.14 francs ($935.64) for every percentage point increase in the enrollment of girls out of school. Instiglio, a non-profit impact bond and results-based financing intermediary organization, provided technical assistance to all parties during the design of the DIB and currently provides performance management assistance to Educate Girls on behalf of UBS Optimus Foundation. 

The outcomes will be calculated in 2018, at the end of three years; however, preliminary results for the year since the launch of the DIB (representing multiple months of door-to-door visits and seven weeks of interventions in the classroom) were released last week. The payments for the DIB were structured such that the investor, UBS Optimus Foundation, would earn a 10 percent internal rate of return (IRR) on their investment at target outcome levels, which were based on Educate Girls’ past performance data. The table below presents the metrics, target outcome level, year-one result, and the progress toward the target. 

Table 1: Educate Girls DIB Results from first year of services

The DIB was challenging to implement and required DIB stakeholders to be resourceful.

First, the reliability of government data was a challenge, which necessitated flexibility in the identification of the target population and metrics. Second, given the number of stakeholders engaged and the novelty of this approach, the transaction costs were higher than they would have been for a traditional grant. This meant that strong and regular communication was crucial to the survival of the project.

The role of the outcome funder and investor were significantly different versus a grant.

The outcome funder spent more resources on defining outcomes, but spent fewer resources on managing grant activities. The investor utilized risk management and monitoring strategies informed by the activities in their commercial banking branch, which they have not used for other grants.

The DIB has changed the way the service provider operates.

In the video below, Safeena Husain from Educate Girls’ highlights the ways in which financing a portion of their program through a DIB differs from financing the program through grants. Safeena describes that in a grant, performance data is reported up to donors, but rarely makes it back down to frontline workers. The DIB has helped them to develop mobile dashboards that ensure performance data is reaching the front line and helping to identify barriers to outcomes as early as possible.

Based on the learnings from the implementation of the first DIB for education, this tool can be used to improve the value for money for the outcome funder and strengthen the performance management of a service provider. As the panelists discussed in the webinar, DIBs and other outcome-based financing mechanisms can help differentiate between organizations that are adept at fundraising and those that excel at delivering outcomes. However, service providers must be sufficiently prepared for rigorous outcome measurement if they plan to participate in a DIB; otherwise the high-stakes environment might backfire. In our research, we have closely examined the design constraints for impact bonds in the early childhood sector.

There are countless lessons to be learned from the stakeholder’s experience in the first DIB for education. We applaud the stakeholders for being transparent about the outcomes and true challenges associated with this mechanism. This transparency will be absolutely critical to ensure that DIBs are implemented and utilized appropriately moving forward.

Editor's Note: The introduction to the 2015 Brown Center Report on American Education appears below. Use the Table of Contents to navigate through the report online, or download a PDF of the full report.

TABLE OF CONTENTS

Part I: Girls, Boys, and Reading

Part II: Measuring Effects of the Common Core

Part III: Student Engagement

INTRODUCTION

The 2015 Brown Center Report (BCR) represents the 14^th edition of the series since the first issue was published in 2000.  It includes three studies.  Like all previous BCRs, the studies explore independent topics but share two characteristics: they are empirical and based on the best evidence available.  The studies in this edition are on the gender gap in reading, the impact of the Common Core State Standards -- English Language Arts on reading achievement, and student engagement.

Part one examines the gender gap in reading.  Girls outscore boys on practically every reading test given to a large population.  And they have for a long time.  A 1942 Iowa study found girls performing better than boys on tests of reading comprehension, vocabulary, and basic language skills.  Girls have outscored boys on every reading test ever given by the National Assessment of Educational Progress (NAEP)—the first long term trend test was administered in 1971—at ages nine, 13, and 17.  The gap is not confined to the U.S.  Reading tests administered as part of the Progress in International Reading Literacy Study (PIRLS) and the Program for International Student Assessment (PISA) reveal that the gender gap is a worldwide phenomenon.  In more than sixty countries participating in the two assessments, girls are better readers than boys. 

Perhaps the most surprising finding is that Finland, celebrated for its extraordinary performance on PISA for over a decade, can take pride in its high standing on the PISA reading test solely because of the performance of that nation’s young women.  With its 62 point gap, Finland has the largest gender gap of any PISA participant, with girls scoring 556 and boys scoring 494 points (the OECD average is 496, with a standard deviation of 94).   If Finland were only a nation of young men, its PISA ranking would be mediocre.

Part two is about reading achievement, too. More specifically, it’s about reading and the English Language Arts standards of the Common Core (CCSS-ELA).  It’s also about an important decision that policy analysts must make when evaluating public policies—the determination of when a policy begins. How can CCSS be properly evaluated? 

Two different indexes of CCSS-ELA implementation are presented, one based on 2011 data and the other on data collected in 2013.  In both years, state education officials were surveyed about their Common Core implementation efforts.  Because forty-six states originally signed on to the CCSS-ELA—and with at least forty still on track for full implementation by 2016—little variability exists among the states in terms of standards policy.  Of course, the four states that never adopted CCSS-ELA can serve as a small control group.  But variation is also found in how the states are implementing CCSS.  Some states are pursuing an array of activities and aiming for full implementation earlier rather than later.  Others have a narrow, targeted implementation strategy and are proceeding more slowly. 

The analysis investigates whether CCSS-ELA implementation is related to 2009-2013 gains on the fourth grade NAEP reading test.  The analysis cannot verify causal relationships between the two variables, only correlations.  States that have aggressively implemented CCSS-ELA (referred to as “strong” implementers in the study) evidence a one to one and one-half point larger gain on the NAEP scale compared to non-adopters of the standards.  This association is similar in magnitude to an advantage found in a study of eighth grade math achievement in last year’s BCR.  Although positive, these effects are quite small.  When the 2015 NAEP results are released this winter, it will be important for the fate of the Common Core project to see if strong implementers of the CCSS-ELA can maintain their momentum.

Part three is on student engagement.  PISA tests fifteen-year-olds on three subjects—reading, math, and science—every three years.  It also collects a wealth of background information from students, including their attitudes toward school and learning.  When the 2012 PISA results were released, PISA analysts published an accompanying volume, Ready to Learn: Students’ Engagement, Drive, and Self-Beliefs, exploring topics related to student engagement.

Part three provides secondary analysis of several dimensions of engagement found in the PISA report.  Intrinsic motivation, the internal rewards that encourage students to learn, is an important component of student engagement.  National scores on PISA’s index of intrinsic motivation to learn mathematics are compared to national PISA math scores.  Surprisingly, the relationship is negative.  Countries with highly motivated kids tend to score lower on the math test; conversely, higher-scoring nations tend to have less-motivated kids. 

The same is true for responses to the statements, “I do mathematics because I enjoy it,” and “I look forward to my mathematics lessons.”  Countries with students who say that they enjoy math or look forward to their math lessons tend to score lower on the PISA math test compared to countries where students respond negatively to the statements.  These counterintuitive finding may be influenced by how terms such as “enjoy” and “looking forward” are interpreted in different cultures.  Within-country analyses address that problem.  The correlation coefficients for within-country, student-level associations of achievement and other components of engagement run in the anticipated direction—they are positive.  But they are also modest in size, with correlation coefficients of 0.20 or less. 

Policymakers are interested in questions requiring analysis of aggregated data—at the national level, that means between-country data.  When countries increase their students’ intrinsic motivation to learn math, is there a concomitant increase in PISA math scores?  Data from 2003 to 2012 are examined.  Seventeen countries managed to increase student motivation, but their PISA math scores fell an average of 3.7 scale score points.  Fourteen countries showed no change on the index of intrinsic motivation—and their PISA scores also evidenced little change.  Eight countries witnessed a decline in intrinsic motivation.  Inexplicably, their PISA math scores increased by an average of 10.3 scale score points.  Motivation down, achievement up.

Correlation is not causation.  Moreover, the absence of a positive correlation—or in this case, the presence of a negative correlation—is not refutation of a possible positive relationship.  The lesson here is not that policymakers should adopt the most effective way of stamping out student motivation.  The lesson is that the level of analysis matters when analyzing achievement data.  Policy reports must be read warily—especially those freely offering policy recommendations.  Beware of analyses that exclusively rely on within- or between-country test data without making any attempt to reconcile discrepancies at other levels of analysis.  Those analysts could be cherry-picking the data.  Also, consumers of education research should grant more credence to approaches modeling change over time (as in difference in difference models) than to cross-sectional analyses that only explore statistical relationships at a single point in time. 

Downloads

• Download the report