Ninety-three years ago yesterday, the Boston police force went on strike, leaving the city unprotected while the state scrambled to find replacements. Governor Calvin Coolidge’s declaration of support for the city—he said that “There is no right to strike against the public safety, anywhere, anytime”—established his national reputation that ultimately led to the presidency.

Public outrage at labor actions that compromise public safety has historically been a bipartisan affair.  Coolidge was a Republican but his actions earned the respect of Democratic President Woodrow Wilson, who hailed his re-election as Massachusetts governor as “a victory for law and order.” Nearly 20 years later, President Franklin Roosevelt shared his view that a strike by public employees of any sort is “unthinkable and intolerable.”

The impacts of the Chicago teacher strike that began today may not be as immediately obvious as the looting and vandalism that descended on Boston in 1919, but they are just as serious. Research from a large, urban school district found that teacher absenteeism has a negative impact on student learning in math.

But a strike doesn't leave students with substitute teachers—it leaves them without any school at all. Research on summer learning loss shows that being out of school has a disproportionate effect on low-income students. One recent study found that “while all students lose some ground in mathematics over the summer, low-income students lose more ground in reading, while their higher-income peers may even gain.” In other words, the consequence of being out of school is to increase the already unacceptably large achievement gap between low-income students and their affluent peers.

The American labor movement has made important contributions in areas ranging from workplace safety to child labor to employment discrimination. There are good reasons to believe that the public ought to accept higher coal prices resulting from a strike to protect the lives of miners. But the public should not tolerate damage to the education of disadvantaged students resulting from a strike over disagreements about teachers’ salaries, benefits, job security, and method of evaluation.

The Chicago Teachers Union’s differences with the city over how the public schools ought to be run may well be legitimate. But those battles should be fought in the court of public opinion and ultimately at the ballot box, not through strikes that come largely at the expense of poor children.

Donald Trump has made waves during this year’s election cycle by taking a hard line on illegal immigration. This, however, builds on years of heated debate among policymakers. It is also an enduring hot-button issue in Arizona, which has passed several immigration laws over the years.  In 2010, the passage of SB 1070 brought national attention to this debate.  Deemed the strictest immigration law to date, SB 1070 sought to achieve “attrition [of illegal immigrants] through enforcement” by requiring law enforcement to detain any person whom they believed to be residing in the country illegally. Although SB 1070’s effects on individuals and families have been well documented, little is known about its impact on students and schools. To this end, we sought to estimate the relationship between the passage of SB 1070 and school-level student achievement.

We anticipated that anti-immigration policies would primarily affect children from the families of undocumented immigrants. Such effects could be observed in different ways. For instance, the emotional and psychological distress of these children could result in a decline in average test scores at the school-level. On the other hand, students might have left the country or the state under the threat of being deported in which case school-level test scores would rise (since these students often perform below their peers). To this end, we considered three scenarios: 

1. Immigrant children remain in the state but experience higher levels of stress.  As a result, average school-level test scores will drop while Hispanic enrollment remains the same.
2. Children of undocumented immigrants leave the state, which results in a drop in Hispanic enrollment accompanied by an increase in school-level test scores.
3. Or, the first two scenarios occur simultaneously and we do not observe any change in test scores since the two effects would cancel each other, but note a slight decrease in Hispanic enrollment.

In order to see which of these hypothetical scenarios is supported by the data, we first estimated the relationship between the passage of SB 1070 and average school-level reading test scores. We then attempted to unpack the mechanism through which such an effect might have taken place. To this end, we used publicly available data on school-level achievement and enrollment collected by the Arizona Department of Education (ADE). Given the targeted nature of the policy and the demographics of immigrants in Arizona, the majority of whom are of Hispanic or Mexican descent, we focused on schools that traditionally enroll large proportions of Hispanic students. We identified schools with high (more than 75 percent) shares of Hispanic students as those whose average achievement and student composition are most likely to be affected by immigration reform. We contrasted changes in school-level achievement and enrollment in those schools with schools that enroll less than 25 percent Hispanic students, as these schools are less likely to experience any changes as a result of tightening immigration laws.

Figures 1 and 2 show trends in the average percentage of students passing the state reading test and average Hispanic enrollment at these schools between 2006-2007 and 2011-2012.           

Figure 1. Average Percent of Students Passing AIMS Reading

Figure 2. Average Hispanic Student Enrollment

Clearly, the rate of growth in school-level reading scores was much higher for high Hispanic schools after the passage of SB 1070 in 2010 (Figure 1). At the same time, there was a significant decrease in Hispanic enrollment in these schools (Figure 2). Thus, it appears the second scenario is likely driving the patterns we observe.

The data also suggest that the trends for high Hispanic and low Hispanic schools started diverging before the passage of SB 1070 - after the 2007-2008 school year.  This happens to be the year that Arizona passed an even more restrictive, though less controversial, immigration law – the Legal Arizona Workers Act (LAWA). LAWA required Arizona business owners to verify the legal status of their employees using E-Verify, an online tool managed by the federal government. Although LAWA used a different mechanism, similar to SB 1070 it sought to achieve the attrition of undocumented immigrants from the state. 

We then would anticipate both laws to have similar effects on school-level achievement and Hispanic enrollment. Indeed, we estimated that LAWA likely led to an average increase of roughly 4 percent of students passing the state reading test at high Hispanic schools. This was accompanied by an average loss of 38 Hispanic students per school. Because the passage of SB 1070 was preceded by the passage LAWA as well as a language policy that would have affected treatment schools, disentangling the effects of these two policies is not straightforward. However, based on our analysis, we estimate that SB 1070 is associated with an average increase of between 1.5 percent and 4.5 percent of students passing the state reading test at the school-level accompanied by an average loss of between 14 and 40 Hispanic students. 

Despite the fact that we cannot pin down the exact magnitude of SB 1070’s effect on school-level achievement, our analysis shows that when Arizona passed restrictive immigration laws in 2008 and 2010, it looked as if the state’s lowest performing schools were improving rapidly. This, however, likely had more to do with the changing composition of schools as an indirect though anticipated effect of immigration policies than with policies aimed at improving student achievement. 

Despite this, the Arizona Department of Education took credit for these gains. Similarly, Arizona was recently recognized as one of the nation’s leaders in growth on the National Assessment of Educational Progress (NAEP) over the last ten years. Although wrongly attributing these gains may seem harmless at first glance, it is important to remember that Arizona is viewed by many as a model for controversial education reforms like school choice and high-stakes accountability. It is easy to imagine how policymakers might look at increasing test scores in Arizona and wrongly attribute them to these kinds of reforms. That’s not to say that these policies don’t have merit. However, if other states adopt education policy reforms under the assumption that they worked in Arizona, then they might find that these policies fail to deliver.

Over the course of my career, I have watched Egypt’s transformation from an authoritarian state to a revolutionary one and back again. But last month’s murder of Italian graduate student Giulio Regeni (with some pointing fingers at Egyptian security forces) illuminates that today’s Egypt is even less safe, less free, and less tolerant than it was under Hosni Mubarak—an impressive feat. The disintegration in Egypt’s security environment could haunt the country and its leaders, as it will only push international travelers and researchers further from its shores.

Fear and loathing in Cairo

In 2010, shortly before the 2011 revolution, I lived in Cairo interviewing civil society activists and government officials on the ability of NGOs to challenge the Mubarak regime. I returned a few months after the uprising to a very different Egypt. 

In some ways, the environment had become more hospitable for discussing democracy and seeking honest assessments of the regime. Egyptians were still brimming with hope that the revolution would bring them the Egypt they had fought for and expressed overwhelming pride in their accomplishments in Tahrir Square. They were forthcoming with critiques of the former regime and inspired to begin by participating in politics, overturning the draconian NGO law, and founding innovative organizations to help usher in an era of democracy in Egypt. 

But in other ways, the conditions in Egypt had become dangerous. The security situation was precarious, as a post-revolutionary crime wave and general lawlessness keeping Egyptians at home and tourists away. For the first time, I hired a driver to ensure my safety. I was afraid to walk alone at night, ride the metro, or hang out in the same cafes I had frequented during my trips to Mubarak’s Egypt. 

Ironically, I was also far more cognizant of the security services in this new “freer” Egypt than I had been in past visits. The vestiges of Mubarak’s security apparatus remained, but they were operating under different and far more arbitrary and kinetic rules, making it challenging to identify—and avoid—redlines. I heard stories of NGO raids that were no different from the Mubarak era and possibly more punitive, with pro-regime security forces hoping to exact revenge on the activists who unseated their leader. Frustration and anger towards foreigners—governments, donor organizations, and even researchers—had emerged among civil society actors, who believed that Washington, in particular, was meddling in a process that was home-grown. Civil society activists whose NGOs had been fully reliant on international funding vowed to no longer take USAID money, for example. And although I was a full-time doctoral student with no ties to the U.S. government, some of those whom I interviewed distrusted my motives and saw me and other foreign scholars as inextricably linked to our governments. 

I heard stories of NGO raids that were no different from the Mubarak era and possibly more punitive, with pro-regime security forces hoping to exact revenge on the activists who unseated their leader.

Pining for yesterday

But the atmosphere in the immediate aftermath of the revolution was nothing like that of today’s Egypt. The murder of Italian national and Cambridge University student Giulio Regeni, who was last seen alive in Cairo on January 25 (the five-year anniversary of the Egyptian revolution), has sparked outrage around the world. The Italian ambassador to Egypt has said that Regeni’s autopsy revealed “clear, unequivocal marks of violence, beating and torture.” Egyptian security officials have admitted taking Regeni into custody. And while the Ministry of Interior subsequently denied such reports, Egyptian State Prosecutor Ahmed Nagi would not rule out police involvement in his murder. 

Despite the similarity of Regeni’s case to “widespread” reports of torture and forced disappearances by the Egyptian security services, we do not know for sure who is responsible for Regeni’s murder. Scholars across the globe have called on the Egyptian government to conduct a thorough and honest investigation. But regardless of the outcome, the very perception that students are no longer safe in Cairo has caused great harm to Egypt. The very fact that scholars, some of whom have studied Egyptian politics for decades, believe that the Egyptian Security Services could have committed this crime speaks volumes about the state of repression there. 

The very fact that scholars, some of whom have studied Egyptian politics for decades, believe that the Egyptian Security Services could have committed this crime speaks volumes about the state of repression there.

Not all press is good press

Regeni’s violent and tragic death and the Egyptian government’s response have far-reaching implications for Egypt. First, the sheer volume of attention on the Regeni case has caused harm to Egypt’s already decaying reputation. Abdel-Fattah el-Sissi’s regime is engaged in a crackdown on freedom of expression surpassing that of Mubarak. As the leadership of the Middle East Studies Association (MESA)—the most prominent academic organization on the Middle East—rightly note in an open letter to the Egyptian regime, Regeni’s case is not an exception, but rather the latest example of an increasingly vicious attack on freedom of expression in Egypt. As the MESA letter states, “human rights reports suggest that academics, journalists and legal professionals are in greater danger of falling victim to arbitrary state repression today than at any time since the establishment of the republic in 1953.” This was particularly true in the weeks leading to the anniversary of the Egyptian revolution, as the state sought to quiet any public discontent before it started. 

But unlike the hundreds of cases of forced disappearances and systematic torture of Egyptians in custody, the sheer brutality of Regeni’s murder and his status as a young, Western scholar, have made it difficult for Western states to ignore and have shed much needed light on the escalating attack on the rights and freedoms of both foreigners and Egyptians. Most clearly, Egypt’s relationship with an important political and economic partner, Italy, is tarnished. And the suspected state involvement in torture is now an issue that Western interlocutors must raise with their Egyptian counterparts, obliging the Egyptian government to address, or at least find a way to dance around, the issue.

the suspected state involvement in torture is now an issue that Western interlocutors must raise with their Egyptian counterparts, obliging the Egyptian government to address, or at least find a way to dance around, the issue.

Egypt’s foreign minister Sameh Shoukry happened to be in Washington when the circumstances of Regeni’s death was made public. His tone-deaf public responses were telling. He not only flatly denied that Egypt is engaged in a widespread crackdown on freedom of expression, he even compared Egypt’s critics, including internationally respected human rights organizations, to Nazi propaganda minister Joseph Goebbels. Shoukry’s response, so undiplomatic and divorced from reality, is unlikely to quiet Egypt’s critics. Rather, it will keep Regeni’s death (and the issue of security service abuses) in the international press even longer. 

This sort of public attention is something that the Mubarak regime would have taken seriously. Mubarak regularly acknowledged and attempted to diffuse, albeit often ineffectively, accusations of human rights abuses under his watch, often justifying repression in the name of security. But the Sissi regime’s response has been far less strategic, and this has potentially dangerous consequences. By ignoring the festering wound the regime has created for itself by torturing, jailing, disappearing, and killing those who speak out against it, the infection will spread, not disappear. 

Fading from view?

Another outcome of Regeni’s murder is that universities will steer their students away from studying in Cairo, traditionally one of the most popular destinations for American students of Arabic, and may discourage faculty from visiting as well. For the American University in Cairo (AUC), an institution known for high standards and academic freedom, the loss of foreign students and researchers could pose serious financial problems. 

That may not concern the regime, but it is not only AUC that will suffer from a deterioration of foreign contacts. Even prior to Regeni’s murder, some Western scholars believed it was too difficult and risky to conduct serious research in Egypt, and this trend will increase. Other scholars may still study Egypt, but will do so from a distance, rather than risking their lives on the ground there. 

This sort of public attention is something that the Mubarak regime would have taken seriously.

A dramatic decline in international academic contacts should worry the Egyptian government. This will greatly harm the world’s understanding of what is happening in a country that has proven time and again its importance to the region’s economy and political trajectory. Egyptian students and scholars will suffer as well, missing out on the important information and cultural education that comes from cross-border academic exchange. 

Not to mention that Egypt is in the midst of an economic crisis. Regeni’s death will likely keep Western tourists away, harming the tourism industry, which makes up over 10 percent of Egypt’s GDP, and which has failed to recover from dramatic declines during the revolution. 

A continued crackdown on freedom of expression and an increasingly dangerous environment for American and European visitors also has implications for Egypt’s diplomatic relationships. While Egypt’s history, size, and political role in the region will keep it on Washington’s radar, it risks joining the ranks of Somalia or Yemen or Libya—states with a limited (if any) diplomatic presence, and even more limited economic assistance package. The robust U.S.-Egyptian relationship—including several high-profile visits each year and a $1.5 billion aid package--is based, in part, on Egypt’s portrayal of itself as the “leader” of the Arab world and a country on the path toward democracy. If the Sissi regime continues to jail, torture, and murder its critics, including Western scholars, it will make it very challenging for the United States to continue this level of support. 

As Secretary of State John Kerry said last month following his meeting with Shoukry, Egypt is “going through a political transition. We very much respect the important role that Egypt plays traditionally within the region--a leader of the Arab world in no uncertain terms. And so the success of the transformation that is currently being worked on is critical for the United States and obviously for the region and for Egypt.”

The Egyptian government is underestimating the negative repercussions of Regeni’s death. Scholars like Regeni and me study Egypt and visit Egypt are driven by Egypt’s incredible history and because of its important cultural, economic, and political role in the modern Middle East. On my very first day in Cairo back in 2002, a kind Egyptian man took my hand and helped me cross the street amidst the infamously crazy Cairo traffic. When we safely made it across and the look of trepidation fell from my face, he told me to repeat after him, “Ana b’hib Masr” (I love Egypt). It was the first colloquial Egyptian phrase I learned and one I have repeated many times. But sadly, it is not one that I or other international researchers will likely be able to repeat in Egypt any time soon.

As every good teacher knows, education is not just about academics. It is about broadening horizons and discovering passions. (The root of education is the Latin e ducere, meaning “to draw out.”) From this perspective, extra-curricular activities count for a great deal. But as Robert Putnam highlights in his book Our Kids, there are growing class gaps in the availability of music, sports, and other non-classroom activities.

Fewer field trips?

Schools under pressure may also cut back on field trips outside the school walls to parks, zoos, theaters, or museums. In the 2008-09 school year, 9 percent of school administrators reported eliminating field trips, according to the annual surveys by the American Association of School Administrators (AASA). That figure rose through the recession:

Just 12 percent of the administrators surveyed about 2015-16 said they had brought back their field trips to pre-recession levels. Museums around the country report hosting fewer students, from Los Angeles and Sarasota, to Minneapolis, and Columbia, Missouri. None of this is definitive proof of a decline in field trips, since we are relying on a single survey question. But it suggests a downward trend in recent years.

Museums help with science tests

If some children are missing out on field trips, does it matter? They may be nice treats, but do they have any real impact, especially when they take time away from traditional learning? There is some evidence that they do.

Middle school children with the chance to go on a field trip score higher on science tests, according to a 2015 study by Emilyn Ruble Whitesell.

She studied New York City middle schools with teachers in Urban Advantage, a program that gives science teachers additional training and resources—as well as vouchers for visiting museums. In some schools, the Urban Advantage teachers used the field trip vouchers more than others. Whitesell exploits this difference in her study, and finds that attending a school with at least 0.25 trips per student increased 8^th grade scores by 0.026 standard deviations (SD). The odds of a student passing the exam improved by 1.2 percentage points. There were bigger effects for poor students, who saw a 0.043 SD improvement in test scores, and 1.9 percentage point increase in exam pass rates.

Art broadens young minds

Students visiting an art museum show statistically significant increases in critical thinking ability and more open-minded attitudes, according to a randomized evaluation of student visits to the Crystal Bridges Museum in northwest Arkansas. One example: those who visited the museum more often agreed with statements like: “I appreciate hearing views different from my own” and “I think people can have different opinions about the same thing.” The effects are modest. But the intervention (a single day at the museum) is, too. Again, there were larger effects for poor students:

All this needs to be put in perspective. In comparison with the challenge of closing academic gaps and quality teaching, field trips are small beer. But schools create citizens as well as undergraduates and employees. It matters, then, if we have allowed field trips to become a casualty of the great recession.

On July 5, the results from the first year of the world’s first development impact bond (DIB) for education in Rajasthan, India, were announced. The Center for Universal Education hosted a webinar in which three stakeholders in the DIB shared their perspective on the performance of the intervention, their learnings about the DIB process, and their thoughts for the future of DIBs and other results-based financing mechanisms.

Approximately 3 million girls ages 6 to 13 were out of school in India according to most recent data, 350,000 of which are in the state of Rajasthan. Child marriage is also a large issue in the state; no state-specific data exists, but nationwide 47 percent of girls ages 20 to 24 are married before age 18. According to Educate Girls, a non-governmental organization based in Rajasthan, girls’ exclusion is primarily a result of paternalistic societal mindsets and traditions. Given the evidence linking education and future life outcomes for girls, this data is greatly concerning.

The DIB finances a portion of the services provided by Educate Girls, which has been working to improve enrollment, retention, and learning outcomes for girls (and boys) in Rajasthan since 2007. The organization trains a team of community volunteers ages 18 to 30 to make door-to-door visits encouraging families to enroll their girls in school and to deliver curriculum enhancement in public school classrooms. Their volunteers are present in over 8,000 villages and 12,500 schools in Rajasthan. The DIB was launched in March of 2015 to finance services in 166 schools, which represents 5 percent of Educate Girls’ annual budget. The DIB is intended to be a “proof of concept” of the mechanism using this relatively small selection of beneficiaries.

The investor in the DIB is UBS Optimus Foundation, who has provided $238,000 in working capital to fund the service delivery. ID Insight, a non-profit evaluation firm, will evaluate the improvement in learning of girls and boys in the treatment schools in comparison to a control group and will validate the number of out of school girls enrolled. The Children’s Investment Fund Foundation serves as the outcome funder, and has agreed to pay UBS Optimus Foundation 43.16 Swiss francs ($44.37) for each unit of improved learning and 910.14 francs ($935.64) for every percentage point increase in the enrollment of girls out of school. Instiglio, a non-profit impact bond and results-based financing intermediary organization, provided technical assistance to all parties during the design of the DIB and currently provides performance management assistance to Educate Girls on behalf of UBS Optimus Foundation. 

The outcomes will be calculated in 2018, at the end of three years; however, preliminary results for the year since the launch of the DIB (representing multiple months of door-to-door visits and seven weeks of interventions in the classroom) were released last week. The payments for the DIB were structured such that the investor, UBS Optimus Foundation, would earn a 10 percent internal rate of return (IRR) on their investment at target outcome levels, which were based on Educate Girls’ past performance data. The table below presents the metrics, target outcome level, year-one result, and the progress toward the target. 

Table 1: Educate Girls DIB Results from first year of services

The DIB was challenging to implement and required DIB stakeholders to be resourceful.

First, the reliability of government data was a challenge, which necessitated flexibility in the identification of the target population and metrics. Second, given the number of stakeholders engaged and the novelty of this approach, the transaction costs were higher than they would have been for a traditional grant. This meant that strong and regular communication was crucial to the survival of the project.

The role of the outcome funder and investor were significantly different versus a grant.

The outcome funder spent more resources on defining outcomes, but spent fewer resources on managing grant activities. The investor utilized risk management and monitoring strategies informed by the activities in their commercial banking branch, which they have not used for other grants.

The DIB has changed the way the service provider operates.

In the video below, Safeena Husain from Educate Girls’ highlights the ways in which financing a portion of their program through a DIB differs from financing the program through grants. Safeena describes that in a grant, performance data is reported up to donors, but rarely makes it back down to frontline workers. The DIB has helped them to develop mobile dashboards that ensure performance data is reaching the front line and helping to identify barriers to outcomes as early as possible.

Based on the learnings from the implementation of the first DIB for education, this tool can be used to improve the value for money for the outcome funder and strengthen the performance management of a service provider. As the panelists discussed in the webinar, DIBs and other outcome-based financing mechanisms can help differentiate between organizations that are adept at fundraising and those that excel at delivering outcomes. However, service providers must be sufficiently prepared for rigorous outcome measurement if they plan to participate in a DIB; otherwise the high-stakes environment might backfire. In our research, we have closely examined the design constraints for impact bonds in the early childhood sector.

There are countless lessons to be learned from the stakeholder’s experience in the first DIB for education. We applaud the stakeholders for being transparent about the outcomes and true challenges associated with this mechanism. This transparency will be absolutely critical to ensure that DIBs are implemented and utilized appropriately moving forward.

Editor's Note: The introduction to the 2015 Brown Center Report on American Education appears below. Use the Table of Contents to navigate through the report online, or download a PDF of the full report.

TABLE OF CONTENTS

Part I: Girls, Boys, and Reading

Part II: Measuring Effects of the Common Core

Part III: Student Engagement

INTRODUCTION

The 2015 Brown Center Report (BCR) represents the 14^th edition of the series since the first issue was published in 2000.  It includes three studies.  Like all previous BCRs, the studies explore independent topics but share two characteristics: they are empirical and based on the best evidence available.  The studies in this edition are on the gender gap in reading, the impact of the Common Core State Standards -- English Language Arts on reading achievement, and student engagement.

Part one examines the gender gap in reading.  Girls outscore boys on practically every reading test given to a large population.  And they have for a long time.  A 1942 Iowa study found girls performing better than boys on tests of reading comprehension, vocabulary, and basic language skills.  Girls have outscored boys on every reading test ever given by the National Assessment of Educational Progress (NAEP)—the first long term trend test was administered in 1971—at ages nine, 13, and 17.  The gap is not confined to the U.S.  Reading tests administered as part of the Progress in International Reading Literacy Study (PIRLS) and the Program for International Student Assessment (PISA) reveal that the gender gap is a worldwide phenomenon.  In more than sixty countries participating in the two assessments, girls are better readers than boys. 

Perhaps the most surprising finding is that Finland, celebrated for its extraordinary performance on PISA for over a decade, can take pride in its high standing on the PISA reading test solely because of the performance of that nation’s young women.  With its 62 point gap, Finland has the largest gender gap of any PISA participant, with girls scoring 556 and boys scoring 494 points (the OECD average is 496, with a standard deviation of 94).   If Finland were only a nation of young men, its PISA ranking would be mediocre.

Part two is about reading achievement, too. More specifically, it’s about reading and the English Language Arts standards of the Common Core (CCSS-ELA).  It’s also about an important decision that policy analysts must make when evaluating public policies—the determination of when a policy begins. How can CCSS be properly evaluated? 

Two different indexes of CCSS-ELA implementation are presented, one based on 2011 data and the other on data collected in 2013.  In both years, state education officials were surveyed about their Common Core implementation efforts.  Because forty-six states originally signed on to the CCSS-ELA—and with at least forty still on track for full implementation by 2016—little variability exists among the states in terms of standards policy.  Of course, the four states that never adopted CCSS-ELA can serve as a small control group.  But variation is also found in how the states are implementing CCSS.  Some states are pursuing an array of activities and aiming for full implementation earlier rather than later.  Others have a narrow, targeted implementation strategy and are proceeding more slowly. 

The analysis investigates whether CCSS-ELA implementation is related to 2009-2013 gains on the fourth grade NAEP reading test.  The analysis cannot verify causal relationships between the two variables, only correlations.  States that have aggressively implemented CCSS-ELA (referred to as “strong” implementers in the study) evidence a one to one and one-half point larger gain on the NAEP scale compared to non-adopters of the standards.  This association is similar in magnitude to an advantage found in a study of eighth grade math achievement in last year’s BCR.  Although positive, these effects are quite small.  When the 2015 NAEP results are released this winter, it will be important for the fate of the Common Core project to see if strong implementers of the CCSS-ELA can maintain their momentum.

Part three is on student engagement.  PISA tests fifteen-year-olds on three subjects—reading, math, and science—every three years.  It also collects a wealth of background information from students, including their attitudes toward school and learning.  When the 2012 PISA results were released, PISA analysts published an accompanying volume, Ready to Learn: Students’ Engagement, Drive, and Self-Beliefs, exploring topics related to student engagement.

Part three provides secondary analysis of several dimensions of engagement found in the PISA report.  Intrinsic motivation, the internal rewards that encourage students to learn, is an important component of student engagement.  National scores on PISA’s index of intrinsic motivation to learn mathematics are compared to national PISA math scores.  Surprisingly, the relationship is negative.  Countries with highly motivated kids tend to score lower on the math test; conversely, higher-scoring nations tend to have less-motivated kids. 

The same is true for responses to the statements, “I do mathematics because I enjoy it,” and “I look forward to my mathematics lessons.”  Countries with students who say that they enjoy math or look forward to their math lessons tend to score lower on the PISA math test compared to countries where students respond negatively to the statements.  These counterintuitive finding may be influenced by how terms such as “enjoy” and “looking forward” are interpreted in different cultures.  Within-country analyses address that problem.  The correlation coefficients for within-country, student-level associations of achievement and other components of engagement run in the anticipated direction—they are positive.  But they are also modest in size, with correlation coefficients of 0.20 or less. 

Policymakers are interested in questions requiring analysis of aggregated data—at the national level, that means between-country data.  When countries increase their students’ intrinsic motivation to learn math, is there a concomitant increase in PISA math scores?  Data from 2003 to 2012 are examined.  Seventeen countries managed to increase student motivation, but their PISA math scores fell an average of 3.7 scale score points.  Fourteen countries showed no change on the index of intrinsic motivation—and their PISA scores also evidenced little change.  Eight countries witnessed a decline in intrinsic motivation.  Inexplicably, their PISA math scores increased by an average of 10.3 scale score points.  Motivation down, achievement up.

Correlation is not causation.  Moreover, the absence of a positive correlation—or in this case, the presence of a negative correlation—is not refutation of a possible positive relationship.  The lesson here is not that policymakers should adopt the most effective way of stamping out student motivation.  The lesson is that the level of analysis matters when analyzing achievement data.  Policy reports must be read warily—especially those freely offering policy recommendations.  Beware of analyses that exclusively rely on within- or between-country test data without making any attempt to reconcile discrepancies at other levels of analysis.  Those analysts could be cherry-picking the data.  Also, consumers of education research should grant more credence to approaches modeling change over time (as in difference in difference models) than to cross-sectional analyses that only explore statistical relationships at a single point in time. 

Downloads

• Download the report

Part III of the 2015 Brown Center Report on American Education

Student engagement refers to the intensity with which students apply themselves to learning in school.  Traits such as motivation, enjoyment, and curiosity—characteristics that have interested researchers for a long time—have been joined recently by new terms such as, “grit,” which now approaches cliché status.  International assessments collect data from students on characteristics related to engagement.  This study looks at data from the Program for International Student Assessment (PISA), an international test given to fifteen-year-olds.  In the U.S., most PISA students are in the fall of their sophomore year.  The high school years are a time when many observers worry that students lose interest in school.

Compared to their peers around the world, how do U.S. students appear on measures of engagement?  Are national indicators of engagement related to achievement?  This analysis concludes that American students are about average in terms of engagement.  Data reveal that several countries noted for their superior ranking on PISA—e.g., Korea, Japan, Finland, Poland, and the Netherlands—score below the U.S. on measures of student engagement.  Thus, the relationship of achievement to student engagement is not clear cut, with some evidence pointing toward a weak positive relationship and other evidence indicating a modest negative relationship. 

Education studies differ in units of analysis.  Some studies report data on individuals, with each student serving as an observation.  Studies of new reading or math programs, for example, usually report an average gain score or effect size representing the impact of the program on the average student.  Others studies report aggregated data, in which test scores or other measurements are averaged to yield a group score. Test scores of schools, districts, states, or countries are constructed like that.  These scores represent the performance of groups, with each group serving as a single observation, but they are really just data from individuals that have been aggregated to the group level.

Aggregated units are particularly useful for policy analysts.  Analysts are interested in how Fairfax County or the state of Virginia or the United States is doing.  Governmental bodies govern those jurisdictions and policymakers craft policy for all of the citizens within the political jurisdiction—not for an individual.  

The analytical unit is especially important when investigating topics like student engagement and their relationships with achievement.  Those relationships are inherently individual, focusing on the interaction of psychological characteristics.  They are also prone to reverse causality, meaning that the direction of cause and effect cannot readily be determined.  Consider self-esteem and academic achievement.  Determining which one is cause and which is effect has been debated for decades.  Students who are good readers enjoy books, feel pretty good about their reading abilities, and spend more time reading than other kids.  The possibility of reverse causality is one reason that beginning statistics students learn an important rule:  correlation is not causation.

Starting with the first international assessments in the 1960s, a curious pattern has emerged. Data on students’ attitudes toward studying school subjects, when examined on a national level, often exhibit the opposite relationship with achievement than one would expect.  The 2006 Brown Center Report (BCR) investigated the phenomenon in a study of “the happiness factor” in learning.^[i]  Test scores of fourth graders in 25 countries and eighth graders in 46 countries were analyzed.  Students in countries with low math scores were more likely to report that they enjoyed math than students in high-scoring countries.  Correlation coefficients for the association of enjoyment and achievement were -0.67 at fourth grade and -0.75 at eighth grade. 

Confidence in math performance was also inversely related to achievement.  Correlation coefficients for national achievement and the percentage of students responding affirmatively to the statement, “I usually do well in mathematics,” were -0.58 among fourth graders and -0.64 among eighth graders.  Nations with the most confident math students tend to perform poorly on math tests; nations with the least confident students do quite well.   

That is odd.  What’s going on?  A comparison of Singapore and the U.S. helps unravel the puzzle.  The data in figure 3-1 are for eighth graders on the 2003 Trends in Mathematics and Science Study (TIMSS).  U.S. students were very confident—84% either agreed a lot or a little (39% + 45%) with the statement that they usually do well in mathematics.  In Singapore, the figure was 64% (46% + 18%).  With a score of 605, however, Singaporean students registered about one full standard deviation (80 points) higher on the TIMSS math test compared to the U.S. score of 504. 

When within-country data are examined, the relationship exists in the expected direction.  In Singapore, highly confident students score 642, approximately 100 points above the least-confident students (551).  In the U.S., the gap between the most- and least-confident students was also about 100 points—but at a much lower level on the TIMSS scale, at 541 and 448.  Note that the least-confident Singaporean eighth grader still outscores the most-confident American, 551 to 541.

The lesson is that the unit of analysis must be considered when examining data on students’ psychological characteristics and their relationship to achievement.  If presented with country-level associations, one should wonder what the within-country associations are.  And vice versa.  Let’s keep that caution in mind as we now turn to data on fifteen-year-olds’ intrinsic motivation and how nations scored on the 2012 PISA.

PISA’s index of intrinsic motivation to learn mathematics comprises responses to four items on the student questionnaire:  1) I enjoy reading about mathematics; 2) I look forward to my mathematics lessons; 3) I do mathematics because I enjoy it; and 4) I am interested in the things I learn in mathematics.  Figure 3-2 shows the percentage of students in OECD countries—thirty of the most economically developed nations in the world—responding that they agree or strongly agree with the statements.  A little less than one-third (30.6%) of students responded favorably to reading about math, 35.5% responded favorably to looking forward to math lessons, 38.2% reported doing math because they enjoy it, and 52.9% said they were interested in the things they learn in math.  A ballpark estimate, then, is that one-third to one-half of students respond affirmatively to the individual components of PISA’s intrinsic motivation index.

Table 3-1 presents national scores on the 2012 index of intrinsic motivation to learn mathematics.  The index is scaled with an average of 0.00 and a standard deviation of 1.00.  Student index scores are averaged to produce a national score.  The scores of 39 nations are reported—29 OECD countries and 10 partner countries.^[ii]  Indonesia appears to have the most intrinsically motivated students in the world (0.80), followed by Thailand (0.77), Mexico (0.67), and Tunisia (0.59).  It is striking that developing countries top the list.  Universal education at the elementary level is only a recent reality in these countries, and they are still struggling to deliver universally accessible high schools, especially in rural areas and especially to girls.  The students who sat for PISA may be an unusually motivated group.  They also may be deeply appreciative of having an opportunity that their parents never had.

The U.S. scores about average (0.08) on the index, statistically about the same as New Zealand, Australia, Ireland, and Canada.  The bottom of the table is extremely interesting.  Among the countries with the least intrinsically motivated kids are some PISA high flyers.  Austria has the least motivated students (-0.35), but that is not statistically significantly different from the score for the Netherlands (-0.33).  What’s surprising is that Korea (-0.20), Finland (-0.22), Japan (-0.23), and Belgium (-0.24) score at the bottom of the intrinsic motivation index even though they historically do quite well on the PISA math test.

Let’s now dig a little deeper into the intrinsic motivation index.  Two components of the index are how students respond to “I do mathematics because I enjoy it” and “I look forward to my mathematics lessons.”  These sentiments are directly related to schooling.  Whether students enjoy math or look forward to math lessons is surely influenced by factors such as teachers and curriculum.  Table 3-2 rank orders PISA countries by the percentage of students who “agree” or “strongly agree” with the questionnaire prompts.  The nations’ 2012 PISA math scores are also tabled.  Indonesia scores at the top of both rankings, with 78.3% enjoying math and 72.3% looking forward to studying the subject.  However, Indonesia’s PISA math score of 375 is more than one full standard deviation below the international mean of 494 (standard deviation of 92).  The tops of the tables are primarily dominated by low-performing countries, but not exclusively so.  Denmark is an average-performing nation that has high rankings on both sentiments.  Liechtenstein, Hong Kong-China, and Switzerland do well on the PISA math test and appear to have contented, positively-oriented students.

Several nations of interest are shaded.  The bar across the middle of the tables, encompassing Australia and Germany, demarcates the median of the two lists, with 19 countries above and 19 below that position.  The United States registers above the median on looking forward to math lessons (45.4%) and a bit below the median on enjoyment (36.6%).  A similar proportion of students in Poland—a country recently celebrated in popular media and in Amanda Ripley’s book, The Smartest Kids in the World,^[iii] for making great strides on PISA tests—enjoy math (36.1%), but only 21.3% of Polish kids look forward to their math lessons, very near the bottom of the list, anchored by Netherlands at 19.8%. 

Korea also appears in Ripley’s book.  It scores poorly on both items.  Only 30.7% of Korean students enjoy math, and less than that, 21.8%, look forward to studying the subject.  Korean education is depicted unflatteringly in Ripley’s book—as an academic pressure cooker lacking joy or purpose—so its standing here is not surprising.  But Finland is another matter.  It is portrayed as laid-back and student-centered, concerned with making students feel relaxed and engaged.  Yet, only 28.8% of Finnish students say that they study mathematics because they enjoy it (among the bottom four countries) and only 24.8% report that they look forward to math lessons (among the bottom seven countries).  Korea, the pressure cooker, and Finland, the laid-back paradise, look about the same on these dimensions.

Another country that is admired for its educational system, Japan, does not fare well on these measures.  Only 30.8% of students in Japan enjoy mathematics, despite the boisterous, enthusiastic classrooms that appear in Elizabeth Green’s recent book, Building a Better Teacher.^[iv]  Japan does better on the percentage of students looking forward to their math lessons (33.7%), but still places far below the U.S.  Green’s book describes classrooms with younger students, but even so, surveys of Japanese fourth and eighth graders’ attitudes toward studying mathematics report results similar to those presented here.  American students say that they enjoy their math classes and studying math more than students in Finland, Japan, and Korea.

It is clear from Table 3-2 that at the national level, enjoying math is not positively related to math achievement.  Nor is looking forward to one’s math lessons.  The correlation coefficients reported in the last row of the table quantify the magnitude of the inverse relationships.  The -0.58 and -0.57 coefficients indicate a moderately negative association, meaning, in plain English, that countries with students who enjoy math or look forward to math lessons tend to score below average on the PISA math test.  And high-scoring nations tend to register below average on these measures of student engagement.  Country-level associations, however, should be augmented with student-level associations that are calculated within each country.

The 2012 PISA volume on student engagement does not present within-country correlation coefficients on intrinsic motivation or its components.  But it does offer within-country correlations of math achievement with three other characteristics relevant to student engagement. Table 3-3 displays statistics for students’ responses to: 1) if they feel like they belong at school; 2) their attitudes toward school, an index composed of four factors;^[v] and 3) whether they had arrived late for school in the two weeks prior to the PISA test. These measures reflect an excellent mix of behaviors and dispositions.

The within-country correlations trend in the direction expected but they are small in magnitude.  Correlation coefficients for math performance and a sense of belonging at school range from -0.02 to 0.18, meaning that the country exhibiting the strongest relationship between achievement and a sense of belonging—Thailand, with a 0.18 correlation coefficient—isn’t registering a strong relationship at all.  The OECD average is 0.08, which is trivial.  The U.S. correlation coefficient, 0.07, is also trivial.  The relationship of achievement with attitudes toward school is slightly stronger (OECD average of 0.11), but is still weak.

Of the three characteristics, arriving late for school shows the strongest correlation, an unsurprising inverse relationship of -0.14 in OECD countries and -0.20 in the U.S.  Students who tend to be tardy also tend to score lower on math tests.  But, again, the magnitude is surprisingly small.  The coefficients are statistically significant because of large sample sizes, but in a real world “would I notice this if it were in my face?” sense, no, the correlation coefficients are suggesting not much of a relationship at all.    

The PISA report presents within-country effect sizes for the intrinsic motivation index, calculating the achievement gains associated with a one unit change in the index.  One of several interesting findings is that intrinsic motivation is more strongly associated with gains at the top of the achievement distribution, among students at the 90^th percentile in math scores, than at the bottom of the distribution, among students at the 10^th percentile. 

The report summarizes the within-country effect sizes with this statement: “On average across OECD countries, a change of one unit in the index of intrinsic motivation to learn mathematics translates into a 19 score-point difference in mathematics performance.”^[vi]  This sentence can be easily misinterpreted.  It means that within each of the participating countries students who differ by one unit on PISA’s 2012 intrinsic motivation index score about 19 points apart on the 2012 math test.  It does not mean that a country that gains one unit on the intrinsic motivation index can expect a 19 point score increase.^[vii]    

Let’s now see what that association looks like at the national level.

PISA first reported national scores on the index of intrinsic motivation to learn mathematics in 2003.  Are gains that countries made on the index associated with gains on PISA’s math test?  Table 3-4 presents a score card on the question, reporting the changes that occurred in thirty-nine nations—in both the index and math scores—from 2003 to 2012.  Seventeen nations made statistically significant gains on the index; fourteen nations had gains that were, in a statistical sense, indistinguishable from zero—labeled “no change” in the table; and eight nations experienced statistically significant declines in index scores. 

The U.S. scored 0.00 in 2003 and 0.08 in 2012, notching a gain of 0.08 on the index (statistically significant).  Its PISA math score declined from 483 to 481, a decline of 2 scale score points (not statistically significant).

Table 3-4 makes it clear that national changes on PISA’s intrinsic motivation index are not associated with changes in math achievement.  The countries registering gains on the index averaged a decline of 3.7 points on PISA’s math assessment.  The countries that remained about the same on the index had math scores that also remain essentially unchanged (-0.09) And the most striking finding: countries that declined on the index (average of -0.15) actually gained an average of 10.3 points on the PISA math scale.  Intrinsic motivation went down; math scores went up.  The correlation coefficient for the relationship over all, not shown in the table, is -0.30.

The analysis above investigated student engagement.  International data from the 2012 PISA were examined on several dimensions of student engagement, focusing on a measure that PISA has employed since 2003, the index of intrinsic motivation to learn mathematics.  The U.S. scored near the middle of the distribution on the 2012 index.  PISA analysts calculated that, on average, a one unit change in the index was associated with a 19 point gain on the PISA math test.  That is the average of within-country calculations, using student-level data that measure the association of intrinsic motivation with PISA score.  It represents an effect size of about 0.20—a positive effect, but one that is generally considered small in magnitude.^[viii]

The unit of analysis matters.  Between-country associations often differ from within-country associations.  The current study used a difference in difference approach that calculated the correlation coefficient for two variables at the national level: the change in intrinsic motivation index from 2003-2012 and change in PISA score for the same time period.  That analysis produced a correlation coefficient of -0.30, a negative relationship that is also generally considered small in magnitude.

Neither approach can justify causal claims nor address the possibility of reverse causality occurring—the possibility that high math achievement boosts intrinsic motivation to learn math, rather than, or even in addition to, high levels of motivation leading to greater learning.  Poor math achievement may cause intrinsic motivation to fall.  Taken together, the analyses lead to the conclusion that PISA provides, at best, weak evidence that raising student motivation is associated with achievement gains.  Boosting motivation may even produce declines in achievement.

Here’s the bottom line for what PISA data recommends to policymakers: Programs designed to boost student engagement—perhaps a worthy pursuit even if unrelated to achievement—should be evaluated for their effects in small scale experiments before being adopted broadly.  The international evidence does not justify wide-scale concern over current levels of student engagement in the U.S. or support the hypothesis that boosting student engagement would raise student performance nationally.

Let’s conclude by considering the advantages that national-level, difference in difference analyses provide that student-level analyses may overlook.

1. They depict policy interventions more accurately.  Policies are actions of a political unit affecting all of its members.  They do not simply affect the relationship of two characteristics within an individual’s psychology. Policymakers who ask the question, “What happens when a country boosts student engagement?” are asking about a country-level phenomenon.

2.  Direction of causality can run differently at the individual and group levels.  For example, we know that enjoying a school subject and achievement on tests of that subject are positively correlated at the individual level.  But they are not always correlated—and can in fact be negatively correlated—at the group level. 

3.  By using multiple years of panel data and calculating change over time, a difference in difference analysis controls for unobserved variable bias by “baking into the cake” those unobserved variables at the baseline.  The unobserved variables are assumed to remain stable over the time period of the analysis.  For the cultural factors that many analysts suspect influence between-nation test score differences, stability may be a safe assumption.  Difference in difference, then, would be superior to cross-sectional analyses in controlling for cultural influences that are omitted from other models.

4.  Testing artifacts from a cultural source can also be dampened.  Characteristics such as enjoyment are culturally defined, and the language employed to describe them is also culturally bounded.  Consider two of the questionnaire items examined above: whether kids “enjoy” math and how much they “look forward” to math lessons.  Cultural differences in responding to these prompts will be reflected in between-country averages at the baseline, and any subsequent changes will reflect fluctuations net of those initial differences.

Downloads

• Download the report

Tom Loveless, a nonresident senior fellow in Governance Studies, explains his latest research on measuring achievement of American students.

“The bottom line here: the implementation of the common core has appeared to have very little impact on student achievement,” Loveless says. In this episode, he discusses whether the common core is failing our students, whether AP achievement is indicative of student success, and the role of principals as instructional leaders.

Also in this episode: Get to know Constanze Stelzenmüller, the Robert Bosch Senior Fellow in the Center on the United States and Europe, during our "Coffee Break” segment. Also stay tuned to hear the final episode in our centenary series with current and past Brookings scholars.

Show Notes:

The Brown Center Report on American Education

Brookings Centenary Timeline

Subscribe to the Brookings Cafeteria on iTunes, listen in all the usual places, and send feedback email to BCP@Brookings.edu.