Improving data in the field of workforce development is a necessary step to evaluating programs and replicating success. What does current data tell us about the populations served? What outcomes should we measure to ensure programs are meeting America’s workforce development needs?

Earlier this month, we convened an expert group of policy makers, practitioners and scholars to address this problem, along with other challenges in workforce development. Previously, we interviewed Kate Blosveren Kreamer on the need to strengthen bridges from school to work. Next up in our Q&A series is Beth Weigensberg, a researcher at Mathematica Policy Research.

A: Although there is increasingly more rigorous research to assess effectiveness of programs, I feel a missing piece is understanding how to replicate and scale-up effective strategies. Often times workforce development programs that are deemed effective in one place do not always succeed when implemented in another. Research that evaluates effectiveness of programs should assess the role of contextual factors (including organizational, leadership, community, and political factors) to identify what is needed to successfully implement, replicate, and scale successful programs.

A: The workforce development field has an unfortunate history of “creaming”—programs selectively work with individuals most likely to succeed at finding employment, leaving those “harder-to-serve” individuals struggling to find assistance. Individuals that are often considered “hard-to-serve” include those who are homeless, disabled, formerly incarcerated, older workers, non-English speakers, low-income, and youth who are disconnected from school and employment. Increasing efforts to focus on these “harder-to-serve” populations include specialized targeted programs and strategies to help address the complex needs of these individuals, which often extend beyond skill development and finding a job. These specialized programs often provide additional support services to help address their complex needs, which can serve as additional barriers to obtaining and retaining employment.

A: Intermediate measures of engagement and skill development would provide interim measures of progress, while the ultimate objectives are obviously employment and educational attainment. Ongoing evaluation on interim measures allows for earlier acknowledgment of achievement and identification of those struggling to progress. Assessing outcomes in ways that control for different populations or barriers to employment, such as using risk-adjusted methodologies, can help us evaluate workforce development programs in an equitable manner.

One of the biggest challenges in the field is ensuring we have valid and reliable data to accurately estimate outcomes. The data available to assess outcomes are usually limited by what is collected in management information systems, which are often developed to be responsive to reporting requirements of publically-funded programs. But these siloed data do not allow for comprehensive assessment of workforce development outcomes within a state, locality, or even within a community-based employment and training organization that relies on numerous funding sources. Efforts are needed to integrate data and assess standardized outcome measures across program and funding silos to allow for more comprehensive assessment of outcomes within the field.

In the course of my graduate work at Harvard University, I paid hundreds of Americans living in poverty the equivalent of about $2 an hour. It was perfectly legal for me to do so, and my research had the approval of my university’s ethics board. I was not alone, or even unusual, in basing Ivy League research on less-than-Walmart wages; literally thousands of academic research projects pay the same substandard rates. Social scientists cannot pretend that the system is anything but exploitative. It is time for meaningful reform of crowdsourced research.

This is what crowdsourced research looks like. I posted a survey using Mechanical Turk (MTurk), a website run by Amazon.com. Across the country, hundreds of MTurk workers (“turkers”) agreed to fill out the survey in exchange for about 20 cents apiece, and within a few days I had my survey results. The process was easy, and above all, cheap. No wonder it is increasingly popular with academics; a search on Google Scholar returns thousands of academic papers citing MTurk, increasing from 173 in 2008 to 5,490 in 2014.

Mechanical Turk is a bargain for researchers, but not for workers. A survey typically takes a couple minutes per person, so the hourly rate is very low. This might be acceptable if all turkers were people with other jobs, for whom the payment was incidental. But scholars have known for years that the vast majority of MTurk tasks are completed by a small set of workers who spend long hours on the website, and that many of those workers are very poor. Here are the sobering facts:

• About 80 percent of tasks on MTurk are completed by about 20 percent of participants that spend more than 15 hours a week working on the site. MTurk works not because it has many hobbyists, but because it has dedicated people who treat the tasks like a job.
• About one in five turkers are earning less than $20,000 a year.
• A third of U.S. turkers call MTurk an important source of income, and more than one in ten say they use MTurk money to make basic ends meet.

Journal articles that refer to Mechanical Turk. Source: PS: Political Science and Politics

It is easy to forget that these statistics represent real people, so let me introduce you to one of them. “Marjorie” is a 53-year-old woman from Indiana who had jobs in a grocery store and as a substitute teacher before a bad fall left her unable to work. Now, she says, “I sit there for probably eight hours a day answering surveys. I’ve done over 8,000 surveys.” For these full days of work, Marjorie estimates that she makes “$100 per month” from MTurk, which supplements the $189 she receives in food stamps. Asked about her economic situation, Marjorie simply says that she is “poverty stricken.”

I heard similar stories from other MTurk workers—very poor people, often elderly or disabled, working tremendous hours online just to keep themselves and their families afloat. I spoke to a woman who never got back on her feet after losing her home in Hurricane Rita, and another who had barely escaped foreclosure. A mother of two was working multiple jobs, plus her time MTurk, to keep her family off government assistance. Job options are few for many turkers, especially those who are disabled, and MTurk provides resources they might not otherwise have. But these workers that work anonymously from home are isolated and have few avenues to organize for higher wages or other employment protections.

Once I realized how poorly paid my respondents were, I went back and gave every one of my over 1,400 participants a “bonus” to raise the survey respondent rate to the equivalent of a $10 hourly wage. (I paid an additional $15 to respondents who participated in an interview.) This cost me a little bit more money, but less than you might imagine. For a 3-minute survey of 800 people, going from a 20-cent to a 50-cent payment costs an additional $240. But if every researcher paid an ethical wage, it would really add up for people like Marjorie. In fact, it would likely double her monthly income from MTurk.

Raising wages is a start, but it should not be up to individual researchers to impose workplace standards. In this month’s PS: Political Science and Politics, a peer-reviewed journal published for the American Political Science Association, I have called for new standards for crowdsourced research to be implemented not only by individual researchers, but also by universities, journals, and grantmakers. For instance, journal editors should commit to publishing only those articles that pay respondents an ethical rate, and university ethics boards should create guidelines for use of crowdsourcing that consider wages and also crowdsourcers’ lack of access to basic employment protections.

The alternative is continuing to pay below-minimum-wage rates to a substantial number of poor people who rely on this income for their basic needs. This is simply no alternative at all.

This post originally appeared on the TechTank blog.

In the course of my graduate work at Harvard University, I paid hundreds of Americans living in poverty the equivalent of about $2 an hour. It was perfectly legal for me to do so, and my research had the approval of my university’s ethics board. I was not alone, or even unusual, in basing Ivy League research on less-than-Walmart wages; literally thousands of academic research projects pay the same substandard rates. Social scientists cannot pretend that the system is anything but exploitative. It is time for meaningful reform of crowdsourced research.

This is what crowdsourced research looks like. I posted a survey using Mechanical Turk (MTurk), a website run by Amazon.com. Across the country, hundreds of MTurk workers (“turkers”) agreed to fill out the survey in exchange for about 20 cents apiece, and within a few days I had my survey results. The process was easy, and above all, cheap. No wonder it is increasingly popular with academics; a search on Google Scholar returns thousands of academic papers citing MTurk, increasing from 173 in 2008 to 5,490 in 2014.

Mechanical Turk is a bargain for researchers, but not for workers. A survey typically takes a couple minutes per person, so the hourly rate is very low. This might be acceptable if all turkers were people with other jobs, for whom the payment was incidental. But scholars have known for years that the vast majority of MTurk tasks are completed by a small set of workers who spend long hours on the website, and that many of those workers are very poor. Here are the sobering facts:

• About 80 percent of tasks on MTurk are completed by about 20 percent of participants that spend more than 15 hours a week working on the site. MTurk works not because it has many hobbyists, but because it has dedicated people who treat the tasks like a job.
• About one in five turkers are earning less than $20,000 a year.
• A third of U.S. turkers call MTurk an important source of income, and more than one in ten say they use MTurk money to make basic ends meet.

Journal articles that refer to Mechanical Turk. Source: PS: Political Science and Politics

It is easy to forget that these statistics represent real people, so let me introduce you to one of them. “Marjorie” is a 53-year-old woman from Indiana who had jobs in a grocery store and as a substitute teacher before a bad fall left her unable to work. Now, she says, “I sit there for probably eight hours a day answering surveys. I’ve done over 8,000 surveys.” For these full days of work, Marjorie estimates that she makes “$100 per month” from MTurk, which supplements the $189 she receives in food stamps. Asked about her economic situation, Marjorie simply says that she is “poverty stricken.”

I heard similar stories from other MTurk workers—very poor people, often elderly or disabled, working tremendous hours online just to keep themselves and their families afloat. I spoke to a woman who never got back on her feet after losing her home in Hurricane Rita, and another who had barely escaped foreclosure. A mother of two was working multiple jobs, plus her time MTurk, to keep her family off government assistance. Job options are few for many turkers, especially those who are disabled, and MTurk provides resources they might not otherwise have. But these workers that work anonymously from home are isolated and have few avenues to organize for higher wages or other employment protections.

Once I realized how poorly paid my respondents were, I went back and gave every one of my over 1,400 participants a “bonus” to raise the survey respondent rate to the equivalent of a $10 hourly wage. (I paid an additional $15 to respondents who participated in an interview.) This cost me a little bit more money, but less than you might imagine. For a 3-minute survey of 800 people, going from a 20-cent to a 50-cent payment costs an additional $240. But if every researcher paid an ethical wage, it would really add up for people like Marjorie. In fact, it would likely double her monthly income from MTurk.

Raising wages is a start, but it should not be up to individual researchers to impose workplace standards. In this month’s PS: Political Science and Politics, a peer-reviewed journal published for the American Political Science Association, I have called for new standards for crowdsourced research to be implemented not only by individual researchers, but also by universities, journals, and grantmakers. For instance, journal editors should commit to publishing only those articles that pay respondents an ethical rate, and university ethics boards should create guidelines for use of crowdsourcing that consider wages and also crowdsourcers’ lack of access to basic employment protections.

The alternative is continuing to pay below-minimum-wage rates to a substantial number of poor people who rely on this income for their basic needs. This is simply no alternative at all.