About that Harvard Instagram study… You may have heard about the Harvard study wherein researchers “ .” The paper’s subject is perfectly weaponized to make the media rounds, combining data, AI, health, a popular social network, and an enticing question to encourage clicks ( ). , , , , and others all hit post. The story has been lighting up Twitter for nearly a week now. trained a machine to spot depression on Instagram what filter means you’re depressed? MIT Technology Review Wired The Next Web The Independent _One of the curious things about color is that we associate it with emotions. Intuitively, we tend to link darker…_www.technologyreview.com A machine can tell whether you're depressed just by looking at your photos on Instagram But once the depression filter was revealed (Inkwell, ), I’m pretty sure everyone stopped reading. If they had, they’d have found a different story about depression: of course the crowdsourced workers who fuel the algorithms, which will evaluate us, are very depressed. To find this sad story let’s run down the numbers cited in the : MIT Technology Review article The researchers asked workers on Amazon’s crowdsource platform, , to complete a survey which contained a standard clinical depression survey. 500 Mechanical Turk The researchers asked those workers if they would share their Instagram posts for the study. workers agreed. 500 170 Out of those workers, were clinically depressed, based on their survey responses. 170 70 The researchers sorted the Instagram photos shared with them(again, with Mechanical Turk workers) and used that data to train a . machine learning algorithm The researchers tested their algorithm on the photos of 100 individuals and correctly identified of those who are depressed. (Who these individuals are, how they got their photos, and how they diagnosed them is not specified — I assume they repeated the steps above.) 70% 70% accuracy sounds pretty good! Allegedly, this hit rate is better than general practitioners. But it is statistically significant. A test group of 100 is laughably small and the paper has yet to be peer reviewed. ( , atoning for the publication’s earlier breathlessness.) hardly Nick Stockton covers this on Wired But they’ve buried the real story. . The depression rate among adults in the United States is 6.7% The depression rate among the crowdsourced workers who shared their photos is . Over the national norm. 41.2% six times Working on Mechanical Turk, it appears, is crushing. Mechanical Turk does not pay well. Because of their status as independant contractors, Turkers (as they are called) are not covered by labor laws. . Their hourly pay ranges from $1-$5 _The wages offered on Mechanical Turk do not seem very enticing. Three cents for copying the product codes and prices…_priceonomics.com Who Makes Below Minimum Wage in the Mechanical Turk Sweatshop? But poverty does not appear to be the driver for this high depression rate. , poverty doubles the average US depression rate. Mechanical Turk, according to the Instagram study, multiplies it by six. According to the CDC With the , Mechanical Turk has become a training ground for algorithms. Turkers sort data which will be used to create machine learning products. The best summary of Mechanical Turk, its workers, and the machines they train is . recent rise of deep learning this episode of NPR’s Planet Money _They are hundreds of thousands of people out there doing stuff to your internet that you probably think is automatic…_www.npr.org Episode 600: The People Inside Your Machine Listening to Planet Money, its easy to see how crowd work can spur frustration and feelings of helplessness beyond poverty itself. There are no bosses or structure, just rapidly cycling tasks. Pay for repetitive work is generally insultingly low. There are no avenues for recourse other than which generate no response. self-organization and open letters to Amazon When we discuss the issues inherent with AI and machine learning we usually . We rarely discuss the people whose work or attention create the algorithms themselves. focus on the perils of allowing computers to make decisions humans currently own This is a mistake. Crowd work will only grow in the future, either through sharing-economy applications or online work. It’s existence without appropriate, modern regulation is worth discussion. In an ironic twist, the decisions made by the powerless people on Mechanical Turk will be amplified in algorithms which will eventually have power over us all. If the people training these machines do not represent us, we will cede decisions to algorithms with which we will likely disagree. The case discussed here regarding Mechanical Turk is even worse: the work of sorting itself could turn a representative population into a depressed one, making skewed decisions unavoidable. Do the depressed judge depression or photos differently than the happy? It is a missed opportunity that crowd work remains largely invisible while its output, machine learning, is a topic du jour.
