“The Turk” was a machine that was used to trick the likes of Napoleon Bonaparte and Benjamin Franklin into thinking they were beaten by an automated chess player.
In fact, there was a hidden human chess master calling the shots within the machine. Very sneaky!
Amazon leverages this historical “machine” in the naming of one of their services called Mechanical Turk. This service allows “Requesters” the ability to create and manage online human intelligence tasks (HITs) that are to be completed by human workers.
Since it is managed by Amazon, it can be very convenient to create, deploy, and compensate workers using an existing Amazon account for much less money than it can take to organize and deploy your own experiments or surveys.
I am working on an upcoming article that will try to compare human intuition against an artificial neural network . I needed real people to take my online intuition test in order to compare human performance against the artificial neural network.
( Follow me on Twitter for the results! )
In this book, Duncan Watts talks about using Mechanical Turk for some of his studies so I decided to try it out for my project. (There are quite a few insightful conclusions he makes about payment and other goodies around Mechanical Turk so be sure to check out his book).
After I researched it a bit, I realized that using Mechanical Turk would be an efficient way to gather the human data I needed (Obviously)! I also learned that a lot of the functionality within Mechanical Turk is actually geared towards collecting data for machine learning. Classification problems, transcription, and sentiment analysis are just a few of the ways Mechanical Turk can be used to gather machine learning data.
I found Mechanical Turk to be extremely useful, but ran into a few gotchas that existing or future “Requesters” should know about.
In addition, I did a follow up poll on actual workers for feedback on some rookie mistakes “Requesters” make that they should avoid.
So let’s dive in to Mechanical Turk!
Turkin’ is super easy, especially if you already have an account set up with Amazon.
There is of course a lot of details in between but the site itself provides a lot of good information as you navigate around. The remainder of this article aims to help fill in the gaps for “Requesters” to help them avoid common pitfalls that are often overlooked.
There are a lot of choices in how to structure your HIT. The screenshot below highlights some templates that you can start out with:
Mechanical Turk Survey
As shown above, there are simple surveys that you can create with a variety of input types. There are also setups for common tasks that have some extra configuration and functionality like “Transcription from A/V” and “Categorization Wizard”.
Once you pick a particular type, it’s time to configure some details about your HIT so the workers know what to expect:
Give your Mechanical Turk HIT a name and description
Once, you have given it a name and a nice description, it’s time to determine some important details about distributing your task to workers and how much you will be paying:
Setup the numbers for distribution of your Mechanical Turk Task
This step is important. The instructions they provide are helpful here so read carefully. Be sure to pay extra attention to the estimated total cost you will end up with!
Also, note that it is possible to individually approve each “Worker” once your HIT is up and running, but there is also an “Auto-approve” window that will approve everyone in a certain number of days.
The last section of the initial setup of your HIT is determining what type of workers should be allowed to do your task.
The “Turk Masters” option allows you to specify you want only workers that have “Masters Qualification” to do your HIT. Amazon says workers who have demonstrated excellence across a wide range of HITs are awarded the Masters Qualification”.
Determine which Mechanical Turk Workers will do your HIT
Additional Qualifications for Turk Workers
You can also add up to 5 additional qualifications that will also filter the available workers based on some criteria. There are quite a few, and some are shown here.
After you figure out who you want your workers to be, it’s then time to work on the layout for your HIT.
Your layout of course will very by type, but you will have the opportunity to do some manipulation in their WYSIWYG editor to customize instructions and form fields. WARNING: A common problem and complaint from workers is when the markup for collecting the data is not correct so the data is not submitted properly. This is where clicking “SOURCE” in the editor is helpful to make sure your form elements have unique names to collect the data.
Click “Source” to view the markup underneath
jQuery and Twitter Bootstrap are injected and can be used, so if you have some web development experience you can go nuts. If you aren’t super savvy with developing a form in HTML, then be sure to ask someone who is to look over your work.
Once your layout is complete, you have a chance to preview your HIT before finishing. You can always come back and edit this HIT and you’ll have the opportunity to do so before you deploy it.
Now that your HIT is created and saved, you can deploy it out to the workers and start to manage it. Simply find your existing project in the “Create” tab and click “Publish Batch”
You’ll go through a few quick steps and will be asked to validate your Amazon account once again for payment purposes.
Now it’s time to watch in real time as workers complete the HIT!
Monitor Progress, Results, and Approve Payment in Real Time!
Be sure to keep an eye on your email you have associated with your account. workers who run into problems will often send out a quick email.
You can also view the results that have come in and manually approve workers for payment at this time.
During my first few deployments, a few thoughtful workers emailed me to let me know I had not set the time window high enough for the task and they were unable to complete the task and therefore unable to get paid!
I had to cancel my deployments and adjust the time window… twice!
So the first piece of advice is to make sure all workers (not just the average worker) will have enough time to complete your task and as one worker mentioned, you might as well throw in a little time for a potential bathroom break :-)
I wanted to know what some other gotchas might be that I did not run into yet to avoid future frustrating situations for the workers.
I prepared another HIT with a few simple questions for 50 workers to see what common problems they have experienced.
Here are the results from the survey of 50 Mechanical Turk Workers:
Question 1: Estimate the percentage of HITs that you have done that have some type of flaw
On average, workers estimate 15.3% of HITs have some type of flaw
Question 2: Please list advice you have for future HIT creators to help them make good HITs and avoid common mistakes
I received over 130 comments from 50 workers and they were all thoughtful and honest. Some of the comments are specific to the “Survey” type tasks, while others can apply to any type of HIT. I have grouped them by category below:
Worker Comments Categorized With Frequencies
The biggest issue based on the feedback seems to be when things are generally broken and workers end up unable to complete the HIT and get paid.
Avoid obstructions by following these guidelines:
Some reported issues were not obstructions, but make for confusing, frustrating, or inefficient situations.
Avoid frustrating situations by following these guidelines:
I won’t comment on what “fair pay” should be, I will however, mention that just like the original “Turk” there is a real person hiding behind the machine (possibly trying to pay for grad school) so try to keep the “Golden Rule” in mind. :-)
I hope this helped you either learn more about Mechanical Turk and inspired you to consider using this incredibly valuable tool!
Like this article? Click the 💚 below and FOLLOW for future posts.