Intro to Mechanical Turk, Applications to Machine Learning, and Painless Data Collection
What’s covered in the article:
- Explain “The Turk” and Mechanical Turk
- Discuss the relationship between Mechanical Turk & Machine Learning
- Set up your first Human Intelligence Task (HIT) on Mechanical Turk for data collection!
- Reduce issues for both you and the workers that accept your Human Intelligence Tasks (HITs)
Origins and Evolution of “The Turk”
http://poe.olin.edu/poe2014/knights/subsystems.html
“The Turk” was a machine that was used to trick the likes of Napoleon Bonaparte and Benjamin Franklin into thinking they were beaten by an automated chess player.
In fact, there was a hidden human chess master calling the shots within the machine. Very sneaky!
Amazon leverages this historical “machine” in the naming of one of their services called Mechanical Turk. This service allows “Requesters” the ability to create and manage online human intelligence tasks (HITs) that are to be completed by human workers.
Since it is managed by Amazon, it can be very convenient to create, deploy, and compensate workers using an existing Amazon account for much less money than it can take to organize and deploy your own experiments or surveys.
Mechanical Turk and Machine Learning
I am working on an upcoming article that will try to compare human intuition against an artificial neural network . I needed real people to take my online intuition test in order to compare human performance against the artificial neural network.
( Follow me on Twitter for the results! )
Luckily, I had just listened to “Everything is Obvious. *Once you Know the Answer” by Duncan Watts.
In this book, Duncan Watts talks about using Mechanical Turk for some of his studies so I decided to try it out for my project. (There are quite a few insightful conclusions he makes about payment and other goodies around Mechanical Turk so be sure to check out his book).
After I researched it a bit, I realized that using Mechanical Turk would be an efficient way to gather the human data I needed (Obviously)! I also learned that a lot of the functionality within Mechanical Turk is actually geared towards collecting data for machine learning. Classification problems, transcription, and sentiment analysis are just a few of the ways Mechanical Turk can be used to gather machine learning data.
I found Mechanical Turk to be extremely useful, but ran into a few gotchas that existing or future “Requesters” should know about.
In addition, I did a follow up poll on actual workers for feedback on some rookie mistakes “Requesters” make that they should avoid.
So let’s dive in to Mechanical Turk!
How to Get Your Turk On…
Turkin’ is super easy, especially if you already have an account set up with Amazon.
- Navigate to: https://www.mturk.com/mturk/welcome
- Follow the instructions to either earn money being a “Worker” or to create your own Human Intelligence Task (HIT) as a “Requester” for workers to complete
There is of course a lot of details in between but the site itself provides a lot of good information as you navigate around. The remainder of this article aims to help fill in the gaps for “Requesters” to help them avoid common pitfalls that are often overlooked.
Creating Your First HIT (Human Intelligence Task)
There are a lot of choices in how to structure your HIT. The screenshot below highlights some templates that you can start out with:
Mechanical Turk Survey
As shown above, there are simple surveys that you can create with a variety of input types. There are also setups for common tasks that have some extra configuration and functionality like “Transcription from A/V” and “Categorization Wizard”.
Please note that the setup process can affect the results of your HIT, so in addition to this article, be sure to checkout Lydia Chilton’s work for a more details around effective HITs.
Once you pick a particular type, it’s time to configure some details about your HIT so the workers know what to expect:
Give your Mechanical Turk HIT a name and description
Once, you have given it a name and a nice description, it’s time to determine some important details about distributing your task to workers and how much you will be paying:
Setup the numbers for distribution of your Mechanical Turk Task
This step is important. The instructions they provide are helpful here so read carefully. Be sure to pay extra attention to the estimated total cost you will end up with!
Also, note that it is possible to individually approve each “Worker” once your HIT is up and running, but there is also an “Auto-approve” window that will approve everyone in a certain number of days.
The last section of the initial setup of your HIT is determining what type of workers should be allowed to do your task.
The “Turk Masters” option allows you to specify you want only workers that have “Masters Qualification” to do your HIT. Amazon says workers who have demonstrated excellence across a wide range of HITs are awarded the Masters Qualification”.
Determine which Mechanical Turk Workers will do your HIT
Additional Qualifications for Turk Workers
You can also add up to 5 additional qualifications that will also filter the available workers based on some criteria. There are quite a few, and some are shown here.
After you figure out who you want your workers to be, it’s then time to work on the layout for your HIT.
Your layout of course will very by type, but you will have the opportunity to do some manipulation in their WYSIWYG editor to customize instructions and form fields. WARNING: A common problem and complaint from workers is when the markup for collecting the data is not correct so the data is not submitted properly. This is where clicking “SOURCE” in the editor is helpful to make sure your form elements have unique names to collect the data.
Click “Source” to view the markup underneath
jQuery and Twitter Bootstrap are injected and can be used, so if you have some web development experience you can go nuts. If you aren’t super savvy with developing a form in HTML, then be sure to ask someone who is to look over your work.
Once your layout is complete, you have a chance to preview your HIT before finishing. You can always come back and edit this HIT and you’ll have the opportunity to do so before you deploy it.
Publishing Your New HIT
Now that your HIT is created and saved, you can deploy it out to the workers and start to manage it. Simply find your existing project in the “Create” tab and click “Publish Batch”
You’ll go through a few quick steps and will be asked to validate your Amazon account once again for payment purposes.
Now it’s time to watch in real time as workers complete the HIT!
Monitor Progress, Results, and Approve Payment in Real Time!
Be sure to keep an eye on your email you have associated with your account. workers who run into problems will often send out a quick email.
You can also view the results that have come in and manually approve workers for payment at this time.
Turk Gotchas
During my first few deployments, a few thoughtful workers emailed me to let me know I had not set the time window high enough for the task and they were unable to complete the task and therefore unable to get paid!
I had to cancel my deployments and adjust the time window… twice!
So the first piece of advice is to make sure all workers (not just the average worker) will have enough time to complete your task and as one worker mentioned, you might as well throw in a little time for a potential bathroom break :-)
I wanted to know what some other gotchas might be that I did not run into yet to avoid future frustrating situations for the workers.
I prepared another HIT with a few simple questions for 50 workers to see what common problems they have experienced.
Here are the results from the survey of 50 Mechanical Turk Workers:
Question 1: Estimate the percentage of HITs that you have done that have some type of flaw
On average, workers estimate 15.3% of HITs have some type of flaw
Question 2: Please list advice you have for future HIT creators to help them make good HITs and avoid common mistakes
I received over 130 comments from 50 workers and they were all thoughtful and honest. Some of the comments are specific to the “Survey” type tasks, while others can apply to any type of HIT. I have grouped them by category below:
Worker Comments Categorized With Frequencies
Avoid Obstructions
The biggest issue based on the feedback seems to be when things are generally broken and workers end up unable to complete the HIT and get paid.
Avoid obstructions by following these guidelines:
- Test the HIT before you deploy in the sandbox, then TEST IT AGAIN!
- Ensure any links you provide work. This means across different browsers and that the link you provide can handle the load of a lot of workers!
- Be sure that is a survey codes is provided at the end that it actually works.
- Verify the HTML is correct and that you can submit the proper data.
- Ensure that instructions are clear by having someone else test or do a small trial run and disclose it is a trial run and pay well. :-)
- Oh, and TEST, TEST, TEST to avoid OBSTRUCTIONS!
Avoid Frustration
Some reported issues were not obstructions, but make for confusing, frustrating, or inefficient situations.
Avoid frustrating situations by following these guidelines:
- Provide estimate of time it will take to complete in title or description
- Configure that actual time limit to be much bigger to ensure everyone can finish. Like way more time!
- Pay fairly and approve payments quickly. Workers will notice how much you are paying and it can effect quality (Litman, Robinson, & Rosenzweig, 2015)
- Be honest, transparent, and set clear expectations from the start.
- Make sure instructions are clear and concise and provide examples.
- Check your email frequently and respond to feedback from workers.
- Use the “Exclusion” qualification if you don’t want the same workers repeating the HIT at a later date (this keeps workers from wasting time looking for valid HITs).
- Screening should be up front and clear to avoid wasting worker time.
- Check mturkcrowd.com to view and request feedback on HIT proposals.
- You can also check out these guidelines that have been generated by the “Worker” and “Requester” community.
- More information can be found in this vibrant reddit community.
I won’t comment on what “fair pay” should be, I will however, mention that just like the original “Turk” there is a real person hiding behind the machine (possibly trying to pay for grad school) so try to keep the “Golden Rule” in mind. :-)
Keep Calm and Turk On!
I hope this helped you either learn more about Mechanical Turk and inspired you to consider using this incredibly valuable tool!
Like this article? Click the 💚 below and FOLLOW for future posts.