Mechanical Turk is a cheap way to get lots of qualitative feedback
Recap
I recently quit my hedge fund job to pursue entrepreneurship. My hypothesis is that in the long term entrepreneurship outperforms high paying jobs, and that most risk of failure can be minimized when starting a business by choosing conservative (easy) problems, and applying rational thinking as well as creativity to business problems. I started chronicling my success and failures here, this is article 4. The next article is here
My first project is a group email business. I’m going to go into how I tried to make my landing page better with Mechanical Turk.
GroupMail enables users to easily create email groups by forwarding existing threads. There is a built in viral component, where people create groups with their friends, and hopefully their friends create more groups with more friends, etc. However converting from my website is my first starting point. Up until now I didn’t have very much traffic, but enough to know that no one is converting and my landing page must be terrible.
If you’ve never used Mechanical Turk before, Mechanical Turk is an Amazon product where you can pay random people on the internet to do something for you for a fixed fee (I was offering 5 cents). It’s an incredibly cheap way to do market research, but the downside is there is some selection bias in who you’re going to get. I setup a survey which asks the following
- Go to https://www.groupmail.in. In your own words describe what this business does
- Would you use this service
- If so, give me an example of how you would use this service
- If not, why wouldn’t you use this service.
I’m only offering 5 cents per survey, so I got back some ridiculous answers. For example for question 1 about what this business does — here are some of the nonsense answers I got.
- Infrastructure to hold email groups in the cloud
- FUN
- deal with your mail
The good news is, most people who weren’t giving me garbage did know what this business does. There was also a small sample that confuses GroupMail with online marketing mailing lists like MailChimp.
So the problem is there is no quantitative benchmark for what is a good score here this would be amazing as a conversion rate but since these are paid survey takers it’s hard to anchor these results. Slightly less than half said they would use GroupMail . 24/102 people said they don’t use email groups, and 23/102 people said they didn’t see any value to the service. I don’t think there is much I can do for those people.
I was able to get some interesting qualitative information. Here are the interesting reasons for rejection.
- I run my own mailserver
- GMAIL CAN HANDLE THIS
- I would not be comfortable giving out my peers/co-workers email addresses to an external service, and also I can just create groups in outlook or gmail and make my own grouped messages that way.
- Sending group emails is never difficult or time consuming. My standard email client handles this perfectly well.
I’m surprised that someone who runs their own mail server still wants to fill out my survey for cash, but there were a few key takeaways
- The website does not inspire trust
- People don’t understand how this is different than client side email groups
- People don’t understand how this service is differentiated from google and yahoo groups
- Marketing a service that provide marginal benefit but not a huge benefit is harder than I thought
Using Mechanical Turk to Benchmark against a competitor
I also wanted to baseline these results, so I created another survey from the perspective of a competitor, gaggle email. I asked the following questions
- Please go to https://gaggle.email/. In your own words, what does this company do?
- Would you use this service?
If they answer yes
- How would you use it
- Please go to https://www.groupmail.in. which is a free competitor to gaggle.email, would you use groupmail.in
If they answer no
- Why wouldn’t you use it
- Please go to https://groupmail.in, which is a free competitor to gaggle.email, would you use groupmail.in
This was pretty interesting to me because it allows me to benchmark the performance of my landing page against the performance of a competitor and find out what sets them apart from me.
Slightly less than half said they would use the service — which is basically the same performance as GroupMail.
Only 36 percent would switch to GroupMail — which is pretty damaging to me considering my service is free. Here are the reasons cited:
- Seems more established
- I liked the design of the page better
- It seemed more legitimate
- The website seemed simpler and more appealing.
- gagglemail actually seems like it would be far easier to use which I like.
Wow, so I have a very tiny impact on people who don’t like gaggle.email.
One person got what I was going for!
Groupmail feels soothing to the nominal of all the painful google’s and gaggle just seems to add on to it.
Unfortunately it’s only one person.
Here are my takeaways from this exercise
- Professionalism and trustworthiness matters, gaggle.email has that, groupmail does not
- There were many comments about how gaggle and groupmail are both low value propositions compared to free alternatives. This is encouraging because it shows to me that you can build a successful business out of a marginal improvement (gaggle.email is profitable, and I would classify them as a marginal improvement over google groups)
- In terms of benchmarks, GroupMail got a similar rejection rate to gaggle.email, so it’s not terrible, but there are definitely things that can be improved
Based on this feedback, I’ve made text changes to differentiate GroupMail from common alternatives, both client side email groups, as well as mailing lists from Google and Yahoo. I haven’t made any changes regarding trustworthiness or professionalism, as those require much more work.
Conclusions
So Mechanical Turk can be a very useful way to get feedback — both on your site, and how you compare against competitors. Extracting any quantitative information out of mechanical turk for these purposes is a bit difficult however, since there is a significant selection bias and since these people paid.
Some Practical Details
If you want to do this yourself, here are a few things boring details to know. If you will never do this, skip this paragraph. Amazon Mechanical Turk requires workers to give some sort of “answer” to amazon to get credit for their work. Most people who use Turk for surveys generate a code per survey and have Turkers submit that so you can verify that they did the job. Google Forms does NOT let you vary the code, you just have to output some code at the end of the survey. If you click the gears icon that will allow you to choose what is displayed after submission.
You can also have a question asking for their amazon worker ID (several people online recommend this). That would allow you to confirm who has taken your survey, and if anyone clearly did not actually take your survey you can refuse payment. However if you’re only charging 5 cents, it’s not worth your time to do this (the next time I do this, I’m not going to ask for worker IDs). DO NOT ask for email addresses. That is a violation of policy and you will get banned. Finally Jobs you create in mechanical turk are called batches.
Here are 2 batches
Both batches I have are setup for 50 tasks. I can edit one of them, and ask for more tasks, or change the price per task. The important thing is that if you want to gather more results — do NOT create a new batch. Amazon has controls in place so you can prevent workers from taking multiple tasks from a batch(I do not want the same person taking my survey twice). Creating multiple batches circumvents this control
Thanks for reading!. My past projects are about simple easy email groups, and getting help for open source projects. My current project provides cloud hosted Jupyter notebooks. My next article is about moving from groupmail to a different business — a support model for open source developers