Lessons learned - My nightmare using Amazon Mechanical Turk

Hey all!

Figured I'd share my learnings after a long stressful day 😓

The Problem:
I had located a great source of speed test data (to be used in my project to surface internet speeds on Airbnb listings). To be useful, however, I needed to filter out the data coming from mobile phones. Sounds easy enough?

The only way to do this with the dataset was to figure out the network provider of each speed test. These were recorded as AS numbers (https://en.wikipedia.org/wiki/Autonomous_system_(Internet)), which can be mapped to the name of the provider using this website (https://bgpview.io/).

Unfortunately, there are thousands of these numbers! I made it my goal to get the top 1000 in the United States properly mapped. I did the first ~70 manually, and then figured that this would be a great use case for Amazon Mechanical Turk!

The Setup:
I created a project on Mechanical Turk that would provide users a link, and then they were to categorize the provider as either a) Residential provider b) Cellular provider, or c) Business/other. What could go wrong?

I set my payout to 10 cents per HIT, 2 HITs per job, and uploaded my credit card info, and sent it off!

The Deception:
Before I knew it, the entire project had been completed within 10 minutes of submission. I thought "that can't be right?"

I went through and started auditing the answers to find that hundreds of submissions had been completed within 10 seconds. Clearly, the bots had gotten their grimy hands on my project! Their answers were complete garbage...simply random guessing at my multiple-choice options.

Luckily, mechanical turk gives you the option to approve or reject submissions before the payout is guaranteed. Therefore, I just had to reject all of the bad submissions and block all of the bots. Unfortunately, the only thing worse than 1000 blank responses, is 1000 completed responses with no way of knowing which are legitimate.

Nonetheless, I felt I was $230 in the hole and needed to salvage the situation. Therefore I started the lengthy process of manually attempting to determine which users were bots and then blocking their submissions.

The Fallout:
I had blocked over twenty accounts before the emails started piling into my inbox.

"Good afternoon
Please, you could kindly check this incorrect rejection, as the hit was made as requested, a google search was performed with the name of the provider to check the type of service, satellite, wifi, commercial and etc.
I kindly ask you to check because I always try to do my work in the best possible way! and I appreciate the job opportunity."

"You are accusing me of being a bot?
I expect this is a joke."

UGH, now I've really done it. In my effort to reject the bots, I had mistakenly rejected what appear to be real people as well.

Ok, I can fix this! I'll just go back and re-approve these people's submissions. So one-by-one I went through and corrected/audited the submissions relating to each complaint. I submit the corrections in mechanical turk...yet the system appears to be taking a while to update the records 😬

As it stands now, I have thousands of submissions, but the only way to verify their quality is to go through each one manually. Great.

Lessons Learned:

• If you plan on using mechanical turk make sure your limit your posting to only show to verified, or high-quality users on the platform.
• In retrospect, 10 cents per hit was likely too much to pay. I made the mistake of pricing based on what I thought was reasonable given the task instead of taking the time to learn more about what the cost of labor is in this marketplace.
• When I set up the project I decided to only have two people complete each assignment. Next time, to guard against poor quality, I will instead have more users complete each task.

So that's it! Hope I'm not forever banned by Amazon. Live and learn ✌🏻