Everybody is spamming everybody else on Mechanical Turk

Two months ago , an NYU professor who studies anthropological aspects of online slavery as well as doing nerdy computer science stuff, pointed out in his blog post that 41 percent of mechanical turk HITs are posted by spammers. These requesters are typically those who ask turkers to click on a link, friend somebody on facebook or trash somebody on facebook. These tasks may not look like spam to us as I personally friend and/or trash people on facebook on regular basis so no big deal, but I never get paid for it:)

Anyways as a curious researcher who  doesn’t want to accept anything until he has done his due diligence I posted a spam HIT with a generous 1 dollar reward asking people to write a review about interesting places to visit in the  “Shiraz City, France”. Shiraz is known for its ridiculously hot girls and great wine but the problem is that it is not city in France, surprisingly it is in Iran.

I exclusively asked for those turkers who have lived in “Shiraz City, France” and made sure to highlight that if the turker has never lived in that city the work is going to be rejected. (For those who would like to bring up ethical issues for  this experiment I would like to point out that turkers have screwed me over and over on countless number of occasions so I reserve the right for myself to mess with them occasionally, additionally you can look at this as the Milgram experiment of the crowdsourcing era. Who doesn’t want to be called Philip Zimbardo of the online slavery market? Now that I am thinking about it I actually kind of like it.

Antoine Dodson

The title of this post was inspired by a poem by Antoine Dodson, a frustrated community observer who once shouted out of frustration "Hide yo' keeds, hide yo' wife, and hide yo' husband cause they rapin'everybody out here" He was the first one to voice his frustration about the kind of society that we all have become, a community in which people don't care about each other, the society that leaves no choice for people but hiding!

Anyways if the market was fair I assume two things would have never happend

1) I would have never been allowed to submit such an obscene work

2) I would have never received any responses

To my surprise both happened.

Nobody really polices the mturk market so I was able to get my HIT through. but the more interesting thing was that I was actually receiving suggestions for visiting a city that didn’t even exist!

I receive three types of responses:

  • 1- Random excerpts from our old friend “the internet” about cities in france (mostly Paris)
  • 2- Email addresses of the turkers
  • 3- Random user names

We all know that Mechanical Turk challenges the whole “Junk-in, Junk-out” dilemma and makes it more like “Always junk-out, regardless of the input process” but I feel that users posted junk for my junk HIT assuming that there is no quality assurance process working behind the task and I would probably just accept them all (which I have done for many jobs before)

Anyways I just wanted to highlight that the spamming goes both ways. Turkers spam requesters, requesters spam turkers, everybody wins!

(On a side note, I really need some behind the scene data from Amazon, how can I convince amazon to give me some data for research? In return I am willing to give them a high quality travel booklet that I have curated about a city in France, the city is called “Shiraz”    : )

About these ads

~ by marksalen on March 9, 2011.

5 Responses to “Everybody is spamming everybody else on Mechanical Turk”

  1. Totally agree

    I think the key in the reduction of spam is comments connected to a real identity, ironically I have also been doing a little research in this as well, but slightly different, the idea was to avoid crappy spam comments without having to moderate as much

    http://www.philgainley.com/2011/03/what-comment-system-should-i-use/

  2. There are other crowd sourcing providers (who I won’t mention here for fear of being a spammer myself) who professionally manage the quality control of their workers. This typically works in a system where more trusted workers are paid to review the work of junior turkers. They also don’t take just anyone with a heartbeat and a US credit card as either a turker or requester :)

    Yes, this is typically more expensive. But in addition to “Junk-in, Junk-out” one must also remember another adage – “You get what you pay for”

  3. This is a very interesting read and thanks for posting the link to Panos’ blog. Loved the last line !!

  4. I use Mechanical Turk for my research into computational linguistics. Although quality control is very important, with careful HIT design and a few control questions you can get really excellent results. I think that these articles about spam on MTurk are overblown. It’s not magic. You can’t expect it to work without putting any effort into it. Once I learned how to use it, it became an invaluable tool for research. If anyone wants tips on how to get better results, feel free to contact me (my info is on my web page).

    Chris Callison-Burch
    Associate Research Professor
    Johns Hopkins University
    Computer Science Dept.

  5. hehe funny experimentation
    Thinking of using turk machine myself

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

Join 37 other followers

%d bloggers like this: