The Fascinating World Of "Human Powered" Annotation Services That Support Modern AI Algorithms

I recently tried and got rejected when I applied to be a worker for Amazon's MTurk services. It wasn't so much for the money given that tasks usually pay in cents but I was just curious as to what sort of tasks were being posted up.

So what's MTurk and why should you care ?

So much of the media focus has been on the amazing things that machine learning algorithms can do (Looking at you GPT3!) but less well covered is the fact that a lot of Machine Learning (*) still relies heavily on supervised learning techniques that require LABELLED/ANNOTATED training samples to work properly.

(*...for now, let's side step the latest developments in self-learning and unsupervised machine learning techniques)

So I thought I'd share a few articles I collated around "crowd-sourced" platform services like Amazon's Mechanical Turk that are providing these services and some of the issues surrounding them. Amazon first coined the term artificial artificial intelligence for the process of outsourcing some parts of a computer program to humans, for those tasks carried out much faster by humans than computers. (Although , I personally prefer my own sobriquet - A.I.'nt )

Challenge No 1: Data Privacy & Security Concerns

One of the problems with this is related to how companies process the data you may have given them access to collect and analyse as the practice of crowd sourcing the data annotation may end up causing inadvertent privacy issues.

One such example was in 2017 where a paperless business expense management service was found to have been using humans to transcribe receipts the company's own computer vision system couldn't decipher.

This article from the Guardian also covers a few other examples elsewhere in big tech companies including Facebook.

Challenge No 2: Ethical Concerns Around Microtask Worker Welfare

Another challenge is related is how these sort of platforms services have essentially created a way for companies to circumvent the regulations that would normally protect workers against low paying wage.

The Atlantic and Wired have covered this here:-

Challenge No 3: Quality Of Work Completed By Micro-Taskers

Although this article is a bit dated (2018), this piece from Wired is an interesting read as it covered a case where a number of psychologists suspected that their MTurk requests were being replied to not by real people but bots that were all coming from the same Virtual Private Server (Try to wrap your head around that, Amazon created a service to use people to train machines and people tried to cheat by using machines to automate their responses)

Following that incident, Amazon has since revved up it's controls in vetting it's workers (Which is probably why I got rejected). There have also been a number of academic studies that have shown that the Wired article may have exaggerated the scale of the problem. However 'fraudulent' respondents aside, a more recent (2020) study , had this amusing statement : While find no evidence of a “bot epidemic,” significant portions of the data—between 25%-35%—are of dubious quality. ....Furthermore many respondents appear to respond humorously or insincerely, and this behavior increased over 200% from 2018-2020. Which only just highlights that most people are dicks.

Ending On A Tangential Note.....

I conclude on a somewhat related topic - what I find most annoying is that many of us may be performing similar type of micro-tasks for no remuneration every time we complete a captcha to verify that we are not a robot.

Google has ingeniously turned the concept around it's head and has reaped the double benefit of charging other companies to use it's reCaptcha services and also "making positive use of this human effort by channeling the time spent solving CAPTCHAs into digitizing text, annotating images, and building machine learning datasets."

I leave you with this gem from xkcd