Poisoning attacks

- [Instructor] If you see a picture of a skull and crossbones on a bottle, chances are, you know that that bottle has something in it that could hurt or even make you sick or kill you. In AI and ML, poison data makes the system or model sick too. Machine learning systems learn using data. If that data's been tampered with, the system will not learn what was intended. Consider a machine learning system that's being used to identify polar bears in Greenland to study their migration patterns. If the images of polar bears were poisoned and replaced with images of penguins the system would not be able to track the polar bears and not be able to serve the function it was created for. Marcus Comite of the Harvard Kennedy School Belfer Center described poisoning attacks as corrupting the process during which the AI system is created so that the resulting system malfunctions in a way desired by the attacker. When machine learning systems continue to learn in production, they're being trained on live, and in some cases, un-vetted data sets. This can also lead to unintended consequences or failure. In March, 2016, Microsoft researchers launched an AI bot on Twitter that they named Tay. Tay was meant to learn from interactions on Twitter how to acquire more natural language. There were no constraints on what people could say to Tay. And unfortunately, a subset of the Twitter-verse quickly figured out how to poison Tay's learning by tweeting offensive racist phrases at the bot, which the bot then repeated. So rather than being able to study how the bot learned from Twitter due to the poisoning, Microsoft had to take the bot down within 24 hours of launch. Data poisoning can impact other types of AI and ML, too. Regression analysis is often used in financial services to help manage investment portfolios. It's also used in the pharmaceutical sector to determine things like dosage for medications. Researchers in Germany recently published a paper on data poisoning attacks in regression learning and analysis. The Warfarin Dataset is used to predict the correct therapeutic dose for patients. The data set is a joint effort with 59 contributors each contributing about 1.7% of the data. By submitting around 2% poison data, about the same percentage as all the other contributors the researchers were able to increase the median error rate for recommended dosage and decrease the acceptable dosage rate. In other words they would've then been predicting the wrong dosage for patients. Poisoning attacks don't have to be targeted to work. They succeed as long as the system fails in any way to function properly. A form of machine learning denial of service is then achieved. Tay saying offensive things got a lot of attention, but if attackers had simply succeeded in having the bot spout gibberish that would've been a failure too.