Adversarial machine learning isn't new. Attacks like data poisoning (manipulating an algorithm through its data) is a very real risk and one that we've been defending against as long as "AI" has been around (criminals were using it to avoid spam sensors in the early 2000's and search engine autocomplete attacks have a whole industry around them). But it's started popping up recently at various conferences so I figured I'd drop a quick note on some of the things we advise clients to do to protect themselves.
I'm not going to go much into the how of attacks - just search using DuckDuckGo for "adversarial machine learning" and you can get down to whatever level you feel comfortable, but suffice to say it's going to become one of the more favored and disruptive attacks as "advanced" AI becomes more and more embedded into our day to day lives.
If you want a simple example of how an algorithm can be manipulated by its data, you need go no further than the folks at Microsoft, who introduced Tay, a chatbot. Within 24 hours it had been turned into a misogynistic racist asshole because of what was being tweeted at it. Was Tay deliberately manipulated or was it just the fact social media is a cesspit? Opinions differ. Either way, it was yanked offline pretty quickly.
Now Microsoft can afford such an expensive failure. But can you?
So what are some things we can do to defend against these and other types of attacks?
- Get your team up to speed. Security teams haven't historically had a lot of interaction with AI / ML programs. But understanding a technology and its risks is half the battle. So make sure your security folks are integrated into your AI projects as early as possible, and given the opportunity to learn its weaknesses and strengths. Get them training in how models can be compromised, and bring AI into your overall security program, not just as a technology you can use, but as one you need to defend.
- Protect and understand your data sets (both in development & production environments). This sounds relatively simple, but be aware that only a small sample of data has to be manipulated in order to trigger the response that the attacker wants (shifting the model accuracy or installing a backdoor for example), and finding that data in a massive dataset may be prohibitively expensive. Plus datasets may be sourced outside your business so you may not know the origin.
- Talking of knowing the origin, know where your models came from and how they work (which sometimes isn't as simple as it sounds) - models are often open sourced and attackers love to get in on that supply chain. If you must use a 3rd party model, ensure you have the expertise to understand how it has been constructed, what components it uses and its development lineage. Did it come from a reputable source or was it found on some random GitHub repository? If the bad guys have a white box (they know or have already manipulated the algorithm) they have a lot more options than a black box attack (where they know nothing about the model).
- Find reputable developers & data scientists who are aware of the risks. This is DevSecOps, the revenge. Same issues, same solutions. Ideally they should also have access to experienced security practitioners internal to their team, in the enterprise security team or have solid 3rd party consultancy relationships.
- Don't think your current anti-malware solutions will help. They won't. You will need new tools and approaches (sorry, more work). There's a whole ecosystem starting to develop around this area with major players looking at solutions. And don't overlook the potential for your models to be used to deploy malware. Sound far fetched? It isn't and is being actively considered as a future vector.
- Be prepared to analyze and filter all data coming into your model. Regular or large datasets coming from a limited number of IP addresses, users or domains may be a warning flag (and trigger rate limiting, automated / manual moderation or quarantining). You can also use statistical analysis (regression testing etc.) and input validation. As always, ensure it is appropriate to your operational requirements, risk appetite and technical capabilities. You don't want your AI baby to fail because you don't understand the impact of your protection mechanisms.
- Maintain good operational security. The more an attacker knows about your model and how it has been constructed, the more efficient and effective their attacks will be. Designs, documentation, development platforms, source code, engineering notes etc. should all be secured as with any intellectual property. And don't go bragging about your fancy AI at some conference. Many a drunk developer or product manager has let slip way too much at get togethers, and it's not just the good guys attending!
- Practice good cyber hygiene. Least privilege access is a great idea. Segment those networks before someone segments them for you. Multi factor authentication for the development platforms and repositories would be a service to all. Configure the logs, and for the sake of your children, keep the logs for longer than a week! (a year would be nice if you can spare some extremely cheap, but secure, offline storage). And don't forget the old stalwart of development - versioning. You can version your code AND your data. I know, call me crazy, but knowing exactly what version of data was used on a model will put you leaps and bounds above your run of the mill machine intelligence mob. You will be a king amongst AI practitioners.
- Consider how you will remove "bad" data from your solutions. Whilst not yet a science (it's more of an academic study) the concept of machine unlearning is starting to take hold. In this scenario you would be able to annex off any corrupted data and instruct the model to "undo" the effects and weights of that data without having to completely retrain the model (which can be ridiculously expensive). As I said not quite there yet, but one to watch.
This isn't a comprehensive list my any means, and people better than I will be able to provide you with even more insight. But it's a good start on a difficult subject, and we all have much to learn.
Good luck, and may the Singularity speed you to a healthy and prosperous artificial future.
