"Smart" is not enough - AI needs to be robust!

Aspects related to robustness and security of AI

Executive leadership will usually scrutinize proposed AI initiatives and solutions from the perspectives of finance (return on investment) and functionality (improvements made, for example efficiency or revenue generation). It is important, though, that from a risk perspective, the issues of security and robustness are also considered. In this respect, the two key questions regarding an AI based solution are:

• Is it solid: always delivering what it was planned to deliver?
• Is it safe: have vulnerabilities been assessed, and are the security measures both adequate, and acceptable (e.g. from the company policy and ethics perspectives)? also from an ethics and company policy point of view?

Robust and Secure AI

Robustness and security of AI participate in multidimensional trade-offs with other requirements including ethics, regulation, bias, explainability and governance. Ultimately, AI solutions represent business decisions rather than simply technical ones. That is why security and robustness related issues and solutions should also always be considered primarily from a business perspective, and that will drive the selection from multiple technical solutions available to secure the best possible fit.

Robust AI

Robustness means that the system can perform in a consistent way under different environmental circumstances. There are several way to test robustness:

• Does the system produce the same results repeatedly with the same data? If input data is not changing, the result should not change either.
• Are results as expected with similar data? Data close to the? original input should produce results falling into the same category without too large variation.
• Examine the border values to see how sensitive the model is and how large a change in inputs is needed to make changes in end results.

The outcome should be predictable. AI solutions are commonly built around machine learning models, and If these are overfitted to the original training data, the system might overreact with even minor changes on inputs leading to obsolete or irrelevant outputs. In diagnosing why a system is not performing as expected, investigating possible issues with original data used while creating the system is a good place to start.

Secure AI

Security for a system means that it is protected from abuse. AI systems are usually automated parts of decision-making processes of some sort. Abusers of the system could exploit susceptibilities in the system by working out the rules of play, entering the criteria the AI program is looking for to “play” the system and gaining desired decisions. In designing the interaction with AI, detection approaches and thresholds for such inputs need to be considered.?

Another dimension of security is the capability to protect the IP embodied in the system. As user interaction will in some cases expose aspects of the design and build of the solution, careful planning is needed in (as for any other type of software) to minimize IP exposure risks.?

Adversarial Threats

All systems are vulnerable to external threats. Often they are compromised by gaining unauthorized access to systems, but in the case of AI based systems tampering can also be? done through manipulation of input data. If AI systems are not planned properly to resist influencing through tampered data, they form an attractive target for manipulation. Basically, attack patterns can be divided into four main categories: poisoning, interference, extraction, and evasion. (These will be discussed more deeply in a separate article.) Manipulation aims to find a way to influence behaviour of an AI system.?

An adversarial attack is a threat for critical applications of AI. Such attacks can be carried out even without exact knowledge of the architecture of the architecture or how it has been implemented, so hiding these details from potential attackers is not enough. Basically, there are three different approaches to tackle the challenge: Analyzing the model robustness with measurements revealing weaknesses; making the model more resilient against attacks through training; or detecting anomalies from input data while running the system.

Deep Neural Networks (DNNs) form a specific case for adversarial threats due
to lack of transparency in the way decisions are derived from input data. AI
systems based on DNNs can achieve human-level performance on different
cognitive tasks like image recognition, object detection or converting
speech to text. DNNs are complex machine learning models simulating at a
certain level the interconnections of neurons in the human brain. DNNs are
equipped to handle high-dimensional inputs (like pixels in high-resolution
images), representing patterns in input at different levels of abstraction,
and relating the representations to high-level semantic concepts. While DNNs
in general can be very accurate, they are vulnerable to adversarial input
data i.e.? input that is deliberately modified to produce desired response
by a DNN. This can be achieved, for example, by adding adversarial noise to
the picture. The resulting picture may be indistinguishable from the original
to the naked eye, but the DNN’s? processing and reasoning is misled by the
added noise. Creators of adversarial input can use this approach to make a
DNN-based solution behave in an undesired way, causing misclassification
or a specific incorrect prediction.??

As the changes to input are hard to catch or even undetectable for humans,
they need be captured by computer. While running the model, detection
methods can be applied to identify inputs that have been tampered with by 
adversaries. Detection methods typically try to exploit abnormal activations
in the internal representation layers of a DNN initiated by these modified
inputs.

A DNN model can also be designed to become more robust against adversarial
inputs. One way to achieve this is to preprocess the inputs and thus
identify potentially maliciously modified ones. Synthetic adversarial
examples could also be utilized while training the DNN model. This approach
would enable the model to possibly identify and classify adversarial inputs
separately from normal inputs. Changes can also be made to the DNN
architecture to prevent propagation of adversarial signals through the
internal representation layers.

The robustness of a DNN against adversarial attack can be assessed by
recording the loss of accuracy while receiving adversarial altered inputs.
Another way to assess robustness is to make small changes to input and
measure the delta between internal representation and the output of a DNN.        

Accuracy vs Robustness

While training AI the machine learning models at the heart of an AI solution, a typical goal is to maximise their accuracy. Without considering robustness of the solution at the same time, however, the end result could be highly accurate but more vulnerable to adversarial attacks. A study carried out by Dong Su et.al. : “Is Robustness the Cost of Accuracy? A Comprehensive Study on the Robustness of 18 Deep Image Classification Models”, ?reveals an interesting fact about relation between accuracy and robustness:

• Model architecture is more critical to robustness than model size, and the disclosed accuracy-robustness Pareto frontier can be used as an evaluation criterion for model designers.

Increasing accuracy is often a tradeoff for robustness, but there are some opportunities to mitigate that impact. The best option is naturally to control both axes of this equation while building the models to find a balance point, where decision making performance is acceptable and residual risk is bearable from a business point of view.

Conclusions

All AI related decisions are primarily business decisions. Technology is just a tool to fulfill the targeted business goal. The desired end state also dictates the measures needed for robustness and security of the planned solution.

Decision makers must use their judgment to balance the benefits of using artificial intelligence against potentially emerging issues related to security and robustness. Normal risk management processes should be carried out to mitigate the risks where feasible. It is important to understand the residual risk after selected mitigation actions have been carried out.

The intended purpose of any AI system and the delivered solution’s fit to it must be carefully analyzed. The system should be able to work dependably in a variety of circumstances, with the responses produced validated as replicable and reliable. It must be resilient against being purposefully misled through altered inputs.?

Increasing accuracy is often a tradeoff for robustness. While teaching the model and planning the architecture of the AI system, you need to consider what is the right balance from a business point of view. It might also be difficult to justify why results are accurate. Technical inerrability is not enough. Careful consideration is needed to understand how and to whom results need to be explained. The ethical and legal implications need also to be considered while thinking about the explainability of the results. (We will be covering the issues around explainability in more detail in a future article.)

The security of an AI system should be designed at a level capable of protecting the business process it is supporting from both manipulation and misappropriation. Also, the security of the IP embedded into the AI system has to be considered, given its likely value as a core business asset.

Adversarial attacks are a relevant threat which AI systems need to be prepared to cope with. There are three different approaches to fight against adversarial attacks:?

• While implementing: Prepare an AI system against adversarial attacks by training the model to identify it.?
• When implementation is ready: Find weaknesses by running measurements and analyzing the AI system.?
• While running: Detect anomalies from input data before delivering that to the AI system.