Untangle AI Model's Security Assessments

Artificial intelligence (AI) is a rapidly growing field with the potential to revolutionize many aspects of our lives. AI is already being used in a variety of applications, including healthcare, finance, transportation, and security. As AI becomes more sophisticated, it is important to consider the security implications of this technology. In this article, I've attempted to explain how security experts should perform assessment on AI models and review security challenges faced by AI models.

Biased Intension??-?Biased intention refers to the case when an AI system is trained on data with systematic errors. This can lead to the system making unfair or discriminatory decisions. For example, an AI system that is trained on data that is biased towards men may be more likely to recommend jobs to men than women. Example; In 2018, Amazon was found to have a hiring algorithm that was biased against women. The algorithm was trained on data from previous hiring decisions, and it found that men were more likely to be hired for technical roles. Amazon discontinued the use of the algorithm after the bias was discovered.

Backdoor Attacks?- Backdoor attacks are targeted poisoning attacks. A backdoor attack attempts to create a predetermined response to a trigger in an input while maintaining the performance of the system in its absence. Attack triggers in the image domain can take the form of patterns or difficult-to-see projections onto the input images like canaries. Example; This attack was first demonstrated in 2017. The attacker was able to train an image classification model that would misclassify images of cats as dogs if they contained a specific pattern. The pattern was hidden in the training data, and it was only visible to the attacker.

How dealing with those ?

Dealing with such attacks could be tedious, multiple approaches should be considered such as manually review training data, you may automate use clustering or unsupervised algorithms, cross correlational analysis by doing EDA on data to identify model biases towards target variable. Contextualize the target outcome which model is trying to solve and evaluate with some basic understanding. Bias sometimes can be unintentional an example dataset for HR Promotion from Kaggle, in this case, EDA demonstrates women are likely to be promoted than men for few roles as per analysis performed by experts, and you could see from submission people consider gender as dependent variable and that calls for biased predictions based on the data.

Evasion Attacks?- An evasion attack is an attacker's attempt to cause a misclassification during the inference phase of a machine learning model. It can be targeted or untargeted, and can take place in the physical or digital world. A brief overview of methods to create such attacks and possible defences is provided. The same attacks can be easily applied with transfer learning techniques.

Example: Change model prediction to unexpected category by adding random noise to the input like images (sample here)

Model Interference Attacks?- Interference attacks are commonly used for information extraction attacks, also known as privacy or reconstruction attacks, encompass all attacks whose objective is to reconstruct the model or information from its training data. These attacks include model theft, attribute inference, membership inference, and model inversion. Information extraction attacks frequently necessitate prior knowledge of the training dataset or access to its public portions. Outcome could be, creating dataset by fuzzing model which will help in reconstructing model decoding logic, in simple words you can connect it to known-plaintext attacks.

AI Solution Risk Assessment -?Implications of outcome should be carefully reviewed, and threat modeled before considering decision made by AI engine for critical operations. Governance has to be established around various levels allowing applications adoptions for such decision-making models. In many applications, machine learning models use sensitive information as training data or make decisions that affect people in critical areas, like autonomous driving, cancer detection, and biometric authentication.

Experts with security and risk should review model with keeping outliers in mind, what if input goes beyond expected, how does that influence outcome of the model ? Example - During medical dose prediction, if the patient's age is provided as greater than 100, which is unexpected, does this affect the dose of medication and can it cause harm?

Man-in-the-middle Attacks??- To thwart attacks, it is also essential to protect the input and output of the AI system from tampering, using measures on the hardware, operating system, and software level (in particular, installing security patches as soon as possible) as appropriate for the respective threat level. Mostly, this is a very simple attack vector that any security professional can easily attribute and attack goes beyond AI model.

Membership Inference Attacks?- In membership inference attacks primarily targeting privacy and caused by utilizing the source of the training datasets, the attacker attempts to figure out whether a data sample was included in the training data for a model. Determining the membership of an individual's data in a dataset or restoring its attributes can be sensitive from a privacy standpoint. The attack exploits differences between model behavior on fresh input data and training data. Training an attack model to recognize such differences is one way to implement such an attack.

Example - In 2016, researchers at the University of California, Berkeley, demonstrated that they could detect whether a particular image was in a training dataset for a facial recognition model.?They did this by feeding the model a set of images, some of which were in the training dataset and some of which were not. The model was able to correctly identify which images were in the training dataset with an accuracy of 90%.

Transparency, Expandability, User Concern mechanism??- The datasets used for training are not disclosed for the vast majority of published models, including LLMs and only a handful of other models has disclosed training sources. absence of fairness, transparency, and expandability of inputs, model behavior, and outputs of AI systems makes AI systems a blackbox toolkit.

Conclusion

While security and risk concerns have been discussed prior to this, it is essential to look at complete deployment for any supply chain-related risks from the following perspectives:

1. How model data is collected, analyzed, filtered and selected for trained ?
2. How model is deployed, modified and version controlling ?
3. How model is being used for interference and how performance and efficiency is constantly monitored.
4. Possibilities of any Side channel or JTag inference attacks against model incase deployed on hardware device
5. How outliners are monitored ? And input validation of model from applications.
6. Rate-limiting from application availability perspective

You have some open source tools and commercial for governance on AI models from build to deployment stages such as neptune.ai, deepchecks, roboflow etc.. and few other tools performing analysis on AI & ML models quality, performance, such insights can be easily combined and used together to get better context.

I hope this helps you get started with securing AI's models and threat modeling.