Managing Risks associated with AI

The recent Standford publication (Stanford CRFM) and Whitehouse directive (On the heels of Biden's executive order, agencies get White House directive on implementing AI - Nextgov/FCW) made last week an action filled week over and above all that is keeping us excited on the rapidly evolving world of AI. I wanted to use these events to step back and look at the managing risks with the AI models we're using (confession of a tesla fan), implementing in products or experimenting with the intent to implement.

What is AI?

I'm sure we've read it over and over again but let us define it to remove any scope for assumptions. Artificial Intelligence is a branch of computer science that aims to create machines capable of intelligent behavior. In other words, it's about making computers do things that would need intelligence if done by humans. This includes tasks like understanding language or images, making decisions, and learning from experience.

What are Foundation Models?

Foundation models are new AI technologies that have gained prominence due to their ability to learn (by themselves) from a vast amount of data on a wide range of topics. Unlike older AI models, that are designed for specific tasks, foundation models can adapt, learns from feedback and apply their knowledge to many different tasks. They are called "foundation" because they serve as a base for building other specialized AI models.

Examples of Foundation Models:

1. BERT: An AI that understands the context of words in search queries.
2. DALL-E: An AI that can create images from textual descriptions.
3. GPT-3: An AI that can write essays, poems, or even computer code, based on a prompt.

Why are Foundation Models Important?

The Foundation models represent a significant shift in AI because they can be fine-tuned to perform tasks they are not explicitly trained to do - not narrow AI. For example, GPT-3 can write an essay on a topic it has never explicitly learned about because it understands language patterns.

Risk Management in AI

As AI continues to become prevalent in our lives (as Apps, experiences and tools), it is important to ensure that AI systems are reliable, secure, and safe. This demands a complete risk management approach such that all possible risks that may arise from the use of AI are considered/mitigated before going live. Integrating risk management from inception to development process of AI systems is essential for ensuring that AI applications are used responsibly and ethically.

Challenges with Foundation Models:

• Transparency: It's often unclear how these models make their decisions. Since they are so complex, even the designers may not fully understand every detail of their decision-making process. Imagine you're using an app to find the best route to a new restaurant. You trust the app because it usually gets you to your destination. But what if one day it takes you through a series of complicated turns and you end up at a dead end? (has happened to many of us in other countries, do comment ??) You'd want to know why it chose that route, right? Transparency in AI is similar. With foundation models, it's like having a powerful app that can do a lot more than just navigate – it can write, understand speech, and even create images. But just like with most apps, we often can't see "under the hood" to understand why it makes the decisions it does. This can be a large problem if these AI models make mistakes when they're used in critical areas like medicine, operating machinery/vehicles or law where the stakes are high.
• Explainability: Related to transparency, explainability is about being able to describe how the AI arrived at a particular decision in a way that humans can understand. This is important when AI is used in areas like healthcare, operating machines or legal decisions to name a few, where understanding the rationale is important. Let's take the example of playing a game of chess against a computer, and it makes a move that causes it to lose a piece. One might wonder, "Why did it do that?". If the computer/AI program could explain its strategy or move, it can be a great learning moment for the human involved. Explainability in AI is about making the computer's "thought process" clear to us. With foundation models, the "thought process" is so complex that even the creators might struggle to explain why the AI made a certain choice. This is tricky because if AI is going to be a part of our daily lives, we need to understand why it does what it does (why take route-A vs route-B, something we see everyday), just like we'd expect a human expert to explain their reasoning.
• Ethics: There are concerns about AI perpetuating biases present in the data they are trained on. For example, if an AI is trained on historical hiring data, it might learn and replicate past discriminatory practices. Ethical AI should be fair and not reinforce negative stereotypes or biases. Ethics is all about making sure AI behaves or recommends in a way that is fair and just or unbiased. For example, consider a hiring tool that's supposed to help companies find the best candidates. If this tool was trained on data that included biased human decisions from the past, it might continue to favor or discriminate against certain groups of people without any human telling it to do so. We need to be very careful about the data we use to train them and constantly check to make sure they're not picking up and spreading these "bad precedence". Another area brewing trouble is the topic of IP ownership with code generated by AI, something that's going to soon be a new debate.

Risk management being the practice of identifying, assessing, and mitigating potential risks associated with the use of a given technology, when it comes to AI, risk management involves identifying and addressing potential risks associated with the use of AI systems and applications. We should have an approach to manage the risks associated with development, deployment, use, debugging and maintenance of AI based systems/applications.

To appreciate the need for risk management in AI, let's compare AI to a new driver. A new driver can learn the rules of the road and how to operate a car, but they lack experience. Similarly, AI can learn tasks but may not fully understand the complexities of real-world applications (remember the statement in most foundation models?). Risks can range from minor errors, like misinterpreting a voice command, to major issues, like an autonomous vehicle failing to recognize a stop sign or worse yet a pedestrian / pet.

Once the risks associated with an AI system/application are identified, it is important to develop a plan to mitigate them. The mitigation plan should include approaches for reducing the risk of data breach, ensuring ethical use, and assessing the impact on humans. The plan must include mitigations against legal and regulatory risks, which are being sought to be regulated by Governments across the globe too.

Exploring Risk Management for AI

Incorporating risk management into AI is like giving our new driver a detailed map, a well-maintained car, and a set of rules to follow for a safe journey. As AI evolves, imbibing these practices into our development process will help harness its potential while safeguarding against the risks. We can confidently move towards a future where AI contributes positively to society, driving progress and innovation in a responsible way. Let us look at key pillars to this approach -

1. Transparency: To achieve transparency in AI, developers can adopt open-box models whenever possible, where the inner workings of the AI are accessible and understandable (thinking equations, weights, neural networks?). This involves using algorithms that are inherently more interpretable, such as decision trees, which mimic human decision-making processes. For more complex models, like neural networks, techniques such as layer-wise relevance propagation can help illustrate which inputs most significantly affect the output. Documentation is also crucial; maintaining comprehensive records of the AI's development process, including data sources, model choices, and version histories, can provide clarity on how the AI system operates.
2. Explainability: Explainability requires AI to articulate its decision-making process in a way that is understandable to the average user. This can be achieved by developing companion explanatory models that translate complex AI decisions into simpler terms. For instance, after an AI model makes a decision, a secondary model could generate a human-readable explanation. Another method is to use feature importance scores, which highlight the most influential factors in the AI's decision. Interactive tools that allow users to tweak inputs and observe changes in outputs can also help demystify the AI's reasoning.
3. Ethical Considerations: Managing ethical risks involves embedding ethical decision-making within the AI's operational framework. This starts with diverse and inclusive training data to minimize biases. Regular audits by interdisciplinary teams, including ethicists, can assess whether the AI's decisions align with ethical standards. Ethical guidelines should be established, reflecting societal values and moral principles, and these guidelines should be programmatically integrated into the AI's functionality. Additionally, mechanisms for human intervention should be in place to correct the AI's course when needed.
4. Robustness and Safety: To ensure robustness and safety, AI systems should undergo rigorous testing, including stress tests and scenario analyses, to evaluate how they perform under various conditions. Techniques like adversarial training, where the AI is exposed to difficult or deceptive inputs, can strengthen its ability to handle real-world unpredictability. Redundancy can also be built into the system, so if one part fails, another can take over. Safety protocols, such as emergency stop functions, can provide an additional layer of security.
5. Data Security: Protecting the data involved in AI systems is critical. Encryption can secure both the data used to train AI and the data generated by AI. Access controls, such as authentication and authorization mechanisms, ensure that only authorized personnel can interact with the AI system. Regular security audits and penetration testing can identify and rectify vulnerabilities. Additionally, data anonymization techniques can be used to protect individual privacy when AI processes personal information.
6. Monitoring and Evaluation: Continuous monitoring involves setting up systems to oversee the AI's performance in real-time, flagging any anomalies or drifts in behavior. Evaluation should be an ongoing process, with performance metrics reviewed regularly against benchmarks. Feedback loops where the AI's outputs are assessed and used to refine its models can help maintain accuracy. User feedback can also be a valuable source of information for evaluating the AI's effectiveness and user-friendliness.
7. Legal Compliance: Staying compliant with legal standards requires staying informed about the evolving landscape of AI regulation. Legal experts, along with AI developers, should work together to interpret how laws apply to AI systems (the paper Stanford CRFM seems to be heading in that direction). Compliance checklists and regular legal audits can ensure that AI systems adhere to all relevant laws and regulations. When international use is involved, AI systems must be adaptable to comply with the legal frameworks of different countries.

The report from the Center for Research on Foundation Models at Stanford University (Stanford CRFM) presents a comprehensive discussion on the critical aspects of foundation models, from technical underpinnings to societal consequences. It stresses on the importance of a proactive and holistic approach to identifying/managing the risks associated with AI systems. The ubiquity and versatility of these models make them both exciting and concerning due to their task agnosticism — their ability to perform a wide range of tasks without being specifically programmed for them. By addressing each of the seven pillars —transparency, explainability, ethical considerations, robustness and safety, data security, monitoring and evaluation, and legal compliance — stakeholders can work towards a responsible development / deployment of AI, ensuring that it serves for the greater good while mitigating potential harms. It is clear that these models/technologies are a gamechanger in the AI space. There is a need for implementors/users to be watchful to ensure that these models are developed, used, and managed responsibly. The report provides a good foundation to build towards a responsible approach to the development / deployment of AI. It'll be interesting to see how this impacts policy and shapes the course for legislature towards IP & responsible use of AI.

* All views represented in this blog are purely personal & not those of my employer. Please comment?or InMail me.

References

1. Stanford CRFM
2. report.pdf (stanford.edu)
3. HAI Policy & Society Issue Brief - Improving Transparency in AI Language Models.pdf (stanford.edu)