Article 2: Spotting and Defining Generative AI Risks in Your Projects

By Shantanu Singh, in creative collaboration with multiple AI Systems.

Executive Summary

Identifying risks early is half the battle in managing generative AI. This article (2/5) explores how to spot and define risk issues during project development before they escalate. I explain why early identification is crucial, offer guidance on recognizing "risk clues" in product plans, and provide techniques for defining the context and scope of AI risks. Readers will learn how to overcome common hurdles in the risk identification phase and gain insights from NIST's workshops and industry examples. Finally, this article outlines actionable steps for teams to define their AI risk landscape, detect warning signs, analyze issues, make informed decisions, and mitigate risks effectively.

Introduction: The Value of Early Risk Identification

In AI projects involving generative models, catching issues early saves time, money, and reputation.

Generative AI systems (like chatbots, image generators, or code assistants) can produce unexpected failures. If risks aren't identified until after deployment, consequences range from minor glitches to major incidents.

NIST's AI Risk Management Framework emphasizes contextual understanding as the first step in its "Map" function.

Before building an AI system, teams should map intended use, environment, and stakeholders to derive potential risk scenarios. For generative AI, this mapping exercise must be especially comprehensive due to the models' versatility and far-reaching impacts.

NIST's Generative AI Profile provides multiple risk areas unique to generative AI, giving organizations a head start on potential issues. However, teams must relate these generic risks to their specific projects, requiring technical understanding, domain knowledge, and creative thinking about worst-case scenarios.

Whether you're a product manager, lawyer, or data scientist, you should ask:

"What are we missing? Could this feature introduce problems later?" While perfect foresight is impossible, the goal is to push the horizon as far as possible using available knowledge and set up processes to catch unforeseen issues.

Why Early Identification is Critical

Catching AI issues early offers several benefits:

1. Cost-effectiveness: Mitigating risks during design is cheaper than fixing problems after deployment.
2. Regulatory compliance: Many AI regulations expect organizations to perform risk assessments before release.
3. Trust building: Both internal teams and external stakeholders gain confidence in products with thorough risk management.

Generative AI specifically benefits from early risk mapping because:

• Issues discovered during development can be addressed by adjusting training data or fine-tuning strategies
• The general-purpose nature of these models means users might employ them in unexpected ways
• Learning from previous incidents (what NIST calls "transfer learning in risk") allows teams to anticipate problems

Microsoft's integration of GPT-4 in Bing search illustrates the value of early risk identification. Learning from their earlier "Tay" chatbot incident, the Bing team anticipated potential issues and implemented safeguards before release. When users still managed to extract odd responses, Microsoft had already prepared mitigation plans.

Listening for Risk Clues in Product Features

Certain features and requirements serve as "risk clues" signaling potential problems.

"The AI will operate autonomously without human oversight." This signals risks of uncontrolled behavior or lack of fail-safes. Mitigations might include human-in-the-loop processes or robust monitoring.

"Uses customer data to personalize responses." This indicates privacy and security risks, including potential data leakage or unauthorized use. Consider data anonymization, differential privacy, or strict access controls.

"Generates content in the style of [X]." If X is copyrighted material, this raises intellectual property risks. Consider narrowing the feature or securing proper licensing.

"Real-time interaction with users." This increases risk of unpredictable inputs leading to harmful outputs. Implement content moderation, response filtering, and escalation paths.

"AI output feeds into downstream automated processes." Errors can have cascading effects when outputs trigger further actions. Build checkpoints and validation processes.

"Continuous learning or self-update." Models that retrain on new data risk degradation or manipulation over time. Implement continuous evaluation and constraints on learning.

Many organizations now use "AI risk checklists" for product managers, similar to privacy impact assessments. These structured questions prompt teams to document risks systematically.

Defining the Risk in Context

After spotting a risk, you must define it clearly by articulating:

• The nature of potential harm
• Conditions under which it might occur
• Stakeholders or assets impacted
• Estimated likelihood and severity

This transforms abstract concerns into concrete scenarios that can be analyzed and addressed. NIST's ARIA program emphasizes this contextualization—understanding not just that a risk exists, but how it manifests in a specific setting.

For example, with an AI medical symptom checker, you might define a risk as:

• Nature: AI might misdiagnose or give incorrect medical advice
• Conditions: Could occur with gaps in medical knowledge, misleading symptom descriptions, or model confabulation
• Stakeholders: Patients (health risks), company (liability), healthcare providers (disrupted care)
• Severity/Likelihood: Moderate likelihood but high severity, making it a high-priority risk

This definition clarifies that the risk requires significant controls, such as medical disclaimers, conservative design that errs toward recommending professional care, or limiting scope to non-critical issues.

Leveraging established taxonomies like NIST's Generative AI Profile can help sharpen your definitions. Operational environment and dependencies also provide important context—an AI running on a mobile device faces different risks than one in a secure cloud.

Many organizations now use "AI risk checklists" for product managers, similar to privacy impact assessments. These structured questions prompt teams to document risks systematically.

Defining the Risk in Context

After spotting a risk, you must define it clearly by articulating:

• The nature of potential harm
• Conditions under which it might occur
• Stakeholders or assets impacted
• Estimated likelihood and severity

This transforms abstract concerns into concrete scenarios that can be analyzed and addressed. NIST's ARIA program emphasizes this contextualization—understanding not just that a risk exists, but how it manifests in a specific setting.

For example, with an AI medical symptom checker, you might define a risk as:

• Nature: AI might misdiagnose or give incorrect medical advice
• Conditions: Could occur with gaps in medical knowledge, misleading symptom descriptions, or model confabulation
• Stakeholders: Patients (health risks), company (liability), healthcare providers (disrupted care)
• Severity/Likelihood: Moderate likelihood but high severity, making it a high-priority risk

This definition clarifies that the risk requires significant controls, such as medical disclaimers, conservative design that errs toward recommending professional care, or limiting scope to non-critical issues.

Leveraging established taxonomies like NIST's Generative AI Profile can help sharpen your definitions. Operational environment and dependencies also provide important context—an AI running on a mobile device faces different risks than one in a secure cloud.

Overcoming Unknowns in Risk Identification

Even with thorough processes, generative AI will inevitably surprise us. To mitigate "unknown unknowns":

1. Diverse Brainstorming: Include people with different backgrounds to broaden perspectives.
2. Analogous Cases: Study failures of similar systems and past incidents.
3. Iterative Prototyping: Build prototypes and test them specifically to uncover hidden issues.
4. Broad Monitoring: Set up feedback channels to catch unexpected behaviors early.
5. Multiple Techniques: Combine methods like checklists, scenario analysis, and adversarial testing.

You won't catch everything, but a strong early identification process minimizes unknowns and establishes a risk-aware culture that responds quickly to emerging issues.

Examples from NIST and Industry

Bias in Hiring AI: In NIST workshops, participants highlighted risks of historical bias in hiring algorithms. By identifying this early, a company collected additional training data from underrepresented groups and implemented bias mitigation techniques. They defined the risk as "algorithmic monoculture" and added diversity testing requirements to their development process.

Hallucination in Financial Chatbot: A bank deploying an AI chatbot identified the risk of "confabulation"—the bot fabricating answers about account information or policies. They constrained the bot to only answer from a verified FAQ database and implemented verification checks. This risk was identified not just by engineers but by call center staff who understood customer question patterns.

Data Poisoning via User Input: A company developing a code generator that learns from user input identified the risk of malicious actors feeding harmful patterns. Early identification led them to disable continuous learning in the released version, implement periodic retraining with data cleaning, and add monitoring for poisoning attempts.

5 Actionable Steps for Teams

1. Define: Hold cross-disciplinary risk brainstorming sessions during design. Document each risk with context, priority, and potential triggers in a living risk register.
2. Detect: Create test plans aligned to your risk list. Design QA processes to specifically look for defined risky behaviors. Update risk assessments based on test results.
3. Analyze: Promptly investigate issues encountered during testing. Determine if they confirm known risks or reveal new ones. Involve appropriate experts to diagnose root causes and recommend adjustments.
4. Decide: For each significant risk, choose to mitigate now, accept temporarily, or avoid entirely. Document these decisions and their rationales.
5. Act: Implement decisions during development. Add necessary controls, improve processes, and train teams. Before launch, conduct "tabletop simulations" of worst-case scenarios to ensure preparedness.

Continue this cycle after deployment. The practices developed during early identification will help teams remain alert to new issues in production.

Conclusion

Spotting and defining risks in generative AI projects requires both creative anticipation and structured analysis. By recognizing early warning signs in product plans and rigorously defining risks in context, risk professionals can transform vague concerns into manageable problems.

NIST's frameworks illuminate potential pitfalls based on collective experience. As AI professionals, we should incorporate risk identification into the DNA of project management—making it part of kickoff meetings, design documents, and development sprints.

This approach isn't about discouraging generative AI use but ensuring its thoughtful implementation. By diligently mapping potential problems before deployment, we navigate the uncharted waters of generative AI more safely, balancing innovation with responsible risk management.

The content is for educational purposes and not intended as legal advice or to establish an attorney-client relationship. No content represents advice provided to past or current clients.

Article 3 will explore overcoming challenges in implementing AI risk controls.