Azure OpenAI Shield: Strengthening Security Infrastructure with Advanced Monitoring and Logging for Enterprise Deployments

In the realm of enterprise solutions, the paramount significance of logging and monitoring cannot be overstated. These critical components form the bedrock of a secure and resilient system, particularly in the context of deploying the Azure OpenAI Service API.

For sizable corporations employing generative AI models, it is imperative to establish a system for auditing and logging the utilization of these models. This measure is crucial for fostering responsible use and aligning with corporate compliance standards.

The proposed solution offers a comprehensive logging and monitoring framework tailored for enterprise needs, effectively tracking all interactions with AI models. This not only serves to mitigate any potential misuse but also ensures adherence to rigorous security and compliance standards. Notably, the solution seamlessly integrates with established APIs for Azure OpenAI, requiring minimal modifications to leverage existing code bases. Administrators additionally gain the capability to monitor service usage for comprehensive reporting purposes.

This amalgamation not only facilitates advanced tracking of API usage and performance but also establishes robust safeguards to shield sensitive data and proactively deter malicious activities.

Workflow

1. To engage in text generation (completions) and model training (fine-tuning), client applications interface with Azure OpenAI endpoints.
2. The Azure Application Gateway serves as the singular entry point for Azure OpenAI models, furnishing load balancing functionalities for APIs. It's essential to note that load balancing for stateful operations, such as model fine-tuning, deployments, and inference of fine-tuned models, is not supported.
3. Facilitating comprehensive security controls, auditing, and monitoring for Azure OpenAI models, Azure API Management employs the following measures: a. Enhanced-security access is granted in Azure API Management through Microsoft Entra groups, utilizing subscription-based access permissions. b. Auditing is activated for all interactions with the models by utilizing Azure Monitor request logging. c. The monitoring feature supplies detailed KPIs and metrics for Azure OpenAI model usage, encompassing prompt information and token statistics, enhancing traceability of usage.
4. Azure API Management establishes connections to all Azure resources through Azure Private Link, enhancing security by routing traffic via private endpoints and confining it within the private network.
5. Ensuring high availability and disaster recovery for the service, multiple instances of Azure OpenAI are deployed, enabling scalable API usage and robust redundancy measures.

Components

1. Application Gateway: Functions as an application load balancer, ensuring optimal response times and maximum throughput for users accessing Azure OpenAI APIs, particularly for model completions.
2. API Management: Serves as a comprehensive platform for accessing backend Azure OpenAI endpoints, offering monitoring and logging capabilities not inherently available in Azure OpenAI. Facilitates seamless integration with enterprise-scale applications.
3. Azure Virtual Network: Establishes a private cloud network infrastructure, providing network isolation for models. Ensures that all network traffic related to the models is privately routed to Azure OpenAI, enhancing security and confidentiality.
4. Azure OpenAI: Hosts models and delivers generative model completion outputs. Serves as the core service for AI model functionalities in the enterprise environment.
5. Monitor: Offers end-to-end observability for applications, granting access to application logs via the Kusto Query Language. Features dashboard reports and robust monitoring and alerting capabilities, ensuring a comprehensive understanding of system performance.
6. Azure Key Vault: Functions as an enhanced-security storage solution for keys and secrets utilized by applications. Safeguards sensitive information, contributing to overall application security.
7. Azure Storage: Provides cloud-based storage for applications, offering accessibility to model training artifacts for Azure OpenAI. Enhances the efficiency of model training processes.
8. Microsoft Entra ID: An enhanced-security identity management solution, facilitating user authentication and authorization for applications and supporting platform services. Implements Group Policy to enforce the principle of least privilege, ensuring secure access for all users within the enterprise.

Detailed Implementation

Alternative Solution

Azure OpenAI comes equipped with integrated logging and monitoring tools. While these built-in features enable the tracking of service telemetry, it's crucial to recognize that the default cognitive service logging lacks the capability to record inputs and outputs, such as prompts, tokens, and models.

These particular metrics play a vital role in compliance adherence and verifying the service's expected functionality. Moreover, by scrutinizing interactions with the extensive language models deployed on Azure OpenAI, organizations can gain insights into usage patterns, aiding in the identification of cost factors and informing strategic decisions related to scaling and resource distribution.

Query to Track Usage Monitoring

The below query retrieves and analyzes usage information for the Azure OpenAI service from the ApiManagementGatewayLogs metric table.

The query focuses on logs related to the 'completions_create' operation, extracts relevant information from the logs, and then summarizes usage metrics such as total prompt tokens, total completion tokens, total tokens, and average total tokens for each unique combination of IP address and model.

ApiManagementGatewayLogs
| where OperationId == 'completions_create'
| project model = tostring(parse_json(BackendResponseBody)['model']),
          prompttokens = todecimal(parse_json(parse_json(BackendResponseBody)['usage'])['prompt_tokens']),
          completiontokens = todecimal(parse_json(parse_json(BackendResponseBody)['usage'])['completion_tokens']),
          totaltokens = todecimal(parse_json(parse_json(BackendResponseBody)['usage'])['total_tokens']),
          ip = CallerIpAddress
| summarize
    TotalPromptTokens = sum(prompttokens),
    TotalCompletionTokens = sum(completiontokens),
    TotalTokens = sum(totaltokens),
    AverageTokens = avg(totaltokens)
    by ip, model
        

Output:

Prompt Usage Monitoring Query

ApiManagementGatewayLogs
| where OperationId == 'completions_create'
| project model = tostring(parse_json(BackendResponseBody)['model']),
          prompttokens = todecimal(parse_json(parse_json(BackendResponseBody)['usage'])['prompt_tokens']),
          prompttext = substring(parse_json(parse_json(BackendResponseBody)['choices'])[0], 0, 100)
        

Output:

Implementation Considerations

The below factors embody the principles outlined in the Azure Well-Architected Framework, serving as foundational guidelines to enhance the quality of a workload.

Reliability:

In the context of enterprise-scale utilization of Azure OpenAI, the focus on reliability is paramount. This entails maintaining a high level of availability for the expansive language models, crucial for serving the diverse needs of enterprise users.

The Azure application gateway plays a pivotal role in delivering a robust layer-7 application mechanism, ensuring swift and consistent access to applications. API Management comes into play for configuring, managing, and monitoring access to models, contributing to the overall reliability of the system.

Security:

Security considerations are paramount to safeguarding against deliberate attacks and protecting valuable data and systems within an enterprise. In this scenario, best practices are implemented for both application-level and network-level isolation of cloud services, effectively mitigating the risks associated with data exfiltration and leakage. Specifically, all network traffic containing potentially sensitive data input to the model is isolated within a private network, eliminating exposure to public internet routes.

The inherent high availability of fundamental platform services such as Storage, Key Vault, and Virtual Network further fortifies the reliability of the application. Introducing multiple instances of Azure OpenAI adds an extra layer of resilience, safeguarding against potential application-level failures. Collectively, these architectural components collectively contribute to establishing and maintaining the reliability of the enterprise-scale application.

Accountability:

Accountability in the context of Azure OpenAI at an enterprise scale involves establishing clear responsibility and tracking mechanisms to ensure transparency and traceability of actions. This entails implementing practices that enable the identification of individuals or entities responsible for specific activities within the system. Network isolation and accountability go hand in hand, as the former ensures secure and controlled access, while the latter involves tracking and attributing actions to specific actors. By fostering a culture of accountability, enterprises can enhance their ability to detect and respond to security incidents, ultimately contributing to a more robust and secure operational environment.