Logging Cost Optimization: How to Save Money and Improve Your Monitoring

If you are using Azure Monitor to collect and analyze data from your applications and infrastructure, you might be wondering how to optimize your logging costs and get the most value out of your data. Logging costs can vary depending on how much data you ingest, how long you retain it, and what tier you choose for your Log Analytics Workspaces (LAWs). In this article, we will explore some best practices and tips to help you reduce your logging costs and improve your monitoring capabilities.

Azure Monitor Overview

Azure Monitor is a comprehensive service that provides a unified view of the health and performance of your applications and infrastructure across different Azure services and on-premises environments.

Azure Monitor consists of several components, such as:

• Log Analytics Workspace: A central repository where you can store, query, and analyze data from various sources, such as Azure resources, applications, virtual machines, containers, and more. You can use the powerful Kusto Query Language (KQL) to run complex queries and visualize the results in dashboards and reports.
• Application Insights: A service that monitors the availability, performance, and usage of your web applications and services, regardless of whether they are hosted on Azure or elsewhere. You can use Application Insights to track metrics, events, exceptions, dependencies, and more. You can also use Application Insights to perform end-to-end tracing of requests across distributed systems and diagnose issues with code-level insights.
• Metrics: A service that collects numerical data from Azure resources and custom sources at regular intervals. You can use metrics to monitor the current state of your resources and detect trends and anomalies. You can also use metrics to create alerts and notifications when certain conditions are met or thresholds are crossed.
• Alerts: A service that notifies you when something important happens in your monitored environment. You can create alerts based on metrics, logs, or other criteria. You can also configure actions to take when an alert is triggered, such as sending an email, calling a webhook, or executing a logic app.
• Notifications: A service that delivers alerts and other messages to various channels, such as email, SMS, voice call, or mobile app. You can use notifications to keep yourself and your stakeholders informed of the status and health of your monitored environment.

How Azure Monitor Components Work Together

The different components of Azure Monitor work together to provide a complete monitoring solution for your applications and infrastructure. For example:

• You can use Application Insights to monitor your web applications and services and send the data to a Log Analytics Workspace for further analysis and correlation with other data sources.
• You can use Metrics to monitor the current state of your Azure resources and send the data to a Log Analytics Workspace for historical analysis and trend detection.
• You can use Alerts to notify you when something important happens in your monitored environment and trigger actions based on the alert criteria.
• You can use Notifications to deliver alerts and other messages to various channels and recipients.

Logs vs Metrics vs Alerts vs Notifications

One of the common questions that users have when using Azure Monitor is what is the difference between logs, metrics, alerts, and notifications. Here is a brief summary of each concept:

• Logs: Logs are records of events that happen in your monitored environment. Logs can contain any type of data, such as text, numbers, JSON objects, etc. Logs are typically used for diagnostic purposes, such as troubleshooting issues or investigating incidents. Logs are stored in Log Analytics Workspaces and can be queried using KQL.

• Metrics: Metrics are numerical values that represent the state or performance of a resource or a system at a given point in time. Metrics are typically used for operational purposes, such as monitoring the current status or detecting trends and anomalies. Metrics are stored in Metrics Explorer and can be visualized using charts or graphs.

• Alerts: Alerts are notifications that something important has happened or is about to happen in your monitored environment. Alerts are typically used for proactive purposes, such as notifying you of potential problems or taking corrective actions. Alerts are created based on metrics, logs, or other criteria and can trigger actions when they are fired.

• Notifications: Notifications are messages that are delivered to various channels or recipients based on alerts or other events. Notifications are typically used for informational purposes, such as keeping you and your stakeholders informed of the health and status of your monitored environment. Notifications are configured using action groups and can be sent to email, SMS, voice call, mobile app, etc.

How to Save Money on Log Analytics Workspaces

Log Analytics Workspaces are one of the main cost drivers for Azure Monitor. The cost of a Log Analytics Workspace depends on three factors:

• Data ingestion: The amount of data that you send to a Log Analytics Workspace from various sources.
• Data retention: The length of time that you keep the data in a Log Analytics Workspace before deleting it or moving it to another tier or storage account.
• Data tier: The pricing tier that you choose for your Log Analytics Workspace, which determines the features and capabilities that you get.

Here are some tips on how to optimize these factors and save money on Log Analytics Workspaces:

Monitor what you need

One of the easiest ways to reduce your data ingestion costs is to monitor only what you need and avoid collecting unnecessary or redundant data. For example, you can use filters, sampling, or custom fields to reduce the amount of data that you send from your applications or resources to a Log Analytics Workspace. You can also use diagnostic settings to control what types of data you send from your Azure resources to a Log Analytics Workspace. You should also review your data sources regularly and remove any that are no longer needed or relevant for your monitoring scenarios. Additionally, you can use the Monitor what you need feature in Azure Monitor to get recommendations on how to optimize your data ingestion based on your usage patterns and monitoring needs. For example, you can use this feature to identify and remove any unused or duplicate data sources, adjust your sampling rates, or apply filters to exclude unwanted data.

Moreover, you can use the must have, nice to have, and not needed framework to prioritize and categorize your data sources based on their value and importance for your monitoring scenarios. For example, you can use this framework to determine which data sources are essential for your monitoring goals, which ones are useful but not critical, and which ones are irrelevant or redundant. By using this framework, you can reduce your data ingestion costs by focusing on the must have data sources and eliminating or minimizing the nice to have and not needed data sources.

Choose the right tier

Log Analytics Workspaces offer three pricing tiers: Basic Logs, Analytics Logs, and Archive Logs. Each tier has different features, capabilities, and costs. You should choose the tier that best suits your monitoring needs and budget. For example, if you only need to store your data for a short period of time and do not need advanced features such as machine learning or anomaly detection, you can choose the Basic Logs tier, which is the cheapest option. If you need to store your data for a longer period of time and use advanced features such as machine learning or anomaly detection, you can choose the Analytics Logs tier, which is more expensive but offers more value. If you need to store your data for compliance or archival purposes and do not need to query or analyze it frequently, you can choose the Archive Logs tier, which is the most cost-effective option for long-term storage.

Optimize your data retention and archive policies

Another way to reduce your data ingestion costs is to optimize your data retention and archive policies. Data retention is the length of time that you keep your data in a Log Analytics Workspace before deleting it or moving it to another tier or storage account. Data archive is the process of moving your data from a Log Analytics Workspace to another tier or storage account for long-term storage. You should set your data retention and archive policies based on your monitoring needs and compliance requirements. For example, if you only need to keep your data for a few days or weeks for troubleshooting purposes, you can set a short retention period and delete your data after that. If you need to keep your data for months or years for compliance or archival purposes, you can set a long retention period and move your data to the Archive Logs tier or another storage account after that. You can use Azure Monitor Logs retention policies and Azure Data Factory copy activity to automate these processes.

Use commitment tiers for analytics logs

One of the benefits of choosing the Analytics Logs tier for your Log Analytics Workspace is that you can use commitment tiers to get discounts on your data ingestion costs. Commitment tiers are prepaid plans that offer discounts based on the amount of data that you commit to ingest per day. Commitment tiers are only available for analytics logs and not for basic logs or archive logs. Commitment tiers start at 100 GB per day with a 15% discount and go up to 50 TB per day with a 36% discount. You should use commitment tiers if you have predictable and consistent data ingestion patterns and want to save money on your analytics logs costs.

How to Align Your Logging Costs with the CAF

The Cloud Adoption Framework (CAF) is a set of best practices and guidelines that help you plan and execute your cloud migration and adoption journey. The CAF suggests using one central Log Analytics Workspace in a separate management subscription within a separate management group for all your monitoring needs.

This approach has several benefits, such as:

• Simplifying your monitoring architecture and governance by having a single point of access and control for all your data sources and users.
• Improving your monitoring performance and scalability by leveraging the high availability and elasticity of Azure Monitor.
• Enhancing your monitoring security and compliance by isolating your monitoring data from your production data and applying consistent policies and standards across all your resources.

To align your logging costs with the CAF, you should follow these steps:

1. Create a separate management subscription within a separate management group for your monitoring needs.
2. Create a central Log Analytics Workspace in the management subscription and configure it with the appropriate tier, retention, and archive policies based on your monitoring needs and budget.
3. Connect all your data sources from different Azure services and on-premises environments to the central Log Analytics Workspace using diagnostic settings, agents, connectors, etc.
4. Configure access control and role-based access control (RBAC) policies for the central Log Analytics Workspace based on the principle of least privilege.
5. Configure alerts, notifications, dashboards, reports, etc. based on the central Log Analytics Workspace to monitor the health and performance of all your applications and infrastructure.

Summary

In this article, we have learned how to use Azure Monitor to collect and analyze data from our applications and infrastructure and how to optimize our logging costs and improve our monitoring capabilities. We have covered the following topics:

1. The overview of Azure Monitor and its components, such as Log Analytics Workspace, Application Insights, Metrics, Alerts, and Notifications.
2. The difference between logs, metrics, alerts, and notifications and how they are used for different monitoring purposes.
3. The tips on how to reduce our data ingestion costs by monitoring only what we need, choosing the right tier, optimizing our data retention and archive policies, and using commitment tiers for analytics logs.
4. The benefits of using one central Log Analytics Workspace in a separate management subscription within a separate management group as suggested by the Cloud Adoption Framework (CAF) and how to align our logging costs with the CAF.

By following these best practices and tips, we can save money on our logging costs and get the most value out of our data using Azure Monitor.

Thank you for reading this article. I hope you found it useful and informative. If you have any questions or feedback, please leave a comment below.

References

1. Log Analytics workspace overview - Azure Monitor | Microsoft Learn
2. Azure Monitor cost and usage - Azure Monitor | Microsoft Learn
3. Pricing - Azure Monitor | Microsoft Azure
4. Application Insights overview - Azure Monitor | Microsoft Learn
5. Analyze metrics with Azure Monitor metrics explorer - Azure Monitor | Microsoft Learn
6. Overview of Azure Monitor alerts - Azure Monitor | Microsoft Learn
7. Plan your Alerts and automated actions - Azure Monitor | Microsoft Learn
8. Azure Monitor Logs - Azure Monitor | Microsoft Learn
9. Metrics in Azure Monitor - Azure Monitor | Microsoft Learn
10. Analyze usage in a Log Analytics workspace in Azure Monitor - Azure Monitor | Microsoft Learn
11. Data retention and archive in Azure Monitor Logs - Azure Monitor | Microsoft Learn
12. Set a table's log data plan to Basic Logs or Analytics Logs - Azure Monitor | Microsoft Learn
13. Configure data retention for logs in Microsoft Sentinel or Azure Monitor | Microsoft Learn
14. Azure Monitor Logs cost calculations and options - Azure Monitor | Microsoft Learn
15. Microsoft Cloud Adoption Framework for Azure - Cloud Adoption Framework | Microsoft Learn
16. What is an Azure landing zone? - Cloud Adoption Framework | Microsoft Learn
17. Azure Landing Zone Conceptual Architecture Visio File