Analyze usage and cost in Log Analytics?-?Microsoft?Sentinel

Azure Monitor costs can vary significantly based on the volume of data being collected in your Log Analytics workspace. This volume is affected by the set of solutions using the workspace and the amount of data collected by each. This article provides guidance on analyzing your collected data to assist in controlling your data ingestion costs. It helps you determine the cause of higher-than-expected usage and also to predict your costs as you monitor additional resources and configure different Azure Monitor features.

Causes for higher than expected?usage

Each Log Analytics workspace is charged as a separate service and contributes to the bill for your Azure subscription. The amount of data ingestion can be considerable, depending on the following factors:

• Set of insights and services enabled and their configuration
• Number and type of monitored resources
• The volume of data collected from each monitored resource

An unexpected increase in any of these factors can result in increased charges for data retention. The rest of this article provides methods for detecting such a situation and then analyzing collected data to identify and mitigate the source of the increased usage.

Usage analysis in Azure?Monitor

You should start your analysis with existing tools in Azure Monitor. These require no configuration and can often provide the information you require with minimal effort. If you need a deeper analysis of your collected data than existing Azure Monitor features, you use any of the following log queries in Log Analytics.

Data volume by?solution

Analyze the amount of billable data collected by a particular service or solution. These queries use the Usage table that collects usage data for each table in the workspace.

Billable data volume by solution over the past month

Usage
| where TimeGenerated > ago(32d)
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())
| where IsBillable == true
| summarize BillableDataGB = sum(Quantity) / 1000. by bin(StartTime, 1d), Solution
| render columnchart        

Billable data volume by type over the past month

Usage
| where TimeGenerated > ago(32d)
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())
| where IsBillable == true
| summarize BillableDataGB = sum(Quantity) / 1000. by bin(StartTime, 1d), DataType
| render columnchart        

Billable data volume by solution and type over the past month

Usage
| where TimeGenerated > ago(32d)
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())
| where IsBillable == true
| summarize BillableDataGB = sum(Quantity) / 1000 by Solution, DataType
| sort by Solution asc, DataType asc        

Billable data volume for specific events If you find that a particular data type is collecting excessive data, you may want to analyze the data in that table to determine particular records that are increasing. This example filters particular event IDs in the Event table and then provides a count for each ID. You can modify these queries using the columns from other tables.

Querying for common data?types

If you find that you have excessive billable data for a particular data type, then you may need to perform a query to analyze data in that table. The following queries provide samples for some common data types:

Security Solution

SecurityEvent
| summarize AggregatedValue = count() by EventID        

Use a custom date range.

We can clearly see in the “SecurityEvent” table that windows filtering events are in huge sums. To mitigate this we just need to reconfigure our windows data connector to collect minimal logs.

Data Collection Filtering Capabilities

The old connector is not flexible enough to choose what specific events to collect. For example, these are the only options to collect data from Windows machines with the old connector:

• All events — All Windows security and AppLocker events.
• Common — A standard set of events for auditing purposes. The Common event set may contain some types of events that aren’t so common. This is because the main point of the Common set is to reduce the volume of events to a more manageable level, while still maintaining full audit trail capability.
• Minimal — A small set of events that might indicate potential threats. This set does not contain a full audit trail. It covers only events that might indicate a successful breach and other important events that have very low rates of occurrence.
• None — No security or AppLocker events. (This setting is used to disable the connector.)

Log Management Solution

Usage
| where Solution == "LogManagement" and iff(isnotnull(toint(IsBillable)), IsBillable == true, IsBillable == "true") == true
| summarize AggregatedValue = count() by DataType        

As we can see our log management solution is not aggregating a lot of events so we’ll leave it there.

Price Calculator

Calculate your estimated hourly or monthly costs for using Azure, this is highly recommended to plan and manage costs across your Azure cloud environment.

Calculate your estimated hourly or monthly costs for using Azure using Microsoft Azure Pricing Calculator.

Conclusion

Microsoft Sentinel costing can be tedious if left unchecked, cost optimization is a huge topic to cover in Azure Cloud however this article will give some level of insight into how to check billable data and what remediation actions can be taken to reduce the cost burden.

Do you like to know more about how we manage costs in Azure Cloud and Sentinel? Feel free to share your thoughts and feedback at info@securiment.com.