Reducing telemetry cost with Data Collection Rules for Application Insights

So, you've created an Application Insights resource in Azure, set up the instrumentation key in the Business Central admin portal, connected the Dynamics 365 Business Central Usage Analytics-Power BI application, you sit back and relax. Telemetry starts rolling in, common user, system and integration errors are easily fixable and the customer is happy. Life's good, right?

About a month later, you'll receive an Azure invoice from Microsoft. What?! Over 90 GB of data ingestion at a rate of € 2,80 per GB for a total of 260 euro's? How is that even possible?

As it turns out, every telemetry signal generated by Business Central is logged into Application Insights. For example when you have external webservice consumers, every "Authorization Succeeded (Pre Open Company)" and every "Authorization Succeeded (Open Company)" is logged... It gets big quite fast. The easiest way to avoid unexpected cost, is by setting a 'Daily cap' inside of the related Log Analytics workspace. However, you might miss out on some crucial telemetry data after the daily cap has been reached.

Data Collection Rules (DCR)

Thankfully, we can control the amount of data ingested to Application Insights by using Data Collection Rules. Data Collection Rules provide a consistent and centralized way to define and customize data collection scenario's. Using the Kusto Query Language (or KQL for short) we can specify exactly what type of signals we want in our Application Insights account.

Decide what you care about

The first step is to decide what you care about. Do you care about certain event IDs? There's a pretty good list available here. For example, if you don't care about "RT0003" and "RT0004", which are Pre Open Company and Open Company events, you can use the Event IDs to filter them out of the ingestion.

Create a new Data Collection Rule (DCR)

The next step is to create a new Data Collection Rule. Go to the Azure Portal and navigate to the Application Insights resource. DCR's are configured based on a workspace. You should be able to navigate to the related Workspace on the Overview-page of the Application Insights resource. If it's not there, you're probably using a non-workspace based Application Insights resource. Create a workspace based one instead.

When in the Log Analytics workspace, navigate to Settings -> Tables. It should contain a table called "AppTraces". This is where the Business Central data is stored. Click on the three dots next to it, and select "Create transformation".

This opens a new page called "AppTraces transformation". Click on "Create a new data collection rule". Give it a name and hit Done. I call mine Severity2AndAbove, because I only want signals with a severity level greater than 1 in my telemetry logs. Click Next.

The next screen shows some of the current rows and a button called "Transformation editor". This is where we can set our KQL-query to filter the incoming signals. I set-up mine like this:

source |
where SeverityLevel > 1        

More examples can be found in the BCTech repository on Github, created by Kennie Nybo Pontoppidan . Hit apply when you're happy with the result.

Now for the last part, review the changes you've made and click Create if it all looks good.

Eventually, you're Application Insights logs should be limited to only contain signals with severityLevel 2 or higher. So much better!

More info on controlling telemetry cost can be found here.