Dynamics 365 F&SCM custom services logs and tracking using Azure application Insights

Recently I have been working in a migration project from AX2012 to D365 F&SCM that has a high number of integrations with external systems that were implemented using AIF.

After re-implement the AX2012 AIF inbound services to custom services in D365 F&SCM I started to look at the requirements related with log & trace of the service requests.

While in AX2012 we had a good logging mechanism implemented as part of the framework where AIF requests & responses were tracked using a set of tables (AIFMessageLog) and available in the product UI,

In D365 F&SCM this is not available any more in the standard.

In the current version this has been replaced with LCS telemetry that can be used to identify what endpoint is hit and how much time it took but It won't have the payload, logging of incoming message, correlation to RecId etc…

In addition to this LCS does not provide mechanisms to expose this logs, alert on them or analysis features.

So what options do we have if we want to have full message log of the service requests for tracking and troubleshooting?

Here are our options:

Option 1 – Utilize Azure API management.

This will encapsulate the Dynamics 365 URL.

Azure API provide logging functionalities.

Azure API Management overview and key concepts | Microsoft Docs

This can be a good option and provides advanced functionality, but adds an extra layer that requires configuration and management effort.

Option 2 - Add table/form in D365 F&SCM

When call is made to custom service, log to the form / table etc. Later utilize Batch job to clean-up the history.

This option is kind of replicate the old logging mechanism provided in AX2012 where we can create a set of custom tables and store request and response info for each service call.

This option will require to perform extra processing & database operations as part of service calls.

Will increase our database size and will require implementation of some clean-up process to clean-up this tables with some periodicity.

And will require F&SCM access to see the logs or expose them in some way to be consumed externally.

Option 3 – Utilize Application insights

Emit custom telemetry from Dynamics 365.

What is Azure Application Insights? - Azure Monitor | Microsoft Docs

Explore .NET trace logs in Application Insights - Azure Monitor | Microsoft Docs

Application insights is a great tool for logging & monitoring and provide analytic tools, alerting, integration with DevOps among other features.

It provides a REST API that can be used to work with the logs programmatically

The impact on your app's performance is small. Tracking calls are non-blocking, and are batched and sent in a separate thread.

Using Application Insights from D365 F&SCM

In my case I selected Application insights from the options above because fits with my requirement for logging custom service requests but as well because it could be used to log other aspects of the product that can give more insights and help to troubleshoot different issues, monitor processes and track events and usage.

Application Insights can be used to log:

• Events Azure Application Insights Telemetry Data Model - Event Telemetry - Azure Monitor | Microsoft Docs
•  Exceptions Azure Application Insights Exception Telemetry Data model - Azure Monitor | Microsoft Docs
• Requests Data model for request telemetry - Azure Application Insights - Azure Monitor | Microsoft Docs
•  Metrics Data model for metric telemetry - Azure Application Insights - Azure Monitor | Microsoft Docs
• Traces Azure Application Insights Data Model - Trace Telemetry - Azure Monitor | Microsoft Docs

So despite my requirement is only related with request, once D365 F&SCM is integrated with Application Insights it could be leveraged in the future for other requirements.

Another reason for me to choose this option is that the other systems that interact with D365 F&SCM use application insights as well as logging mechanism and provide a unique correlation ID in each communication.

So this option will allow us to implement full traceability of the messages across systems in a single place by correlation ID

Implementation

There are different options for implementation, see details in the link below

Explore .NET trace logs in Application Insights - Azure Monitor | Microsoft Docs

In my case I will be using the Trace API directly

Application Insights API for custom events and metrics - Azure Monitor | Microsoft Docs

For this we just need to get the instrumentation key of our application insights.

And reference the Microsoft.ApplicationInsights.dll.

In my case I have created a custom C# dll to implement the methods I need to consume the Microsoft.ApplicationInsights.dll to make it easier to use from X++.

The handler DLL will enable to:

•  Set context properties like user, operation, version & custom global properties.
• Instance and track telemetry for Events, Exceptions, Requests, Metrics, Traces
• Add custom properties & custom metrics related with Events, Exceptions, Requests, Metrics, Traces.

You can get the full code for my DLL class library project here:

kiko-zanon/AppInsightsHandler (github.com)

(this is just an initial version of the DLL for proof of concepts that can be improved in many ways)

to use the DLL from C# we just need to add reference to Microsoft.ApplicationInsights.dll and AppInsightsHandler.dll

And use it as shown below passing the instrumentation key and the tracking type as string.

The example below sends an event with an additional property and metric related

Traces sent can take a few minutes to appear in Application Insights.

The below shows how to send a request telemetry

Like that we can use the DLL to send the other types of traces.

Implementation in D365 F&SCM

Once we have the DLL we can proceed with the implementation in F&SCM.

First I created a parameters table & form to store the integration key and other parameters.

Added the DLL’s as references in AOT

Then I created a class hierarchy that leverages the DLL implementing for the different telemetry types.

As I want to use trace for all my services I implemented a Service base class that will implement the tracking and all service classes will extend from it.

in my service base class i will have a method that will get as parameter required details that we want to log as properties (request and response) and other details to identify the service.

Once we have this we just need to call this method passing request and response to it just before return the response in each of our services.

Here we have some examples of how the requests look in Application Insights

As we can see we will have the request body and response body serialized as JSON as we added them as properties.

We can as well additionally send ExceptionTelemetry messages if we want to save the exceptions.

in my case if there is some error i loop the infolog where exception type is error and i send a ExceptionTelemetry object with the same correlation ID as the order so we can relate the error request with exceptions.

note that there are size limitations depending on the type of tracking we send.

you can check the limitations for each type here:

Azure Application Insights Telemetry Data Model - Azure Monitor | Microsoft Docs

I hope this can be useful for anyone looking for monitoring options for D365 F&SCM.

Enjoy!