Parser Management: How to disable Unused ASim Parsers

A parser is a function which follows a Schema (a standard set of fields) defined by Microsoft. That function will be used to normalize events into a meaningful human-readable dataset. That normalized dataset then can be used to define Analytic rules and automation Etc.

Microsoft Sentinel provides a number of Parsers built into the Sentinel Platform, However, users can also opt to create and deploy their own set of Parsers. There’s no limit to creating and reusing the parsers or the functions.

Although a parser function can be called into another function. Usually, low-level source-specific parsers are called into the high-level union parser.

Union parser can have many functions called into it, and at the time of query it's highly possible that source-specific parser function is queried regardless of the incoming events from that source.

In this scenario, the best practice is to disable unwanted/unused parser functions to reduce the load on the query.

For Example

Let’s look into the _ASim_DnsBuiltIn parser.

let DisabledParsers=materialize(_GetWatchlist('ASimDisabledParsers')
| where SearchKey in ('Any', 'Exclude_ASim_Dns')?
| extend SourceSpecificParser=column_ifexists('SourceSpecificParser','')?
| where isnotempty(SourceSpecificParser)?
| distinct SourceSpecificParser);
let builtInDisabled=toscalar(toscalar(DisabledParsers) has_any ('Exclude_ASim_DnsBuiltIn', 'Exclude_ASim_Dns', 'Any'));
union isfuzzy=true
_ASim_Dns_AzureFirewallV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_AzureFirewall' in (DisabledParsers)))),
_ASim_Dns_CiscoUmbrellaV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_CiscoUmbrella' in (DisabledParsers)))),
_ASim_Dns_CorelightZeekV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_CorelightZeek' in (DisabledParsers)))),
_ASim_Dns_GcpV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_Gcp' in (DisabledParsers)))),
_ASim_Dns_InfobloxNIOSV05(disabled= (builtInDisabled or('Exclude_ASim_Dns_InfobloxNIOS' in (DisabledParsers)))),
_ASim_Dns_MicrosoftNXlogV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_MicrosoftNXlog' in (DisabledParsers)))),
_ASim_Dns_MicrosoftOMSV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_MicrosoftOMS' in (DisabledParsers)))),
_ASim_Dns_MicrosoftSysmonV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_MicrosoftSysmon' in (DisabledParsers)))),
_ASim_Dns_NativeV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_Native' in (DisabledParsers)))),
_ASim_Dns_VectraAIV01(disabled= (builtInDisabled or('Exclude_ASim_Dns_VectraAI' in (DisabledParsers)))),
_ASim_Dns_ZscalerZIAV04(disabled= (builtInDisabled or('Exclude_ASim_Dns_ZscalerZIA' in (DisabledParsers)))),
_Im_Dns_EmptyV03?        

We can see there are a number of different source-specific parsers listed in _ASim_DnsBuiltIn.

If we just simply run _ASim_DnsBuiltIn in the query we are getting Cisco Umbrella Logs at the moment.

Here are query details without disabling any parser, keep in mind the Total CPU usage in milliseconds to process this particular function.

Disabling Unused Parsers

Using the below method we can disable unused parsers without altering the function code.

Reference link:

Deploy Watchlist

To disable unused ASim Parsers, make sure you have deployed the ASimDisabledParser watchlist.

• To deploy the watchlist simply follow the GitHub link:

Once the watchlist is deployed, we can list the parser which isn’t required. Listed parsers will be excluded from the query. If you want to disable ASim instead of Im here is how the actual entry will look like:

• CallerContext : Exclude_ASim_Dns
• SourceSpecificParser : Exclude_ASim_Dns_CiscoUmbrella

This entry will disable the CiscoUmbrella parser. This was just to show how to disable “a“ parser. In actuality, you are supposed to disable all the parsers and leave only the one you want to make use of.

If you disable all the parsers and leave the CiscoUmbrella, that is how the watchlist going to look like:

And the query time will improve 4x times!

Note: The _ASim_DnsBuiltIn functions seem to have an issue. That is how the function looks like

let DisabledParsers=materialize(_GetWatchlist('ASimDisabledParsers'
| where SearchKey in ('Any', 'Exclude_ASim_Dns')
| extend SourceSpecificParser=column_ifexists('SourceSpecificParser','')
| where isnotempty(SourceSpecificParser)
| distinct SourceSpecificParser);
let builtInDisabled=toscalar(toscalar(DisabledParsers) has_any ('Exclude_ASim_DnsBuiltIn', 'Exclude_ASim_Dns', 'Any'));
union isfuzzy=true
_ASim_Dns_AzureFirewallV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_AzureFirewall' in (DisabledParsers)))),
_ASim_Dns_CiscoUmbrellaV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_CiscoUmbrella' in (DisabledParsers)))),
_ASim_Dns_CorelightZeekV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_CorelightZeek' in (DisabledParsers)))),
_ASim_Dns_GcpV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_Gcp' in (DisabledParsers)))),
_ASim_Dns_InfobloxNIOSV05(disabled= (builtInDisabled or('Exclude_ASim_Dns_InfobloxNIOS' in (DisabledParsers)))),
_ASim_Dns_MicrosoftNXlogV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_MicrosoftNXlog' in (DisabledParsers)))),
_ASim_Dns_MicrosoftOMSV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_MicrosoftOMS' in (DisabledParsers)))),
_ASim_Dns_MicrosoftSysmonV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_MicrosoftSysmon' in (DisabledParsers)))),
_ASim_Dns_NativeV03(disabled= (builtInDisabled or('Exclude_ASim_Dns_Native' in (DisabledParsers)))),
_ASim_Dns_VectraAIV01(disabled= (builtInDisabled or('Exclude_ASim_Dns_VectraAI' in (DisabledParsers)))),
_ASim_Dns_ZscalerZIAV04(disabled= (builtInDisabled or('Exclude_ASim_Dns_ZscalerZIA' in (DisabledParsers)))),
_Im_Dns_EmptyV03)        

This function does not return any value even if CiscoUmbrella is not excluded.

I think this is a bug in the function.

This function should be without “Exclude_ASim_Dns“ in the below statement, and should look like this:

let builtInDisabled=toscalar(toscalar(DisabledParsers) has_any ('Exclude_ASim_DnsBuiltIn', 'Any'));        

Conclusion

In our exercise, we saw huge potential and use of exclusion in the parser functions. This article describes the best practice to use the ASimDisabledParsers watchlist and its benefits in increasing the query processing time and overall speed of the platform if used properly.

Hope it helps the Microsoft Sentinel community in general.

In case you have any questions or feedback feel free to send us an email at [email protected].