Sharing my first experiences on how to build custom KQL plugins for Copilot for Security

Hold on reader who is rightly in a hurry but also a bit curious! At least read this initial 5-points summary that I wrote just for you :-)!

Below is a summary of what I have written in this article:

1. Preamble: anything I write in this article is my personal reflection and may be incorrect.
2. In the grounding mechanism, the results of KQL queries can be truncated by the Copilot for Security orchestrator. For this reason, when creating KQL queries to be included in the custom plugins of Copilot for Security, it is not enough to ensure that they give the correct results when launched in Log Analytics. It is necessary to have a clear understanding of the user prompts to which they will have to respond, that is, it is necessary to have very clear the use cases they intend to satisfy. In this way, they can be written in a way that reduces the risk of returning partial results.
3. The custom plugin that I have created and describe in this article aims to allow the addition, to the original “Incident Investigation” promptbooks of Copilot for Security, the possibility of querying the Attack Paths identified by Defender for Cloud and that include at any point any of the resources present in the incident, also providing the recommendations reported in these Attack Paths.
4. The ultimate goal is to allow organizations to have a clear evidence of how the Attack Paths and their Recommendations, when proactively addressed, can help reduce the risk of suffering the latest incidents that have occurred.
5. The new version of the custom plug-in can be downloaded here.

Now, reader who is in a hurry, if you are still curious and have a few more seconds, take a look at the figures that I have put at the end of the article to get an idea of what I have exposed.

Now that I have summarized everything I describe below, I take a little calm to better expose these concepts, …also to try to clarify my thoughts for myself :-).

Preamble

I start with a very necessary premise. Everything I write in my posts and everything I share on GitHub should always be considered material resulting from my experiences and reasoning reported on a personal basis. Anything I publish may contain errors and, certainly, in no way wants to represent an approved communication from my company. This is even more true for what I am writing in this article, as it deals with a new technology and, as far as I can ascertain, it is still only partially documented (...if not still completely undocumented...).

That being said, I share here my experience in creating this custom plugin for Copilot for Security dedicated to the integration of the Attack Paths identified by Defender for Cloud, now in its second version.

I am well aware that, at the time of writing these notes, the official Microsoft plugin for Defender for Cloud is about to be released in public preview. However, from what I have been able to see in the documentation, it does not seem to include the features that make the usage scenarios described here possible. Even if they were included, these notes of mine could still be useful for reasoning about some aspects that may not be immediately evident related to the creation of custom plugins, especially if they are of the KQL type.

Looking back

I created and published a first version of this custom plugin about a week ago. I started with the simple idea that it might be useful to give Copilot for Security the ability to reason with the data from the Attack Paths discovered by Defender for Cloud. In my lab, I set up the Continuous Export feature to have Attack Paths and Recommendations in Log Analytics and I wrote a complex KQL query capable of returning aggregated information on these Attack Paths.

Well, I didn’t realize right away that, although this query was perhaps great for a table in a Log Analytics workbook, it made little sense exposed in a custom plugin. The basic problem is that it potentially returned too many results, making the risk very high that these would be truncated - as rows and/or columns - by the Copilot for Security orchestrator in the grounding phase, I guess to contain the number of tokens exchanged with the LLM (…I report this thought more as a deduction than as a certainty!!).

So I had to ask myself: how can the data from the Attack Paths be useful to a user who is using Copilot for Security? And, subsequently: how can I realize this type of interaction, possibly foreseeing more than one prompt from the user but minimizing the risk of seeing the data truncated?

The new idea

The question about the possible use cases of using Attack Paths in Copilot for Security will certainly have several possible answers. I focused on what seemed to me to be of most value: enriching the investigation of security incidents carried out in the standalone portal of Copilot for Security (for example, using the appropriate promptbooks), allowing the user to have evidence of the possible presence of the cloud resources involved in the incident itself within the Attack Paths detected by the Defender CSPM component of Defender for Cloud in the last N days. I thought about the need to make this time window flexible, or parametric, as an investigator may want to choose the time span in which to do the aforementioned search. I also thought about the opportunity to immediately include, within the incident, the most valuable final element of the identified Attack Paths: the Recommendations associated with these paths. These are, in fact, when addressed, able to neutralize the Attack Paths otherwise available to the attacker.

With these evidences in mind, I reflected on what the minimal input to consider for this type of interaction was. I realized that I had to provide the user with the possibility to simply ask:

1. Looking at the time period X (e.g. last 10 days from today or, again, 2 days prior to the date of the incident, etc…), considering that the incident contains the cloud resources (on Azure, AWS and GCP) that you have already identified by name in the previous steps of the analysis, what are the Attack Paths that include at any point any or more of these cloud resources?
2. Looking at the same time period, considering the same cloud resources and the Attack Paths identified at the previous point, what are the associated Recommendations?

For both of these prompts, the inputs are, therefore:

1. The time period (date to be considered as the end and number of preceding days).
2. The list of names of the cloud resources.

The implemented solution

So, I wrote the KQL code capable of answering the aforementioned questions with the aforementioned inputs. Finally, I inserted this code into two skills of my custom plugin, making sure to describe them in a complete and clear way, in natural language, in the manifest file.

I also created two other skills:

• One skill is used to get the details of a specific Attack Path identified by its “simplified” ID (i.e., from the “only” last 64 characters of its very long ID). These IDs can be obtained from the skill that identifies the Attack Paths to which the cloud resources of the incident belong. In these details, in addition to all the information present in the pages of the Attack Paths in Defender for Cloud (MITRE T&T, descriptions, etc...), the names and types of all the nodes involved in the path are also reported in textual form.
• Another skill finds the Recommendations starting from one or more (simplified) identifiers of Attack Paths. Initially, I thought of searching for the Recommendations within the promptbook in this way, but then I realized that it was always convenient to start from the list of involved cloud resources.

What I tested

Having absolutely not done extensive tests - also because in my laboratory environment I can only simulate a limited number of attacks and vulnerabilities - I cannot be certain that the behavior of my KQL queries and the skills that use them is completely correct. I can only state that, in my tests, the results of the prompts were always consistent with the results of the KQL queries made directly in Log Analytics: there were no truncated results. I say this knowing that I was able to limit myself to only about ten Attack Paths and Recommendations. It is very likely that on much higher numbers - assuming it makes sense to think of finding cloud resources of an incident in a number much higher than a dozen Attack Paths!! - there is still a truncation.

I also verified that these data were consistent with those found:

1. in the pages of the Attack Paths and Recommendations in Defender for Cloud, for the only paths detected more recently and, therefore, still visible on the portal;
2. in the Log Analytics tables for Attack Paths and Recommendations, for the results searched in time windows further back in time.

The result

I can finally show you, as a demo, the images of what I have accomplished.

I start with the evidence of how I thought to modify the original incident investigation promptbook present in Copilot for Security, specifically that of Defender XDR. In details, I replaced the very last prompt with the following 3 prompts.

A suggestion: please disregard the prompt numbers as they do not accurately represent their actual position in the final promptbook. I also made some final adjustment to the text. Please refer to the bulleted list below for the final version.

In my laboratory environment, I find myself with only one active Attack Path with two Recommendations. To save time on simulating an attack on those specific resources included in that path, I “tricked” Copilot into considering as part of the cloud resources present in the incident also one of the resources present in that active Attack Path (the “juiceshop” container) and a virtual machine (named “vm-hpv1”) that days ago was present in another Attack Path. In practice, just before the 3 prompts shown above, I added to the promptbook a prompt functional only to this testing requirement:

For your convenience, here is the final text of these last 4 prompts in my modified incident investigation promptbook:

• Consider as list of "cloud resources" (servers, clusters, storage, etc... in Azure, AWS or GCP environments) involved in this incident all the cloud resources extracted from the incident itself in the second prompt (but only those for which you have their names) plus the following additional resources: <AdditionalCloudResources>. Write this complete list of cloud resources as a bulleted list of only resource names. Do not add any additional detail.
• Considering the last <NumberOfays> day/days starting from <StartingDateForSearch>, search and list any Attack Path that includes any of the cloud resources involved in this incident. For each Attack Path identified, write its name, its date of first and last detection and its simplified ID. Write the list as a table.
• By searching in the same time window, search for all the Recommendations identified by Defender for Cloud and related to any of the Attack Paths including any of the cloud resources involved in this incident. For each recommendation, write the title directly as a hyperlink to the page, in the Azure portal, showing the details of that specific Recommendation. Include also the name and type of the cloud resource that the Recommendation is targeting. Write everything in a table.
• Write a report summarizing this investigation. Include also a paragraph with the evidence of any possible Attack Path and Recommendation identified during this investigation. The report should be suited for an audience of <AudienceType>.

Finally, I launched the promptbook giving the required inputs in the parameters:

• incident number in Defender
• type of audience for the final report
• additional cloud resources to consider as part of the incident - as just mentioned, it was a trick to force the match with the Attack Paths
• Time interval (end date = today and past days = 10).

The responses to my new prompts

These are the images of the response to the aforementioned prompts.

Here you can see that, indeed, Copilot has added to the incident the two resources that I indicated to force the match with the Attack Paths present in my environment.

Here it can be seen that Copilot finds 8 Attack Paths for those resources.

Here it can be seen that it is, indeed, the same number of attack paths returned by the KQL query.

Here it can be seen that Copilot finds 11 Recommendations associated with those Attack Paths or those cloud resources.

The table of Recommendations related to the Attack Paths identified for this incident is, perhaps, the most value-added content to the investigation. As can be seen, for each Recommendation, a link is provided that directly opens the Defender for Cloud page that provides details for that recommendation.

Here it can be seen that it is, indeed, the same number of recommendations returned by the KQL query.

Conclusion

Beyond the usefulness of the custom plugin presented here - a value that seems high to me, although perhaps an official plugin capable of doing the same things better will soon be available - I hope that the experience described here can be useful to those who, like me, are embarking on the creation of their first custom KQL plugins for Copilot for Security.

I repeat once again: the reasoning and evidences that I have described in this article may be more or less wrong as I wasn't able to find official references in the documentation. Probably, at least some of these concepts may be useful also for you. I am sure that soon we will have much more material available in literature to know how to carry out these developments!