WDAC and ADO Pipelines
Background
#Azure #DevOps #ADO pipelines are an amazing integration tool, and great for #CICD processes.
However, they have been around a while, and sometimes don't follow modern best practices. I recently ran into a case like this, a small bit of code that was 7 years old and causing headaches.
The ADO pipeline agent died on one of my team's development servers when attempting to use the powershell@2 handler on a #WDAC / Constrained Language Mode device.
After reading the error, it was obvious that there was unsafe code being passed via the CLI. Since it isn't signed, it trips Constrained Language Mode requirements and results in the code failing to execute. I've run across this a few other times in the WDAC deployment, and was able to get things going again by disabling enforcement on this endpoint. That wasn't ideal from a security perspective.
I have worked long and hard in getting WDAC and CLM deployed, and had great success in the process; it is nearly universally applied for my organization. I wasn't going to have half of our servers unable to be protected due to a preventable bug in an open source product.
At this point we had a definitive cause of the behavior.
Researching ADO and CLM
From here, I started digging into what options were available for a solution. A number of tickets over the last several years showed up in my search, all trying to do the same type of thing. The first reference to which is six years old, so I wasn't expecting a ready made resolution.
Some folks seemed to think this was an issue in the tasks being executed by the ADO agent, myself included to start with.
Others started looking at the ADO agent itself.
At this point, I spent some quality time reviewing the logs and procmon, and was able to verify that the issue exists not within the tasks being executed, but the pipeline agent itself, specifically agent.worker.exe. We know what the problem is, and where to look for it. Awesome, on to the next step!
Creating the Fix
After getting approval from my CSO (Mike, thanks if you are reading this!) I was able to track this problem down to a single source file in the ADO agent repository. In short, I was right! The code was indeed being built dynamically via the ADO agent, stored in a string within the ADO agent binary, and passed directly into an argument block for powershell.exe when needed.
To correct the issue, I pulled the PowerShell code out from the embedded string (from PowerShell3Handler.cs for the curious) and created a parameterized script. It takes the same values initially used to build the CLI arguments, so there is no change in upstream or downstream behavior, just where the code is stored. Now for the fun part - let's see if this bug fix actually works!
领英推荐
Testing the Fix
Once I got the build details worked out, I was able to deploy a locally built copy of the ADO agent to a WDAC and CLM enabled device. After adding a quick once over with a signing script (both the binaries and script files included) and I had a pretty good replication of what MS provides in the official ADO pipeline download.
No fuss, no muss; the agent compiles, and runs in CLM! Now I needed to test known previous failure conditions. To accomplish this, I concurrently created a simple "wdac-enforcement-test" ADO repository. This was set to run the pipeline on each commit, and target only my CLM enforcement mode agent.
The tests evolved over time as the fix got closer to completion. By the time it was said and done, I verified single line and multi line scripts embedded into the YML files, the legacy powershell: alias, and of course the main powershell@2 handler. Everything works as anticipated! To be fair, I was expecting CLM on my scripts.
Caveats when FullLanguage is required
The updates made here to support WDAC and CLM do not mean that there is no requirement for work from the teams that maintain the ADO pipelines. If tasks exist that would fail under CLM, they would need to be migrated from inline content to files, and properly signed. This should be a tiny minority.
Take this example, which uses powershell@2 and is multi line, inline script. Towards the bottom if the image, it is obvious that this is running in CLM. Will the vast majority of pipelines work without an issue? Yeah, sure. Copying files around, starting and stopping services, rest calls.. most of that doesn't require adding custom types and will just work.
If however you really do need full language mode all hope is not lost. Since signatures for PowerShell are stored as a comment - which share the same pound sign delimiter as YML files you will be hard pressed to use full language mode inline. It might be possible but even then I would be concerned about character limits.
On the plus side however, you can 100% run a signed script that from within your repository, and get full language mode. All it takes is a wrapper file that doesn't require CLM and can be dot sourced, which will then launch your full language mode script independently.
The following is an example of what I'm talking about.
Launcher scripts like this are basic, especially if you are using environment variables to pass your data.
Senior VP IT | Infrastructure and Operations @One Call
1 年Great work Tim! Glad you are part of our team and keeping our environment safe.
Chief Technology Officer
1 年You didn't mention if it was a bug you created ??
SVP and CISO at One Call
1 年Awesome work and great walk-thru!