Filters, filters everywhere and not an end in sight......Part 1 By: Larry Brasher

If assessed wisely, the good: monitoring, management, peace of mind. If not assessed wisely,?the bad: sluggishness, failures, BSOD, loss of connectivity.

The balance…..this is our quest.

Its almost impossible these days to have a functioning production server, workstation or laptop without having some form or another of third party filter drivers in place. Simply put, it’s a necessity as a result of mandatory compliance, regulation and overall computer health. With antivirus, monitoring or management software to gauge performance, activities, access, health and compliance levels, the list of what you can put on a server is endless. Is throwing everything but the kitchen sink on a server in the name of monitoring and security wise though? Can you have too much? Sometimes this question is never asked.?What could possibly be the cons???While all good questions to be asked, often times there is confusion on the path for these answers.?Focusing just on the Microsoft platform, I'd like to share my insight and opinion.

What are filter drivers? As per Microsoft:?

"A file system filter driver is an optional driver that adds value to or modifies the behavior of a file system. It is a kernel-mode component that runs as part of the Windows executive.

A file system filter driver can filter I/O operations for one or more file systems or file system volumes. Depending on the nature of the driver, filter can mean log, observe, modify, or even prevent. Typical applications for file system filter drivers include antivirus utilities, encryption programs, and hierarchical storage management systems."

How can I confirm if filters are in place??This can easily be done with the built in fltmc command. This command has to be ran with Admin privileges.?An example from my own home computer shown below.?

Note the “Altitude” column.?The lower the altitude number the sooner its loaded into the I/O stack. Lets look at the first three rows.

Notice how the MBAMProtection (Malwarebytes) with the altitude number of 328800is loaded or applied before the mtihint (Micron) filter driver with the altitude number of 370030.

Malwarebytes filter altitude of 328800, this falls in line with Microsoft’s list of allocated filter altitudes.

Note: Above is an example from my own personal computer.?

What's wrong with having filter drivers? Nothing, as mentioned before, in some measure they will usually be in place if in production or even at home for example with antivirus installed. The question of more concern is "what is the cost?"?Usually there is always cost which can be measured via resource consumption and overall performance.?Sometimes its minimal and sometimes not and much more damaging.

Key Takeaway #1

These filters can have their hooks into the file system, registry and protocol stack. You can easily see by using procmon.?So now the question becomes how much is too much?

KEY TAKEAWAY #2

Example:?Think of each filter as a sponge and the processing as water. Stack three sponges on top of each other on a plate, now pour water on the top until it reaches the bottom sponge.?Naturally with no sponges in play you can pour the water directly onto the plate.?With three sponges to go through the water reaches the bottom just give it a little time but not as fast as it would with no sponges to filter through though.

Now lets stack eight sponges on top of each other. These represent:

two types of antivirus software

two?types of IPS

two types of log monitoring software

?two types of resource monitoring?software

Now pour water on top until it reaches the bottom sponge. This, as you can imagine, will take much longer.

Now assume this is a server under a moderate to heavy load with such aggressive monitoring for each process, each file, each task underway.?I can assure you, your I/O will be much longer, and processing slower if not outright fail in some cases.?

Just to show you how invasive some can be, even when disabled or stopped these filters can still be in place. For some, I’ve even seen some filters so invasive?that it causes , winsock32 corruption. The result would be connectivity would be sporadic or just outright?fail. In this case ,using the netsh command was used to reset winsock, which in itself after rebuilding the winsock32 catalog, you can lose the IP settings?not to mention that any other filters (applications) that have their hooks into winsock may not work after resetting it.?To save the worse for last, the dreaded BSOD.?I’ve read too many crash dumps in the past which clearly have shown the culprit to be a wayward *.sys file which pointed to a third party filter causing the crash after the server crossed a certain work load threshold. Once removed, no issue, no crash. Such can be the cost if not properly assessed and tested.

KEY TAKEAWAY #3

Ok, so there’s a cost for throwing too much onto a server, what do you suggest?

This answer is easy but time consuming.?It comes down to proper benchmarking and assessment BEFOREHAND.?This word is capitalized and bolded as deployment of such things into a live environment may have unintended consequences and is never suggested.??Also, placing a server in a vacuum with no load, installing your software with 3rd party filters and looking at it for 30 minutes does not suffice either.?

Defining “proper benchmarking” means different things to different people but I’ll outline my thoughts on the matter for you to consider.

1.) Have a test lab\server and place it under a normal work load. Take basic fundamental perfmon counters over time and save the output.?

2.) Run the lab\server under a heavy load and take the same perfmon counters over time and save the output.?

3.) Install your choice of management\monitoring software, just one, and place it under a normal work load. Take basic fundamental perfmon counters over time and save the output.?

4.) With your previous software installed, run the lab\server under a heavy load and take the same perfmon counters over time and save the output.?

Compare your perfmon outputs against each other and ask yourself the following questions: Do you see a sluggishness in any manner? How much RAM is consumed? Is there excessive paging? Is the CPU spiked? Excessive I/O? During all this time, if you take a backup and\or virus scan of the drives, do operations fail? Does the server crash?

After all this is done, install your next management\monitoring software?and repeat the process all over again after each time you add one.?As you install the full suite of management\monitoring filters, the comparisons may surprise you.?Keep in mind is all it takes is just one ultra aggressive?or?poorly written filter to cause a problem.

While this whole process is cumbersome to say the least and maybe even a little expensive as you dedicate a person to focus on this, but in my humble opinion, its preferred to do such testing at a minimal in a test lab vs. installing it in production at the onset with fingers crossed and hope for the best.

Most important part, the decision.

When done, its decision time for which answers may vary.?You have to ask yourself: “Do I need ‘x’ installed??is this redundant monitoring \management or does this truly provide a function that has value that we don’t currently have?”

Can you remove third party filters? Yes, this is can be done.?A very common misunderstanding is that by disabling such filters (example: stopping antivirus) will remove the filters in question. This is a wrong assumption. These filters are still in place and have their hooks into the protocol, registry, file system and often times memory even if disabled.??A clear example is this screenshot below.

This is a from a procmon capture with Malwarebytes disabled and stopped. Selecting the svchost and then the process tab, here you see that malwarebytes, although stopped and disabled is still there with its hooks into svchost.exe. More procmon use to help analyze things in the followup article below.

To fully remove them a good old fashion uninstall and reboot often does the trick.?You can use fltmc command to unload the filter but this is a per servers action and will be needed after each reboot. The bottom line, if not needed, just remove it altogether.

Filters, filters everywhere and not an end in sight. Part 2

Larry Shane Brasher

References:

https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/filter-drivers

https://docs.microsoft.com/en-us/windows-hardware/drivers/ifs/allocated-altitudes

https://www.microsoftpressstore.com/articles/article.aspx?p=2201309&seqNum=3

https://docs.microsoft.com/en-us/sysinternals/downloads/procmon