OPSman Auditing

After doing audits and assessment for more than 5 years now, I've had a pleasure and privilege to visit, and not only visit but to really deep dive into different companies and to the processes that make them tick.

From the heart, I'm an OPS man. Operations is the layer of IT where all the magic happens, where plans, policies and procedures meet the real world of cables, hard drives, CPU's, OS layers and .. the bad guys. I grew up (in my IT life) in a world-class data center, lifting black boxes to rack and pulling RJ45's to switches. My part was to build the defenses - from hardware wiring and firewalling to system hardenings, IDS systems and patch management processes. Naturally, since companies need repeatability, accountability and reliability, planning was essential part of this, as well as building working methods and policies.

Maybe this background twists my perspective, please feel free to disagree with this writing, but I keep on finding myself surprised how much companies in general seem to stumble with their OPS processes.

I've assessed companies with different business models and fields of interests, and companies with great variations in their system architectures, network structures, levels of outsourcing and main technology choices. Some companies have everything in cloud, some have everything running on their basement, with armed guards on the door. Linux houses, mainframe houses, Microsoft houses.. With one thing surprisingly common - disproportionately weak OPS.

Please don’t get me wrong, many companies have great operations teams, with beautiful, harmonized and secure operational platforms with everything under control and running “like a well-greased engine”. Auditing a company like that puts a nice smile on my face for the rest of the day. But that’s not unfortunately the case every time.

Companies can have great policies and guidelines. They might have well organized architectural design teams, good change management, nice well documented IT processes. They might be involved in very delicate, very money-centric or very responsible lines of business.

But it might be that their OPS teams only patches OS levels, and firewalls are running 4 year old versions. They might for example have great central logging, with no centralized time and no server local accounts involved in logs at all. They might have huge amount of effort put into building 2-factor authentication to their VPN, but operational web servers are published by opening firewall TCP443 from production environment to the internet, with server running OS defaults without effective hardening. And in worst cases, OPS teams and their management completely unaware of these issues, or without seeing any risk in not patching .NET or having RC4 and MD5 as options in TLS. Or never testing their incidence response- or disaster recovery plans.

Planning things is basically pretty easy, naturally requiring you to have certain level of knowledge of the subject and enough time and coffee to do the required brainwork.

Building guidelines and building (meaning in many cases, writing) processes is pretty easy, of course requiring you to be pretty familiar with your business model, organizational needs and you being given the talent to put your insights into understandable written form.

But when it comes to OPS, things start to get complicated. Even if companies get the building-the-OPS part right, the big problem with OPS is that things evolve really, really fast. There is no such thing as stable OPS, it's all just continuous monitoring, adjusting, learning and upkeeping. And this is where I see that companies tend to drop the ball.

Companies might have very talented OPS teams, but lacking communication where OPS teams cannot influence the decision making process enough to make effective changes.

They might be separated to a "working class organization" that is lacking the motivation to really try to make a difference, since after all, why bother? Company is still ran by business suits in faraway offices, and they don't really care about heartbleeds or dirty cows.

Or then, OPS is bought from the outside. And we have a contract stating "service provider takes care of our OPS". Oh boy. :)

If you find this writing somehow touching your own life and environment, please take a moment to think about the functionality and gain/loss ratios of different decisions made in your organization. Since every company is running low on resources, running low on funding and cost reductions must be done everywhere.

Everyone knows that companies cannot outsource responsibility, but OPS is surprisingly often the place where many companies seem to try.

OPS is where your plans meet reality, where you policies meet TCP/IP stack. That's where the problems are no longer administrative problems, they are security problems. Where neglectance might not make you lose your compliance certification, but to lose your company.

It is the end result of your fine-tuned information security management system, but don't get fooled on thinking that it's just like a train wagon - it's not enough that you just pull it along.

Make it live, make it part of your decision making, and most importantly - know your OPS. Since they are the reacting muscle of your IT when you need it.

Build visibility to your OPS. Make your OPS feel pride in their work. If you're outsourcing some of your OPS, remember- now their OPS is your OPS. Make sure you know what they are doing, and be sure that's what you want them to do.

On many cases I see that general opinion from management is that "these are details, these don't really matter, and we have them covered anyway". My normal response (now when working at F-Secure CSS) is to pull a little polite grin and ask "would you then like our tech guys to give it a try?". After a short silence, the conversation usually continues with much more accommodative tone. :)

When you see the shoal approaching in front of your ship and you need to make a swift turn, it's not a good time to find out that there's no rope between your helm and the rudder.