Security Monitoring: A Conversation About SIEM, Logs, and What Actually Matters

Is it just me or the cyber security industry has really made SIEM, logging and monitoring hard to wrap your head around? I've been staring at vendor websites, reading white papers, and sitting through demos, and honestly - I'm not sure if it's really that complicated or if all this vendor marketing and vague compliance requirements have introduced artificial complexity.

I'm writing this one to think... and mostly talking to myself, but I hope you also get some clarity from reading it. I’m trying to answer "which SIEM should we buy?"

What is a log and why do we log?

If we strip away all the fancy marketing terms and complicated architectures, logging started with a simple need: figuring out what happened.

Logging exists to answer four basic questions:

• What happened?
• When did it happen?
• Who (or what) did it?
• Where did it happen in our system?

Back in the early days of the web, this was pretty straightforward. You had a server hosting a web application, it wrote down what it was doing in a log file, and when something went wrong, you'd SSH in and read the log. Simple, right?

But then things got...interesting. And by interesting, I mean complicated.

This "simple" web application? Well, it's not so simple anymore. Let me paint you a picture of what a modern web app looks like:

A user clicks on your website. Seems straightforward, right? But that request probably hits:

• A CDN (which logs the request)
• Then a load balancer (more logs)
• Through a Web Application Firewall (yep, more logs)
• Into a Kubernetes cluster (where both the orchestrator and container runtime generate logs)
• Hitting a web server running in a container (application logs)
• Which sits on a VM (operating system logs)
• Running on some hypervisor (virtualization logs)
• On physical hardware (infrastructure logs) and I almost forgot networks (netflow logs and packets)

And that's just the path in! Each of these layers exists for good reasons - scalability, security, reliability. But each one is also generating logs, and each type of log speaks its own language. A WAF log looks nothing like a Kubernetes log, which looks nothing like an Apache access log.

Now, let's talk about why this matters for security. We're not just logging for troubleshooting anymore - we're trying to spot the bad guys.

The challenge isn't just collecting all these logs (though that's hard enough) - it's making sense of them. How do you know what's normal and what isn't? What patterns should trigger alerts? When is a failed login just a forgotten password, and when is it the start of an attack? So we need a SIEM..

But If you make your SIEM do all of this across everything you have, depending on your business type, you're likely going to spend a small fortune, especially if you're heavy on cloud. Why? Because cloud storage isn't cheap, and data transfer costs can make your eyes water. The vendors promise you'll need fewer engineers since you're not maintaining infrastructure, but ask any hybrid-cloud organization how that's working out for them. Spoiler alert: not quite as promised.?

What logs to monitor in your SIEM?

This is a question that's haunted me throughout my career, and my answer has evolved with each role I've had. Let me take you through that journey:

As an engineer, my answer was simple: "Everything! How else will we catch the bad guys?" It made perfect sense from a technical perspective. More data means better correlation, better detection, better everything... right?

Then I became a lead and had to deal with budgets. Suddenly my answer changed to "Everything - but only for our crown jewels and critical systems." Sounds reasonable, right? Focus on what matters most. Not really - when you're responding to incidents, you quickly realize you're missing crucial pieces of the puzzle. That suspicious traffic from your crown jewel system? Well, it would be nice to know what other systems it talked to, but those logs aren't in your SIEM.

Now, as an architect, I'm looking at this through yet another lens: aligning security with business objectives. And let's be honest - for business, it's mostly about making money and achieving business objectives. So now I'm thinking about ROI on SIEM spending, and this is where it gets really interesting.

All these different perspectives have led me to ask some uncomfortable questions: Do we even need a SIEM solution? Why are we storing petabytes of logs in an expensive SIEM when we could just buy cheap storage and dump all the logs there? Wouldn't that satisfy most of our compliance requirements?

And what's a SIEM anyways?

According to Google: SIEM, stands for Security Information and Event Management, is a cybersecurity solution that helps organizations detect and respond to threats. It does this by collecting and analyzing security data from various sources across an IT environment.

Hold up a second. Let's focus on that phrase "security data." What exactly is security data? This is where most organizations get tripped up, because here's the uncomfortable truth: almost ANY data can be security relevant. That failed database query? Could be an SQL injection attempt. That successful login at an odd hour? Could be an attacker. That spike in CPU usage? Could be crypto mining malware.

See the problem? We're back at square one - logging everything. Or at least, almost everything!

This is where most SIEM projects go off the rails. Organizations start dumping all their firewall logs, all their Windows event logs, their Linux logs into their SIEM. They end up paying a small fortune for what? A system that still has significant visibility gaps and an ROI that's harder to find than a secure password in a "12345" string.

Here's the real kicker - after spending all that money, you still might miss critical security events because:

• Your SIEM is so noisy with irrelevant data that important alerts get lost
• Your analysts are drowning in false positives
• You're paying for storage and processing of logs that you'll never actually use
• And despite all this data, you're still missing crucial context when you need it

So how do we choose a SIEM platform and what logs do we store in it, and for how long?

Let's break this down, because we're actually asking several questions here. And surprise - it doesn't start with technical requirements or vendor feature comparisons.

?

You need clarity on your business requirements and technology strategy:

Let's break this down into three critical areas:

Technology Strategy: What's your organization's technology philosophy? Are you a "buy before build" shop that prefers commercial off-the-shelf solutions? Maybe you're all about reusing existing tools before buying new ones? Or perhaps you're that rare breed that prefers to build custom solutions? Understanding this is crucial because it impacts everything from vendor selection to integration approaches. And let's not forget the whole cloud versus on-prem debate - are you cloud-first, hybrid by design, or keeping those servers humming in your data center?

Budget: What's your actual budget over 3-5 years? And I mean real numbers, not wishful thinking. This isn't just about the initial purchase - you need to factor in storage costs that grow over time, training, maintenance, and the inevitable "we need just a bit more capacity" conversations. Trust me, your CFO will appreciate this level of forethought.

Risk and Compliance Landscape: What's keeping your executives up at night? Maybe you're in a heavily regulated industry where compliance is king, or perhaps you're more concerned about protecting intellectual property. Your risk appetite and regulatory requirements will dramatically influence both your SIEM strategy and budget. After all, there's a big difference between "nice to have" and "the auditors are coming."

But here's where most organizations go wrong - they jump straight from these high-level requirements to a policy document that says something useless like "thou shalt log all application events." That's not a strategy, that's just shifting the problem downstream.

What you actually need is a monitoring strategy. Think of it like planning security for a bank branch (because who doesn't love a good physical security analogy?). You don't just throw cameras everywhere and call it a day. You think strategically:

• You put cameras at entry/exit points (your network perimeter)
• You monitor the vault extra carefully (your crown jewels)
• You watch the teller areas (your user-facing applications)
• But you don't waste resources monitoring every bathroom and closet

To do this well in your digital world, you need some foundational elements in place:

• Asset management (you can't protect what you don't know about)
• Asset criticality (not all assets are created equal)
• Network topology (understanding your digital terrain)
• Sensor placement (making sure you have visibility where it matters)

Here's the key mindset shift: Instead of asking "what devices produce logs that I can monitor?" start asking "what do I actually need to monitor to protect my business?"

It's like the difference between installing security cameras based on where you have power outlets versus installing them based on what you actually need to protect. One approach is convenience-driven, the other is strategy-driven.?

So how do we turn this strategic thinking into actual implementation?

Let me break this down into a practical approach that I've seen work in the real world.

Start with Business Context

Before you touch a single log configuration, you need to map out your business context. This isn't just a checkbox exercise - it's about understanding what makes your business tick:

• What systems directly generate revenue?
• Where does your sensitive data live?
• What processes would cripple your business if they went down?

Think Like an Attacker, Monitor Like a Defender

Here's a practical way to approach this - forget generic threat intel for a moment. Go to MITRE ATT&CK, click on CTI > Groups, and do a simple search for keywords relevant to your industry. Are you a bank? Search for "bank". Tech company? Look for groups targeting intellectual property. This isn't just an academic exercise - you're looking at the actual techniques that real threat actors are using against organizations like yours.

This is what you need to monitor before anything else. You need the logs that allow you to create these detections. Why? Because these are the proven paths attackers have used successfully. It's like having a blueprint of how criminals have broken into similar buildings - wouldn't you want to monitor those specific entry points first?

For each of your critical assets and processes, think through:

• Which of these TTPs (Tactics, Techniques, and Procedures) could be used against us?
• What logs and monitoring would we need to detect these specific techniques?
• Where are our blind spots in detecting these known attack patterns?

This is way more practical than trying to boil the ocean with generic monitoring. You're focusing on actual, documented attack patterns that are relevant to your industry, not theoretical threats.

For example, if you see that financial threat actors heavily use phishing followed by PowerShell abuse, guess what? Your monitoring strategy just got its first priority - email logs and PowerShell command logging. If you notice they typically move laterally through Windows admin shares, well, there's another critical log source identified.

This helps you identify what you actually need to monitor, rather than what you could monitor.

Build Your Monitoring Tiers

Instead of the old "log everything" approach, create monitoring tiers based on criticality:

Tier 1: Crown Jewels

• Full logging and real-time monitoring
• Longer retention periods
• More sophisticated detection rules
• Example: Payment processing systems, customer databases

Tier 2: Important Systems

• Selective logging focused on security-relevant events
• Standard retention
• Basic detection rules
• Example: Internal applications, employee systems

Tier 3: Everything Else

• Minimal logging
• Shorter retention
• Log only significant events
• Example: Development systems, test environments

Start Small, Scale Smart

Here's a practical implementation approach that I've seen work:

?Begin with a pilot of your most critical system

• Set up proper logging
• Establish baseline behaviors
• Create initial detection rules
• Validate your assumptions

Learn and adjust

• What logs are actually useful?
• What generates too much noise?
• What detection rules need tuning?

?Document and standardize

• Create logging standards based on your learnings
• Define clear integration requirements
• Establish response procedures

?Expand methodically

• Move to the next most critical system
• Apply lessons learned
• Adjust standards as needed

The key here is to avoid the big bang approach. You don't need to monitor everything on day one. Start with what matters most, get it right, then expand.

See how we haven't even talked about technology or SIEM products yet?

That's not an oversight - it's intentional. Because here's the truth that vendors don't want you to hear: successful security monitoring that actually reduces risk isn't about the product you use. Sure, some products might be faster. Others might have fancier correlation capabilities or prettier dashboards. And yes, some might be easier to manage than others. But these are all secondary considerations. It's like focusing on which brand of security camera to buy before you've even figured out what you're trying to protect and why.

The hard truth is that even the most expensive, feature-rich SIEM will fail if you don't have:

• A clear understanding of what you're trying to protect
• A solid strategy for what you need to monitor
• A practical approach to implementation
• The right processes and people in place

In addition to that, there are several engineering challenges you'll have to solve along the way

• Data Volume, Velocity, and Log Source Variety
• Log Normalization Complexity
• Data Enrichment and Context
• Storage and Scalability
• Integration with Other Security Tools

Your SIEM doesn't exist in isolation. It needs to play nice with your ticketing system, your EDR, your threat intel platform, your orchestration tools - the list goes on. Each integration is another potential point of failure, another system to monitor, another thing that can break during an upgrade.

I've seen organizations running multi-million dollar SIEM deployments that couldn't detect basic attacks, while others running relatively simple setups were catching sophisticated threats. The difference? It wasn't the technology - it was their approach.

Think about it like this: if you don't know what you're looking for, it doesn't matter how powerful your telescope is. You'll still be looking in the wrong direction.

This is why when I’m asked "which SIEM should we buy?" my first response is always "let's take a step back and talk about what you're trying to achieve." Because the tool should fit your strategy, not the other way around.