How can you use log analysis for cloud security monitoring?

I find it fascinating that many companies do not even log various events of interest for security as well as application performance insights. The US Department of State decided they wanted to log MailItemsAccessed which was _the only_ reason they found out about China’s Storm-0558 reading their emails for a month undetected. Anomaly detection doesn’t work if you don’t log system activity!

In my opinion, while it is very easy to provide teams with high flexibility to build amazing solutions using the cloud, it is very important to create a powerful, scalable and reliable mechanism for analyzing logs. There are different situations to observe: sensitive data scattered, inadvertent access modifying the environment and generating unexpected situations and operations harmful to the company's operations and data.

Agreed. It definitely matters in the cloud. AI is being used with the SIEM tools to determine behavior. This is much harder to do manually. For instance, was there a increase in some event (like out of memory, increased cpu usage, higher logins to the public website, etc.) from say 6 months ago and today. Does it mean we have more business and are the increased transactions legitimate? Comparing logs from different systems and pulling out patterns and connections using AI enhanced SIEM tools is how we can benefit.

Log analysis in cloud security involves centralizing logs from various services. Define security use cases, create custom alerts, and correlate logs to detect anomalies. Utilize User Behavior Analytics for advanced threat detection. Logs aid in incident response, providing a timeline for forensic analysis. Ensure compliance by monitoring and auditing logs regularly. Continuous improvement, and adapting to emerging threats, make log analysis integral for proactive cloud security, safeguarding against unauthorized access and potential risks.

This is not only true for cloud computing systems but also for any system that aims to track what is happening. Cloud computing elevated that to a new level of concern once it brought a lot of new tools and a more distributed system. Now with bigger teams and a lot more interactions between services, logs analysis not only turns out to be critical but also vital for any application that wants to improve and move fast.

Third-party tools like Splunk, Loggly, and Sumo Logic. These tools can collect, analyze, and visualize log data, making it easier to monitor your cloud security

There are third party tools such as splunk , datadog, graffana which can be used for log analysis. Apart from these all major cloud platforms provided their inhouse tool which can be used for log collection and analysis l. For example, in Azure we can use Azure Monitor to collect logs and then Log Analytics to drill down. In AWS, it can be configured using Amazon OpenSearch .

Combining solutions is a good way to provide good quality information quickly and economically. CloudTrail + Elastic Stack is a good example of an EKS platform audit

Log analysis for cloud security means looking at records of activities in your cloud services to find and fix potential security problems.

No single tool can address all the needs. It has to be a mix and match. Understanding what to monitor, analyse and at what frequency is one key aspect. The other is, leveraging the threat intelligence. Today, enterprise tools have proprietary intelligence sources. There's a need to have a threat intelligence lake for wider benefits.

I’ve often found the cloud native solutions get you 90% of the way there, AWS CloudTrail, Azure Monitor, Google Cloud Logging. They integrate seamlessly and are often cost effective.

It's always an option to store locally and use rsyslog, fluentd, etc if it's enough for your purposes. When doing that you should compress your logs with zstd (or xz). But often you want to retain them, so again your options are simple - scp to a remote server, or use S3. Syslog is also transferrable over TCP. Think about log retention, most log collectors have settings for that. Have software write logs to stdout, in contrast to having special logfile writes in them, this will give you flexibility in the future to change the logging layer.

All the information used by the product I use is stored in a workspace supported by a high speed storage system based on Log Analytics supported at the same type for different types of storages, like blobs, tables, files, etc. When you need long term storage you can move all the signals to a Blob storage to maintain more than 90 days of signals.

Vale a men??o de que a análise de logs permite identificar padr?es de comportamento suspeito, como tentativas de acesso n?o autorizado, atividades incomuns ou acessos fora do horário comercial. Ao monitorar os logs, é possível detectar indícios de atividades maliciosas. Isso permitirá que a organiza??o se prepare, principalmente, de maneira preventiva e n?o reativa.

It's usually okay to grep information from your logs if you have a few components, but this doesn't scale. But even then, eventually you need to implement some kind of strict format for critical loglines - a special "audit" format that's structured. This way developers are free to log however they want, but critical junctions will always get logged in a predictable format. This kind of structured logging opens up possibilities to send the information to Grafana (and alternatives), so meaningful graphs can be drawn about important metrics.

Understand how log data evolves over time by employing time series analysis tools is one of my favorite ways to visualize the data. This is particularly valuable for identifying trends, seasonal patterns, or sudden spikes in activity. Grafana is usually my go to solution but there are others like Prometheus or the TICK stack.

To analyze the data we can use different mechanism. Azure Kusto Query is the main one, it permites you to ask for detailed data stored in the Log Analytics and look for information. At the same time you can use AI to ask using natural language everything you need to your data.

Consider correlating logs from distributed sources by passing an appropriate context. It will help you see the full picture of what's happening, making it easier to spot and respond to security issues in the cloud.

The major challenges would be to analyze the volume and safeguarding sensitive information. It could also mean sanitising the logs and filtering out duplicates or similar pattern data

The challenges are related with the volume of the data meanwhile you safeguard all the PII information related with the services monitored. The benefits are intrinsically related with the security of your system. Basically a good system can help you to identify not just only the threats but also maintain a good security posture in your systems.

Biggest challenge that I have faced was “logging standardization”, in the big legacy platforms there are often multiple log sinks exists. And sometimes it’s almost impossible to modify logging output, even if current one is inefficient and overloads network. Another problem was bloated logging, when you have logged events for each and every action inside application, without proper leveling and structure. Logs usability going down, and amount of efforts to fix entire platforms sometimes incredibly high.

A análise de logs fornece insights valiosos sobre a eficácia das medidas de seguran?a existentes. As organiza??es podem usar essas informa??es para ajustar e melhorar continuamente suas estratégias de seguran?a.

Too many logs isn't a showstopper problem - you can ship them off-server with the help of your Ops team. The real issue with logging is that logs contain information for very different layers. Sometimes you need a very high-level understanding of a flow, sometimes you need to know which if-else branch in the code was taken in a very specific function. It's hard to know beforehand where logging needs to be added because fresh logic hides unknow failure modes. Taking all this into account, it's always a win if the logs are granular enough to pinpoint an unexpected error without needing new code deploys. Also, it's critical to be able to follow high-level events across your components. This requires agreements between teams.

Real-Time Monitoring - Implementing real-time log analysis for immediate threat detection. I recall a time when this approach helped us identify a breach attempt instantly. Anomaly Detection - Using log analysis to spot unusual patterns. A project I led used this to uncover a subtle, ongoing data leak. User Activity Tracking - Monitoring user activities through logs to identify unauthorized access. Once, this method revealed an insider threat in a client's system. Compliance Audits - Regular log reviews for compliance. I've used this to ensure adherence to GDPR and other regulations in past projects.

If you want to detect security incidents, don’t do it from logs. If you have found something in logs, it’s too late. Logs are not synchronous way to look at the state of your service, g they always have delivery time, and all event-based processing comes with latency. Logs = Latency = Time for attacker.

It is also important to add traceability to track requests and their payload and how they are flowing through the system. Sometimes anomalous requests can try to inject malicious payloads and this must be traced across network and application layers to understand the response of the firewall, network access rules, application layers etc. If a malicious payload does make it through, the nature of the request will provide information on the nature of the threat and plug the layers that allowed such a request to penetrate the system. Invariably analysing terabytes of log data is a continuous activity that needs to make use of algorithms (even ai based tools) to continuously monitor traffic and payloads and figure out which ones to reject.

Muitas organiza??es est?o sujeitas a regulamenta??es específicas que exigem a manuten??o de registros detalhados de atividades. A análise de logs ajuda a garantir que as práticas de seguran?a estejam em conformidade com requisitos legais e regulatórios.