How do you deal with log data from cloud-based or distributed systems?

1 Identify relevant sources

The first step is to identify the relevant sources of log data for your investigation. Depending on the type and scope of the cloud-based or distributed system, you may need to access logs from different layers, such as application, platform, infrastructure, or network. You may also need to obtain logs from different providers, such as cloud service providers, third-party vendors, or external partners. You should consult the system documentation, contracts, and policies to determine what logs are available, where they are stored, how they are accessed, and how long they are retained.

2 Collect log data securely

The next step is to collect the log data securely and preserve its integrity and authenticity. You should follow the standard procedures and tools for forensic acquisition, such as using write blockers, hash values, chain of custody, and encryption. You should also comply with the legal and ethical requirements for accessing log data, such as obtaining authorization, consent, or warrants, and respecting the privacy and confidentiality of the data owners. You should document the sources, methods, and timestamps of your log collection process.

3 Normalize and parse log data

The third step is to normalize and parse the log data to make it consistent and readable. Log data from cloud-based or distributed systems may come in different formats, such as JSON, XML, CSV, or syslog. You should use tools and scripts to convert the log data into a common format, such as a database or a spreadsheet, and extract the relevant fields, such as date, time, source, destination, action, or status. You should also account for the time differences and synchronize the log data to a standard time zone, such as UTC.

4 Analyze and correlate log data

The final step is to analyze and correlate the log data to find patterns, anomalies, or evidence of malicious activities. You should use tools and techniques to filter, sort, group, and visualize the log data, such as queries, charts, graphs, or timelines. You should also use tools and methods to correlate the log data across different sources, such as IP addresses, MAC addresses, usernames, or session IDs. You should look for indicators of compromise, such as unusual events, failed logins, unauthorized access, or data exfiltration.

5 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?