Microsoft Warns Enterprise Customers of Critical Log Loss Due to Bug

Microsoft has issued a warning to its enterprise customers, alerting them to a serious bug that caused critical log data to be partially lost for almost a month, potentially compromising the ability of businesses to detect unauthorized activity on their networks. The bug affected security logs between September 2 and September 19, 2024, creating vulnerabilities for organizations that depend on this data for monitoring suspicious traffic, login attempts, and other security-related behaviors.

The problem, first reported by Business Insider, highlights a significant gap in the collection of security logs, with Microsoft acknowledging that the missing data could increase the risk of attacks going undetected. Further details were revealed in a Preliminary Post Incident Review (PIR), indicating that some services continued to experience disruptions in log collection until as late as October 3.

Impact on Key Microsoft Services

The PIR sheds light on how various services were affected by the bug, with each experiencing different levels of disruption:

• Microsoft Entra: Potentially incomplete sign-in and activity logs, affecting integrations with security products like Microsoft Sentinel, Purview, and Defender for Cloud.
• Azure Logic Apps: Gaps in telemetry data for Log Analytics, Resource Logs, and Diagnostic settings.
• Azure Healthcare APIs: Incomplete diagnostic logs.
• Microsoft Sentinel: Gaps in security-related logs, which may have hindered threat detection and alert generation.
• Azure Monitor: Reduced query results from affected log data, impacting alerting capabilities.
• Azure Trusted Signing: Incomplete signing logs, leading to billing discrepancies.
• Azure Virtual Desktop: Minor gaps in Application Insights, though overall functionality was not impacted.
• Power Platform: Discrepancies in data affecting various reports, including analytics and activity logging.

Microsoft explained that the issue was caused by a bug inadvertently introduced during an attempt to resolve a different problem in its log collection service. The fix, which aimed to address a limit in the service, triggered a deadlock condition during the telemetry upload process. As a result, some logging data exceeded the cache limit and was overwritten, becoming unrecoverable.

This latest incident comes on the heels of previous criticism aimed at Microsoft, particularly after Chinese hackers stole a Microsoft signing key in 2023, leading to breaches of corporate and government Microsoft Exchange and Microsoft 365 accounts. At the time, Microsoft faced backlash for not providing adequate logging data to customers for free, with critical logs only available through premium services. In February 2024, Microsoft expanded its free logging capabilities following pressure from the U.S. government.

The recent log loss has renewed concerns over Microsoft’s logging infrastructure, as businesses and cybersecurity experts stress the importance of reliable log data in defending against sophisticated cyber threats.