Introducing M3 The Time-Series Infrastructure that Powers Metrics at Uber

Time series analysis is one of the fastest growing areas across disciplines such as machine learning or probabilistic programming. The emergence of markets such as the internet of things(IOT) or social networks has increased the relevance of time series infrastructure to power analysis over real time data. The importance of time series analysis have influenced the release of open source stacks such as Graphite or Prometheus. However, many of the top internet have regularly outgrown those stacks and pursued the path of building their own time series infrastructure. Uber is one of the companies that have contributed the most to the time series data infrastructure space. Earlier this year, the transportation giant decided to open source the stack that has been powering their time series analysis for years: M3.

Time is a core element of the Uber experience across its different apps. As a result, time series analysis seems to be a multiple more relevant than on other types of large scale businesses. Initially, Uber relied on traditional time series stacks such as Graphite, Nagios, StatsD and Prometheus to power their time series metrics. While that technology stack worked for a while, it was not able to keep up with Uber’s stratospheric growth and, by 2015, the company was in need of a proprietary time series infrastructure. That was the origin of M3 which was designed with five key guiding principles:

• Improved reliability and scalability: to ensure we can continue to scale the business without worrying about loss of availability or accuracy of alerting and observability.
• Capability for queries to return cross-data center results: to seamlessly enable global visibility of services and infrastructure across regions.
• Low latency service level agreement: to make sure that dashboards and alerting provide a reliable query latency that is interactive and responsive.
• First-class dimensional “tagged” metrics: to offer the flexible, tagged data model that Prometheus’ labels and other systems made popular.
• Backwards compatibility: to guarantee that hundreds of legacy services emitting StatsD and Graphite metrics continue to function without interruption.

M3

High scalability and low latency are key principles of the M3 architecture. Any given second, M3 processes 500 million metrics and persists another 20 million aggregated metrics. Extrapolating those numbers to a 24-hour cycle indicate that M3 processes around 45 TRILLION metrics per day which is far beyond the performance of any conventional time series infrastructure. To handle that throughput, M3 relied on an architecture based on the following components:

· M3DB: M3DB is a distributed time series database that provides scalable storage and a reverse index of time series. It is optimized as a cost effective and reliable real-time and long term retention metrics store and index.

· M3Query: M3 Query is a service that houses a distributed query engine for querying both real-time and historical metrics, supporting several different query languages. It is designed to support both low latency real-time queries and queries that can take longer to execute, aggregating over much larger datasets, for analytical use cases.

· M3 Aggregator: M3 Aggregator is a service that runs as a dedicated metrics aggregator and provides stream based down sampling, based on dynamic rules stored in etcd.

· M3 Coordinator: M3 Coordinator is a service that coordinates reads and writes between upstream systems, such as Prometheus, and M3DB.

· M3QL: A query language optimized for time series data.

The relationship between the core M3 components is shown in the following figure:

Let’s explore some of the previous architecture building blocks in more details.

M3DB

M3DB is the core storage model in the M3 infrastructure. The stack was built in Go and designed for large-scale time series analysis from the ground up. The storage model is both distributed and strongly consistent which facilitates scalabilities while maintaining robust write dynamics. M3DB uses both in-memory and disk storage models depending on whether the records are frequently accessed or just used for long-term calculations respectively. From the management standpoint, M3DB is highly configurable and supported on a wide range of runtime environments.

One of the main contributions of M3DB is its clever storage model. Most transformations within a specific query are applied across different series for each time interval. For that reason, M3DB stores data in a columnar format facilitating the memory locality of the data. Additionally, data is split across time into blocks, enabling most transformations to work in parallel on different blocks, thereby increasing our computation speed.

M3QL

Since the early days, M3 supported Prometheus Query Language(PromQL) and Graphite’s path navigation language. To extend the data access capabilities of M3, Uber decided to build M3QL a pipe-based language that complements the capabilities of path navigation with richer data access routines. Just like other pipe-based languages, M3QL allows users to read queries from left to right offering a rich syntax as shown in the following figure.

M3 Query Engine

Just like other M3 components, the query engine was written in Go from the ground up and optimize for high throughput. Recent metrics from Uber are benchmarking 2500 queries per second being processed by M3’s query engine. The query engine workflow is structured into three main phases: parsing, execution and data retrieval. The query parsing and execution components work together as part of a common query service, and retrieval is done through a thin wrapper over the storage nodes. To support multiple query languages such as M3QL or PromQL, M3 introduces an intermediate representation based on a directed acyclic graph(DAG) which abstracts the query that needs to be executed. The current implementation of the query engine is tied to M3DB but the design can support other time series databases.

M3 Coordinator

M3 is a very complete platform but also enables the integration with mainstream time series analysis systems such as Prometheus. M3Coordinator is a service which provides APIs for reading/writing to M3DB at a global and placement specific level. It also acts as a bridge between Prometheus and M3DB. Using this bridge, M3DB acts as a long term storage for Prometheususing the remote read/write endpoints.

Getting started with M3 is relatively easy as the entire platform is packaged as Docker containers. The infrastructure has been tested on major cloud platforms such as Google Cloud and the entire source code is available in their GitHub repository.

M3 is certainly one of the most advanced infrastructures for time series analysis in the current market. While M3 might lack the support of commercial alternatives, it comes with the robustness developed during years of supporting Uber’s time series analysis processes. Doesn’t get much better than that.