Quick Intro to Time series databases (TSDBs)

Companies like Amazon, Uber, and Facebook use time series databases (TSDBs) to store and analyze large volumes of time-series data. But what exactly are TSDBs, and what use-cases are they best suited for?

For instance,

1. Amazon: Amazon uses a TSDB called Timestream to store and analyze data from IoT sensors in its warehouses. Timestream enables Amazon to track environmental factors such as temperature, humidity, and other metrics to optimize warehouse operations and reduce energy usage.
2. Uber: Uber uses a TSDB called M3 to store and analyze real-time data from its ride-hailing platform. M3 enables Uber to track the location of drivers and riders, analyze traffic patterns, and optimize its system to reduce wait times for riders and improve overall efficiency.
3. Facebook: Facebook uses a TSDB called Gorilla to store and analyze metrics related to its user base, such as the number of daily active users, page views, and ad impressions.

What is Time Series Data ?

In simple terms, data that is generated over time at regular intervals.

You can think of it as a collection of observations that are recorded in chronological order. Each observation is associated with a specific timestamp or time period, such as seconds, minutes, hours, days, weeks etc.

Internals in brief

TSDBs have two primary components: a storage engine and a query engine.

It uses combination of indexing, compression, caching, and aggregation techniques to efficiently store and query time-series data, while also providing scalability and fault-tolerance through sharding and replication.

For example, TSDBs may use an index to quickly locate the data for a specific time range or use pre-computed aggregates to reduce the amount of data that needs to be processed for a query.

Popular Use Cases

TSDBs can be used to store and analyse :

1. Financial data: tracking stock prices, exchange rates, and other financial metrics over time.
2. IoT sensor data: such as temperature, motion, humidity etc.
3. Application performance metrics: performance data such as response times, resource usage, and error rates.
4. Log data: such as server logs, application logs, and other types of log data that are generated over time.

Popular OpenSource TSDBs that you can explore ??

1. InfluxDB: InfluxDB is optimized for high write throughput and efficient querying.
2. TimescaleDB: TimescaleDB is built on top of PostgreSQL, providing SQL-based querying and horizontal scaling capabilities.
3. OpenTSDB: OpenTSDB is built on top of HBase, it is also designed to scale horizontally. (by adding more nodes to the cluster)
4. Prometheus: Prometheus is optimized for monitoring and alerting, providing a powerful query language and integration with other monitoring tools.

These DBs uses a query language that is specifically designed for time-series data, such as PromQL, InfluxQL, or OpenTSDB's Query Language. These query languages support time-based filtering and aggregation operations.

TSDBs are used along with other big data technologies, such as Apache Kafka and Apache Spark, to handle large volumes of time-series data in real-time.

References and Good Reads

• Gorilla: A Fast, Scalable, In-Memory Time Series Database (Research Paper published by Facebook) : https://www.vldb.org/pvldb/vol8/p1816-teller.pdf
• Amazon Timestream: https://aws.amazon.com/timestream/
• Uber M3: https://eng.uber.com/m3/
• InfluxDB: https://www.influxdata.com/products/influxdb/
• OpenTSDB: https://opentsdb.net/
• Prometheus: https://prometheus.io/
• TimescaleDB: https://www.timescale.com/

_______________________________________________________________

If you found this article informative, don't forget to leave a like and subscribe to the newsletter for more snackable system design concepts. The newsletter has already reached 5.1k+ subscribers!

Feel free to connect with me here : https://linktr.ee/asmah98