?? Redefining Redis: Why We Chose It as Our Primary Database for P0 Services ??

It's not every day you hear about Redis being used as the primary database. Many consider it a cache, and that's where it shines, right? Well, let me share why it's more than acceptable, even beneficial, to have Redis as your main data source.

Multi-model database :

We need different database services for different use cases like Relation DB to store Data, Elastic search for search and filtering etc..

It's obvious that this is a pretty complex setup . which could have the following challenges

• Each data service needs to be deployed and maintained
• Know-How needed for each data service
• Different scaling & infrastructure requirements
• More complex application code for interacting with all these different DBs
• Higher Latency (Slower), because of more network hopping

Redis provides Multi-model database: where you just need to run and maintain single database service.

Redis is Modular: Redis isn't just a one-trick pony. It supports multiple data types and extends its core functionality with modules tailored to different data needs. Think of RediSearch for robust search or Redis Graph for efficient graph data storage. Redis's modularity empowers your application to handle diverse data requirements effortlessly.

Redis is Fast: Redis thrives in its in-memory database glory. Data stored in RAM means blistering speed and high performance. It's not just about Redis; it's about making your entire application faster, resulting in a superior user experience.

Out-of-the-Box Caching: Redis is a powerhouse of caching. When you use Redis as your primary database, you don't need an additional caching layer—it's already baked in. Less complexity in your application, fewer worries about cache management.

Now, here comes the kicker.

Data Persistence. You might ask, "How can an in-memory database ensure my data is safe if the Redis process or server crashes?" Excellent question!

Data Persistence in Redis

1. Replication: The simplest way to safeguard your data is through replication. When your primary Redis instance falters, the replicas take the baton. With replicated Redis, your data remains intact, ensuring continuity.
2. Snapshotting & AOF: Redis offers two powerful mechanisms for data persistence. Snapshots: These are periodic data backups that you configure based on time or other criteria. Snapshots are stored on disk, and they become your lifeline for data recovery if the Redis database is compromised. Keep in mind that you might lose the last few minutes of data, depending on your snapshot interval. Append Only File (AOF): AOF is a continuous data saver. Every change is written to the disk, ensuring durability. After an outage, Redis replays AOF logs to rebuild the state. It's more robust, although it can be slower than snapshotting.

??The Best Approach? Use both AOF and snapshots. AOF continuously persists data from memory to disk, while snapshots serve as checkpoints in case you need to recover your data state.

Now, let's talk about scaling a P0 service with Redis.

Scaling Redis Databases

Redis offers a couple of strategies:

1. Clustering: Redis supports clustering, which involves a primary (master) Redis instance for reading and writing data and multiple replicas for reading. This not only scales Redis to handle more requests but also enhances high availability.

Fun fact: Clustered Redis have a decentralised architecture for multiple nodes talking to each other using an internal protocol which supports:

1. Fault tolerance: the majority of nodes decide the health of each node based on read pings.
2. O(1) complexity for get calls.Find more details on Clustered Redis: here

2. Sharding:

Well that seems good enough, but what if

1. your dataset grows too large to fit in a memory on a single server.
2. Plus we have scaled the reads in the database, so all the requests that basically just query the data. But our master instance is still alone and still has to handle all the writing.

So what is the solution here? ??

For that, we use the concept of sharding, which is a general concept in databases and which Redis also supports.

So sharding basically means that you take your complete data set and divide it into smaller chunks or subsets of data, where each shard is responsible for its own subset of data.

So that means instead of having one master instance that handles all the writes to the complete data set, you can split it into say 4 shards, each of them responsible for reads and writes to a subset of the data. ??

And each shard also needs less memory capacity, because they just have a fourth of the data. This means you can distribute and run shards on smaller nodes and basically scale your cluster horizontally:

All this is fine... but what about the cost?????

Here are some cost optimisation strategies which we have implemented.

In-Memory Cuckoo Filter: We've implemented in-memory Cuckoo Filters (an extension of Bloom Filters) to store eligible Redis keys. Before querying Redis, we check the Cuckoo Filter to reduce cache misses and IOPS on the Redis cluster. Updates are handled in near real-time through pubsub events, ensuring data accuracy.

Cuckoo Filter and bloom filter are space-efficient probabilistic data structures, Its primary purpose is to test whether a specific element belongs to a set or not.

Protobuf Compression: We compress data with Protobuf before storing it in Redis, reducing memory usage by approximately 30%.

TTL Optimization: We've fine-tuned TTL settings and minimized payload size to keep Redis operations efficient and cost-effective.

Mindful use of Pubsub events: Redis Pubsub events can get costly we need to use them mindfully on specific key patterns to reduce the cost.

Redis isn't just for caching—it's a robust primary database. Redis empowers us to deliver high-performance applications with data safety and cost efficiency. ?? #Redis #Database #DataPersistence #Scaling #CostOptimization