Caching - In Code or External?

When writing applications, often we think about Caching in all sorts of applications - API services, databases, consumer apps, Android apps and what not. Caching is a beautiful technique to have your data available instantly with minimum overhead and maintenance.

Is Caching always cheap?

One thing to consider when building scalable systems is that when you build a caching layer, how costly is it going to be? Now this question has so many scenarios and variables that we can't cover them here fully. Let's try to check a few major things, and you can solve this system design Hands-On on my coding platform - Eazy Develop

Amount of Data

Obviously, the first cost factor would be the amount of data stored in any cache layer. Now as simple as it seems, it poses a few design issues.

Let's say we are building an application which caches User data (name, dob, other basic info). Now let's say we are having 100 Users in the system, we deploy say 2 instances of our application. Each User's average data size (data in cache) is about 1KB.

Now, we can easily store this cache In Memory using an In-Code approach. Using this, a library or some custom code (using Hash maps and lists and whatnot) to store the data in the application code's memory. Given 2 instances of application, both the instances would use 100KB of extra memory (RAM) to store this cache. Total memory consumption = 200KB.

Scaling to 10k Users

Now, let's say we have got scale and it scaled to 10k users and we scaled our instances to say, 5, to support more concurrent users of the application. Now given our previous approach, we use 10K * 1KB * 5 instances = 10^4 * 1000B * 5 = 5 x 10^7B = 50MB (let's assume 1000KB = 1MB for simplicity)

Total extra cache memory used by our application is 50MB for 10k users and 5 instances. Now even this looks good and might suffice. Let's increase the scale more.

Scaling to 10M Users

Now our application is widely famous and has 10M users using the application with approximately 100 instances of our application running at the backend for high availability and failover.

Using our previous method of using In-Code library, we are going to use - 10M * 1KB * 100 = 10^7 * 1000B * 100 instances = 10^14 B = 1TB of Memory (RAM), divided as 10GB of memory per instance.

Now you might think, how can I get >10GB RAM Virtual machines? Even if we get it on some cloud platform, how costly would that be?

The Problem of In Code Caching

One thing you might have noticed is that we have data duplication 100x because we are storing all users all information in every instance. You might argue that why not include stickiness and only have those User's data in those instance which the User would be accessing. Theoretically, yes you can do it. Should you? Definitely not. This is too much overhead to maintain this stickiness, maintain replica for each User's instance for high availability, distributing the data fairly and evenly and what not. It'll be a nightmare.

External Caching Layer

Let's revisit the problem and use of Caching again before dwelling into the the solution. We cache our data (mostly Database calls / API calls) and those information which is not static (but stay same for some timeframe). So technically, there is 1 SOT (source of truth) being your database which being a persistent layer becomes slow on heavy read usage (Using disk instead of Memory/RAM).

So why not have an external layer / application which stores the data in its memory but makes it available for every instance of your application? It's like outsourcing the Caching logic and management to external system. It will decrease the memory required by your application by 100x factor (above example, let's ignore cache replication) and still have the same performance (network call overhead yes, but let's ignore as not very prominent).

So now with 10M users, you have 1TB of cache layer and each instance of your application would use this cache layer to fetch the data. Later, you can use a distributed cache (topic for separate article) to distribute this cache horizontally to have lower order machines (say 100 machines of 10GB memory).

Some very famous Distributed Caching layers (battle tested) are Redis, Memcached, etc.

Conclusion

So as you saw, it depends on your application use case and scenario, what type of caching system would be beneficial for you. In order to learn it with a Hands-On experience, do SignUp on my Coding Platform - Eazy Develop, which will have much more in-depth problems and challenges for you to code and solve.