USING REDIS WITH NODE.JS

Article written by Dumitru Bereghici .

In today’s fast-paced digital landscape, user expectations for application performance are higher than ever. As developers, we strive to deliver seamless and lightning-fast experiences to our users. However, achieving optimal performance can be challenging, especially as applications scale and handle increasing loads.

One powerful technique for improving the performance of web applications is caching. Caching involves storing frequently accessed data in a temporary storage layer, allowing subsequent requests for the same data to be served quickly without the need to recompute or retrieve it from the original source.

Why is Caching Important?

1. Faster Response Times: By caching frequently accessed data, we can reduce the time it takes to fulfill user requests. This results in faster response times and a more responsive user experience.
2. Scalability: Caching helps alleviate the load on backend resources, allowing applications to handle more concurrent users and scale more efficiently.
3. Reduced Latency: With caching, data can be served from memory, which typically has lower latency compared to retrieving data from disk or over the network.
4. Improved Efficiency: Caching reduces the need to repeatedly fetch or compute the same data, leading to more efficient resource utilization and cost savings, particularly in cloud environments.

Now that we understand the importance of caching, let’s explore Redis, a popular in-memory data store, and how it can be leveraged in Node.js applications.

What is Redis?

Redis, short for Remote Dictionary Server, is an open-source, in-memory data store. It is often referred to as a data structure server due to its ability to store and manipulate various data structures like strings, lists, sets, and more. Redis is renowned for its exceptional speed, making it an ideal choice for caching and real-time applications.

Why use Redis?

Redis is an open-source in-memory data repository that serves as a key-value database. It provides extreme performance because it stores data in memory instead of on disk like other database technologies. In the context of a backend application, Redis is an excellent choice for implementing a caching system. Its speed and efficiency are critical for caching the most requested endpoint responses to reduce the server response times.

Redis is fast, primarely because of three reasons:

1. Speed: Redis stores data in memory, making it incredibly fast for read and write operations
2. Efficient Data Structures: Redis employs simple and efficient low-level data structures
3. Simplicity: Redis offers a straightforward feature set, simplifying implementation and usage

Implementing Redis Cache in a Node.js Application

Now, let’s dive into the practical implementation of Redis caching in a Node.js application. Let’s start by introducing a demo application built with Node.js and Mongoose (a MongoDB object modeling tool), and then demonstrate how to integrate Redis as a caching layer to improve performance and scalability.

Prerequisites

You can run the application either locally or with Docker.

Running locally

• Node.js 18+: If you don’t have Node.js installed on your machine, download the installer from the official site, launch it, and follow the wizard.
• Redis: For running redis, there are two options:You can create a free instance of Redis on redis.com. This is a publicly available website that you can use to run RedisYou can install and run Redis on your local computer. How you install Redis depends on your operating system. See the official documentation to install redis on local machine: Install Redis locally

Running with Docker

Make sure you have Docker and Docker Compose installed on your machine.

App Overview

The application we are going to work on is a simple client-server app with React and Node.js with Express, MongoDB and mongoose. You can clone the repository on the following link: Blogs App. Follow the steps described in the readme file to set up your project.

The Client application is a React app with basic functionality for managing blog posts. Users can log in with Google. Then they can see a list of blog posts on the home page and can add new blog posts.

The Backend application is an express app with MongoDB, mongoose and uses passport for handling authentication.

Inside the backend application, you’ll find a folder routes with two files: auth.js and blogs.js. The auth.js file contains the routes used for the authentication flow.

Now let’s have a look on what’s inside the blogs.js file. It contains two endpoints, one for fetching blogs based on user ID and one for adding new blogs.

const mongoose = require('mongoose');

const requireLogin = require('../middlewares/requireLogin');

const Blog = mongoose.model('Blog');

module.exports = app => {

app.get('/api/blogs', requireLogin, async (req, res) => {

const blogs = await Blog.find({ user: req.user.id });

res.send(blogs);

});

app.post('/api/blogs', requireLogin, async (req, res) => {

const { title, content } = req.body;

const blog = new Blog({

title,

content,

user: req.user.id

});

try {

await blog.save();

res.send(blog);

} catch (err) {

res.send(400, err);

}

});

};

Adding Redis to our GET endpoint

Now, let’s see how we can integrate Redis to cache this list of blogs associated with an user. To leverage Redis for caching, you first need to install the Redis npm package. Once installed, you can integrate Redis into your codebase using the following approach:

const mongoose = require('mongoose');

const requireLogin = require('../middlewares/requireLogin');

const Blog = mongoose.model('Blog');

const redis = require('redis');

const redisUrl = 'redis://127.0.0.1:6379';

const client = redis.createClient(redisUrl);

const util = require('util');

client.get = util.promisify(client.get);

module.exports = app => {

app.get('/api/blogs', requireLogin, async (req, res) => {

// Do we have any cached data in redis related to this query?

const cachedBlogs = await client.get(req.user.id);

// If yes, then respond to the request right away and return

if (cachedBlogs) {

return res.send(JSON.parse(cachedBlogs));

}

// If not, we need to respond to request and update our cache to store the data

const blogs = await Blog.find({ authorId: req.user.id });

res.send(blogs);

client.set(req.user.id, JSON.stringify(blogs));

});

};

This code snippet demonstrates a caching mechanism using Redis. It first checks if the requested data is available in the cache. If it is, the cached data is immediately returned to the client. Otherwise, it fetches the data from the database, sends it to the client, and then updates the cache with the fetched data for future requests.

Note

We use Promisify function to wrap Redis get function with a promise. It is also possible to use a callback, but it is better to avoid them as much as possible, as the callbacks are a kinf of a dying feature of node.

const cachedBlogs = client.get(req.user.id, () => {});
        

Improving our cache solution

With the current setup, there are a few issues with the current code.

1. Cache keys won’t work when we introduce other collections or query
2. Caching code isn’t easily reusable anywhere else in our codebase
3. Cached values never expire

Solution:

1. Figure out a more robust solution for generating cache keys
2. Hook into Mongoose’s query generation and execution process
3. Add timeout to values assigned to redis. Also add ability to reset all values tied to some specific event

Key creation

Right now our caching key relies solely on user ID. Let’s imagine that later on we want to implement an API functionality for storing Tweets, which also have an user. If we were to use user ID as cache key, our Redis implementation won’t work.

To ensure uniqueness and versatility, we can use the following strategy for generating a unique key. Instead of user ID, we can use both the MongoDB query and the collection involved. This allows for a robust caching mechanism that remains effective across various data entities.

    const cacheKey = JSON.stringify({
        query: req.query,
        mongooseCollection: 'blogs' // Assuming 'Blog' is the Mongoose model for blogs
    });
        

The ‘query’ parameter encapsulates the conditions used in the MongoDB query to retrieve data. In the context of our application, it represents the filtering criteria applied to fetch blogs from the database. For instance, it could include conditions like the user’s ID, date range, or any other relevant filters specified in the request. Example of a query:

const query = Person
  .find({ occupation: '/host' })
  .where('name.last').equals('Doe')
  .where('age').gt(17).lt(65)
  .limit(10)
  .sort('-occupation');

        

Using this strategy ensures uniqueness and prevents potential key colissions and ensures that cached data remains distinct for each unique query.

Patching mongoose’s ‘exec’

Next, in order to make the caching code reusable anywhere in the app, we are going to add a cache layer by ‘patching’ mongoose Query exec function via prototype.

Add a new file cache.js and add the following code:

In the provided code snippet, the functionality of Mongoose queries is extended to incorporate caching mechanisms. Mongoose queries are typically constructed using the Query class and executed using the exec function. Leveraging JavaScript’s prototype feature, the code enhances the exec function to include caching logic. This means that before executing a query, the system first checks if the result for a similar query has been cached. If a cached result exists, it is returned directly, avoiding unnecessary database operations. If not, the query is executed as usual, and the result is cached for future use.

Now we are able to query in an efficient way our DB and cache for every query we want without to change our code.

Cache Expiration

When caching data, you need to know how often the data changes. Some API data changes in minutes; others in hours, weeks, months, or years. Setting a suitable cache duration ensures that your application serves up-to-date data to your users. When the cache expires, your application will send a request to the API to retrieve recent data.

To set the expiry time for the cache, you can use the following snippet. This will set the expiry time to 10 seconds.

  client.set(key, JSON.stringify(result), 'EX', 10);
        

Toggleable Cache

So far, the caching solution works pretty well, but there is still an issue. Currently, the caching logic resides within the exec function, resulting in all queries being cached automatically upon execution. While this approach may suffice in most cases, there are scenarios where caching every single query may not be ideal, especially considering the potentially high cost of Redis storage. Let’s explore how we can implement a more selective caching mechanism, allowing us to specify which queries should be cached and which should not with ease.

We are going to modify the cache.js file by extending the Query prototype with a new function called cache. Then we will set the useCache property of the query instance to true. This property is used to indicate whether caching should be enabled for the query.

Nested hashes

Currently, our keys for Redis are comprised of query parameters and the collection name. This approach can present several disadvantages, especially when the data structure is complex:

• it is challenging to represent and manage data with multiple levels of relationships
• cache invalidation strategies may become less precise
• as the dataset grows, scalability becomes a concern

Now, let’s re-implement our Redis schema and rather than implementing a flat data store, we should instead store the data in separated nested hashes.

Automated Cache Clearing with Middleware

Clearely, after adding a new post to the database, it’s important to note that subsequent GET requests may not return the up-to-date data. To ensure that the latest information is always served, it’s important to clear the cache after any data modification operations. To do that, we can create a middleware function.

This middleware function clears the cache associated with the user ID after the route handler has been executed. By using the await keyword, it ensures that the cache clearing process occurs asynchronously, after the completion of the route handling code. As a result, this middleware function serves as the final piece in the middleware chain, ensuring that the cache is cleared only after all other route-related operations have been performed.

Conclusion

The scope of this blog post was to show some practical examples of how to use Redis effectively with Node.js. If you follow all steps above carefully, your project will now contain the same code as this repository.

With Redis integrated into our application, it now boasts improved performance and speed.

Keep coding!