How to implement daily active users metric using Redis

There is so much you can do with Redis but where and when to use it can be a bit of a puzzle to some. The Remote Dictionary Server (yeah that’s right, that’s what the word Redis stands for ??) will incredibly increase the performance of your app by reducing the latency of your app’s reading and writing operations to a ridiculous less than a millisecond through millions of operations per second — yes, that wasn’t a typo.

Redis has a vast variety of data structures familiar to you to meet your application needs. It lets you store your application’s data as a string, list, hash, set, JSON…and more. And guess what? Redis is super easy to use. With Redis, you write fewer lines of code to store, access, and use data in your applications. Additionally, you can use the Redis API without worrying about managing a separate cache, database, or the underlying infrastructure.

But hey! Enough of how Redis can benefit my application. I am already in love. Pray, tell me then, what really is Redis? Relax…??

According to the official documentation, Redis is an open-source, in-memory [key-value] data structure store used as a database, cache, message broker, and streaming engine. You can use Redis from most programming languages like JavaScript, Java, Python, C++, C#, Swift, PHP, etc.

Psst: This is just a high-level overview of Redis. To learn more, visit the official site.

Okay now please, and please, tell me what I can do with Redis ??. Sure, no problem! ??Redis use cases are as follows:

• Caching to decrease data access latency, increase throughput, and ease the load off your relational or NoSQL database and application.
• Chat, messaging, and queues by leveraging a variety of data structures such as lists, sorted sets, and hashes.
• Session store to manage session data such as user profiles, credentials, session state, and user-specific personalization.
• Gaming leaderboards by leveraging the Redis Sorted Set data structure.
• Management of real-time geospatial data using commands such as GEOADD, GEODIST, GEORADIUS, and GEORADIUSBYMEMBER to store, process, and analyze geospatial data in real-time.
• Fast in-memory data store to power live streaming.
• Fast in-memory data store to build, train, and deploy machine learning models quickly.
• Real-time analytics use cases such as social media analytics, ad targeting, personalization, and IoT.

And so much more!

How to implement getting daily active users from Prometheus metrics by leveraging the power of the Redis caching feature

So, now, let me show you how to maximize the Redis caching feature to solve the business problem of fetching unique active users from a real-world application in real-time. That’s probably why you are here, isn’t it?

The Problem

Let’s highlight the problem once more. You are expected to scrape a metric from a Prometheus client integrated into your Node.js application. This metric is a count of daily active users, that is, every user that explored your website at least once for the day. So, no matter if a user opens your website 1 or 1 million times a day, his activities on the website are recorded as 1 for the day.

The Plan

We will create a Prometheus counter. If a unique user opens the website and uses a protected route, the counter is accumulated by 1. That increment will remain so and will not change for that same user until the end of that day. This mechanism will apply to every unique user that opens the website for the day. The counter, therefore, keeps increasing for each unique user throughout the day. Then, on the next day, the counter will increase by 1 for each of the same unique users and will remain so until the end of the day. In a nutshell, the counter will increase by 1 per unique user each day the user visits the website.

The Blocker

In order to recognize that a user has already opened the site at least once for the day and thus not increment the counter the subsequent time he opens another protected route sooner or later that same day, we have to store the user’s information which we can use to validate the next time our application receives another request to a protected route. We do not want to store this information in our database. No way!?? ♂? What we need is a caching system.??

Redis to the Rescue!

Redis is the perfect solution. Using Redis caching, we will do the following on protected routes:

• Use the user’s ID as a key.
• Check if there are cached data based on the key.
• If there is no data, that must have been the first time the user opened the website for the day. Thus, store the user’s ID along with the result of…
• calculating how much time is left in seconds until the end of the day from the time he opened the site and using it as the expiration time for Redis.
• Then, increment the daily active users' counter by 1

Prerequisites

• Prometheus: Since this is about Redis, I assume that you already know at least the basics of Prometheus. If you don’t, I’d recommend that you first take a crash course here.?
• Grafana: A little knowledge of Grafana is ideal. But if you don’t know Grafana, take a look at their documentation. It’s pretty straightforward.?
• Redis: You should know some basic concepts of Redis. But that’s not a problem — I have already introduced you to some of them at the onset. Moreover, I’ll walk you through the installation and some basic Redis commands with the redis-cli.
• Existing Node.js Project: In this text, I will only share code snippets and screenshots of Redis, Prometheus, and Grafana implementations to achieve the subject goal, which is to fetch daily active users.

All right, let’s begin! ??

Installing Redis

How you install Redis depends on your operating system. See the guide here that best fits your needs.

Let’s explore Redis with the CLI

• After successfully installing Redis, open your terminal and execute the following command to start the Redis server.

?? redis-server        

That should result in an output that looks like this:

If you get an error, then you probably don’t have Redis installed or, for Windows users, you are not on a WSL terminal. Make sure to install Redis following the installation guide here. And if you are on Windows, make sure to open your WSL terminal. Then, try the command again.

Note that part of the information printed on your terminal says that Redis is running on port 6379 on your local machine. That is the default port number for the Redis server.

Now, while the Redis server is running, open up a new terminal and execute the following command to start the CLI and access Redis.

? redis-cli        

This should open a command prompt pointing to 127.0.0.1:6379.

Now, let’s play around with some Redis commands. Type and execute the following command to get all keys matching a pattern.

127.0.0.1:6379 ? KEYS *        

Redis maps data to keys named by you at the point of storage. Think of keys as IDs for your data. To retrieve stored data, you have to pass a key. In the above command, if the pattern specified after the “KEYS” keyword matches a key, it returns the key or keys if the pattern is “*” which signifies “all keys”.

The above command should return an “empty list or set” message…unless you have some existing data stored in the past. So, let’s create a new one. Execute the following command:

?127.0.0.1:6379 ? SET name Kenneth        

SET is a keyword for setting new data. It requires a key (name) and a value (Kenneth). Note that data stored like this are stored as strings. Now check that your key “name” exists.

127.0.0.1:6379 ? KEYS name        

You should see a list of keys including your new key (name). But, what if I am interested in the value? I want to see the value stored with a key. The command for that is:

127.0.0.1:6379 ? GET name        

GET is a keyword for getting data. It requires a key, which in this case is “name”. This will output the value stored by that key. So, you should see “Kenneth”.

Our key (and value) will be stored forever because we did not specify an expiration time on creation. However, we can delete it using this command:

127.0.0.1:6379 ? DEL name        

That should return “(integer) 1” denoting that it was successfully deleted. But what if we wanted to set an expiration time for new data on creation? The syntax for that is:

127.0.0.1:6379 ? SETEX name 15 Kenneth        

SETEX sets a new key and value with an expiration time as the argument before the value. The expiration time has to be in seconds. Thus, the above command will set a new key called “name” and assign a value to it called “Kenneth” and starts a timer on it for 15 seconds. To get information about how much time is left to expire, the syntax is:

127.0.0.1:6379 ? TTL name        

TTL stands for “Time To Live”. It requires the name of the key to which it should ascertain how much time it is set to live. Executing the above code again and again will produce different output as the time to live for the key (name) is counting down. Once it gets to -2, it means the key is gone.

Prometheus

As mentioned earlier, I will focus only on the part of the Prometheus code where the metric logic is implemented. When implementing Prometheus, I usually create a “metrics.js” file where I define my metrics. Copy and paste the following code snippet into your metrics file:

?import client from 'prom-client'

?export const dailyActiveUsersGauge = new client.Gauge({

??name: 'my_daily_active_users',

??help: 'Active users metrics on a daily basis.',

??labelNames: ['user_id', 'year', 'month', 'day'],

?});        

Usually, you would have a middleware that validates and verifies the request “jwt token”. For this, I would create a “protect.js” middleware file. It would contain my jwt verification logic. If the verification is passed, I would get the user’s ID (which will have been used as the id required when issuing the jwt token) from the decoded token. Then, I would find the user whose ID matches the decoded token ID and attach the user’s details to a new property on the request object called “user” (so, req.user). Finally, the next() middleware in the stack is called.

All right, that’s just my opinion. But, however you designed your authentication logic, as long as you can get the user’s ID from the request object, mount the following snippet:

?import { dailyActiveUsersGauge } from 'path/to/metrics.js'

?dailyActiveUsersGauge.inc(

????{

???????user_id: currentUser.uid,

???????year,

???????month,

???????day,

????},

????1,

?);        

This would create a new gauge metric with the name “my_daily_active_users” and some labels (year, month, and day) with a cumulative sample value of 1.

Redis

We can’t rely on our current implementation without the help of Redis because the sample value will keep increasing every time the same user uses a protected route and the “dailyActiveUsersGauge” metric is invoked. That would produce an inaccurate result at the end of the day. So, let’s use Redis to cache the user’s ID.

Let’s install the “redis” npm package

? npm i redis        

Create a file in your project folder (I would call it redisCacheHandler.js) and paste the following code into it:

 import { createClient } from 'redis'

?let client;

?if (process.env.NODE_ENV === 'production') {

??client = createClient({

????url: `redis://${process.env.REDIS_USER}:${process.env.REDIS_PASSWORD}@${process.env.REDIS_HOSTNAME}:${process.env.REDIS_PORT}`,

???});

?} else {

??// You should install redis and run the service command ----> redis-server

???client = createClient();

?}?

?client.on('error', (err) => console.log('Redis Client Error', err));

?async function connectRedis() {

???await client.connect();

?}

?connectRedis();

?const defaultExpirationTime = 60 * 10; // 60 seconds times 10 -> 10 minutes

?const getOrSetCache = (cb) => cb();

?export const getCache = (key) =>

???new Promise((resolve, reject) => {

?????try {

???????getOrSetCache(async () => {

????????const data = await client.get(key);

????????if (data) {

??????????console.log('SENDING CACHE...??');

????????}

????????resolve(JSON.parse(data));

??????});

????} catch (error) {

??????reject(error);

????}

?});

?export const setCache = (key, data, expTime) => {

??if (typeof expTime === 'undefined') {

????expTime = defaultExpirationTime;

? }

 return new Promise((resolve, reject) => {

????try {

??????getOrSetCache(async () => {

????????const isOk = await client.set(key, JSON.stringify(data), {

??????????EX: expTime,

????????});

????????if (isOk) {

??????????console.log('CACHED! ?');

????????}

????????resolve();

??????});

????} catch (error) {

??????reject(error);

????}

??});

?};        

Briefly explaining the code:

• Import the redis library.
• Check which environment the application is running in. If in development, use the local redis-server (make sure it is running). If in production, connect to Redis Cloud.
• Export two functions:
• getCache: checks for stored data via given key. Returns a Promise of the data (could be undefined).
• setCache: sets a new key-value data and returns a Promise. Expects key, data, and an expiration time in seconds arguments. If no expiration time is provided, use the default expiration time which is 10 minutes.

Now, go back to where we implemented the “dailyActiveUsersGauge” function and modify your code as follows:

?import { dailyActiveUsersGauge } from 'path/to/metrics.js'

?import { getCache, setCache } from 'path/to/redisCacheHandler.js';


    // .. ..

??  const cacheKey = currentUser.uid;

??  const date = new Date();

????const year = date.getFullYear();

????const month = date.getMonth();

????const day = date.getDate();

????const now = Date.now();

????const endOfDay = date.setUTCHours(23, 59, 59, 999);

????const secondsLeftTillEndOfDay = Math.floor((endOfDay - now) / 1000);

????const cachedData = await getCache(cacheKey);

????if (!cachedData) {

??????dailyActiveUsersGauge.inc(

????????{

??????????user_id: currentUser.uid,

??????????year,

??????????month,

??????????day,

????????},

????????1,

??????);

??????// cache

??????await setCache(cacheKey, '_', secondsLeftTillEndOfDay);

???}

// .. ..        

Briefly explaining the code:

• Import our custom redis modules. Get the day, month, and year and pass them to the dailyActiveUsersGauge function. We’ll need this information, especially the “day”, for grouping our metrics with Prometheus. That is exactly how we intend to get the daily count in the end.
• Use the user’s ID as a key
• Check if there’s a cached data based on the key
• If there is no data, that must have been the first time the user opens the website for the day. Thus, store the user’s ID along with the result of…
• Calculating how much time is left until the end of the day from the time he opens the site in seconds as the expiration time
• Then, increment the daily active users' gauge by 1

The above implementation will make sure that we have an accurate daily users count as the counter will only increase if there is a new user or new key thanks to Redis.

Grafana

On your Grafana dashboard, create a new panel and structure the Prometheus metrics browser like this

Conclusion

Redis is a potent tool that can be leveraged almost anywhere in your web applications. Hopefully, this text has been helpful and more importantly an eye-opener to the power of Redis.

This post is in collaboration with Redis.

To learn more, explore the following reference links:

• Try Redis Cloud for free
• Watch this video on the benefits of Redis Cloud over other Redis providers
• Redis Developer Hub - tools, guides, and tutorials about Redis
• RedisInsight Desktop GUI