Refresher Project — Part 2: Explaining how to build a simple infrastructure with Django, Celery, Redis, Grafana and Prometheus

Refresher Project?: A glimpse of System Design and Architecture explains an overview — bird’s eye view — of the architecture behind a simple refresher project. Although it wasn’t much of a glimpse of design, more of putting a lot of components out there and imagining how it all works together. In this article, I will dive deeper into the actual components of the project and how I approach building an infrastructure.

Brainstorming

Before attempting any implementation, you should always brainstorm. In the brainstorming phase, one tends to always think of the idea itself. The more you brainstorm the idea, the more it grows and expands. You start adding special features, increasing the complexity of the idea itself. You suddenly have created a product’s concept. However, there is a fine line — a virtual threshold — where brainstorming/creativity and functionality meet. In the refresher project case, instead of brainstorming the idea, I brainstormed components and tools. In other words, starting with the bird-eye’s view of a system’s design — starting from a larger scope with objectives and then finding the right tools to make it functioning. I lay out the basic functionality I want in my system like Database Storage, Resources Monitoring and Infographics. Finally, those functionalities are now the system’s objectives.

In this project’s case the objective itself was not an idea or a product, it was a refresher project that has complex components for a technology enthusiastic software developer to refresh their technical skills. With that in mind, any idea or product can easily be added to the project later on.

I start the brainstorming for the project by separating and organizing these three main topics?:

1. The System Architecture
2. Technology Stack
3. Programming

Understanding The Data?Flow

In Refresher Project: Part 1, the focus was the connection between Django and Celery. Django being the web framework tool that provides organized access to data and Celery is the Asynchronous Tasks Manager that executes and organizes scheduled tasks. Finally, Grafana as the analytics and visualization solution for every database.

One of Grafana’s perks is the ability to connect many different datasources. One of those datasources is the PostgresSQL in the refresher system. As the celery task executes, it adds a database entry through Django that adds it to the database.?

Let’s breakdown the components in action to better understand what is actually happening. Using the first main topic System Architecture, I pictured a backend for resources management, a database, a task queuer, and visualization.?

Understanding the data flow by which how the data is transferred from a one resource to another is key to software architecture. Starting from Fig 3 for example and reaching Fig 2 is only by understanding the requirements of the data flow. The data flow however is not always sequential, you can monitor data — you are reading present and past data — while new data is being processed. Those are two different data flow lines. To better understand the data flow of the refresher project, let’s start with the user interaction points. The user can do two major tasks:

1. User can schedule a task
2. User can access dashboards

To accomplish those two major tasks you need to identify the user experience into your components and that leads to the second major topic which is Technology Stack.

Technology Stack choice is a combination of a software engineer skill set, ability to research and execute. The balance between using tools with proficiency and using tools/libraries at a complete beginner state is key. I suggest always checking ProductHunt for some of the latest software tools that can help increase your research skills and keep you updated with the upcoming tools.

So after a research process of compatibility, support and ease of implementation you choose and understand your technology stack. You would probably do that more often that you think! Also, you would follow the same iterative process if you are comparing between API methods, CI tools…etc.?

Finally, the last main topic is combining all those tools together to reach Fig 2 above. Using the major two tasks we have identified above, as a simple objective and start building your dataflow diagram. This will result in a preliminary System Design in which you have a base infrastructure.?

Building a Simple Infrastructure

Well if you look closely there are some tools like redis and Prometheus that are in the design in Fig 2 and Fig 8 but were not included in the data flow diagrams. Well, that’s exactly how those two components ended up in the System Design. It was more a dependency that was needed by another component but was not prepared in the research phase. Let’s take redis as an example.

As in Refresher Project?: A glimpse of System Design and Architecture, it is explained how Django uses a celery library that allows tasks to be stored in the database. This means that there is a defined task that is coded in Django and a reference point to that task is stored in the database. Using Django, you can combine that reference point of the task and choose to execute it at any given time. So, what actually happens during the execution of the task at the specified execution time? That’s the job of Celery Beat, Celery Worker and Redis. The Asynchronous Task Manager — Celery — will have at least two main processes running, one for Celery Beat (think of it as tasks collector) and one or more for Celery Worker(s) (think of it as a task worker). Well, Django helps celery by storing the task reference point in the database, but in runtime (the task execution runtime) where will be the collector and worker store their data? In other words, how will the celery worker know which task to execute?

As shown in Fig 2 above, and in simple terms, a celery beat will collect tasks that are scheduled for execution and store that task into a redis in memory database for pickup. Once the time of the task comes, the Celery Worker will execute that task.

Grafana & PostgresSQL?

Well, so far we covered the connection between Django, PostgresSQL, Celery, Redis and Grafana. Where is Prometheus? or What is Prometheus?

Let’s first go over Grafana and PostgresSQL. As mentioned before, one of the key features in Grafana is the ability to connect to a number of data sources. Different data source of different types as well, like SQL based sources or No-SQL based sources and the ability to apply custom queries on both.?

In our current database design, we have the following tables in the PostgresSQL database.

Where the data in those tables can be turned into a Grafana dashboard by applying some simple queries.

Checkout the public version of this dashboard here

Prometheus?

So I thought of a question that I think every developer should ask:

If I can visualize and monitor my data, why can’t I do the same for my resources?

I wanted to use Grafana to monitor my resources (any computational or storage resource). I iterated the same three main topics discussed above and after research I found a compatible tool called Prometheus — An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.

After adding Prometheus to the system design, there were some exporters missing. Data exporters that take data from one form and translate it to Prometheus language. For example, to collect resources monitoring data from your PostgresSQL database, you will need a translator between PostgresSQL and Prometheus. So, whenever you think of Prometheus, always remember you will need a data exporter for the resource you want to monitor.

Grafana & Prometheus?

After adding Prometheus as a data source to Grafana, Grafana now has access to all the data from the exporters. Meaning you can do the same thing as visualizing your data points as shown in Fig 7.

Thanks!?

Make sure you check out the live dashboard here!

Next: Technical setup of the refresher project!?