Data Warehousing Basics: NFR

What I’m about to tell you today failed a lot of data warehousing projects which is why it’s worth paying attention so you don’t trip over the same thing. It is called NFR, or Non Functional Requirements. ?

Most of us are developers, but only a few of us are engineers. What’s the difference between a developer and an engineer? It’s NFR. We all know about business requirements. To analyse customers by purchasing patterns, to create a dashboard showing the progress of each region on monthly basis, etc. Those are known as “Functional Requirements” (FR). The function of a data warehouse is to do those things for the business.?

But, FRs are only half of the story. You get the requirements, you get the data in, you do the transformation. And voila, you get a warehouse! Fact and dimension tables, full of data. You start creating BI dashboards and job done. The business now gets what they need. You have satisfied the business requirements. The FR.?

But that’s only half of the story. What about the performance? The resilience? The administration, like adding new users and controlling access? What about the security, like encryption and authentication? Do you need to log the user activity? Do you need to monitor the ingestions, and get alert when it fails? How are you going to release it into production? How fresh should the data be? Is initial data loading required??

Has it been tested properly? Where can it be tested? Using what data? Do you need a separate environment? How will it connect to the data sources every day. Do we need a new routing? Does it need to be backed up? How will you test the recovery when it fails? Do you need to use a particular stack of technology when doing all the above? ?

All of those are called NFR (Non Functional Requirements). They are not about the function of the data warehouse. They are about the technicality, the management, the administration and the supportability of the data warehouse.?

As an engineer, you need to do development. You need to build everything from scratch: data extraction from the source, data ingestion, all the transformation, designing the warehouse, creating the report and dashboards. It is hard work, I know. But that’s only half of the story. You are doing your “developer” job.?

As an engineer, you also need to build a system which is resilient. “Production grade” quality. And for that you need security, access control, monitoring, logging, data quality, and many many more. That is our second job my friends, as an engineer.?

And that is the difference between a developer and an engineer. A developer builds a data warehouse according to the business requirements. But that is only half of the story. An engineer, build the NFR so that the warehouse becomes resilient with production grade quality.?

So, are you just a developer who develops, or an engineer who also does the NFR??

As I said in the beginning, a lot of data warehouses failed because of this. Everyone is focused on the business requirements, and forget about the NFR. And the warehouse experienced a lot of issue when it is put into production. All the NFR that I mentioned above. Well now you are aware of what they are, and you can prevent your warehouse project fails because of that.?

Hope this was helpful.?

Keep learning!?