Navigating the Data Landscape: Evolution from Traditional Warehousing to Modern Lakehouse Architecture

For generations, the Data Warehouse has been a crucial tool for generating reports and dashboards. Essentially, it stores vast historical data in a way that focuses on specific subjects. Data warehouses allow organizations to bring together data from various sources into one unified location, keeping analysis separate from transactional activities.

There are two types of database systems that serve different purposes in handling and processing data – OLTP and OLAP. There is a difference between OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing) systems and how OLAP systems assist in meeting reporting needs. But, is the traditional data warehouse still the best solution, and if not, what has changed recently to prompt this question?

Here are some changes in the technology landscape in recent years. Although these changes are not directly related to the data warehouse, they impact data storage and reconciliation strategies.

Cheaper Storage: Storing application data has become more affordable nowadays, and even maintaining historical data in the transactional system is not as costly.

Faster Processing: With faster processing units and memory, applications can perform transactions and analytical queries more swiftly without significantly impacting the application.

Elasticity: Many cloud providers offer elastic capacity, making it more manageable to increase or decrease resources at runtime.

Data Variations: It's not just about relational or structured data anymore; there's a combination of unstructured and semi-structured data.

Considering the changing analytics and reporting needs, there's a shift towards more accurate time analytics than historical ones. Business needs are evolving rapidly. So, does this mean there's no longer a need for a data warehouse? Not exactly.

As systems and businesses evolve, data warehouses are also adapting with new structures. Call it what you want, but a term gaining popularity is "Data Lakehouse."

A Data Lakehouse combines features from both a data warehouse and a data lake. It allows us to store various types of data and, in some cases, access data directly from the source without moving it. In my proposed architecture, I suggest using operational data storage, data marts, and enterprise data warehouses within the data lakehouse.

Operational Data Store (ODS): This might be a traditional system storing raw data from source systems. In the lakehouse, I propose moving the data only if it's necessary. Sometimes, it's better to integrate the data directly with the source system. This is part of the medallion architecture, also called the bronze layer, encompassing structured, unstructured, or semi-structured data.

Data Marts: Why do we need data marts within the lakehouse structure? These are essential for specific business needs and Key Performance Indicators (KPIs). What's different from traditional data marts is the reduced need to move data between systems. Data marts will reside in the ODS (Operational Data Storage) system, enabling businesses to target specific KPIs.

Enterprise Data Warehouse: This still plays a unique role within the lakehouse. This data is refined and only necessary at the enterprise level. It remains canonical or highly structured, facilitating self-reporting in tools like Power BI and Tableau.

In summary, we still require a data warehouse, but in a more evolved form to meet the latest business needs and adapt to the changing technology landscape.

Disclaimer from TECHGINIA

The opinions expressed in this article are those of the guest author and do not necessarily reflect the views of our publication. The information provided in this article is for general informational purposes only, and should not be considered as professional advice. The reader should always conduct their own research and due diligence before taking any action based on the information provided in this article.