What is the best way to prioritize data modeling tasks in a data engineering project?

Even the finest chef faces challenges in a kitchen with mismatched ingredients and inadequate tools. Cooking isn't just mixing; it involves precise chopping, right ratios, and suitable vessels. While delicious meals can emerge from less-than-ideal setups, is it sustainable? Data modeling, akin to an expertly arranged kitchen, prioritizes understanding consumer demand—whether from businesses or internal/external stakeholders. It establishes a foundation that not only caters to present needs but is also future-resilient. Undoubtedly, a well-crafted data model is paramount for the success of "data" in the long run.

Let’s talk in simple terms - first identify all the key business entities and build a conceptual data model. This will help the business analyst to get across more about the data sources, data relationships and to identify a perfect data model for underlying data-warehouse both ER and Dimensional, ODS, Data marts and Data lake. Based on the CDM prepare the source to target mapping across the Staging to Base to Semantic to Presentation layer and prepare Logical Data Model & Physical Data Model. Build a Proper Data Model and it’s structure which will help the business in long term, simplify the data processing across the extraction, consumption, ETL and presentation layer.

The MoSCoW method or the Kano model, to prioritize data modeling tasks. These frameworks provide a structured approach to prioritizing tasks based on their must-haves, should-haves, could-haves, and won't-haves.

Choosing the data modelling approach is very important when you start contributing as a data engineer in a new project. For example once at work, I was asked to create a report which would quantify the productivity of a team , in terms of the document processed by that team. There three things which I had to consider:- 1. How to get the team alignment. 2. How to get the number of hours worked by an employee. 3. How to create a relationship between the team and employee. Answers to these questions helped me to design the initial data model. In this case a traditional data model, would not have helped. I had to create a alignment layer to create the relationship between the teams and employees.

Prioritize data modeling tasks in a data engineering project by having frequent connects with the stakeholders to get each and every detail regarding the requirements/KPIs/data usage/volume and also their reporting needs. All these discussions will help the modeler to design a conceptual model which then can go through multiple iterations until perfected. This conceptual model will create a strong base for developing the logical and physical model. All these detailed discussions will help in identifying perfect relationships among entities and eliminating any kind of data quality and integrity issues. In my view the traditional way of developing data models in still the best way forward.

The most crucial in any data modeling is who will use it, how they will use it, and what they hope it will accomplish. Without it you will never enable its maximum potential. Business and data architecture elements from the business perspective of both customer/user is key to the success of any modeling and should never be done in a vacuum of technology but rather through a collaboration of SMEs to unlock ultimate value and preserve variability for larger reuse and broader value.

Prioritize data modeling tasks in a data engineering project by first understanding business requirements. Identify critical data entities, relationships, and key performance indicators. Focus on high-impact areas that directly align with business objectives. Address data quality and integrity concerns early. Consider scalability and performance needs. Collaborate closely with stakeholders to ensure the model meets analytical and reporting needs. Regularly review and adapt the data model as the project evolves, maintaining flexibility and alignment with business goals.