How do you consider business and user needs when approaching data engineering projects?

For a Data Engineering project, the first and foremost step is to understand the business requirements and the underlying data. Below are the steps one must follow in order to generate effective insights : 1. What the business gains out of the Analysis and how will the analysis output be used 2. Understanding of the business problem 3. Data availability (open source/ internal teams/ TPAs required) 4. Data Quality & Reliability (this is important as the analysis results will rely on the data being used) 5. Defining the key performance indicators (KPIs) relevant to the business requirement 6. Defining the scope of the analysis and how to use the results in such a way that it justifies the business requirement

From my perspective this is where the Conceptual Data Model comes into play, perhaps better named a Conceptual Information Model. It identifies business objects, their inter-relationships and business definitions , it is solution agnostic and allows the business to articulate the information in scope in terms relevant to themselves. As a reference for solution/data architects and developers it provides a clear unambiguous set of statements regarding the information and relationships in scope.

Knowing what the underlying assumptions are going into a meeting is key. If there are misconceptions or confusion about where data exists, definitions of data points, or who has access to what data sources, then it is possible that the goal setting itself will be based on bad data.

Most important design principle is "Know your data". You cannot add value if you dont know the data you are dealing with. You can create data requirements and specifications only if you have "seen" the data. Data Analysis and profiling will help in the first step of your Data Engineering project.

Selection of the right Data Engineering option has to be a "Data Engineering team" function and not a Senior management driven decision taken on a Golf course. Since this has serious long term implications for data migration strategy, data maintenance, data governance, tech resource staffing and support ops. In my experience, most firms don't spend enough time and money on the assessment and then end up spending millions on proving why their (wrong) decision makes sense and works. Just because everyone else is doing, does not mean you have to. Think of what goal will that achieve and if it fits the purpose. Otherwise it is just another thing to be re-migrated in couple of years !

Its important for data engineers to communicate and collaborate with business and technical stakeholders. Using the proper communication you can add the business value.

Communication should be as centralized as possible in order to create a timeline of decisions and decision makers. Create a specific communication channel for each project and share that with the stakeholders up front, then link any related documentation on that channel for future reference.

I believe data engineers need lot of guidance / support from data architects so that they can test the business rules for their pipeline and add the value in QA process. There are many ETL testing methodolgies which must be followed religiously.

Involving QA in testing data requirements will likely involve specialized tools and training. Consider the cost and time for that in all requirements approval processes and test plans.

Technical aspects aside but as per my experience the data engineers have very narrow minded approach in terms of handling the data . They only think from technical engineering perspective not from business perspective. here comes lot of improvements to be required by the data engineers. Also in new world very minimum or no documents produced which creates hell lot of problems for the projects.

Understanding the difference between different data sources (data lake, data warehouse, data store) and different variations of Extracting, Transforming, and Loading (ETL, ELT, ELTL) data is a prerequisite to any project. The need for a shared vocabulary can be met by having a standard project kickoff which introduces these terms along with the interpersonal introductions.