How do you manage data engineering project uncertainties?

First by understanding what the task is and then determine what some of the issues may be. But most importantly I keep good notes and an open line of communication to everyone involved.

From a systems thinking perspective, define the boundaries of your problem space first. Scope should be defined by: (1) a starting point, (2) a result of value / endpoint, with an understanding of the tolerance range around that end, (3) the unmodifiable environmental factors - these surround the value stream and may be captured as assumptions, but are not directly in the value stream, (4) constraints in or around the value stream, and (5) uncertain inputs and outputs. It is exceedingly rare that a failed project has successfully defined each of these up front.

This is teh first that you should ask yourself before starting any project anywhere: "why does this need to get done?". Defining a clear scope and goals in a data engineering project is critical for guiding the project's direction, it sets expectations and provides a roadmap for achieving specific outcomes.

As already stated how important it is to start the project on right fundamentals of planning.I have experienced through projects to inject discovery through proof of Concepts to be really helpful.It is like an MVP approach where we have something to show and grab early feedback to course correct.This helps in better alignment of the business goals and a parallel learnings.

Try and identify the known possible data issues and workout the mitigation in advance and keep the team up to date with clear communications. I like to have a strike team built, multi-talented folks, to address the unidentified data issues that come up. It helps to have the project's goals clearly identified before the project starts and keeping the team focus, while allowing flexibility.

One thing to add to this is to define and build your communication plan, depending on the size of the project. You will need to identify who, what, and how frequently updates need to be communicated as well as what the escalation path and who the escalation approvers/committee is for change requests.

To manage uncertainties in data engineering projects effectively, selecting the appropriate methodology and tools is crucial. Choose a project management methodology that aligns with your project's size, complexity, and need for flexibility. Options include agile for iterative progress, waterfall for linear development, or hybrid models that combine elements of both. Additionally, select data engineering tools and technologies that are compatible with your data sources, formats, quality, volume, and velocity. Ensure these tools meet your requirements for data processing, analysis, and visualization. The right combination of methodology and tools streamlines workflow, minimizes errors, and guarantees quality.

Knowing your team's capabilities and talents is key. I also find it helpful to promote folks working outside their normal areas of expertise. This promotes creativity and teamwork.

Too often, we design projects around data that don't exist. It is better to learn from and leverage existing data. While trying to improve the up-time of a system, one aviation team wasted three years trying to define and gather new technical performance measures, while they overlooked terabytes of existing flight records data. Once they started mining the existing resources, they were able to find the root cause of the challenges and solve the availability issue.

By managing the risks and dependencies you are proactively addressing potential obstacles. Minimizing disruptions helps us maintain project timelines.

Managing risks and dependencies is critical for minimizing uncertainties in data engineering projects. Start by identifying potential risks and dependencies that could impact your project’s scope, schedule, budget, quality, or performance. This includes challenges related to data availability, access, security, privacy, integration, validation, and transformation. For each identified risk and dependency, assess its impact and likelihood. Then, develop strategies to mitigate, avoid, or resolve these risks and dependencies effectively. This proactive approach enables you to prevent or minimize potential delays, failures, or conflicts, ensuring smoother project execution and more reliable outcomes.

In my opinion the use case is important.It defines what managed services to use for the technology stack.If the requirement is real time updates then we have to balance out between over engineering and cost.All this also guides the triple constraints.Managing expectations then becomes important with the key stakeholders.

Testing and constantly validating the released Data Products is one of the most important aspects. Data has a tendency to evolve over time, or we can name it change. Without proper observability issues might cause a cascading effect, which might lead to mistrust in the products. To ensure observability, ensure that all warehouses, lakes, pipeline components (streaming platform, ETL tool, et cetera) are well covered with Performance related Metrics. Ensure your pipeline include monitoring on the Data Latency and in case of real-time data the throughput capacity. As well as ensure that the Data Quality is assessed already as part of the pipeline. Start with the minimum metrics, but constantly iterate and improve. Observability ensures trust.

Continued learning and self-imporvement after delivering a data engineering project is essential for staying updated with evolving technologies and methodologies, ensuring your skills remain rlevenat (and you too on the job market). Rely on making documentation and discuss what happened during the sprint on the retrospective meetings.

The key to successfully delivering a data engineering project is to accept that there will be unknowns, uncertainty, change and risks and therefore an agile mindset offers several advantages: 1. Flexibility to adapt to changing requirements. 2. Delivering in sprints allowing for quick feedback and improvements. 3. Close collaboration between the project team through stand-ups and demos. 4. Prioritising tasks based on their business value. 6. Regular testing and quality checks built into each iteration. 7. Agile allows you to identify and address risks early in the project. 8. Since stakeholders see tangible progress at the end of each iteration, it often leads to higher customer satisfaction.

In my opinion, Change Management is one of the most crucial aspects in managing uncertainties in Data Engineering projects. Data teams cater to a dynamic Business landscape. As such, there is no such "end" to a Data Engineering project. Quite often, there is only the successful implementation of an initial requirement for which changes are iteratively proposed; even within the same scope. But when change management happens in an ad-hoc manner without proper tickets/documentation/tracking, Data Engineering teams end up not being able to showcase their effort/impact. It ends up becoming a vicious cycle of "one small change" after another. Thus, it is crucial to set-up and strictly follow a robust CR process in Data Engineering Projects.

An alternative perspective I'd like to offer is the importance of establishing a solid foundational implementation for data engineering projects, both from a business and technological standpoint. If the initial design and implementation lack foresight for future-proofing, it can lead to complications when critical system design changes become necessary down the road, resulting in time-consuming and costly adjustments. It's worth mentioning that while modularity in a system is beneficial in projects like these, excessive complexity without a well-defined structure and flow can be detrimental. Just as principles for programming code with the aim for low coupling and high cohesion, the same can be applied to overall data flow architecture.

Understand Your Audience to Drive Impact Consider which key decisions and which key decision makers need to see and understand the data - before designing a data strategy or system. Make sure that your findings are clear to the most important stakeholders and are delivered in advance of critical decision points.