Data engineers and business analysts clash on project governance. How will you find common ground?

To find common ground between data engineers and business analysts on project governance: Facilitate Open Communication: Schedule regular meetings to discuss goals, challenges, and expectations. Align on Shared Objectives: Identify mutual goals and emphasize collaborative benefits. Define Clear Roles: Clarify and document roles and responsibilities to prevent overlap. Develop an Inclusive Plan: Create a project plan with input from both teams, addressing key concerns. Use Collaborative Tools: Implement tools and processes for transparency and efficient communication. By focusing on these steps, both teams can work together more effectively and harmoniously.

I think its important for both the Business Analyst and the Data Engineer to understand the requirements and acceptance criteria of the deliverables. In an ideal situation the Business Analyst and Data Engineer should have some knowledge of each other's disciplines. This will lead to better communications and understanding between the two groups. Most important both groups must be flexible and willing to compromise to some degree to reach a common solution. The best Data Engineering solution might not be the best solution for the analyst and vice versa. At the end of the day the project success is determined by the acceptance, benefit, and usability by the business or stakeholders.

Trabalho em equipe é fundamental para que exista um bom relacionamento entre as partes. Os dois profissionais devem identificar que precisam da outra parte para que o trabalho seja realizado de forma efetiva.

1. Understand the business ask. 2. Listen with empathy to each others point of view 3. Understand both viewpoints in terms of alignment with consumer needs and technology / process guidance. 3. Communicate clearly your insights and come to a consensus.

This issue arises when the distinction between data and process isn’t recognized. Data governance differs from owning or defining the process. A business analyst is ideally suited to maintain this separation for the stakeholders’ benefit. When conflicts occur, it highlights the absence of clearly defined roles within the organization.

Encouraging open and honest communication helps data engineers and business analysts understand each other's perspectives. Regular check-ins and updates can identify potential blockers early, allowing teams to address issues proactively. Through this collaborative approach, the gap between technical execution and business requirements can be bridged, leading to more successful project outcomes.

During the frequent communication and daily catchup, business analysts and data engineers could know more each other “language”. Business analysts (BA) could understand more technical language that DEs are pointing at, by that BA can interpret business requirements into technical requirements for DEs while DEs will also know more about the business context for better understanding on datasets

Effective communication between Data engineers and Business analysts is key to success of any project, they should stay composed and level headed during conflicts, Emotional intelligence plays a pivotal role navigating contentions situations.

Cannot emphasis this enough. Being able to communicate and come to a compromise for the better of the business is crucial. Learning data engineering concepts and business analysis concepts can help us understand and communicate with each other better~!

Regular communication checkpoints identifying what objective KPIs are working good for both parties and what KPIs are not working for both parties should be set up. Any challenges being faced by either parties need to be brought up and resolved in collaboration. A proper process should be setup for other interactions- like requests for data extraction, requests for data updates, change in requirements, etc

The tension often arises because both groups have distinct priorities and ways of utilizing and interpreting data; data engineers focus on the technical integrity and scalability of data systems, while business analysts prioritize insights that drive business decisions. The solution lies in establishing a shared framework where both parties collaborate from the beginning. This involves creating cross-functional teams that co-develop project goals, ensuring that technical considerations and business outcomes are mutually understood and valued. Outlining agreed-upon metrics for success can serve as a guidepost, reducing ambiguity and misalignment. This relies on the company culture being an open and communication-driven environment.

Process alignment and Communication,establish a unified process that incorporates the needs and responsibilities of both data engineers and business analysts. This involves creating standardized procedures for data management, reporting, and decision-making. Aligning processes ensures that both teams work with the same methods and criteria, reducing misunderstandings and conflicts. Regular communication and feedback loops between the teams can further enhance collaboration. Clear documentation and updates to the process will help in streamlining workflows, improving efficiency, and ensuring that both technical and business requirements are met cohesively.

One often overlooked aspect of process alignment is the role of continuous feedback loops. Ensuring that both data engineers and business analysts regularly review and refine the process can help adapt to evolving project needs and minimize the impact of unforeseen challenges. This not only enhances collaboration but also allows for proactive adjustments, ensuring that the alignment remains strong throughout the project lifecycle.

Data may be unreliable at different times or locations depending on the viewers own interpretation. Data is typiclly supplied from a source to a target and at each point in its journey between these , it may be viewed at different levels of reliability. If data can begin as raw and progress at each version of itself through different locations, to becomming more reliable at each point then analysts can choose at which point they wish to analyse the data for different purposes. Location rules for reliability need to be present with examples for common understanding from an initial state or Raw to Reliable. From Bronze to Gold as it were. Take care not to over complicate data supply initially.

Understanding requirements alongside business process is essential in defining the business solution and effective execution of project. Better understanding and alignment better will be the outcome!

Allow project time to confirm these shared objectives with data examples. When theoretical ideas and terms are used alone its not surprising mistakes, missunderstandings and gaps occur.

SMART objectives will help in developing a solution that will be timely and effective. less ambiguous the better and well defined scope of work will lead to success in a timely way!

Everyone aligned with the same goal: "Deliver a project with excellence." First, bring the team into the goal, that is, bring them into the excitement of delivering the project. Make it very clear how important each person's participation is. Unite the team with a single goal: the excellence of project delivery.

Establishing shared objectives is crucial for bringing together data engineers and business analysts. By setting clear, mutual goals that are Specific, Measurable, Achievable, Relevant, and Time-bound (SMART), both teams can align their efforts towards the success of the project. This common ground helps prioritize tasks and facilitates compromises, ensuring that all contributions effectively drive the project forward.

Engineers create data systems and manage data movement, while Business analysts analyze data to discover insights and share them.

When engineers and analysts meet, encouraging a culture of compromise is essential to find common ground ... Align goals: Start by identifying and aligning the common goals of both teams. Emphasize the overall impact on the business and how working together can improve results. Facilitate open dialog: Create an environment where both groups can openly discuss their concerns and perspectives. Encourage active listening and mutual respect. Regular meetings and joint workshops can help close the communication gap. Implement solutions: Work towards solutions that strike a balance between technical requirements and business needs. This may require compromises from both sides, but should ultimately be aimed at the success of the project.

A culture of compromise is essential in any tasks, as Business analysts and data engineers have different roles in hand, one can easily bifurcating each role by leveraging both sets of skills. Business analysts are responsible for interpreting data, understanding customer needs, and developing strategies to meet those needs. Data engineers, on the other hand, are responsible for designing, building, and maintaining data systems, such as databases and data warehouses.

I always say: If the team is aligned and aware of the obligation to deliver, commitment comes automatically. In the current project I am working on, the team is so close that when one is absent, the other automatically takes over the other's task. This is called commitment; unforeseen events happen. However, the project never stops.

Creating a culture of compromise is essential when data engineers and business analysts encounter disagreements. Valuing the insights and perspectives of each team leads to stronger solutions. Compromises could involve revising timelines, re-evaluating project scopes, or adopting new tools to better meet everyone's needs. Being open to suggestions and willing to make concessions fosters a more harmonious and productive work environment.

That’s why you need Product Managers who other than their product specific skills can bring in board a bridge between Business Analyst and Software Engineers . It will help draw the line . Business Analyst should be responsible for articulating Business Requirements and scope . They should specify Acceptance criteria and Definition of done . While Software Engineers can have recommendations , they don’t have ultimate authority on the requirements . Now it’s responsibility of Software Engineers to do solution engineering in line with broader solution architecture .Here Business Analyst can have point of view , but don’t have authority as such . They can both work together to adjust requirements , scope and timelines.

As learning takes place during project delivery allow for data and task iterations before a final delivery step. Where change control is added to catch unexpected unkowns, new priorities may lead to an ever growing bucket of 'lets deal with this later'. Ensure these buckets are sized by volume and effort and impact before being sizelined to help delivery and focus and also context understanding. All of which may intern change the project requirements. When learnings are taking place during project delivery take care over timesheet coding. Transparancy and frequent updates keep all invested in the delivery journey. A new project with no problems is just a change control project in waiting.

In my experience, creating data awareness across various stages in various projects is critical. It is crucial to cross-learn from the business that is using the data and the implementers from the data governance organization and vice versa, sharing the business needs and technology needs back to the data governance organization. As the market demands change, processes need to change; while processes change, data needs to change; while data needs to change, the governance around it changes. A continuous collaborative learning will take the organization towards a data culture.