How can product managers and data engineers collaborate for data-driven decision making?

Understanding each other's roles and skills is crucial for effective collaboration between product managers and data engineers. Product managers need to familiarize themselves with the data engineering process, including the types of data, data quality challenges, and the technologies used by data engineers. On the other hand, data engineers should understand the product development cycle, user requirements, key performance indicators (KPIs), and the methodologies used by product managers. This mutual appreciation enhances communication, sets realistic expectations, and allows each party to leverage the other's strengths for successful collaboration.

Working together on data analysis and experimentation, Product Managers and Data Engineers can generate and leverage data-driven insights amp up the product value and user experience. This iterative process involves continuously monitoring user behavior, feedback, and performance metrics to drive product innovation and ensure that user needs are effectively met. Add in a designer and a software engineer and we have the Avengers of Tech Company :p Cheers!

Collaboration between product managers and data engineers is crucial for data-driven decision-making. To foster this collaboration, we propose a "Data Bridge" initiative. This initiative involves co-creating data requirements, establishing shared data infrastructure, and scheduling regular sync meetings. Additionally, it integrates a feedback loop mechanism for data insights to inform product iterations. Cross-training opportunities are encouraged to enhance mutual understanding between teams. Implementing the Data Bridge initiative will lead to informed decision-making and accelerated product innovation.

Understanding each other's roles is crucial across several phases. During POC planning, it aligns goals and expectations. In data collection and pre-processing, it ensures high quality data tailored to the needs. In feature engineering for example, a PM might identify key factors for the model, which the DE then incorporates. The collaboration never stops and continues to model development and training, where both have impact on the decision on algorithms and performance metrics. In testing and validation, this continual understanding helps interpret results against business objectives. Finally, in deployment and maintenance, both roles ensure seamless integration and ongoing support, demonstrating how mutual understanding drives success.

If you aren't able to collect useful data, you can't be data driven. It is much easy to be data driven as a B2C company as there are many approaches that are possible for running A/B tests to collect clear data and data is often big enough to be statistically significant. Unfortunately, I have mostly worked on B2B products so you end up looking at trends for small numbers of clients/actions or long-time windows. Collecting data across many corporate systems most organizations use is not easy and requires strong collaboration between IT, finance, web, and product teams.

Aligning on a data strategy and roadmap is essential for successful collaboration between product managers and data engineers. Together, they should: Identify Data Needs and Priorities: Determine what data is crucial for the product's success and prioritize these needs. Understand Data Sources and Dependencies: Identify where the data will come from and how it interrelates with different components of the product. Define Data Models and Schemas: Agree on the structure of the data and how it will be organized and accessed. Establish Data Governance and Security Policies: Set rules and standards for data usage, quality, and protection.

In my role as a PM, aligning on on the data strategy and roadmap was crucial in key ML project. Working closely with data engineers, we identified data needs, sources, and dependencies, ensuring they aligned with the product vision. We established robust data governance and security policies. This involved setting clear scopes, timelines, and resources for each data project, and using a shared project management tool for tracking progress. This alignment allowed us to focus on impactful data initiatives, preventing duplication and inconsistencies, and fostered strong collaboration between the product management and data engineering team.

Before refinement, these 4 steps are iterative: 1. Jointly design experiments: Product managers and data engineers collaborate to design experiments that test the formulated hypotheses. This includes defining the target audience, experiment setup, data collection methods, and success metrics. 2. Develop and implement analysis pipelines: Data engineers build and maintain reliable data pipelines to collect, clean, and prepare data for analysis. 3. Product managers provide feedback on data quality and ensure it aligns with the experiment's goals. 4.Analyze data and interpret results: Both teams jointly analyze the data, identify trends, and interpret the results to draw meaningful conclusions about the tested hypotheses.

The large language models you’re using to train your software probably are leading you sideways, sometimes at least. There was a really interesting law suit filed by the NYT today about copyright infringement. Will be curious to see how the large cloud companies respond. AI is amazing, though. Keep the faith.

Both teams have to respect each other’s speciality and accept that this is a team sport where each has a role to play. It all starts with context. Data engineers must first understand the business context for the project at hand and product managers needs to appreciate the technology context of what is possible, feasible and viable. After context setting that creates a common ground the joint team agrees on desired outcomes, success factors and approach. Next, collaborative strategy and design sessions are initiated and facilitated by the most knowledgeable team member on the topic at hand regardless of rank, seniority or reporting structure. Finally, ensure co-ownership of all deliverables and avoid any handover between the two sides.

3 suggestions from my experience 1. "Safe to Share" Standards & Data Catalogs: Establish standards for what constitutes "safe to share" data. Implement data catalogs for efficient data management, ensuring that sensitive information is protected while still allowing access to necessary data for project progression. 2. Feedback Loops & Iteration Cycles: Establish a process for continuous feedback and iterations. This ensures that the product evolves in response to real user needs and data insights. 3. Documentation and Knowledge Sharing: Maintain thorough documentation. This includes not just the technical aspects, but also learnings and decision-making rationales. This is crucial for future iterations and for onboarding new team members.