How do you collaborate with data professionals on architecture projects?

Most organization nowadays just care about speed and deadline, but that also needs to balance well with their data quality and data model. Otherwise, if you can't trust your data, there's no point of collecting them in the get-go.

In our organization, collaborating with data professionals on architecture projects is a structured process. It begins with - ~Scoping and defining project objectives ~ Involving data experts from the start ~ Cross-functional teams, including IT architects, analysts, and stakeholders, work together to gather requirements and understand both technical and business aspects. Data architects take the lead in designing the architecture, with input from data engineers and scientists. Post-implementation support is provided, with data professionals working alongside the support team to address any issues. A feedback loop captures insights from data experts, guiding process improvements for future projects.

Bringing product and project management is key. Product management ensures the data architecture aligns with the business vision, while project management orchestrates the timely execution of tasks. It's a two-way street where the exchange of ideas between data engineers and non technical professionals drives the project towards a solution that serves the business objectives.

More and more architects also (have to) work in a product development context. Key topics in this context are: (1) addressing the value case, what is the expected value to be delivered and (2) having a 'proper' discovery phase, not only addressing the immediate need, but aiming to address the underlying business problem in a most fundamental and sustainable way.

Yes it true, first DA team need to understand the Business context. they should get time to connect with source data , do a profiling , raise Q&A with business team get it sorted out them come up with the data model and fine tuned mapping get these docs approval from the governance teams . and hand over to implementation teams

Too generic. Here are a few points that could be considered for enhancement: Clarity: Clearly convey ideas to prevent misunderstandings. Frequency: Communicate regularly to keep everyone updated. Appropriate Tools: Use diagrams, documents, or dashboards to share information effectively. Common Terminology: Employ standard terms to avoid confusion. Active Listening: Respectfully listen to and address feedback or suggestions. Adaptability: Adjust communication methods as per the project or team needs. Engagement: Encourage open discussions for a more inclusive environment.

For communication to be effective, I feel that data architecture should be a 2-way street, actively incorporating feedback and empathizing with business needs and priorities. Necessary prerequisites for data architecture to act in such a way is that it's agile/emerging/incremental, and that it's responsive, which is described in a later section ("Adopt a collaborative approach")

Too generic and very shallow depth. suggest edit to include: Specificity: Provide examples to illustrate role delineation. Flexibility: Mention the need for role adaptability as projects evolve. Conflict Resolution: Briefly address conflict resolution strategies. Feedback Mechanism: Incorporate a feedback system for clarity in task delegation. Process Documentation: Emphasize documenting roles and processes for reference. Regular Review: Highlight the importance of reviewing and realigning roles regularly

I don't think it's up to data architecture to define roles and responsibilities. Having those clear and transparent, so the described benefits can be reaped, for me seems to be part of effective communication.

One thing I've found helpful is when we collaborate with various team members of different user personas such as Leads, Product Owners, Architects, Engineers, or even L1 and L2 Support resources, we get good insights into how the team is aware of the architecture and if any user group is not in sync, they get an opportunity to clear up their basics. This is vital for leveraging the skills and strengths of others where the users lack the basics understanding of the project architecture and life cycle

A few more points for following the best practices : 1) Data Lifecycle Management 2)Scalability: 3) Performance Optimization(caching ,indexing) 4)Storage Technology ( Data lake, nosql , sql) 5) Data backup and restore 6) Monitoring and Logging 7) Metadata Catalogue .

I find this a tricky topic. By relying on concept like "Data Mesh", organizations quickly feel that the architecture is not right-sized, fit-for-purpose for their specific context. So, while I certainly agree with relying on practices and standards, I do think it should be a right blend of proven patterns and situational awareness. Also, I personally don't believe in 'best' practices. There are only better practices, given a specific context.

Below are some considerations for best practices: - Foster collaboration between various data and business stakeholders (data architects, engineers, business stakeholders). - Data Integration - Implementing efficient and effective data integration processes. - Data life cycle management - Data quality management (Monitor and measure the data quality) - Establishing strong data governance model.

Below are some cents : Give them complete overview of functional /business requirements. Do not ignore the non-functional requirements. Before jumping into the hardcore development try to build the business rules in pseudo code etc.