What's your data architecture strategy and roadmap?

The initial assessment of existing infrastructure, tools, and processes reveals inefficiencies that, once addressed, can reduce operational costs and improve performance. It also benchmarks current capabilities against industry standards, ensuring that any roadmap developed positions the business as a leader rather than a laggard in data management.

Knowing your current situation is the main stone in building any strategy. In data architecture strategy you have to measure and understand your organisation's capability and capacity then start comparing these capabilities and capabilities with the data landscape. One of the outputs of this assessment is the gaps that you need to fill and build befor joumping to the next stage in your data management program. Also another output from this assessment is feasibility on your goals and objectives, and assure that you can achieve them or not.

I would also add assessing the company's budget and appetite for a data project. In theory and reality, we could make a case for the usefulness of data projects anywhere, but the deciding factors in real life are an awareness of the needs by stakeholders and the financial constraints like budget, expected return on investment time frames. The fact that some or all of the architecture will be in the cloud means that cost will be a major constraint on the target architecture, so we need to know these constraints from the start.

To develop a data architecture strategy and roadmap , I would start with following : - Assess current data infrastructure. - Identify gaps and challenges. - Define business objectives. - Choose compatible technologies. - Establish data governance. - Plan data integration. - Address scalability and security. - Plan data migration. - Define analytics requirements. - Invest in training. - Implement testing and monitoring. - Document the strategy and roadmap for clarity and reference.

To access current state of your data portfolio, its important to understand how data flow across various business functions. Identify data domains of all of your producers, system of record for each domain is key to building data lineage for your organization. Having this view of your landscape is the key for the success of modernization efforts and reducing data silos. Data Models, Data Catalogues, Data dictionaries and Technical Architecture diagrams will further complement in complete understanding of your landscape.

One of the key deliverables of the assessment phase is the AS-IS architecture which shows the current state . Then based on the architecture requirements, a target architecture need to be defined this is sometime reffered as TO-BE architecture. The gap Between the AS-IS and the TO-BE is then translated into initiatives that need to undertaken as a roadmap

Defining a target state is not a one way street for today's consultant. It should be collaborative approach to define various version of target state which will eventually become your roadmap to incrementally improve or add new capabilities. Your goal of defining target state should not be limited to a static level but delivering incremental value with these various versions of target state data architecture.

Assess current and target states, identify gaps, and prioritize actions based on dependencies. E.g., if current state includes a data warehouse and BI tool, and if target is to further exploit AI/ML capabilities using company's data for business decision-making. Then, gaps identified could be the need for fine-grain data for AI/ML, the need for a common data analytics platform for processing fine-grain data, making certified data discoverable/searchable, and enabling AI/ML experimentations. To address this, while keeping the warehouse and BI tool intact, (1) Set up the data analytics platform, (2) Direct raw data to it and make certified fine-grain data discoverable for AI/ML experiments, (3) Incentivize and train staff to be innovative.

Identifying the gaps may sometime relates to client perception or expectation from current and future state of data flowing across the system. It is imperative that gaps and dependencies should be recorded in client's business tone and should not be mare technical jargon.

Keep in mind during this phase the data Integration and Quality. Focus on integrating data from various sources and maintaining high data quality. This enables to have a single source of truth and make informed decisions based on accurate data.

The Roadmap and action plans should be developed with a clear scope and with business boundary context. Zero-in with client for many milestones early in your roadmap and action plan development stage helps build client trust and confidence.

Developing a roadmap and action plan aligns technical objectives with business goals, ensuring that every step taken contributes to the desired outcome of growth and scalability. A well-defined roadmap provides clarity and direction, setting milestones that keep the data architecture evolution on track with the business's pace. Action plans convert this strategic vision into executable tasks, enabling the organization to respond agilely to market changes and opportunities. This methodical approach to data architecture ensures that the infrastructure is not only robust but also adaptive, directly supporting the business's aim to be data-forward and competitive.

One factor that I found valuable is assessing people and skills. The plan has to take into account current team capacity and capability, as well as balancing the time and effort to upskill and learn with the time and effort to keep the lights on. Work with the existing team on collecting information on the main challenges they face from the current state and where the most time is and effort are spent on maintenance. The future state should be focused on the design more than the technology, as a more modern stack that is built with the same exact design flaws is nothing but a more expensive version of the current state.

Breaking it down into manageable phases and aligning with business goals is vital. Utilizing tools like Gantt charts and Kanban boards for planning and execution is a smart move. Regular reviews and adaptations of the roadmap are crucial for staying on track. Setting clear success criteria ensures we measure progress effectively. It’s all about strategic planning and adaptability to drive value in our data-driven strategies.

a roadmap should cover all the points to be considered for this type of project: Data Governance and Security, Scalable Infrastrucuture, Data integration and quality, Advanced analytics and AI , continuous improvement and alignement with business goals and expectations ! ...one defined, it'll be important to follow this roadmap to be able to harness the full potential of your data, driving innovation and achieve your business goals .....

In an early stage and before start preparing and planning for the strategy, the organization and the strategy team should make sure about readiness of the organisation for the strategy loads and the availability and capacity of what called strategy management system which include the assessment of people, culture and other prime components must be ready as prerequisites for the strategy. Here we are assessing the capabilities before preparing the tools. The organizations should have a good expectation about it's capability to perform the strategy activities and take the heavy load of the strategy on the existing resources.

Successful implementation of a new data architecture strategy often involves significant changes to how an organization operates. This includes new roles and responsibilities, changing processes, and adopting a data-driven culture. Handling this change requires a thoughtful change management strategy to encourage employee buy-in and adoption. This important aspect is missing in the article.

Don't just imagine a shiny marketecture, also imagine and envision a competent and efficient team with the tools they need to operate, maintain, monitor and evolve the platform based on a data driven approach. Priorities should be set based on facts and information from the operations and focus on the performance of the overall operating model. To take ingestion as an example, one tool that does batch ingestion is not going to make an difference if you change it to another tool if your ingestion design doesn't make it easier to monitor, restart, pick up where you left off, cancel and redo batches, all without lots of manual labour.

Managing stakeholders is crucial when implementing or altering data architecture. A robust strategy involves identifying potential resistance, communicating benefits, and engaging those affected to ensure a smoother transition. A good communication plan is useful in addressing concerns and highlighting the positive impacts of the new architecture on their workflows and outcomes. This proactive approach often leads to better acceptance and support from stakeholders