How do you start a data architecture project?

Data Architecture itself is not a Project! Wow, Surprising! DA is part of a Data Strategy. Its a one piece in well oiled data machinery. It has potential to give abstract view of entire landscape of underlying Data Asset. So, start with one use case and build DA. It should be flexible to adopt new use cases and/or scenarios. Data Architecture, addresses, Data Integration, Storage, Processing, Extraction, Governance, Security, Reporting/Visualization, Data Linage, Data Catalog, Data Models There could be multiple starting points, Knowing Business Goal or Defining & Understanding Data or Laying Data Framework. So instead of calling to start a DA project, let's call it, data product project and build Data Architecture for it.

Embarking on a data architecture project begins with a profound assessment of the current data landscape. Go beyond the surface—plunge into the data's soul. Unearth its hidden narratives, anomalies, and latent value. Engage data anthropologists, not just analysts, to understand how it lives and breathes within the organization. Map its journey, from source to consumption. Only by comprehending the present can you sculpt a future-ready architecture. Don't just take inventory; decipher its language, emotions, and untapped potential. This introspective exploration lays the foundation for a transformative data architecture, one that doesn't just change structures but reimagines the data's purpose and power.

Embarking on a data architecture project is a voyage into uncharted waters. Start by assembling a diverse crew of stakeholders to capture organizational needs. Dive into data sources, understand currents, and identify valuable information. Open communication with IT and business teams to build a shared vision. Chart the course for data governance, and establish principles for quality, security, and lifecycle management. This guides through regulatory storms, ensuring safe passage for sensitive information. Successful data architecture is more than design; it's crafting a narrative that aligns with the organization's story. Navigate strategically, and you'll not only reach your destination but also discover islands of innovation.

The title of this topic is wrong also the bullet "Assess the current state " also misleading. I would like to confirm as per my experience it is not necessary you always have luxury of having existing environment. Sometimes you need to start from the scratch this is the big problem in IT that they start the projects without understanding the need to any complex data platform. First step should be to understand the business requirement and then move to the next step.

Starting a data architecture project involves a systematic approach to ensure a seamless integration of data processes within an organization. The first step is to thoroughly understand the organization's needs and objectives. Conducting comprehensive assessments of existing data systems and identifying gaps is crucial.

Let say, there is already a dataware house in an organization, which is already already with business goal or objective. Now IT would like to build a cloud base data platform as they have budget and presented various benefits such as redution in data processing. Now in this case we have data sources inventory as we have already a data ware house and it is like migration or replica project..so my question is that then in this case what is involvement of business. I know it should some where aligned with buiness owner or leader , so we can later define data owner.

Can't disagree with this. Sometimes we just need to catch up with business needs no matter what is going on with our data warehouse or our data pipeline or even with our data engineers, even making our data warehouse do some transactional things in parts. What I'm trying to say is, we have to define business goals with the stakeholders involved in them, that's very important and we need to make sure that they know about the strategy. Sorry, not just conscious but they know that sometimes we need to do things that are "bad" (not ideal circumstances) before we can achieve our goals. And make sure they know what they need to do. If not then prepare for the worst scenario that you will face.

This is the most important aspect of Data Architecture and should be prioritised number one. The business requirements will drive the further high level solution and data architecture. Remember both process and data goes hand in hand so you should have clear understanding about your processes and datasets.

In my experience, the most successful projects have specific use cases in mind before beginning an implementation. If you are implementing a new tool, it's important to have a few use cases from each of the intended end users. Without specific, well-defined anchors, a project can drift significantly off track before being noticed at the end of the road. I recommend getting Jira ticket-level descriptions of how end users will interact with the tool and what data is required for that goal before beginning at all

Engaging key stakeholders and obtaining their input is vital, as it provides valuable insights into specific requirements and expectations. Once the objectives are defined, creating a detailed project plan, including tasks, timelines, and resource allocation, is the next critical step. Collaborative teamwork and effective communication are fundamental throughout the process.

I can’t agree more. IMO all three models are important but logical model cannot be ignored. Instead of creating these three models from scratch we should transform conceptual model to logical model and then logical model to physical model. We can use appropriate tool and architecture framework to develop these models to save time and effort.

Design plays a vital role in data architecture because we are going to represent the architecture in HLD(high level design) and (Low Level Design) as per the business requirement and project scope,if anything gets manipulated, will be facing the consequences in later point of time while implementing.

This is the most important part of data architecture which is mostly misinterpreted and wrongly assumed. What I have observed people do not understand difference between the Conceptual , Logical and Physical model. They say they dont have time to build it so they always build the Physical model and do not update based on changes to the environment which impact badly. Conceptual Data Model should include only business entities and their relationship , in Logical Data Model should have entities and attributes irrespective of technology. Physical model is operational implementation of logical data model. The beauty of business models like Conceptual and Logical model is that it is technology agnostic.

Once you have defined you Business Knowledge and Actions the Logical models (data and applications) are developed in sequential Tactics order as follows: 1. Data-items are associated with each Knowledge class 2. The most effective Logical Data Model (LDM) is developed by deciding which Knowledge class will become a logical table. 3. Generate the LDM - preferably automatically 4. Project Managers will then use the relationship links in the LDM to develop prioritized Subject Areas 5. Logical Applications can now be created using pseudo code. These will test the LDM for quality Once the LDM and pseudo code have proven to be effective the databases can be generated from the LDM and the pseudo code can be turned into computer code

Below important aspects are missing from this title : The physical data model should be implemented using the physical operational model. Non-Functional requirements are missing like Performance and Tuning , Data Format , Data Scheduling , Data Frequency , Data Security and Privacy , Data Compliance (GDPR etc) , Data Governance , Data Management etc

Implementation follows testing: 1. Unit - where each application is thoroughly tested 2. Systems - where a full test of every application is tested to ensure that there are no fundamental problems with hardware and software 3. Stress - where a full scale simulation is run in order to ensure the implementation hardware will cope with the load

To Implement Chalk out clearly how you would transition from Current State Architecture to the Target State Architecture. Map out how the Business Functionality that is not being met currently is fulfilled in the Target State. Then Spell out the Data Architecture components necessary to transition from Current State to Target State. Does it result in : 1) New Data Structures. Then map out how the data will be stored and enabled across the enterprise 2) Integration with other systems. Map out how the data integration will occur. What components are needed. 3) Map the Business Process flow and the data flow both in the Current and the Target systems. 4) Validate the Business Process Flow and the Data Flow in target system overcomes gap.

Actually I think this is the hardest part, when we evaluate architecture it means we need comparison. We can't rely on the metrics we gather from anywhere else. We can say "this section is broken so we need to change the stack" and then when we change it we just run into something we didn't expect to exist. Architectures are not just “mix n match” scenarios or technology stacks, changing architectures means additional costs and time. Don't you dare change the architecture annually.

If you follow a top down approach like the one I have developed the data architecture will have already been fully aligned with the business requirement. That is if you use a repository like the one I provide. When changes are needed then these will be assessed as follows: 1. A new Measure is required; The Knowledge Architect will identify the Knowledge classes that will be needed to support it 2. If a new entity then this will be added to the repository 3. The Data Architect will then identify any new attributes 4. The Database Designer will decide if the new Knowledge class needs a new logical table. 4.1. Yes: A new project to be created 4.2. No: Maintenance of an existing Subject Area will be required

It's essential to remain adaptable, considering the evolving nature of data technologies and business requirements. Regular evaluations and adjustments to the architecture ensure it stays aligned with the organization's objectives, making the data architecture initiative a continuous and evolving process.

Data Architecture is not a project instead this is design component and data strategy at Enterprise level or Project level. It ensures the end to end journey of data flow across the use case , project or enterprise. It spans various subjects of data area not only techical but non technical aspects as well.

Consider the latency of the data needed for the business process vis a vis your infrastructure components. For example if your reports are required in real time, there is little use in sourcing the reports from a data repository in which last weeks data is stored. So evaluate is the current architecture meeting the business functionality desired. If not, what needs to be done. Consider User training on the new systems for widespread usage. Consider systems where is data is not siloed but easily shared between applications. Build in Governance and data validations especially at the Systems of records, or source of truth of data.