What are the most important data architecture concepts to understand?

I have seen people failed to make models without the business understanding , purpose of model and level of integration. It is quite important models should talk in business process not in applications or KPIs. Then what level of integration and data coupling is required to answer the business query and keeping data in model to portray a business process.

Alright, let's put it this way: Think of data modeling as the script of a blockbuster movie; often overlooked, but it's what makes the whole thing work. It's not just about nicely organizing data; it's about translating business lingo and jargons into data speak and vice versa.

Architecting data begins from understanding data in detail with aim to draw meaningful insights, faster query read and write. Having 3 stages ( Conceptual, Logical, Physical ). My tryst with Data modelling started from siebel CRM relational days where db cant be designed without models- Siebel Party model -famous model which still serves the foundation of many systems. 3 major things to know while designing DM. 1. How entity is related to each other . ( 1: m) , ( m:1) , ( m:m). 2 Find the lowest data grain in each entity 3. Define the keys and how that key could be joined to other entities. After that draft how the data would flow in simple data flow diagram, then add relations type , id followed by constraints and data types.

Levels of Abstraction: Data models should be expressed at multiple levels - conceptual, logical, and physical. Each of them provides a distinct perspective, aiding in understanding, design, and implementation. Evolution: Data models should continue to evolve with changing business requirements. Regular updates and modifications ensure that the data model remains aligned with the dynamic needs of the organization.

Data modeling is now becoming a neglected practice, according to reports from various sectors of industry. This is not a good trend. Data modelling techniques, such as defining entity relationship and normalization rules are key to building reliable and scalable data systems. Even in unstructured and NoSQL data systems, data models have to be clearly defined, as well as schemas, data types, and Data Quality metrics.

Enforcement and Validation: The schema serves as a tool for enforcing data integrity rules and validating incoming data ensuring that data adheres to the predefined structure and constraints, maintaining consistency across the board

After formulating the logical model, a significant portion of the work is completed. The subsequent step involves enhancing it through the creation of a physical model, incorporating details like data types, cardinality, and constraints. Tools such as Erwin facilitate a streamlined transition from the entire model to a schema through features like forward engineering. To further optimize the structure, indexes, constraints, and partitions can be added based on specific requirements. Relational databases needs a predefined schema, a set structure to organize and store data. Where as NoSQL databases operate dynamically, adapting to evolving data needs without a rigidly defined schema.

Una buena práctica es tener definido un esquema de entrada y uno de salida y de ser posible tener un repositorio dedicado a esquemas y bien organizado. Los datos al ir pasando por diferentes capas o zonas podrían necesitar cambiar de tipo de dato durante su ciclo de vida y para mantener una buena calidad de los datos debemos de tener definido cómo esperamos recibir los datos y como esperamos que salgan. Ejemplo: Input accountNumber:Int Output accountNumber:String

Dimensions of Data Quality: DQ is multifaceted and can be assessed through different dimensions, including accuracy, completeness, consistency, timeliness, relevance, and usability. Each of them contributes to the overall quality assessment.

Es muy importante establecer un MVP (Minimum Viable Product) cuando se están ingestando grandes volumenes de datos, aquí se establecerían reglas de calidad de los datos para cumplir con las diferentes dimensiones. Así estaríamos seguros de que todo lo que llega al Data warehouse cumple al menos con lo mínimo que el negocio necesita.

Effective data must possess quality. Without it, data loses its significance . Quality data is the foundation for deriving meaningful insights. Thus, it becomes imperative to establish a robust quality framework. This necessitates collaboration with business stakeholders and data domain experts to define precise quality rules. Subsequent steps involve data profiling and reporting, supported by well-designed dashboards. These dashboards play a crucial role in distinguishing between high-quality and subpar records , followed by implementing mechanisms for notifying data sources about subpar records is crucial to ensure continuous improvement in data quality.