登录查看更多内容

How can you master data architecture patterns?

由人工智能和领英社区提供技术支持

Data architecture patterns are reusable solutions to common data problems, such as data integration, data quality, data governance, data security, and data analytics. By mastering data architecture patterns, you can design and implement data systems that are scalable, reliable, and adaptable to changing business needs. In this article, you will learn about some of the most popular and useful data architecture patterns, and how you can apply them to your own data projects.

此文章中的业界达人

由社区从 8 条内容中精选。了解更多

1 Data Lake

A data lake is a centralized repository that stores raw and unstructured data from various sources, such as web logs, social media, sensors, and transactions. A data lake enables you to store and access data in its original format, without imposing any predefined schema or structure. This allows you to explore and analyze data using different tools and methods, such as SQL queries, machine learning, or data visualization. A data lake also supports data ingestion, transformation, and processing using frameworks such as Apache Spark, Apache Hadoop, or AWS Lambda.

添加您的观点

Greg Alvord,CMSP
举报内容
A data lake can easily become a data swamp without a planning pattern around it. I favor a Data Mesh that provides roles, infrastructure, and governance to a set of purpose driven data lakes.

已翻译

赞

2 Data Warehouse

A data warehouse is a specialized database that stores structured and refined data for analytical purposes, such as reporting, dashboards, or business intelligence. A data warehouse organizes data into tables, columns, and rows, following a predefined schema or model, such as star, snowflake, or galaxy. A data warehouse also applies data quality and governance rules, such as validation, cleansing, deduplication, and encryption. A data warehouse typically uses a batch or scheduled process to load data from various sources, such as data lakes, operational databases, or external APIs.

添加您的观点

Vidya Iyer
举报内容
Data warehouse is a storage area of all data in the organization available to be analyzed, interpreted and evaluated to be presented in all areas for all perspectives

已翻译

赞
Greg Alvord,CMSP
举报内容
The downside, and it is a big one of the data warehouse pattern is the need to reorganize it every time a new data resource is added. I have seen a data warehouse be offline more than online due to reorganizations.

已翻译

赞
Kain Nanne

Manager Consultant at Sogeti with a focus on Data Engineering and Sustainability
举报内容
Data warehouses can be built for a specific domain or general purpose (e g. Finance DWH or Analytics DWH). This can often lead to many data warehouses created within an organization with competing strategies, definitions or models of the same data. Consider the data mesh approach as an alternative to this scenario with its decentralized governance model.

已翻译

赞

3 Data Hub

A data hub is a hybrid solution that combines the features of a data lake and a data warehouse, offering both flexibility and performance for data management. A data hub stores data in different zones or layers, such as ingestion, raw, curated, and consumption. A data hub allows you to ingest and store data in any format and structure, while also providing tools and services to transform and enrich data for analysis and consumption. A data hub also enables you to access and query data using different protocols and interfaces, such as REST APIs, GraphQL, or JDBC.

添加您的观点

Greg Alvord,CMSP
举报内容
The success of a data hub depends on the domain expertise of the central data hub team. Typically, their data domain expertise is drawn from the experts of the data domain. They are already a highly sought after skilled team and they view Data Hub work as time way from their primary responsibility.

已翻译

赞

4 Data Mesh

A data mesh is a decentralized and distributed approach to data architecture, where data is owned and managed by different domains or teams, rather than by a central authority or function. A data mesh treats data as a product, rather than as a resource, and empowers data producers and consumers to collaborate and exchange data using common standards and protocols. A data mesh also leverages microservices and cloud-native technologies, such as containers, orchestration, and event-driven architectures, to enable scalability, resilience, and agility for data operations.

添加您的观点

Greg Alvord,CMSP
举报内容
The data product producers are the data domain experts. They are incentivized to provide data products that are needed by management to provide the data needed to implement a "data driven decision" process. The data consumers may provide machine learning, analysis, or visualization to assist in the "data driven decision" process.

已翻译

赞

5 Data Vault

A data vault is a modeling technique that enables you to design and build a data warehouse that can handle complex and changing data sources, such as legacy systems, ERP systems, or CRM systems. A data vault consists of three types of tables: hubs, links, and satellites. Hubs store the unique identifiers or keys of business entities, such as customers, products, or orders. Links store the relationships or associations between business entities, such as customer orders or product categories. Satellites store the attributes or details of business entities, such as customer name, product price, or order date. A data vault separates the structure and semantics of data, allowing you to add or modify data sources without affecting the existing data warehouse.

添加您的观点

Kain Nanne

Manager Consultant at Sogeti with a focus on Data Engineering and Sustainability
举报内容
The data model of a data vault architecture is particularly well suited to a data lake environment on top of raw or unprocessed data. Business logic and reporting transformations would then happen in lightweight data marts, consuming from this data vault, without the need to build out entire and complex data warehouses.

已翻译

赞

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

加载更多内容

Data Architecture

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How can you master data architecture patterns?

1

2

3

4

5

6

1 Data Lake

2 Data Warehouse

3 Data Hub

4 Data Mesh

5 Data Vault

6 Here’s what else to consider

Data Architecture

给文章评分

感谢您的反馈

更多Data Architecture相关文章

更多相关阅读内容

How can you master data architecture patterns?

1

2

3

4

5

6

1 Data Lake

2 Data Warehouse

3 Data Hub

4 Data Mesh

5 Data Vault

6 Here’s what else to consider

Data Architecture

给文章评分

感谢您的反馈

查看其他技能