登录查看更多内容

How do you create a scalable data model?

由人工智能和领英社区提供技术支持

Data models are essential for designing and managing data systems, but they also need to be scalable to accommodate changing requirements, data volume, and performance. A scalable data model is one that can handle growth and complexity without compromising quality, efficiency, or usability. In this article, you will learn some key principles and practices for creating a scalable data model in the context of data architecture.

此文章中的业界达人

由社区从 11 条内容中精选。了解更多

1 Understand the business needs

The first step in creating a scalable data model is to understand the business needs and goals of the data system. What are the main use cases, queries, and analytics that the data system will support? What are the expected data sources, types, and formats? What are the non-functional requirements, such as availability, security, and compliance? By answering these questions, you can define the scope, purpose, and expectations of the data model, and align it with the business value and strategy.

添加您的观点

Tirumalesh Kowdlay

Vice President at J.P. Morgan Chase
举报内容
We have to get Senior Business Leadership sponsor and support the Data Engineering foundational pillar for the Technology solution and allocate members from business who can collaborate provide their problems for us to create an efficient and customer functional model. Additional we also need the support from the Technology leadership to ensure the Software Engineering Team utilizes the data engineering offering and base their development partnering with Data Engineering team.

已翻译

赞
Pankaj Dhillon

AWS Certified Database - Specialty
举报内容
To better understand full picture, we need to get sponsors and their perspective/requirements from different business departments like Finance, Marketing etc. Many times, I have seen that we just have a team in technology department with not much interaction with other teams which led to incomplete understanding of business needs.

已翻译

赞

加载更多内容

2 Choose the right data modeling approach

The next step is to choose the right data modeling approach for your data system. Data modeling approaches can be broadly classified into two categories: relational and non-relational. Relational data modeling is based on the concept of tables, columns, and relationships, and follows a schema-on-write approach, where the data structure is defined before loading the data. Non-relational data modeling is based on the concept of collections, documents, and attributes, and follows a schema-on-read approach, where the data structure is inferred at the time of querying. Depending on the nature, variety, and velocity of your data, you may opt for one or the other, or a hybrid of both.

添加您的观点

Tirumalesh Kowdlay

Vice President at J.P. Morgan Chase
举报内容
Technology is evolving not all the information is stored in Relational Model. We have to choose the data model that will provide the ability to manage the full data pipeline. From inception to consumption. Data can be in lake, cloud or RDBMS we have to help develop solution where the processing of the data from ingestion through consumption does not keep creating islands of data in their own format rather level the data and have processes which can format the data on spot rather then keeping on transforming the data for various needs.

已翻译

赞

加载更多内容

3 Apply normalization and denormalization techniques

Normalization and denormalization are two techniques that can help you optimize the performance, storage, and maintenance of your data model. Normalization is the process of reducing data redundancy and inconsistency by organizing the data into smaller and more manageable tables, and linking them through foreign keys. Denormalization is the process of increasing data redundancy and aggregation by combining the data from multiple tables into larger and more accessible tables, or adding derived attributes to existing tables. Depending on your data system's needs, you may apply different levels of normalization and denormalization to balance the trade-offs between query speed, data quality, and complexity.

添加您的观点

Tirumalesh Kowdlay

Vice President at J.P. Morgan Chase
举报内容
We also need to conform and consolidate data for various business needs and avoid data duplication. For example Putting the data in data lake using snowflake, data bricks. We cannot afford to have RDBMS for analytics, DS or ML needs.

已翻译

赞

加载更多内容

4 Design for scalability and flexibility

The final step is to design your data model for scalability and flexibility, so that it can adapt to changing data volume, variety, and velocity. To achieve this, use modular and modular data models that can be extended, modified, or replaced without affecting other parts of the system. Additionally, consistent and descriptive naming conventions, data types, and formats should be employed to be easily understood and validated by users and applications. Indexes, partitions, and clusters can also improve query performance and data distribution across multiple nodes or servers. Furthermore, data compression, encryption, and archiving techniques should be used to reduce storage costs and enhance data security and retention. Lastly, data validation, cleansing, and transformation tools should be employed to ensure data quality and integrity across the data lifecycle. By following these steps, you can create a scalable data model that can support your data system's current and future needs while delivering value to your business and stakeholders.

添加您的观点

加载更多内容

5 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Zaki E.

Data & AI Leader at Electronic Arts (EA) ?? | Designing Scale & Innovation ??
举报内容
The current modeling landscape for data warehousing poses challenges with options like 3NF and Star Schema. While 3NF's complex primary keys and cascading snapshots create difficult-to-maintain data structures, the Star Schema's performance advantage faces inflexibility in adapting to changing business requirements. Both models require significant effort and tend to lack agility to respond to rapid changes. That been said, Data Vault is system business intelligence system with three pillars of methodology (pattern-based and consistent), a multi-iterative architecture for scalability, and a flexible hub-spoke model to manage diverse data sources effectively. It proves to be a robust and adaptable solution on the long run.

已翻译

赞
Pankaj Dhillon

AWS Certified Database - Specialty
(已编辑)
举报内容
We also need to make sure that proper documentation and data dictionary is maintained for any data model/architecture so that not only it is easy to use and understand the data system, but future updates are easy to be implemented with proper understanding of existing system.

已翻译

赞

加载更多内容

Data Architecture

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How do you create a scalable data model?

1

2

3

4

5

1 Understand the business needs

2 Choose the right data modeling approach

3 Apply normalization and denormalization techniques

4 Design for scalability and flexibility

5 Here’s what else to consider

Data Architecture

给文章评分

感谢您的反馈

更多Data Architecture相关文章

更多相关阅读内容

How do you create a scalable data model?

1

2

3

4

5

1 Understand the business needs

2 Choose the right data modeling approach

3 Apply normalization and denormalization techniques

4 Design for scalability and flexibility

5 Here’s what else to consider

Data Architecture

给文章评分

感谢您的反馈

查看其他技能