登录查看更多内容

How can you build a resilient data warehousing network?

由人工智能和领英社区提供技术支持

Data warehousing is the process of collecting, storing, and analyzing data from various sources for business intelligence and decision making. However, data warehousing is not a static or simple task. It involves many challenges, such as data quality, security, scalability, and performance. Moreover, data warehousing needs to adapt to changing business requirements, data sources, and user expectations. How can you build a resilient data warehousing network that can handle these challenges and deliver reliable and timely insights? Here are some tips to consider.

此文章中的业界达人

由社区从 8 条内容中精选。了解更多

Davide Camera

Head of Financial Institutions & Info Management
Claudio Bonel, M.Sc.

Professor | Pesquisador | Cientista de dados | Desenvolvedor de Machine learning | Inteligência artificial | Business…

1 Choose the right architecture

The architecture of your data warehouse determines how you organize, store, and access your data. There are different types of architectures, such as centralized, decentralized, federated, or cloud-based. Each one has its own advantages and disadvantages, depending on your data volume, variety, velocity, and veracity. You need to choose the architecture that best suits your data characteristics, business goals, and budget. For example, if you have a large amount of structured data that needs to be processed quickly and consistently, you might opt for a centralized architecture. On the other hand, if you have diverse and dynamic data sources that require flexibility and scalability, you might prefer a decentralized or cloud-based architecture.

添加您的观点

Davide Camera

Head of Financial Institutions & Info Management
举报内容
In my experience selecting an architecture that suits the data volume, velocity, and variety is crucial for resilience across different industries. For instance, in ecommerce, a cloud based, elastic data warehouse like Amazon Redshift can handle fluctuating data volumes from online transactions. In contrast, a traditional industry might opt for an on premises solution for robust data control.

已翻译

赞
Claudio Bonel, M.Sc.

Professor | Pesquisador | Cientista de dados | Desenvolvedor de Machine learning | Inteligência artificial | Business Intelligence | Python | SQL | Power BI | PhD Student
举报内容
Penso que antes de "escolher" uma arquitetura certa. Faz-se necessário definir um método de desenvolvimento do seu Data Warehouse. E, no meu ponto de vista, por se tratar de um sistema de apoio à decis?o, a Engenharia de Software é o método científico mais adequado, mais seguro e mais resiliente, para você utilizar. Dentro da Engenharia de software, você encontrará a Engenharia de requisitos, que dará as técnicas para você identificar requisitos funcionais e n?o funcionais, que ser?o o insumo para os requisitos técnicos onde, aí sim, você PROJETARá (e n?o escolherá) sua arquitetura, primeiro a de solu??o, depois a de dados. Dúvidas? Chamaí!

已翻译

赞

2 Implement data governance

Data governance is the set of policies, processes, and standards that define how you manage your data quality, security, and compliance. Data governance ensures that your data is accurate, consistent, complete, and trustworthy. It also protects your data from unauthorized access, modification, or loss. Data governance involves defining roles and responsibilities, establishing data quality rules and metrics, implementing data security measures and audits, and enforcing data compliance regulations. By implementing data governance, you can enhance your data warehouse's resilience and reliability, as well as avoid costly errors and risks.

添加您的观点

Davide Camera

Head of Financial Institutions & Info Management
举报内容
Building a resilient data warehousing network requires implementing robust data governance. This entails establishing policies and procedures for data accuracy, accessibility, and security. For example, in finance, it involves managing sensitive financial information securely. Effective governance fosters trust in the data warehouse as a reliable resource, essential for decision making across various industries.

已翻译

赞
Claudio Bonel, M.Sc.

Professor | Pesquisador | Cientista de dados | Desenvolvedor de Machine learning | Inteligência artificial | Business Intelligence | Python | SQL | Power BI | PhD Student
举报内容
Hoje mesmo estive em uma Organiza??o Privada e a principal "dor" do CTO era os painéis de BI todos descentralizados. N?o tem ninguém tomando conta. Tem dados inconsistentes. Problemas de performance. Painéis conectando direto em tabelas de sistemas em produ??o. E o clássico, n?o tem documenta??o de nada, logo ng sabe como as regras forma implementadas e pq. Problemas clássico de uma organiza??o que n?o tem um área de dados, com um programa de governan?a de dados implementado. "Ahh Bonel, mas dá trabalho" Sim! dá trabalho, mas você pode iniciar, dedicando pessoas a área, iniciando um assessment no ambiente atual, implementando uma estrutura de desenvolvimento, baseada em métodos científicos, como a engenharia de software. Dúvidas? Chamaí!

已翻译

赞
AAMIR P

Senior Software Engineer at Tiger Analytics | Padma Shri Award nominee for the year 2023 | Author of 25+ books | Badminton Player | Udemy Instructor | Public Speaker | Podcaster | Chess Player | Coder | Yoga Volunteer |
举报内容
Establish data quality rules and metrics to monitor and maintain the accuracy and completeness of data. Implement processes for data profiling, cleansing, and validation.

已翻译

赞

3 Optimize data integration

Data integration is the process of combining data from different sources into a unified and consistent format for your data warehouse. Data integration involves various tasks, such as data extraction, transformation, loading, validation, and cleansing. Data integration can be challenging, especially if you have heterogeneous and complex data sources that change frequently. To optimize data integration, you need to use the right tools and techniques, such as ETL (extract, transform, load), ELT (extract, load, transform), or data pipelines. You also need to monitor and update your data integration processes regularly, to ensure that your data warehouse reflects the latest and most relevant data.

添加您的观点

Davide Camera

Head of Financial Institutions & Info Management
举报内容
In my experience, effective data integration is key to maintaining a robust, reliable data warehousing network across different industry sectors. This means efficiently consolidating data from diverse sources while ensuring quality and consistency. For instance, in retail, streamline the integration of sales data from various channels (online, in-store) for accurate inventory management.

已翻译

赞

4 Design data models

Data models are the logical and physical representations of how you structure and organize your data in your data warehouse. Data models help you to define the relationships, hierarchies, dimensions, and measures of your data. Data models also influence how you query and analyze your data, as well as how you present and visualize your data. Data models can be designed using different approaches, such as star schema, snowflake schema, or data vault. You need to design data models that are clear, consistent, and comprehensive, as well as aligned with your business needs and user expectations.

添加您的观点

AAMIR P

Senior Software Engineer at Tiger Analytics | Padma Shri Award nominee for the year 2023 | Author of 25+ books | Badminton Player | Udemy Instructor | Public Speaker | Podcaster | Chess Player | Coder | Yoga Volunteer |
举报内容
Consider using surrogate keys for dimensions to maintain stability in data. Surrogate keys are system-generated unique identifiers that replace natural keys in dimension tables.

已翻译

赞

5 Test and validate data

Testing and validating data is the process of verifying that your data warehouse meets the expected quality, functionality, and performance standards. Testing and validating data involves various methods, such as unit testing, integration testing, regression testing, or performance testing. Testing and validating data helps you to identify and resolve any issues, errors, or anomalies in your data warehouse, before they affect your business outcomes or user satisfaction. Testing and validating data also helps you to ensure that your data warehouse is resilient and robust, as well as compatible and interoperable with other systems and applications.

添加您的观点

AAMIR P

Senior Software Engineer at Tiger Analytics | Padma Shri Award nominee for the year 2023 | Author of 25+ books | Badminton Player | Udemy Instructor | Public Speaker | Podcaster | Chess Player | Coder | Yoga Volunteer |
举报内容
Involve end-users in user acceptance testing to ensure that the data warehouse meets their expectations and business requirements. UAT provides valuable feedback from the perspective of those who will be using the data for decision-making.

已翻译

赞

6 Monitor and improve data

Monitoring and improving data is the process of measuring and enhancing the efficiency, effectiveness, and value of your data warehouse. Monitoring and improving data involves various activities, such as collecting and analyzing data metrics, identifying and addressing data gaps or opportunities, implementing and evaluating data improvements, or soliciting and incorporating user feedback. Monitoring and improving data helps you to keep your data warehouse relevant, responsive, and reliable, as well as to adapt to changing business requirements, data sources, and user expectations. Monitoring and improving data also helps you to achieve continuous improvement and innovation in your data warehouse.

添加您的观点

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Data Warehousing

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How can you build a resilient data warehousing network?

1

2

3

4

5

6

7

1 Choose the right architecture

2 Implement data governance

3 Optimize data integration

4 Design data models

5 Test and validate data

6 Monitor and improve data

7 Here’s what else to consider

Data Warehousing

给文章评分

感谢您的反馈

更多Data Warehousing相关文章

更多相关阅读内容

How can you build a resilient data warehousing network?

1

2

3

4

5

6

7

1 Choose the right architecture

2 Implement data governance

3 Optimize data integration

4 Design data models

5 Test and validate data

6 Monitor and improve data

7 Here’s what else to consider

Data Warehousing

给文章评分

感谢您的反馈

查看其他技能