You need to build a data warehousing application. Which platforms should you consider?

Consider streaming data capabilities and integration with real-time processing frameworks. Evaluate encryption capabilities for data in transit and at rest. Assess the platform's mechanisms for ensuring data availability and continuity.

all of them, bring them all in, baseline them, load them up and see who wins for your particular case. They all do the same thing, some with a single lane Ferrari, others with multiple highways and golf carts (don't let the golf carts fool you). They all can go across country moving your stuff. Only point to add, if you pick the Ferrari, make sure the tracks are clean. If you pick golf carts, make sure each one is loaded up. This is akin to Indexing & cubes vs. multiple parallel table scans. I often choose the latter.

We need to first decide the scope. The purpose of the DWH. If this is meant for a particular LOB, then the data relevant to the use case need to be brought in. In case of the whole enterprise, the approach should be "bring it on, all of it." This will cover the variety part. The business need, retention policy and the relevant regulation should define how much volume of data is to be present in the DWH and how much need to be archived. There may be argument of technology handling large amount of data and not to worry about storage etc. I am of the opinion that there is a trade-off. The current requirement of data and related insight should define the volume to be retained in DWH and rest should be archived.

In my experience, before we choose any DWH platform solution it is really important to understand and evaluate the size of data and its schema. This gives you an idea of data volume that you are going to work upon and whether the data is structured, semi-structured or unstructured. Henceforth, helping organizations in terms of cost estimations and data retention policies

There are three aspects that we need to look into: 1)Data Storage - For this we have traditional RDBMS, NoSQL databases; Data warehousing solutions from Amazon, Google BigQuery etc 2)Data Processing: We have Hadoop and Spark data processing frameworks, For cloud platforms we have GCP, AWS and Microsoft Azure 3)Analytics and Business Intelligence Tools: We have Tableau, Power BI etc The decision on which one to choose depends on a lot of factors like processing speed, ease of integration/ adaptability, cost , if the company is already using/used to a certain solution, the learning curve, data handling capability, security.

When end users access your data (visualization tools ), it is important to bake in role based security access for end users in the data warehouse as well.

Data quality is paramount. A good data warehouse platform will have robust tools for data cleaning, validation, and deduplication, ensuring that the data you work with is accurate and reliable. Equally important is data governance - the ability to manage the availability, usability, integrity, and security of the data. Features like access controls, audit trails, and compliance with data protection regulations are essential.

When choosing a data platform, prioritize data quality and governance. Ensure accuracy, completeness, and consistency through built-in tools or external services. Evaluate data governance aspects, including policies and security roles. Address compliance, audit needs, and explore metadata management capabilities for effective data lineage.

I will go with Informatica Cloud along with the Oracle database. I will go with top to bottom approach. Along with with development of conceptual logical and physical data model.

One of the most important aspects of maintaining a DWH solution is to ensure that your asset which is data is of high quality and is governed at various levels. For this, medallion architecture of data storage has been really effective as we maintain raw->transformed->aggregated layers of datasets. On top of this, making sure that certain business as well as functional rules are applied to enhance data quality on each layer.

For any data solution, at which level you are accessing the data becomes really important to consider while we design the architectural solution. At the same time, how your users are given access to the data plays a major role in controlling what one should see and what should be protected.

Ao criar um aplicativo de armazenamento de dados com foco em Acesso e Análise de Dados, você deve considerar plataformas que ofere?am suporte eficiente para gerenciamento de dados, processamento analítico e interface de usuário amigável. Algumas das principais plataformas a serem consideradas incluem: Plataformas de Nuvem: Amazon Web Services (AWS) Microsoft Azure Google Cloud Platform (GCP) Essas plataformas oferecem servi?os robustos de armazenamento de dados, processamento e análise, além de fornecerem escalabilidade e flexibilidade.

Ease of access to data and the ability to perform complex analyses are key. The platform should offer intuitive ways for users to query and retrieve data. It should also support advanced data analytics capabilities, enabling users to extract actionable insights through data mining, predictive analytics, and machine learning techniques.

Data analysis is process of using data to discover patterns trends insights and knowledge.the reuse of data as a public private or public private platform to support range of upstream social and economic activities has led some experts to consider data as a infrastructure resource.

I go with Snowflake for this also. Data Access: Supports SQL-based querying with a cloud-native architecture for scalable data access. Analysis: Integrates with popular BI tools and supports advanced analytics through Snowflake's native functionalities.

When crafting a data warehousing solution, it's crucial to think about the architecture and design carefully. Imagine your data warehouse as a well-organized library where information is stored neatly for easy access. Start by understanding the types of data you'll store – like books in different genres. Plan how these pieces of information relate to each other, ensuring a logical arrangement, much like organizing books on similar topics. Think about how quickly you can find a specific book – in this case, the speed of retrieving data. Finally, ensure that your "library" can expand gracefully as you acquire more data over time. A well-thought-out architecture and design make your data warehouse efficient.

A well-designed data architecture is the foundation of any effective data warehouse. The platform should support a flexible and scalable architecture, allowing for efficient data storage, retrieval, and management. It should facilitate integration with other systems and databases, and support various data warehouse architectures, like data lakes or federated databases, depending on your needs.

When evaluating data platforms, focus on architecture and design. Examine data warehouse architecture encompassing sources, models, marts, and pipelines. Evaluate design choices like schema, partitioning, indexing, and loading techniques. Platforms may provide templates or require custom builds, impacting flexibility and complexity. Consider your preference for predefined structures or the freedom to craft your own architecture and design using platform-provided tools or third-party solutions.

The Reporting layers. The purpose of Datawarehouse is to source data for reports or generate them. An example is extract or DataMart for third party reports.

Data Warehouse is destination to store data and used it for business purpose. First understand source of the data, what type of data and where data will be used, accordingly we can structure it.

In my experience an effective DWH is not only about type of storage you choose. It also includes effective mechanism of source data capture and data integration, its transformation and conformation and applying curative analytics to offer data in subjected oriented data marts. In old days we used to choose data storage, ETL and analytical tools as a separate disjoint solutions. But now, with advent of big data and cloud technologies we have better platforms like Databricks, snowflake and Azure analytics. These platforms allow you to store data more cost effectively and offer ETL services along with their orchestration as part of overall suit. This is really revolutionary, and new gen DWH systems must take advantage of them.

Data Engineering This is the engineering required to provision efficiently source data into a Datawarehouse and sourcing it for downstream systems.

I think, Teradata Vantage, Snowflake and Databricks are the good options. Basically I would look at the total cost of ownership before suggesting a platform.

Few platforms we can use for Business Intelligence reporting and dashboards : SAP BI , IBM Cognos , Power BI, Tableau , Xcelcius etc. For Extract Transform and Load : Informatica For database : Oracle For query analysis : Toad Sample data dictionaries based on sector like BFSI we have Finacle. We need to also build universes based on the views from business analysts and data architects.

,Amazon Redshift, Google Big Query, Microsoft Azure Synapse Analytics and Snowflake are currently the most in demand platforms. Depending upon the current technical landscape and its suitability, costs for new technology and skills availability and specific functionality needed, platform can be decided.

Factors to consider consider: 1. Scalability: Scale your cluster up or down based on volume and query complexity. 2. Performance: Enable fast query execution, especially on large datasets. 3. Ease of Use: User-friendly interface and simplified management especially for non tech users. 4. Integration Capabilities: Seamless integration with other services, facilitating a unified data ecosystem. 5. Security Features: Robust security measures, including data encryption, access controls, and multi-factor authentication, ensuring data protection. 6. Pricing Model: On demand and reserved pricing allowing flexibility. 7. Support Services: Extensive support system. Popular DW platforms: BQ, AWS Redshift, Snowflake, Azure Synapse

Honestly, the tech itself should be the last of your concerns when designing a central analytical environment, such as a data warehouse. I would recommend you build it with minimal dependencies on the specific technology you choose to start with. Tech changes very fast; most services are in a SaaS purchasing model so that you can juggle. Invest (and maintain!) in the business and logical layer, in the data quality streaming in and used by decision-makers & services. Then, design tech-agnostic architecture and stick to it. Only then, choose best-of-breed tech for each component with an exit strategy to replace it when something better (TCO, performance..). The vendors would not like it -》 your primary signal this is the right approach:)

The Cost of data warehouse: 1. Compute Costs: Plan based on expected compute power and cluster usage; it impacts overall costs. 2. Data Volume Costs: Storing data affects expenses, including compute, clusters, and storage. 3. SQL Complexity Costs: Complex queries increase compute, memory, and network costs. 4. Licensing Fees: Expenses for using specific platforms and tools. 5. Storage Expenses: Storing data and retrieval speed matter; cost is involved. 6. Data Ingestion Costs: Bringing in data adds extra expenses. 7. Archiving and Housekeeping: Deciding data retention influences costs; deleting old data saves money. 8. Maintenance and Monitoring Costs: Ongoing management for regular operations.

Look beyond traditional data warehousing platforms as platforms and industry have come a long way in that path. Look for cheap storage supporting ACID (delta lake standards), easy scale up and scale down compute (spark or serverless sql), multi language support (python, scala, sql,..), good orchestration and monitoring features, multi-capability support (data science, data engineering) in an integrated interface. In sum, a platform that supports end to end lake house design. Mind the vendor lock-in and cloud lock-in problems when desigining your lake house.

1) In my opinion there must be cloud native and fully managed service. 2) Separation of storage and compute resources to scale each independently. 3) Must have elastic scalability(Load balancing techniques) to scale up or scale down based on their workload requirement. 4) Now most of the organisations are looking for a multi cloud support like AWS, Microsoft Azure, GCP, avoid vendor lock-in. 5) Security will be priority for any data warehousing platform must comprises of end-to-end encryption, Role based access control (RBAC), IP whilelisting policies and many more…. Apart from these, it’s good to have global availability for the organisation having global footprints. Secure data sharing policy and pay-as-you-go pricing model.