登录查看更多内容

How can you successfully migrate data warehousing to a cloud-based serverless computing environment?

由人工智能和领英社区提供技术支持

Data warehousing is a process of collecting, storing, and analyzing large volumes of data from different sources to support business intelligence and decision making. Traditionally, data warehouses are built on-premise using dedicated servers, databases, and software tools. However, with the increasing complexity, variety, and volume of data, as well as the need for scalability, flexibility, and cost-efficiency, many organizations are considering migrating their data warehousing to a cloud-based serverless computing environment. Serverless computing is a model where the cloud provider manages the allocation and execution of resources, such as servers, storage, and network, on demand, without requiring the user to provision or manage them. This way, the user only pays for the resources consumed by the application, and can focus on the business logic and data processing rather than the infrastructure. In this article, we will discuss how you can successfully migrate your data warehousing to a cloud-based serverless computing environment, and what are the benefits and challenges of doing so.

此文章中的业界达人

由社区从 8 条内容中精选。了解更多

Anna Abramova

Enterprise Data & AI
Venkata Sai ITHAKOTA

LinkedIn Top Voice Badges for Data Architecture and Data warehousing | Sr.Data Architect @ Cognizant | Snowflake…
Mayank Srivastava

Thought Leader Data & AI, Enterprise Integration, and Technology Transformation, Global Director Architecture and…

1 Assess your current situation

Before migrating your data warehousing to a cloud-based serverless computing environment, it is essential to assess your current situation and identify your goals, requirements, and constraints. To do this, ask yourself questions like: what is the size, type, and source of your data? How often do you need to update, load, and query your data? What are the performance, security, and compliance standards that must be met? How much budget and time do you have for the migration project? And what are the skills and expertise of your team? Answering these questions will help you determine the scope, complexity, and feasibility of your migration project, so that you can choose the most suitable strategy and tools.

添加您的观点

Anna Abramova

Enterprise Data & AI
举报内容
Just talk to Snowflake :) and they will help you, also talk to their Professional Services team - they can help plan a migration

已翻译

赞
Venkata Sai ITHAKOTA

LinkedIn Top Voice Badges for Data Architecture and Data warehousing | Sr.Data Architect @ Cognizant | Snowflake Architect | AWS certified solutions architect
举报内容
In my opinion, migrating to cloud doesn't add much value to the investments. On-Prem to cloud migration is a major change and it has to be planned systematically with all the stakeholders and understand the current pain points(ex: data refresh intervals, adding more insights..) and try to address them. So i would rather say this exercise as a re-engineering program.

已翻译

赞
Divya Jyoti

Solution Architect at NN | Ex-Microsoft | Ex-Oracle | Microsoft Certified Azure Solution Architect Expert | PRINCE2 | SAFe? 5 Agilist | Digital Transformation & Cloud | Architecture
举报内容
I would also consider the ease of migration or re-architecture while considering a cloud provider. For e.g. if the AS-IS is on Microsoft stack, I would incline towards Azure in TO-BE cloud platform. This decision will also help the existing expertise to be reused.

已翻译

赞
Erfan Mahmoodnejad

Senior Data Consultant
举报内容
The following questions need to be answered first: What is the purpose of migrating a DW to the cloud? What is the architecture of the DW? Who are the users that are going to access data? Which services are going to connect to this DW? How much do you want to spend for this DW considering the egress and ingress? What is the type of data (Stream or batch) and what is the refresh interval and how big is the data? By answering these questions, you can make a decision to choose among the available cloud environments for this purpose(Snowflake, Synapse, Fabric, Databricks, etc..). There are many factors to be considered on how to migrate data to destination and it requires designing data flow architecture.

已翻译

赞
Redde Pidugu

Freelance Data Engineer (Azure / AWS Certified)
举报内容
It is completely depends what kind of data you are trying to migrate to cloud batch data or real-time streaming data. For both I prefer to use Databricks or Snowflake platforms on cloud as they are capable to handle both . Databricks is available on Azure & AWS. Snowflake itself is a cloud based Datawarehousing solution.

已翻译

赞

2 Choose a cloud provider and a serverless platform

The next step is to choose a cloud provider and a serverless platform that can support your data warehousing needs. There are many cloud providers in the market, such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), and others, that offer different features, services, and pricing models. You need to compare and evaluate them based on your criteria, such as availability, reliability, scalability, security, compatibility, and cost. Some of the popular serverless platforms for data warehousing are AWS Lambda, Azure Functions, Google Cloud Functions, and AWS Athena. These platforms allow you to run your data processing functions without managing servers, and integrate with other cloud services, such as storage, databases, analytics, and machine learning. You need to choose a serverless platform that can handle your data volume, variety, and velocity, and provide the functionality, performance, and flexibility that you need.

添加您的观点

Venkata Sai ITHAKOTA

LinkedIn Top Voice Badges for Data Architecture and Data warehousing | Sr.Data Architect @ Cognizant | Snowflake Architect | AWS certified solutions architect
举报内容
it is recommend to perform a PoC with major cloud based DWH/DL vendors and assess the platform based on your needs. By performing this assessments one would clearly understand the positives and negatives of the platform and can better choose the solution based on their context and future needs. This will be a fruitful exercise if all the stakeholders are involved.

已翻译

赞

3 Design and implement your serverless data warehouse

Once you have chosen your cloud provider and serverless platform, you need to design and implement your serverless data warehouse. This involves defining your data schema, structure, and format, as well as your data pipeline, workflow, and architecture. You need to consider how you will store, transform, load, and access your data in a serverless environment, and what are the best practices and patterns for doing so. For example, you may need to use a cloud storage service, such as Amazon S3, Azure Blob Storage, or Google Cloud Storage, to store your raw and processed data in a serverless format, such as JSON, CSV, or Parquet. You may also need to use a cloud database service, such as Amazon Redshift, Azure SQL Data Warehouse, or Google BigQuery, to store and query your aggregated and dimensional data. You may also need to use a cloud data integration service, such as AWS Glue, Azure Data Factory, or Google Cloud Data Fusion, to orchestrate and automate your data pipeline and workflow. You may also need to use a cloud analytics service, such as Amazon QuickSight, Azure Power BI, or Google Data Studio, to visualize and explore your data.

添加您的观点

4 Test and optimize your serverless data warehouse

After you have designed and implemented your serverless data warehouse, you need to test and optimize it to ensure its functionality, performance, and quality. You need to verify that your data is accurate, consistent, and complete, and that your data pipeline and workflow are reliable, efficient, and scalable. You also need to monitor and measure your serverless data warehouse's resource consumption, execution time, latency, throughput, and cost, and identify and resolve any issues or bottlenecks. You may need to use some cloud monitoring and optimization tools, such as AWS CloudWatch, Azure Monitor, or Google Cloud Monitoring, to track and analyze your serverless data warehouse's metrics and logs, and to adjust and fine-tune your configuration and code.

添加您的观点

5 Deploy and maintain your serverless data warehouse

The final step is to deploy and maintain your serverless data warehouse in production. You need to ensure that your serverless data warehouse is secure, compliant, and resilient, and that it can handle your expected and unexpected data loads and demands. You also need to keep your serverless data warehouse up to date, and incorporate any changes or enhancements that may arise from your business needs or feedback. You may need to use some cloud deployment and maintenance tools, such as AWS CloudFormation, Azure Resource Manager, or Google Cloud Deployment Manager, to automate and manage your serverless data warehouse's deployment and maintenance processes.

添加您的观点

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Mayank Srivastava

Thought Leader Data & AI, Enterprise Integration, and Technology Transformation, Global Director Architecture and Engineering @ FrieslandCampina.
举报内容
Depends what stack you are using. Databricks for instance lest you choose compute clusters and configure them. They have the Serverless compute for Jobs available now

已翻译

赞
Erfan Mahmoodnejad

Senior Data Consultant
举报内容
It is important to know that the egress cost in the serverless DW would likely be the highest, not the storage or ingress. In order to reduce costs, it is crucial to optimize the egress.

已翻译

赞

Data Warehousing

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How can you successfully migrate data warehousing to a cloud-based serverless computing environment?

1

2

3

4

5

6

1 Assess your current situation

2 Choose a cloud provider and a serverless platform

3 Design and implement your serverless data warehouse

4 Test and optimize your serverless data warehouse

5 Deploy and maintain your serverless data warehouse

6 Here’s what else to consider

Data Warehousing

给文章评分

感谢您的反馈

更多Data Warehousing相关文章

更多相关阅读内容

How can you successfully migrate data warehousing to a cloud-based serverless computing environment?

1

2

3

4

5

6

1 Assess your current situation

2 Choose a cloud provider and a serverless platform

3 Design and implement your serverless data warehouse

4 Test and optimize your serverless data warehouse

5 Deploy and maintain your serverless data warehouse

6 Here’s what else to consider

Data Warehousing

给文章评分

感谢您的反馈

查看其他技能