登录查看更多内容

What are the best practices for loading data into a data engineering portfolio?

由人工智能和领英社区提供技术支持

If you are a data engineer or aspiring to become one, you probably know that building a data engineering portfolio is a great way to showcase your skills and experience. A data engineering portfolio consists of projects that demonstrate your ability to collect, process, store, and analyze data using various tools and techniques. However, before you can start working on your projects, you need to load the data into a suitable environment. How do you do that effectively and efficiently? In this article, we will share some of the best practices for loading data into a data engineering portfolio.

此文章中的业界达人

由社区从 5 条内容中精选。了解更多

Adrian Brudaru

Open source pipelines - dlthub.com

1 Choose your data sources

The first step is to decide what kind of data you want to work with and where to get it from. Depending on your goals and interests, you can choose from a variety of data sources, such as public datasets, APIs, web scraping, streaming platforms, or your own personal data. You should also consider the size, format, quality, and availability of the data, as well as any ethical and legal implications of using it. Ideally, you want to choose data sources that are relevant, reliable, and diverse enough to showcase your skills.

添加您的观点

Adrian Brudaru

Open source pipelines - dlthub.com
举报内容
What to look for when inspecting a source for loading? Extracting a source: * Try to create a generic extractor that can iterate over endpoints to keep your code dry. You could have an authenticated pagination method that could hit endpoints from a list. Loading a source: Look for the increment mode in the data type and source metadata. - If the data has no state, or is immutable, such as events, you can append - If the data has a state, or is subject to change, and it has a way to tell which records change, you can request the new records and upsert or merge them to the destination - If the data has a state but no way to tell what changed, your only option is a full load.

已翻译

赞

2 Select your data storage

The next step is to select a suitable data storage option for your portfolio. There are many factors to consider when choosing a data storage, such as the type, volume, velocity, and variety of the data, as well as the cost, performance, scalability, and security of the storage solution. Some of the common data storage options for data engineering portfolios are cloud-based services, such as AWS S3, Google Cloud Storage, or Azure Blob Storage, relational databases, such as MySQL, PostgreSQL, or SQLite, or non-relational databases, such as MongoDB, Cassandra, or Redis. You should also think about how you will organize your data in the storage, such as using schemas, partitions, indexes, or buckets.

添加您的观点

Adrian Brudaru

Open source pipelines - dlthub.com
举报内容
If the last 10y in data engineering taught me anything, it's that technology is a quickly evolving space. When selecting storage, do not vendor lock yourself into specific use case technologies, instead consider using standard formats like Parquet on fsspec storage, or table formats. Alterantively, use a db-agnostic loader like the open source python dlt library which creates schemas before loading and can thus change destination at the flip of a switch. The benefit of going this route is that you can easily use local envs like DuckDB for testing and cloud DWH for production.

已翻译

赞

3 Use appropriate data loading tools

The third step is to use appropriate tools and methods to load the data from the sources to the storage. Depending on the complexity and frequency of the data loading process, you can use different tools and methods, such as ETL (extract, transform, load), ELT (extract, load, transform), or streaming. Some of the popular tools for data loading are Apache Airflow, Apache Kafka, Apache Spark, AWS Glue, Google Cloud Dataflow, or Azure Data Factory. You should also follow some best practices for data loading, such as validating the data quality, handling errors and exceptions, logging and monitoring the process, and documenting the data lineage and metadata.

添加您的观点

Adrian Brudaru

Open source pipelines - dlthub.com
举报内容
There are many vendor specific solutions in the ETL space, just find the vendor you like with the soruces you need. If you are looking to code pipelines, there is only one python library made for data people loading data: pip install dlt, the library that's open source, and made to boost developer productivity 5-10X by automating things like normalisation and schema inference, evolution, migration, lineage. There are also frameworks or other tools for creating pipelines, which have their own shortcomings - frameworks are for software developers, while drag and drop tools offer limited options.

已翻译

赞

4 Optimize your data loading process

The fourth step is to optimize your data loading process to make it faster, cheaper, and more reliable. There are several ways to optimize your data loading process, such as using compression, encryption, parallelization, batching, partitioning, or caching. You should also test and benchmark your data loading process to measure its performance and identify any bottlenecks or issues. You can use tools like Apache JMeter, AWS CloudWatch, Google Cloud Monitoring, or Azure Monitor to test and monitor your data loading process.

添加您的观点

Adrian Brudaru

Open source pipelines - dlthub.com
举报内容
Optimising your loading process is not possible in most tools - they generally do not offer these options. If you have custom flows, your options will revolve around making the best use of the hardware. Memory: Make sure you do not exceed memory available by stream processing data with microbatching buffers. Processing: Multithreading is a start, and parallelisation on multiple machines is the more powerful version. If you are using dlt library, you can benefit from these options out of the box.

已翻译

赞

5 Update and maintain your data

The fifth step is to update and maintain your data in the storage to keep it fresh and accurate. Depending on the nature and frequency of the data changes, you can use different strategies to update and maintain your data, such as incremental loading, full loading, or delta loading. You should also implement some data maintenance tasks, such as backup, recovery, archiving, or purging, to ensure the availability and integrity of your data. You can use tools like AWS Backup, Google Cloud Backup, Azure Backup, or Cron to automate and schedule your data maintenance tasks.

添加您的观点

Adrian Brudaru

Open source pipelines - dlthub.com
举报内容
Managing schema changes is the biggest nightmare in data engineering. But should we even do it? I think not - Data engineers should define how the data should be loaded, not what the data is - this is up to the stakeholder and producer. So how to maintain schemas? The most effective way is to detect schema changes, and alert the other parties to curate the event - should it be accepted, or rejected? dlt library offers functionality for detecting schema changes before inseting the data and enabling alerting and curation by the analyst. The data engineer can leave the pipeline in "evolution" mode to load to the raw layer regardless, or apply a contract to reject changes.

已翻译

赞

6 Access and analyze your data

The final step is to access and analyze your data in the storage to generate insights and value. Depending on the type and purpose of the analysis, you can use different tools and techniques to access and analyze your data, such as SQL, Python, R, or BI tools. You should also follow some best practices for data analysis, such as defining the problem and objectives, exploring and cleaning the data, applying appropriate methods and models, visualizing and communicating the results, and evaluating and improving the analysis. You can use tools like Jupyter Notebook, Google Colab, RStudio, or Tableau to perform and present your data analysis.

添加您的观点

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Data Engineering

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

What are the best practices for loading data into a data engineering portfolio?

1

2

3

4

5

6

7

1 Choose your data sources

2 Select your data storage

3 Use appropriate data loading tools

4 Optimize your data loading process

5 Update and maintain your data

6 Access and analyze your data

7 Here’s what else to consider

Data Engineering

给文章评分

感谢您的反馈

更多Data Engineering相关文章

更多相关阅读内容

What are the best practices for loading data into a data engineering portfolio?

1

2

3

4

5

6

7

1 Choose your data sources

2 Select your data storage

3 Use appropriate data loading tools

4 Optimize your data loading process

5 Update and maintain your data

6 Access and analyze your data

7 Here’s what else to consider

Data Engineering

给文章评分

感谢您的反馈

查看其他技能