登录查看更多内容

What are the most commonly used data storage and retrieval technologies in data engineering?

由人工智能和领英社区提供技术支持

Data engineering is the process of designing, building, and maintaining data pipelines that collect, transform, and deliver data for various purposes. Data storage and retrieval technologies are essential components of data engineering, as they enable data engineers to store, access, and manipulate large volumes of data efficiently and effectively. In this article, we will explore some of the most commonly used data storage and retrieval technologies in data engineering, and how they differ in terms of architecture, functionality, and use cases.

此文章中的业界达人

由社区从 4 条内容中精选。了解更多

Leon Combrink

Creative BI Developer/Data Engineer and Architect - All things data

1 Relational databases

Relational databases are based on the relational model, which organizes data into tables with rows and columns, and defines relationships between them using primary and foreign keys. Relational databases use structured query language (SQL) to perform operations on data, such as inserting, updating, deleting, and querying. Relational databases are widely used for transactional and operational data, as they provide high consistency, integrity, and security. Some examples of relational databases are MySQL, PostgreSQL, Oracle, and SQL Server.

添加您的观点

Leon Combrink

Creative BI Developer/Data Engineer and Architect - All things data
举报内容
This is by far the most common data you find in the bulk of the businesses out there. MySQL is common for websites. MSSQL Server is common for websites and Data Warehouses. PostgreSQL is common for scientific applications and often found in Massively Parallel Appliances such as Greenplum. Oracle appears to be big in ERP and Hotel group management. So, no matter if you are an Analyst or a Developer it is best to get to know SQL!

已翻译

赞

2 NoSQL databases

NoSQL databases are non-relational databases that do not follow the relational model, and use different data structures and formats to store and access data. NoSQL databases are often categorized into four types: key-value, document, columnar, and graph. NoSQL databases use various query languages or APIs to interact with data, depending on the type and vendor. NoSQL databases are suitable for handling unstructured, semi-structured, or heterogeneous data, as they offer high scalability, flexibility, and performance. Some examples of NoSQL databases are MongoDB, Cassandra, Redis, and Neo4j.

添加您的观点

3 Data warehouses

Data warehouses are specialized databases that store and analyze historical and aggregated data for business intelligence and analytics purposes. Data warehouses use a dimensional modeling approach, which structures data into facts and dimensions, and supports complex and multidimensional queries. Data warehouses use SQL or other query languages to perform analytical operations on data, such as aggregation, filtering, grouping, and joining. Data warehouses are often integrated with other data engineering tools, such as extract-transform-load (ETL) pipelines, data lakes, and data marts. Some examples of data warehouses are Amazon Redshift, Google BigQuery, Snowflake, and Teradata.

添加您的观点

Leon Combrink

Creative BI Developer/Data Engineer and Architect - All things data
举报内容
There is a misconception that Data Warehouses store aggregated data. They do not! They store granular lowest level Transactional Data where one row represents a transaction. The structure allows OLAP technology like SSAS and Pentaho to aggregate the data on the fly. It's is also important to note that there is a Data Warehousing design pattern that guarantees Transactional Integrity. This is called Data Vault.

已翻译

赞

4 Data lakes

Data lakes are large-scale repositories that store raw and unprocessed data in its native format, without imposing any schema or structure. Data lakes use a distributed file system, such as Hadoop Distributed File System (HDFS) or Amazon Simple Storage Service (S3), to store and access data in various formats, such as text, JSON, XML, CSV, binary, or image. Data lakes use various frameworks and tools to process and analyze data, such as Apache Spark, Apache Hive, Apache Pig, or Apache Flink. Data lakes are ideal for storing and exploring diverse and massive data sets, as they offer low cost, high availability, and high flexibility.

添加您的观点

Leon Combrink

Creative BI Developer/Data Engineer and Architect - All things data
举报内容
Yes they are dynamic, but the most important part of them lies in the current technology growth we see in the industry today. Data Lakes enable Data Scientists and Data Engineers to leverage off the same set of source data!

已翻译

赞

5 Data streams

Data streams are continuous and real-time flows of data that are generated by various sources, such as sensors, devices, applications, or social media. Data streams use a publish-subscribe model, where producers publish data to a stream, and consumers subscribe to a stream and consume data as it arrives. Data streams use various technologies and protocols to transmit and receive data, such as Apache Kafka, Apache Flume, MQTT, or WebSocket. Data streams are useful for handling and analyzing time-sensitive and dynamic data, as they enable low latency, high throughput, and high scalability.

添加您的观点

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Leon Combrink

Creative BI Developer/Data Engineer and Architect - All things data
举报内容
There are 2 newer concepts - or rather data patterns - on the block. Namely Lake House and Data Fabric. They both have their place in the world of Data. Go have a look as they may just firt the current situation or organisation you are in.

已翻译

赞

Data Mining

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

What are the most commonly used data storage and retrieval technologies in data engineering?

1

2

3

4

5

6

1 Relational databases

2 NoSQL databases

3 Data warehouses

4 Data lakes

5 Data streams

6 Here’s what else to consider

Data Mining

给文章评分

感谢您的反馈

更多Data Mining相关文章

更多相关阅读内容

What are the most commonly used data storage and retrieval technologies in data engineering?

1

2

3

4

5

6

1 Relational databases

2 NoSQL databases

3 Data warehouses

4 Data lakes

5 Data streams

6 Here’s what else to consider

Data Mining

给文章评分

感谢您的反馈

查看其他技能