登录查看更多内容

How do you choose the best data storage for a distributed system?

由人工智能和领英社区提供技术支持

Choosing the best data storage for a distributed system is a complex and important decision that affects the performance, reliability, and scalability of your application. There are many factors to consider, such as the data model, the access patterns, the consistency requirements, the availability trade-offs, and the cost-efficiency. In this article, you will learn about some of the common types of data storage for distributed systems, their advantages and disadvantages, and how to evaluate them for your use case.

此文章中的业界达人

由社区从 8 条内容中精选。了解更多

Paul Fretter

Leader in IT/HPC service development and delivery | 30+ years supporting scientific research.
Raghvendra Mishra

Project Lead : Data Engineer at Birlasoft
Harikrishnan Vijayan Rajeswary

Tech Director | Architect | Digital Transformation | MS Tech Stack | ChatBot | Azure | AI | Cognitive Service |…

1 Relational databases

Relational databases are based on the relational model, which organizes data into tables with rows and columns, and supports structured queries using SQL. Relational databases offer strong consistency, meaning that all nodes see the same data at the same time, and ACID transactions, meaning that operations are atomic, consistent, isolated, and durable. Relational databases are suitable for applications that need complex joins, aggregations, and analytics on structured data. However, relational databases are not very scalable, as they require a lot of coordination and synchronization among nodes, and they may not handle unstructured or semi-structured data well.

添加您的观点

Ackim David Swan

IT Specialist @ Parliamentary Election Office | Threat Detection, Cloud Administration
举报内容
One of the primary considerations for a distributed system is scalability. The chosen data storage solution should be able to handle increasing data volumes and accommodate the system's growth. Look for storage options that can scale horizontally by adding more nodes or partitions to distribute the data effectively.

已翻译

赞

2 NoSQL databases

NoSQL databases are a broad category of data storage systems that do not follow the relational model, and use different data models, such as key-value, document, column-family, or graph. NoSQL databases offer high scalability, as they can distribute data across many nodes with minimal overhead, and high availability, as they can tolerate node failures and network partitions. NoSQL databases are suitable for applications that need fast and simple access to large volumes of unstructured or semi-structured data. However, NoSQL databases sacrifice consistency for availability, meaning that different nodes may see different versions of data, and they do not support ACID transactions, meaning that operations may not be atomic, consistent, isolated, or durable.

添加您的观点

3 Distributed file systems

Distributed file systems are data storage systems that provide a unified view of files and directories across multiple nodes, and allow concurrent access and modification by multiple clients. Distributed file systems offer high fault tolerance, as they can replicate data across nodes and recover from failures, and high throughput, as they can leverage parallelism and load balancing. Distributed file systems are suitable for applications that need to store and process large amounts of binary or text data, such as images, videos, logs, or documents. However, distributed file systems may not offer strong consistency, meaning that clients may see stale or conflicting data, and they may not support complex queries or transactions, meaning that clients may need to implement their own logic or use other data storage systems.

添加您的观点

Paul Fretter

Leader in IT/HPC service development and delivery | 30+ years supporting scientific research.
举报内容
In a distributed file system, you get what you pay for and if you need high dependability (reliable, resilient and robust) to hold the 'crown jewels' of your organisation, then consider established enterprise-grade systems that are well supported. Consistency of unstructured (file) data, e.g. using file locking, should be robustly handled by the storage system software. Consistency of structured data (e.g. databases) may need to be handled by the application (e.g. DBMS). I recommend you have detailed technical conversations with potential suppliers, and craft your specification carefully.

已翻译

赞
Raghvendra Mishra

Project Lead : Data Engineer at Birlasoft
举报内容
In my opinion Hadoop distributed file system is best among all to store disturbed data of any format and at any scale. It provide fast data retrieval and scales big data in such a way which can help clean utilise and transform data according to the use case scenarios. HDFS block size provides more data storage which is also based upon Hbase which provides strong mechanism of NoSQL from Backend.

已翻译

赞

4 In-memory data stores

In-memory data stores are data storage systems that store data in the main memory of the nodes, rather than on disk. In-memory data stores offer high performance, as they can access data much faster than disk-based systems, and high flexibility, as they can support various data models and operations. In-memory data stores are suitable for applications that need low latency, high concurrency, and real-time processing of data. However, in-memory data stores are limited by the memory capacity of the nodes, and they may not persist data across restarts or failures, unless they use replication or snapshots.

添加您的观点

Harikrishnan Vijayan Rajeswary

Tech Director | Architect | Digital Transformation | MS Tech Stack | ChatBot | Azure | AI | Cognitive Service | Solution Designs | Listener & Learner
举报内容
In my experience i have found the use of redis cache really works well if your system requires high performance. It’s capable of storing key value pairs of substantial size but we need to be careful in handling it. It greatly supports pub sub design which makes it ideal for chat applications.

已翻译

赞
I?aki Barba Colás

Infrastructure Services en Axis Data S.L.U.
举报内容
Según mi experiencia con cabinas híbridas la memoria asignada en cache a discos flash enseguida se os quedará peque?a, ya que los sistemas demandan cada vez mayor velocidad en los tiempos de lectura y escritura. En el corto plazo puede ser una solución más económica, pero a la larga la tendencia os empujara a aumentar más y más la capacidad de los discos flash para mantener mayor cantidad de datos en cache por más tiempo, así que a poco que vuestro presupuesto os lo permita, ir directamente a una solución all-flash. Pensad que las nuevas generaciones de cabinas all-flash también incorporan un ratio de compresión mayor, y muchas, como las HPE, cuentan con IA externa que os indicará la tendencia actual, y vuestras necesidades futuras.

已翻译

赞

5 How to choose

When it comes to data storage for distributed systems, there is no one-size-fits-all solution. To choose the best system for your application, you must understand your data and evaluate the data storage systems based on criteria such as the data model, access pattern, consistency, availability, scalability, and cost-efficiency. These criteria will help you determine which system can provide the optimal balance of functionality and performance. You may also consider combining multiple systems to meet your needs.

添加您的观点

Stephen Hopson

Managing Director and Consultant at Corvid Computer Consultancy Ltd
举报内容
The choice of database cannot be made without reference to end user requirements. Distributed applications will require different solutions to that of a central monolithic application. In either case a mixture of data storage and retrieval techniques will be required (data storage, caching, duplication, etc).

已翻译

赞

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Prashant Kadre

Experienced Technology Leader with over 14 years of expertise in DevSecOps, Cloud Migration, and Security.
举报内容
Choosing the best data storage for a distributed system involves considering factors like scalability, consistency, fault tolerance, and performance. Assess your system requirements, data access patterns, and anticipated growth. Options like distributed databases (e.g., Cassandra, MongoDB), cloud storage (e.g., AWS S3, Azure Blob Storage), or distributed file systems (e.g., HDFS, Ceph) offer different trade-offs. Evaluate based on your specific needs, considering factors like data volume, read/write patterns, and geographic distribution. Conduct performance testing and be mindful of maintenance overhead.

已翻译

赞
Ali Husnain

System and Network Administrator | Cisco, Google & Microsoft Cloud Certified IT Professional
举报内容
Benchmark and test: Evaluate different storage solutions based on your specific workload and performance requirements. Consider future needs: Choose a solution that can adapt to your evolving data and system requirements. Seek expert advice: Consult with data storage specialists for tailored recommendations based on your unique needs.

已翻译

赞

Operating Systems

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How do you choose the best data storage for a distributed system?

1

2

3

4

5

6

1 Relational databases

2 NoSQL databases

3 Distributed file systems

4 In-memory data stores

5 How to choose

6 Here’s what else to consider

Operating Systems

给文章评分

感谢您的反馈

更多Operating Systems相关文章

更多相关阅读内容

How do you choose the best data storage for a distributed system?

1

2

3

4

5

6

1 Relational databases

2 NoSQL databases

3 Distributed file systems

4 In-memory data stores

5 How to choose

6 Here’s what else to consider

Operating Systems

给文章评分

感谢您的反馈

查看其他技能