OpenSearch Index, Shards, Nodes and Clusters

Efficient indexing is crucial for optimizing OpenSearch clusters, ensuring scalability, performance, and resource efficiency. To fully harness the power of OpenSearch, it's essential to understand the building blocks of its architecture: indexes, shards, replicas, nodes, and clusters. In this article, we’ll start by breaking down these fundamental concepts, explain the default configurations, and explore optimization strategies. We'll also present a case study on shard reduction and JVM memory optimization to demonstrate practical applications of these principles.

What Are Indexes in OpenSearch?

An index is a collection of documents that OpenSearch uses to organize, store, and retrieve data. It’s the foundational data structure in OpenSearch, similar to a database table. An index is divided into smaller units called shards, which distribute the data across nodes in a cluster for scalability and fault tolerance.

What Are Shards?

A shard is a unit of storage and processing within an index. Shards make it possible to distribute data across multiple nodes in a cluster, allowing OpenSearch to scale horizontally. There are two types of shards:

• Primary Shards: These store the original data. Each document in an index is stored in exactly one primary shard.
• Replica Shards: These are copies of primary shards. They provide redundancy for fault tolerance and improve read performance by spreading query loads across multiple nodes.

What Are Replicas?

Replicas are additional copies of primary shards, designed to enhance:

1. High Availability: If a primary shard or its hosting node fails, a replica shard can take over.
2. Read Performance: Queries can be distributed across both primary and replica shards, reducing response times.

The number of replicas can be adjusted dynamically, but the number of primary shards is fixed at index creation.

Default Shard Configurations in OpenSearch

When creating an index, the default shard configurations differ based on the platform:

• Amazon OpenSearch Service:Default: 5 primary shards and 1 replica, resulting in 10 total shards.
• Open-Source OpenSearch:Default: 1 primary shard and 1 replica, resulting in 2 total shards.

While replica counts can be modified later, the number of primary shards is immutable after index creation. To adjust it, a new index must be created, and data must be reindexed.

Optimizing Shard Size for Performance

Efficient shard management is critical for performance and resource utilization:

1. Ideal Shard Size:
2. Avoid Too Many Small Shards:
3. Right-Sizing Shards:

Case Study: Reducing Shard Count for JVM Optimization

Scenario: An index with:

• 5 primary shards
• 2 replicas (15 total shards)
• 20 GB of data

This configuration results in 1.3 GB of data per shard—far below the recommended minimum.

Solution:

1. Recreate the index with 2 primary shards.
2. Reduce the number of replicas to 1.

Post-Reconfiguration:

• Primary Shards: 2
• Replica Shards: 1
• Total Shards: 4
• Data per Shard: ~10 GB

This adjustment optimizes shard utilization, reduces JVM memory usage, and ensures better performance.

What Is a Node in OpenSearch?

A node is a single instance of OpenSearch running on a machine. Each node serves as a unit of storage and computation within the OpenSearch system. Nodes are responsible for storing data and executing indexing and search operations.

Types of Nodes

Nodes can perform different roles in a cluster:

1. Master Node:
2. Data Node:
3. Ingest Node:
4. Coordinator Node:

Nodes can serve multiple roles or specialize in one, depending on your cluster's setup.

What Is a Cluster in OpenSearch?

A cluster is a collection of nodes that work together to store and analyze data. Clusters enable horizontal scaling, meaning you can add more nodes to distribute the workload as your data grows.

Key Features of a Cluster

1. Single Namespace: The entire cluster operates under a unified namespace, allowing you to treat all the data as part of a single system.
2. Data Redundancy: Clusters replicate data across nodes (using replica shards) to ensure fault tolerance.
3. High Availability: If one node fails, the cluster can continue operating because replica shards take over.

How Nodes and Clusters Work Together

1. Distributing Data: Data is divided into shards and distributed across data nodes. This ensures efficient storage and processing, even for large datasets.
2. Load Balancing: Queries are distributed across the cluster, leveraging multiple nodes to handle requests simultaneously for better performance.
3. Failover and Recovery: If a node goes down, the cluster automatically reroutes queries to replica shards on other nodes, maintaining availability.

Example Scenario

Imagine a cluster with 5 nodes:

• Node 1: Master Node
• Nodes 2–5: Data Nodes

You create an index with 3 primary shards and 1 replica per shard. Here’s how the data is distributed:

• Shard 1 (Primary) is stored on Node 2, with its replica on Node 3.
• Shard 2 (Primary) is stored on Node 3, with its replica on Node 4.
• Shard 3 (Primary) is stored on Node 4, with its replica on Node 5.

This setup ensures:

• Data is spread across nodes for efficient storage and querying.
• High availability, as any single node can fail without data loss.

Key Takeaways

1. Right-Size Shards: Target 10–50 GB per shard for balanced performance and resource efficiency.
2. Minimize Metadata Overhead: Avoid excessive small shards to prevent JVM heap exhaustion.
3. Adjust Shards Dynamically: Use APIs like Reindex and Index Settings to optimize shard allocation.
4. Monitor Shard Utilization: Regularly review shard sizes and performance to maintain an efficient cluster.
5. A node is an individual OpenSearch instance, responsible for storing data and executing tasks.
6. A cluster is a group of nodes working together to manage, store, and analyze data.
7. Clusters distribute data across nodes using shards for scalability and fault tolerance.

By mastering shard and replica configurations, you can ensure your OpenSearch cluster remains scalable, performant, and resource-efficient. Understanding nodes and clusters is essential for designing a scalable and resilient OpenSearch architecture that efficiently handles your data and workloads.

Blog: https://medium.com/@ratnadeepdeyroy/opensearch-index-shards-nodes-and-clusters-8c73f8c71588

Let’s continue the conversation! Share your experiences and strategies for optimizing OpenSearch shards in the comments below.