MongoDB Sharded Cluster Setup on AWS Infrastructure (Traditional Approach)

Introduction: Setting the Stage for MongoDB in the Cloud

In an age where data is at the heart of decision-making, businesses must adopt strategies to handle vast and ever-growing datasets. MongoDB, with its flexibility and distributed design, has become a preferred choice for modern applications. When combined with AWS’s robust infrastructure, the result is a high-performing and scalable solution suitable for enterprise demands. This article delves into building a MongoDB sharded cluster on AWS, highlighting critical components, best practices, and considerations for scalability.

Why Opt for MongoDB Sharded Clusters on AWS?

The combination of MongoDB and AWS offers several advantages:

• Scalability: MongoDB's partitioning of data enables horizontal scaling, while AWS provides the infrastructure to support growth seamlessly.
• Availability: Multi-AZ deployments ensure fault tolerance, while MongoDB’s replica sets enhance resilience.
• Cost Efficiency: Optimized use of AWS resources and sharding reduce operational overhead for large datasets.
• Reliability: AWS services like S3 and Elastic Load Balancer bolster MongoDB’s distributed architecture for uninterrupted operations.

Deep Dive into the Architecture

The architecture incorporates critical MongoDB and AWS components, working together to ensure performance, resilience, and security:

MongoDB Components

• Shards: Each shard is a replica set to ensure high availability. The data is partitioned across shards based on the shard key.
• Config Servers: Responsible for storing the metadata and configuration of the sharded cluster.
• Mongos Routers: These act as query routers, distributing client queries across the shards efficiently.

AWS Components

• EC2 Instances: Hosting the MongoDB components.
• VPC: Isolated network for security.
• Elastic Load Balancer (ELB): Distributing traffic to Mongos instances.
• S3 Buckets: Secure backup storage.

Backup Component

• Percona Backup System: It provide reliable, consistent, and efficient backup solutions that ensure data integrity and can easily be integrated into existing MongoDB environments for disaster recovery and point-in-time restores.

Architecture Workflow

1. Data Distribution: Data is partitioned across shards based on a shard key. Each shard is a replica set with nodes distributed across multiple AWS availability zones (AZs) for high availability.
2. Query Distribution: Mongos routers distribute client queries efficiently across shards by coordinating with config servers for metadata.
3. Backup Workflow: Percona Backup is connected to secondary replicas to offload backup operations, ensuring minimal impact on primary replica performance. Backups are then securely stored in S3 buckets, which offer durability and cost-effective storage.
4. Load Balancing: An AWS Elastic Load Balancer (ELB) manages traffic distribution between Mongos routers to ensure efficient query handling and fault tolerance.

The workflow ensures seamless operation: data is partitioned across shards, queries are routed efficiently by Mongos instances, and backups are offloaded to S3 for durability.

Step-by-Step Guide: Setting Up MongoDB Sharded Cluster on AWS

Provisioning EC2 Instances on AWS

Instance Requirements

For a basic setup with 3 shards, at least 2 Mongos routers, and 3 config servers:

1. Shards (Replica Sets): 3 shards, each with 3 nodes (Primary + 2 Secondaries). Total: 9 EC2 instances.
2. Config Servers: 3 Config Server nodes (Primary + 2 Secondaries).
3. Mongos Routers: At least 2 Mongos instances for high availability.

Instance Types

• Shards: m5.large (optimized for data operations).
• Config Servers: t3.medium (cost-effective for metadata storage).
• Mongos Routers: t3.medium.

Setup

1. Create a VPC with public and private subnets.
2. Configure security groups to allow traffic only between MongoDB components.
3. Assign IAM roles for S3 access and backups.

Installing MongoDB on EC2 Instances

Create a Repository for MongoDB

• Add the MongoDB Repository and Install MongoDB

sudo yum install -y libcurl openssl
cat <<EOF | sudo tee /etc/yum.repos.d/mongodb-org-6.0.repo
[mongodb-org-6.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2/mongodb-org/6.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-6.0.asc
EOF        

Configure MongoDB

• Edit /etc/mongod.conf to enable replica sets, shards, or config server roles.
• Example for replica set configuration:

replication:
  replSetName: rs0
sharding:
  clusterRole: shardsvr        

Start MongoDB

sudo systemctl start mongod
sudo systemctl enable mongod        

Configuring Replica Sets

Initialize Replica Set

• Log in to the primary node of a shard and run:

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "<Primary_Node_IP>:27017" },
    { _id: 1, host: "<Secondary_Node_1_IP>:27017" },
    { _id: 2, host: "<Secondary_Node_2_IP>:27017" }
  ]
})        

• Check Replica Set Status

rs.status()        

Configuring Sharding

Setup Config Servers

• Edit mongod.conf to include:

replication:
  replSetName: configReplSet
sharding:
  clusterRole: configsvr        

• Initialize the config server replica set:

rs.initiate({
  _id: "configReplSet",
  configsvr: true,
  members: [
    { _id: 0, host: "<Config_Server_1_IP>:27017" },
    { _id: 1, host: "<Config_Server_2_IP>:27017" },
    { _id: 2, host: "<Config_Server_3_IP>:27017" }
  ]
})        

Start Mongos Routers

• Edit mongos.conf to include:

sharding:
  configDB: configReplSet/<Config_Server_1_IP>:27017,<Config_Server_2_IP>:27017,<Config_Server_3_IP>:27017        

Start the Mongos service:

sudo systemctl start mongos
sudo systemctl enable mongos        

Add Shards to the Cluster

• Connect to a Mongos instance and run:

sh.addShard("rs0/<Shard_1_Primary_IP>:27017")
sh.addShard("rs1/<Shard_2_Primary_IP>:27017")
sh.addShard("rs2/<Shard_3_Primary_IP>:27017")        

Enable Sharding for a Database

sh.enableSharding("myDatabase")
sh.shardCollection("myDatabase.myCollection", { shardKey: 1 })        

Backup and Disaster Recovery Strategies

Backup Strategies

1. Incremental Backups Use Percona Backup connected to secondary replicas to avoid performance degradation on the primary. Store incremental backups in an S3 bucket for durability and accessibility.
2. Weekly Full Backups Schedule weekly full backups using Percona and store them securely in S3 buckets.

Disaster Recovery

1. Multi-AZ Deployment: Deploy replicas across multiple availability zones to handle failures.
2. Restore Process: Test restoration procedures regularly from S3 backups to ensure data integrity.
3. Automated Alerts: Use CloudWatch and AWS Lambda to set up alerts for replication lag, disk space, and other metrics.

Monitoring and Maintenance

Monitoring Tools

1. AWS CloudWatch: Monitor EC2 health, disk usage, and network performance.
2. Custom Dashboards: Use tools like Grafana integrated with Prometheus for visualization.

Maintenance Tasks

1. Periodic scaling by adding shards or resizing EC2 instances.
2. Reviewing logs for potential issues or security concerns.
3. Regular updates to MongoDB and system packages.

Advantages and Limitations

Advantages of the Traditional Approach

1. Scalability: Sharding and replica sets provide horizontal scalability and high availability.
2. Resilience: Multi-AZ deployment ensures redundancy and fault tolerance.
3. Cost Efficiency: Optimized EC2 instances and S3 for backups minimize costs.
4. Performance: Using secondary replicas for backups ensures uninterrupted primary operations.

Limitations

1. Complexity: Manual setup and configuration require expertise and time.
2. Scalability Limits: Scaling beyond a certain point can become cumbersome without automation.
3. Operational Overhead: Maintenance, monitoring, and backup management require ongoing effort.

Reflecting on the Traditional Approach

This approach demonstrates how a combination of MongoDB’s sharded architecture and AWS’s infrastructure can meet modern demands for scalability and resilience. However, manual setup and maintenance require significant effort, and scaling beyond a certain point may become complex.

To address these challenges, organizations are increasingly adopting containerized solutions. A Kubernetes-based MongoDB deployment simplifies scaling, automates failovers, and reduces operational overhead. This marks a shift toward a more agile and streamlined approach to database management.

Final Thoughts

Building a MongoDB sharded cluster on AWS offers a powerful solution for handling large-scale applications. By carefully planning architecture, deploying across multiple AZs, and implementing robust backup strategies, businesses can achieve a balance of scalability, availability, and cost-effectiveness.

As organizations evolve, the transition to containerized setups represents the next phase of innovation. Stay tuned for our upcoming article, where we’ll explore how Kubernetes can revolutionize MongoDB deployments for future-ready enterprises.

Your Turn

What’s your experience with MongoDB on AWS?

Have you implemented a similar architecture? Or perhaps faced challenges along the way? Share your thoughts or any questions in the comments below I’d love to hear your insights!

And stay tuned for our next article, where we’ll explore how Kubernetes can further simplify MongoDB deployments for modern enterprises.

?